The Error

If you found this page, you probably saw this error after adding a new table or joining it to an existing pipeline:

PipelineException: Circular dependency detected in pipeline graph.
  Dataset 'silver_customer_flags' -> 'silver_order_enriched'
              -> 'silver_customer_flags'
  The pipeline cannot be executed. Resolve the cycle before retrying.

‍

The problem indicates that Spark’s pipeline planner detected a cycle within the DAG of your datasets. Since Spark Declarative Pipelines demand a strict DAG architecture, any loops will render execution impossible, regardless of how logical the design may seem initially.

Minimal Reproducible Example

This scenario is common in medallion architectures where two silver-layer tables need to share data. Consider a pipeline that builds an enriched orders table and a customer flags table at the silver layer:

import pyspark.dp as dp 
from pyspark.sql import functions as F 

# silver_order_enriched needs customer flags to flag high-risk orders
@dp.table
def silver_order_enriched():
    orders = dp.read('bronze_orders')
    flags  = dp.read('silver_customer_flags')  # <-- reads from sibling silver table
    return orders.join(flags, 'customer_id', 'left')

# silver_customer_flags needs order history to compute risk score
@dp.table
def silver_customer_flags():
    enriched = dp.read('silver_order_enriched')  # <-- reads back from sibling
    return enriched.groupBy('customer_id').agg({'risk_score': 'max'})

‍

Both tables are trying to use each other's outputs simultaneously. To solve this, move the shared computation into a separate intermediate silver table. Then, both tables can read from this shared source independently.

How to Fix It: Move Shared Logic to an Intermediate Table

Both tables are trying to consume each other's output simultaneously. To fix this, extract shared computation into a separate intermediate silver table. Both tables then read from this shared source independently.

# Step 1: Intermediate table — computes raw customer order stats
# from bronze only. No dependency on any other silver table.
@dp.table
def silver_customer_order_stats():
    return (
        dp.read('bronze_orders')
        .groupBy('customer_id')
        .agg({'risk_score': 'max', 'order_count': 'count'})
    )
# Step 2: Customer flags reads from the intermediate table only
@dp.table
def silver_customer_flags():
    return (
        dp.read('silver_customer_order_stats')
        .filter('risk_score > 0.8')
    )
# Step 3: Enriched orders joins bronze orders + customer flags
# No back-reference to silver_customer_flags' dependencies
@dp.table
def silver_order_enriched():
    orders = dp.read('bronze_orders')
    flags  = dp.read('silver_customer_flags')
    return orders.join(flags, 'customer_id', 'left')

‍

The dependency graph is now a clean DAG: bronze_orders → silver_customer_order_stats → silver_customer_flags → silver_order_enriched. No cycles, no error.

Root Cause Analysis: Why Does This Happen?

Spark Declarative Pipelines resolve dataset references at definition time. Each @dp.table registers a node, and each dp.read() registers a directed edge. The planner then runs a topological sort to determine execution order. The pipeline fails immediately if a cycle is detected. In a medallion architecture, these cycles typically occur when:

• Sibling tables reference each other.
• Gold-to-Silver feedback: A lower-layer table attempting to filter based on a higher-layer aggregate.
• Duplicated logic loops: Two tables attempting to read from one another to avoid re-computing shared logic.

Quick Verification

You can check this in the Databricks Pipelines UI. After saving, go to your pipeline and click Graph. If you see a valid graph with no red cycle indicators, your DAG is clean. Then, click Start to run a full pipeline update.

Important Caveats

• If you use spark.read.table('LIVE.x') instead of dp.read('x'), it hides the cycle but does not fix it. The pipeline might start, but the table will not update correctly when upstream data changes, and you could end up reading stale data without noticing.
• Moving shared logic into an intermediate table can change how often your pipeline updates. If the intermediate table is expensive to compute, you might want to use @dp.view instead of @dp.table. This way, you avoid saving it to Delta storage every time the pipeline runs.

How v4c.ai Can Help

v4c.ai is a pure-play Databricks services partner with 600+ certifications and 400+ practitioners across Financial Services, Retail, Manufacturing, and Healthcare. We specialize in building resilient, scalable, and modular pipeline architectures that avoid common pitfalls such as circular dependencies. Our data engineering practice can help review your current DAGs and pipeline configurations to identify bottlenecks and structural dependencies before they cause execution errors. We re-engineer complex, interdependent silver-layer tables into scalable, intermediate structures that adhere to strict DAG principles and transition from high-latency table structures to efficient views and incremental ingestion patterns. By introducing comprehensive observability and audit logging, we enable your teams to proactively identify and resolve complex DAG issues before they ever disrupt production systems.

Further Reading

Consult the official Databricks documentation for deeper reference:

• Delta Live Tables — Pipeline Modes & Execution

‍