Why Lineage and Quality Metrics Are the Foundation of Context Graphs
Where data lineage ends and decision context begins
Welcome to Beyond the Traverse. This Substack lives where data lineage ends and decision context begins. If you’ve ever traced a data issue back to its source only to find the trail goes cold at a spreadsheet, a Slack thread, or someone’s head, you’re in the right place. We’ll be writing about context graphs, metadata architecture, and the infrastructure gap between what our tools capture and what our organizations actually need to know.
The next wave of enterprise value won’t come from accumulating more data. It’ll come from capturing the reasoning that makes data actionable. That’s the thesis behind Foundation Capital’s recent piece “AI’s Trillion-Dollar Opportunity: Context Graphs,” co-authored by partners Jaya Gupta and Ashu Garg. Their argument signals a shift in how data engineers should think about metadata architecture: we’re moving from systems that record what happened to systems that capture why it happened and how decisions were made.
For data engineers, this isn’t just VC speculation. Data lineage and quality metrics, long treated as governance overhead, are becoming the primary infrastructure layer for AI-ready data platforms. If you’re designing modern data architectures, knowing how these familiar tools fit into the context graph paradigm matters.
Context Graphs Capture What Lineage and Quality Metrics Miss
Traditional data lineage tells you that Table A feeds Model B, which populates Dashboard C. Quality metrics tell you whether the data in those assets meets defined thresholds. But neither captures the institutional knowledge that actually drives decisions: the exception a VP approved, the precedent that justified a discount, the cross-system synthesis that informed an escalation.
Gupta and Garg call this missing layer “decision traces,” the reasoning, exceptions, and precedents that make data actionable. When a sales leader approves a 20% discount despite a 10% policy cap, your CRM records one fact: “Final price: 20% discount.” Everything else (the PagerDuty incidents that justified the exception, the Slack thread flagging churn risk, the finance approval chain) disappears. The synthesis happened in someone’s head. No system captured it.
A context graph addresses this gap by stitching decision traces across entities and time so precedent becomes searchable. It’s not merely a knowledge graph with extra metadata; it’s a living record of how decisions were actually made, enriched with temporal context, confidence scores, provenance tracking, and reasoning traces. The graph structure enables multi-hop traversal: you can query not just “who owns this data” but “what decisions depended on this data, who approved them, and what exceptions were granted.”
Lineage Provides the Structural Backbone
Data lineage isn’t new, but its role is shifting. For context graphs to function, they need a foundational layer that maps how data flows from sources through transformations to consumption points. Lineage provides this structural backbone.
Modern lineage has standardized around OpenLineage, an LF AI & Data Foundation project that defines a vendor-neutral specification for capturing lineage metadata. The model is elegant: Jobs consume and produce Datasets, Runs represent job executions, and Facets attach atomic metadata pieces to any entity. This architecture maps naturally to graph structures. Nodes represent assets (sources, transformations, consumption points), edges represent data flow relationships, and facets enrich the graph with schema information, quality scores, and transformation logic.
The shift from table-level to column-level lineage represents a critical step forward. Table-level lineage shows that a dashboard depends on a warehouse table; column-level lineage shows exactly which fields feed which metrics, enabling precise impact analysis and root cause tracing. When a data quality incident occurs, column-level lineage lets you trace the issue backward through specific transformation logic to the exact upstream field that introduced the anomaly.
For context graphs, lineage serves multiple functions. It provides the horizontal connectivity between assets, enabling queries like “what downstream systems are affected by this change.” It captures transformation semantics, showing not just that data moved but how it was modified. And it establishes provenance chains that ground AI responses in traceable sources rather than hallucinated connections.
Quality Metrics Add the Health Dimension
If lineage provides the structural map, quality metrics provide the health overlay. The six core dimensions (accuracy, completeness, freshness, consistency, validity, and uniqueness) represent different lenses for evaluating data fitness for purpose. But raw quality scores in isolation have limited value. A 95% completeness score means nothing without context: Is that acceptable for this use case? How does this compare to historical patterns? Which downstream consumers are affected when it drops?
Context graphs turn quality metrics from standalone measurements into contextual health signals. By overlaying quality status on lineage relationships, you can visualize not just which assets are unhealthy but how quality issues propagate through the data ecosystem. A freshness violation in an upstream source becomes immediately visible across all downstream dashboards, prioritized by business criticality and usage patterns.
The integration pattern emerging across modern data observability platforms follows a consistent architecture: quality metrics attach to lineage nodes as facets, lineage edges propagate impact analysis, and business metadata (ownership, classification, SLAs) provides the interpretive context. Tools like Monte Carlo, Bigeye, and Soda have converged on this model, combining automated anomaly detection with lineage-aware alerting that routes incidents to affected downstream owners.
Data contracts represent the governance layer that formalizes this integration. A data contract defines the schema, quality expectations, and delivery SLAs between data producers and consumers. When contracts are encoded alongside lineage metadata, you can answer questions like “which teams are violating their freshness SLAs” or “what downstream contracts will be breached if this upstream change is deployed.” The Open Data Contract Standard (ODCS) provides a specification for encoding these expectations in version-controlled code.
The Architectural Convergence Toward Active Metadata
Gupta’s context graph thesis arrives amid a broader industry convergence. Several parallel movements (active metadata management, data observability, semantic layers, and data mesh/fabric architectures) all point toward the same insight: metadata needs to do more than document; it needs to enable.
Gartner’s active metadata research predicts that organizations implementing active metadata will cut time-to-delivery for new data assets by 70% by 2027. The distinction matters: passive metadata sits dormant in catalogs, waiting to be queried; active metadata drives automation, triggers governance actions, and enables bidirectional synchronization across the data stack.
Context graphs can be understood as the logical next step for active metadata. They don’t just catalog what exists. They capture how assets relate, how they’re used, what decisions they’ve informed, and what trust signals apply. The graph structure provides the flexible, extensible foundation that rigid catalog schemas can’t match.
Semantic layers add the translation capability that makes graphs accessible to business users and AI systems. dbt’s acquisition of Transform (MetricFlow) in 2023 signaled the importance of metrics-as-code, version-controlled business definitions that allow consistent interpretation across tools and teams. For AI applications, semantic layers are becoming a key requirement. They provide the structured context that grounds LLM responses in business reality rather than hallucinated interpretations.
The data mesh movement, championed by Zhamak Dehghani, stresses domain-oriented ownership and data-as-product thinking. In a mesh architecture, each domain treats its data assets as products with owners, SLAs, documentation, and quality metrics. This is exactly the metadata richness that context graphs require. Federated governance keeps these distributed assets discoverable and interoperable through shared metadata standards.
What’s Still Missing
Despite the convergence, current platforms fall short of true context graph capabilities in several ways. These gaps aren’t minor feature requests; they represent structural limitations in how we’ve built metadata infrastructure. This is the terrain Beyond the Traverse will keep returning to: the space past the last node, where the tools stop and the real work begins.
No capture layer for decision context. Lineage tools track data flow. Quality tools track health. Neither captures why a decision was made or what exceptions were granted. When an analyst overrides a model recommendation, or a business user exports data to Excel for manual adjustment, that reasoning evaporates. Current tools have no mechanism to intercept and record these moments. The decision happened outside the system, so the system has nothing to index.
Siloed tools with no unified graph layer. Most organizations run separate systems for lineage (Atlan, DataHub, OpenMetadata), quality (Monte Carlo, Soda), catalogs (Alation, Collibra), and semantic definitions (dbt, LookML). Each maintains its own metadata store. Stitching them together requires custom integration work, and the result is usually a lossy approximation. There’s no shared graph substrate that lets you traverse from a quality incident to the upstream lineage to the business definition to the downstream consumers in a single query.
Lineage stops at system boundaries. OpenLineage and similar specs do well within their scope: Spark jobs, Airflow DAGs, dbt models. But lineage breaks down at the edges. Data exported to a spreadsheet? Gone. Manual uploads from vendor portals? No provenance. The analyst who copy-pasted numbers into a slide deck? Invisible. Most real decisions involve at least one step that happens outside the instrumented perimeter.
Quality metrics lack causal attribution. Observability tools can tell you that a freshness SLA was violated or that null rates spiked. They can’t tell you why. Was it a source system outage? A schema change? An upstream job that failed silently? Root cause analysis still requires manual investigation. The quality signal exists, but it’s disconnected from the causal chain that would explain it.
No temporal reasoning across the graph. Current tools give you a snapshot: here’s the lineage now, here’s the quality now. But context graphs need to answer temporal questions: What did the lineage look like when this decision was made? How has data quality trended over time for this asset? What changed between the last successful run and this failure? Point-in-time queries against a versioned graph are rare. Most platforms overwrite state rather than append history.
Semantic and lineage layers don’t talk. dbt’s semantic layer knows that “revenue” means a specific calculation. Your lineage tool knows which tables feed that calculation. But the two systems don’t share a unified model. You can’t easily ask “show me all dashboards that use the ‘revenue’ metric and trace back to source systems with quality issues.” The semantic meaning and the physical lineage exist in parallel, not in integration.
Adoption fails without workflow integration. The deeper problem is organizational. Metadata tools sit outside the flow of work. Engineers don’t want to context-switch to a catalog to document their changes. Analysts don’t log their reasoning in a governance tool before making a decision. Unless metadata capture is embedded in the tools people already use (IDEs, notebooks, BI tools, Slack), it won’t happen. Current platforms treat metadata as a destination, not a byproduct.
No feedback loop from AI consumption. As organizations deploy LLMs against their data, they need to know which assets are being referenced, which queries are failing, and where the model lacks sufficient context. Current platforms have no mechanism to capture AI agent interactions and feed them back into the graph. The consumption patterns that would reveal gaps in documentation or lineage remain opaque.
These gaps aren’t insurmountable, but they explain why context graphs remain more vision than reality for most organizations. The building blocks exist. The integration layer doesn’t.
Practical Implications for Data Engineers
For data engineers designing modern platforms, the context graph paradigm suggests several priorities:
Invest in column-level lineage as foundational infrastructure. Table-level lineage is no longer sufficient. OpenLineage provides a vendor-neutral path to capturing granular lineage across diverse tooling (dbt, Spark, Airflow, Dagster). The upfront investment pays dividends in impact analysis, compliance, and (increasingly) AI readiness.
Treat quality metrics as first-class metadata, not afterthought reporting. Quality scores should attach to lineage graphs as facets, enabling contextual interpretation. The pattern of quality overlays on lineage visualizations is becoming standard across observability platforms.
Implement data contracts to formalize producer-consumer expectations. Contracts encode quality SLAs, schema expectations, and ownership responsibilities in version-controlled code. They provide the governance backbone that prevents context graphs from becoming garbage-in-garbage-out systems.
Consider graph-native storage for metadata architectures. As metadata relationships grow complex, relational schemas struggle with multi-hop traversal queries. Graph databases (Neo4j, AWS Neptune) or purpose-built knowledge graph platforms enable the flexible, relationship-first queries that context graphs require.
Start capturing decision context manually. If automated capture doesn’t exist yet, build the habit. Tag decisions in Slack with structured metadata. Log exception approvals in a shared Notion database. Create a lightweight “decision journal” that links outcomes to the data and reasoning that informed them. Even manual capture builds the muscle memory and data foundation for future automation.
Push for workflow-embedded metadata. Advocate for tools that capture metadata as a byproduct of work, not a separate task. IDE plugins that auto-generate documentation. Notebook extensions that log data sources. BI tools that record lineage when analysts build dashboards. The less friction, the more complete the graph.
The Path from Documentation to Decision Intelligence
Gupta’s distinction between “systems of record” and “systems of decisions” captures what’s at stake here. Traditional data platforms excel at recording what happened: transactions, events, states. Context graphs capture the reasoning layer above: why decisions were made, what exceptions applied, what precedents governed.
For data engineers, this represents both an opportunity and a challenge. The opportunity: we’re already building the foundational layers (lineage, quality metrics, semantic models) that context graphs require. The challenge: connecting these layers into coherent, queryable knowledge structures demands new architectural thinking.
The organizations that capture decision traces, not just decision outcomes, will compound an institutional memory that makes AI agents increasingly capable over time. Each automated decision adds another trace to the graph; exceptions become encoded knowledge; the organization stops relying on tribal memory. That’s the trillion-dollar opportunity: building the contextual intelligence layer that makes data truly actionable.
Beyond the Traverse is about the metadata infrastructure we don’t have yet. If you’re building at the edge of what current tools can do, or just frustrated by the gaps, subscribe. We’ll be here, mapping the terrain.