Authors: Aspanna Yaswanth, Rajiv Gupta, Akhil Kumar & Vijayaraju Yarra
Track, trust, and transform your data with end-to-end visibility
What is Data Lineage in Snowflake?
Snowflake Data Lineage provides a visual and programmatic map of your data’s journey—showing how it moves, transforms, and relates between objects in your environment. It captures two core relationship types:
- Data Movement Relationships: This tracks instances where data is copied or materialized from one object to another. Think of operations like CTAS (Create Table As Select), INSERT, or MERGE.
- Object Dependencies: This refers to scenarios where an object references a base object without directly materializing new data, such as with VIEWS.
Key Features that Elevate Your Data Governance
Snowflake Data Lineage isn’t just a tracking tool; it’s a robust solution with features designed to enhance your data governance framework:
- Impact Analysis: Swiftly understand the ripple effect of changes by visualizing relationships between various objects.
- Enhanced Monitoring: Gain insights into data movement and object dependencies, ensuring transparency in your data pipelines.
- Compliance Facilitation: Easily track the flow of sensitive data, simplifying compliance audits and regulatory adherence.
- Tag and Policy Management: Seamlessly integrate with Snowflake’s tagging and masking policy features for granular control and consistent data classification.
- Data Trust: Foster confidence in your data by clearly understanding its origins (source objects and columns) and destinations (target objects and columns).
- Delegated Administration: Implement role-based access control to lineage information, ensuring that only authorized personnel can view sensitive data flow.
Setting Up a Sales Analysis Pipeline with Lineage Tracking (POC)
To demonstrate the power of Data Lineage, let’s walk through a practical example: setting up a sales analysis pipeline in Snowflake.
Prerequisites
- Snowflake Enterprise Edition or higher.
- Appropriate access privileges for creating objects and viewing lineage.
Setup Overview
This Proof of Concept (POC) outlines the steps to build a sales analysis pipeline and leverage Snowflake’s data lineage capabilities to visualize and manage data flow.
This Proof of Concept (POC) outlines the steps to build a sales analysis pipeline and leverage Snowflake’s data lineage capabilities to visualize and manage data flow.
1. Database and Schema Setup
First, we’ll set up our database and schema to house our sales data:
2. Stage and File Format Setup
Next, we’ll create a stage to load our CSV files and define the file format.
Note: At this point, upload your CSV files (e.g., US.csv, UK.csv, and India.csv) to the SALES_STG stage using the Snowflake UI.
3. Data Loading
We’ll load data from our regional CSV files into separate tables:
4. Creating a Consolidated Sales Table
Now, let’s combine all regional data into a single GLOBAL_SALES table:
5. Tag Creation and Application
Tags are essential for better data organization and governance. Let’s create and apply some tags:
6. Masking Policy Creation
To protect sensitive data, we’ll implement a masking policy:
7. Creating Analysis Views
Finally, we’ll create various views for sales analysis, applying our masking policy where appropriate:
7.1 Materialized View for Regional Performance
7.2 Product Analytics View
7.3 Time Series Analytics View
Visualizing Data Lineage in Snowsight
Once you’ve executed the setup scripts, the real magic begins in Snowsight:
- Sign in to Snowsight with the necessary privileges.
- Navigate to Data » Databases.
- Select the global_sales table or any of the views you created.
- Click on the Lineage tab.
Here, you’ll see a clear and interactive diagram showcasing:
- How data flows from your regional sales tables into the global_sales table.
- How the global_sales table feeds into the various analysis views.
- The tags and masking policies applied to your objects and columns, providing a holistic view of your data governance.
Working with Tags and Masking Policies for Enhanced Governance
Data Lineage in Snowflake isn’t just for viewing; it’s for actively managing your data’s quality and security.
Identifying and Applying Missing Tags
- Navigate to the Lineage tab for an object (e.g., global_sales).
- Select a column (e.g., unitprice in global_sales).
- Click “View Lineage” and choose “Downstream Lineage” or “Upstream Lineage.”
- Look for columns with missing or mismatched tags, often highlighted with dashed or yellow borders.
- Use the “Review and Apply” button to propagate tags consistently across your data pipeline.
Verifying Masking Policies
- Navigate to the Lineage tab for an object with masked columns.
- Select the object to view its details.
- Look for the mask symbol next to column names.
- Hover over the symbol to see the masking policy details and ensure your sensitive data is protected as intended.
Failure & Error Testing: Ensuring Robustness
Understanding how Data Lineage behaves under different scenarios, especially during failures or misconfigurations, is critical.
1. Intentional Misconfigurations with SNOWFLAKE.CORE.GET_LINEAGE()
Let’s test with some intentional errors to see how the programmatic lineage function responds.
- Test with non-existent objects:
Output:
- Test invalid direction parameter:
Output:
- Test invalid column parameter:
Output:
2. Testing using Snowflake UI (After Object Dropped)
Observe how lineage changes in the UI when a source table is dropped:
Before Object Dropped:
After Object Dropped:
This visual feedback is invaluable for quickly identifying and addressing broken data pipelines.
Programmatic Lineage with GET_LINEAGE Function
For those who prefer programmatic access, Snowflake provides the SNOWFLAKE.CORE.GET_LINEAGE function. This function offers the same lineage data as the visual Lineage tab, but in a table format, allowing for advanced automation and custom reporting.
Syntax:
- object_name: Fully qualified name of the object.
- object_domain: Domain of the object (e.g., ‘COLUMN’, ‘TABLE’, ‘STAGE’).
- direction: ‘UPSTREAM’ or ‘DOWNSTREAM’.
- distance: Number of levels (optional, default 5, max 5).
Examples:
a) Object-level lineage
b) Column-level lineage
c) Distance Parameter
The GET_LINEAGE function allows you to retrieve up to 5 levels of lineage by default. While your data might have more levels, the function provides visibility up to this limit.
Note: The GET_LINEAGE function can return up to 10 million rows. If the lineage exceeds this limit, additional rows are excluded without warning.
Lineage with Streamlit and ML Lineage
Snowflake’s ecosystem extends lineage capabilities even further:
Lineage using Streamlit
A Streamlit application can connect to Snowflake via Snowpark sessions and leverage the GET_LINEAGE function to programmatically fetch data lineage, supporting both upstream and downstream analysis with configurable distance parameters. This allows for customized lineage visualization and integration into your existing applications.
ML Lineage
Beyond traditional data lineage, Snowflake also supports ML Lineage. This specialized feature focuses on tracking how data is used and transformed within machine learning workflows, providing transparency into your AI models.
- ML Quick Start: For a step-by-step guide on setting up ML Lineage, explore the Snowflake Quickstart:
https://quickstarts.snowflake.com/guide/develop-and-manage-ml-models-with-feature-store-and-model-registry/index.html?index=..%2F..index#0
Cost Implications: Transparency Without Surprises
Snowflake Data Lineage is designed to be cost-effective:
- Enterprise Edition Required: Ensure your Snowflake account has the appropriate licensing (Enterprise Edition or higher).
- No Additional Compute Costs: Viewing lineage information does not consume additional compute credits.
- Minimal Storage Impact: The storage of lineage metadata has a negligible cost impact.
- ROI Factors: The return on investment comes from significant cost savings through improved compliance, reduced troubleshooting time, and enhanced data governance.
Limitations and Best Practices for Optimal Use
While powerful, it’s important to be aware of certain limitations and to adopt best practices for maximizing the value of Snowflake Data Lineage.
8.1 Limitations:
- Shared Database Restriction: Lineage is not available for objects in shared databases. Cross-account lineage tracking requires alternative approaches.
- System Objects Restrictions: Lineage is not available for the SNOWFLAKE database or INFORMATION_SCHEMA. System metadata relationships must be tracked separately.
- Dynamic Tables: Limited lineage visibility for dynamic tables. They appear in lineage graphs for other objects but do not have their own Lineage tab.
- Retention Period: A one-year retention period for lineage information. For long-term historical tracking, supplementary archiving is needed.
- External Tools: Snowflake’s data lineage feature does not include external tools, functions, or jobs in its displayed diagrams or the GET_LINEAGE function.
8.2 Best Practices for Maximum Value:
- Consistent Naming Conventions: Use clear, consistent naming for all database objects (e.g., DATALINEAGE_DB instead of just DATALINEAGE).
- Granular Tag Application: Apply tags at both object and column levels for detailed classification (e.g., tagging specific columns like REGION).
- Role-Based Masking: Create masking policies that respect organizational roles, ensuring appropriate data visibility.
- Documentation: Add comments to objects to provide better understanding of their purpose.
- Lineage Review Process: Schedule regular reviews of data lineage (e.g., monthly) to identify and fix any broken links or inconsistencies.
- Integrate with Governance Tools: Where possible, connect Snowflake lineage to external catalogues for a unified governance framework.
Conclusion: Turning Lineage into a Strategic Advantage
Snowflake Data Lineage isn’t just a technical feature—it’s a governance accelerator. By understanding its limitations, applying best practices, and integrating it into your broader governance strategy, you can:
- Build trust in your data through transparent, auditable flows
- Strengthen compliance with clear traceability for sensitive information
- Reduce troubleshooting time with instant impact analysis
- Enable faster, more confident decision-making across the business
When paired with robust tagging, masking, and documentation, Data Lineage becomes more than a map of your data—it’s a blueprint for operational efficiency, regulatory confidence, and analytics at scale.
About kipi.ai
Kipi.ai, a WNS Company, is a global leader in data modernization and democratization focused on the Snowflake platform. Headquartered in Houston, Texas, Kipi.ai enables enterprises to unlock the full value of their data through strategy, implementation and managed services across data engineering, AI-powered analytics and data science.
As a Snowflake Elite Partner, Kipi.ai has one of the world’s largest pools of Snowflake-certified talent—over 600 SnowPro certifications—and a portfolio of 250+ proprietary accelerators, applications and AI-driven solutions. These tools enable secure, scalable and actionable data insights across every level of the enterprise.
Serving clients across banking and financial services, insurance, healthcare and life sciences, manufacturing, retail and CPG, and hi-tech and professional services, Kipi.ai combines deep domain excellence with AI innovation and human ingenuity to co-create smarter businesses. As a part of WNS, Kipi.ai brings global scale and execution strength to accelerate Snowflake-powered transformation world-wide.
For more information, visit www.kipi.ai.
