Yash Narang

Senior Data Engineer @CGI

Toronto, ON, CA

Developed and maintained large-scale data pipelines on Azure Data Factory, orchestrating ingest-transform-load operations from multiple on-prem and cloud sources.• Built real-time and batch ETL/ELT pipelines using Azure Databricks and PySpark to process clickstream, transactional, and user profile data.• Designed and implemented data warehouse models in Snowflake using star and snowflake schemas tailored for performance and reusability.• Automated metadata-driven pipeline development using parameterized ADF templates for scalable ingestion patterns.• Integrated Apache Airflow with Azure Data Factory and Databricks for job orchestration, dependency management, and SLA monitoring.• Provisioned Azure Data Lake Gen2 storage layers with lifecycle policies and RBAC to store raw, curated, and consumption-ready data zones.• Built CI/CD pipelines using Azure DevOps and Git for deploying ADF pipelines, Databricks notebooks, SQL scripts, and access control policies.• Deployed infrastructure as code using Terraform modules to provision Azure SQL, ADF, Databricks, and Key Vault resources.• Implemented job failure alerting, retry logic, and Slack/email notifications through Airflow and Azure Monitor integrations.• Created custom Python packages for logging, error handling, and dynamic data parsing, deployed across all pipeline stages.• Established integration between Snowflake and Power BI/Tableau for business intelligence and self-serve analytics.• Designed secure views and enforced role-based access control (RBAC) policies in Snowflake for data sharing across departments.• Enabled cost optimization by setting auto-suspend and auto-resume policies for Snowflake compute warehouses.• Tuned complex queries in Snowflake by applying clustering keys, optimizing partitions, and analyzing execution plans.• Built reusable PySpark modules for deduplication, windowed aggregations, and time-series enrichment used in multiple pipelines.