Guruprasad Patil

Data Engineer @Tata Consultancy Services

Designed and developed scalable ETL pipelines using Azure Data Factory to ingest structured and semi-structured data into ADLS Gen2.• Built PySpark transformation logic in Azure Databricks for data cleansing, deduplication, null handling, and standardization.• Implemented incremental load frameworks using watermark columns (LastModifiedDate), reducing pipeline execution time by 30%.• Enforced schema validation and schema drift handling across raw, staging, and curated data layers.• Performed source-to-target reconciliation using SQL and PySpark — row count checks, control totals, and column-level quality validation.• Optimized Spark jobs using partitioning, broadcast joins, and efficient filtering to improve pipeline performance.• Developed parameterized ADF pipelines with reusable datasets for multi-environment deployments.• Delivered analytics-ready datasets in Azure SQL Database powering Power BI dashboards for business stakeholders.Tech stack: ADF · Databricks · PySpark · ADLS Gen2 · Azure SQL · MS Fabric · Power BI · Apache Spark