Mahesh N
Senior Data Engineer | Azure, GCP, AWS, ETL, Airflow, Spark, Databricks
- Role
- Senior Data Engineer at FirstEnergy
- Location
- Dallas, TX, US
- LinkedIn followers
- 500 followers
About Mahesh N
Armed with a Bachelor of Engineering in Computer Science from Osmania University, my…
Experience
Senior Data Engineer
May 2023 — Present · US
Designed and implemented metadata-driven ETL pipelines using Azure Data Factory (ADF) to orchestrate data migration from on-premises SQL Server to Azure Data Lake Storage Gen2 (ADLS), Snowflake and Databricks for AI/ML usecases, enterprise analytics and reporting• Developed real-time data pipelines to ingest data asynchronously from Kafka and webhooks over HTTP into Unity Catalog tables and Snowflake using Azure Functions and Spark Streaming on Databricks• Extracted and unified social media data from Twitter, Facebook, Instagram into Databricks Unity Catalog tables and Snowflake using ADF and APIs, enabling reporting dashboards and ML models for sentiment analysis and campaign performance optimization• Engineered PySpark-based transformation pipelines in Databricks Spark Notebooks to populate the silver layer, implementing full load, incremental load, and Slowly Changing Dimension (SCD) Type 2 processes• Designed and implemented pipelines to load gold-layer data into Snowflake Pool for reporting and curated Delta tables in ADLS to support ML workloads• Developed reusable PySpark utility modules to support consistent data validation, enrichment, and transformation logic across ingestion pipelines• Migrated ADF jobs from non-Unity Catalog (Legacy Hive Metastore) to Unity Catalog for better governance, performance improvements, and enhanced security controls• Implemented Dynamic Data Masking (DDM), role-based access control (RBAC), and row-level security filters to protect sensitive attributes (PII/PHI) and restrict data access by user roles• Designed and implemented a Delta Lake architecture on ADLS Gen2 to organize raw (bronze) layer data in Parquet and delta format, enabling scalable ingestion and downstream integration, while supporting ACID transactions and efficient incremental data processing for both batch and streaming workloads
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.