Nomaan Mohammed
Lead Data Engineer @Highmark Health
Signup · Get unlimited contacts
WORK HISTORY
Lead Data Engineer @Highmark Health
Chicago, IL, US
Designed and developed ETL/ELT pipelines using SQL, Python, and PySpark, integrating healthcare data from 30+ payors into enterprise data warehouses. Designed and implemented centralized logging and monitoring solutions with AWS CloudWatch, CloudTrail, and ELK stack, improving observability of large-scale data pipelines. Deployed and managed Prometheus and Grafana dashboards to monitor distributed Spark jobs, cluster health, and application performance. Configured automated alerts and notifications for pipeline failures, SLA breaches, and infrastructure anomalies using AWS SNS and PagerDuty. Established cost monitoring and optimization practices by integrating AWS Cost Explorer and CloudWatch metrics, reducing monthly infrastructure spend. Implemented proactive monitoring and auto-healing strategies for AWS EKS clusters running containerized data processing workloads. Automated infrastructure compliance checks and security monitoring with AWS Config and GuardDuty, ensuring alignment with enterprise governance policies. Built and optimized SQL Server stored procedures, triggers, and queries, improving performance and scalability of healthcare claims processing systems. Implemented dimensional data models and schema designs in Snowflake and Azure Synapse, enabling efficient querying and BI reporting. Developed and deployed pipelines in Azure Data Factory and Databricks, supporting both batch and near real-time ingestion of Claims, Revenue, and Rx Medicare data. Troubleshot complex data quality and transformation issues, implemented validation frameworks, and ensured compliance with healthcare data standards (HIPAA).
EDUCATION
Osmania University, Hyderabad
Bachelor's degree, Computer Engineering
ABOUT NOMAAN MOHAMMED
Senior Data Engineer with 14+ years of experience in designing and implementing scalable data solutions using AWS services, including S3, Lambda, Glue, Athena, and Redshift for efficient data management and processing. Proven expertise in building real-time data pipelines with Apache Kafka and Spark, utilizing PySpark and Python to process large-scale datasets with minimal latency. Extensive experience in deploying serverless solutions using AWS Lambda for automating workflows and reducing operational overhead. Proficient in building and managing distributed systems with AWS, leveraging Spark and Redshift for high-performance data analytics and storage solutions. Integrated Databricks with AWS and Snowflake to create a unified, scalable data architecture for analytics and reporting. Utilized Databricks SQL for building reusable data models, enabling seamless integration with BI tools like Tableau and Looker. Automated data pipeline monitoring and logging in Databricks to proactively identify and resolve performance bottlenecks. Used Spark and Spark-SQL to read the parquet data and create the tables in hive using the Scala API. Hands-on experience with Python, SQL, and Snowflake for data processing, integration, and analysis. Designed and optimized data models to support Looker and Tableau dashboards, ensuring efficient reporting for finance, sales, procurement, operations, and payroll teams. Strong understanding of data warehousing, analytics, and programming, with extensive experience in SQL, including database programming (procedures and functions). Expertise in API integration for data loading processes, with experience in leveraging cloud-based solutions, preferably AWS, for scalable data infrastructure. Implemented the use of Amazon EMR for Big Data processing among a Hadoop Cluster of virtual servers on Amazon related EC2 and S3. Expert in optimizing ETL workflows by identifying and resolving bottlenecks, ensuring efficient data extraction, transformation, and loading processes for high-volume data pipelines. Proficient in analyzing and tuning complex SQL queries by optimizing joins, indexing strategies, and execution plans to enhance query performance across relational databases like Oracle, SQL Server, and Snowflake. Enhanced Snowflake usage for data warehousing, data extraction, and ETL processes, ensuring the scalability of data solutions. Developed SQL stored procedures and functions to streamline data transformation and analysis, ensuring performance optimization.
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.