Harshverdhan Shukla

Big Data Engineer @Vanguard

Charlotte, NC, US

As a Big Data Engineer, I design, build, and operate scalable data pipelines that power analytics and machine-learning platforms. My core responsibilities include:End-to-end pipeline development: Architect and implement batch and streaming ETL workflows using Apache Spark, Kafka, and Airflow to ingest, cleanse, and transform terabytes of data daily.Cloud data infrastructure: Provision and manage data storage and compute on AWS (S3, EMR, Redshift) or GCP (BigQuery, Dataflow), ensuring cost-effective scalability and high availability.Data modeling & warehousing: Define and maintain dimensional models, star schemas, and Delta/Lakehouse tables to enable performant downstream reporting and self-service analytics.Workflow orchestration: Author DAGs in Apache Airflow to schedule complex job dependencies, implement retries and alerting, and monitor pipeline health via built-in and custom metrics.Real-time data streams: Develop Kafka producers and consumers to capture event streams (e.g, user activity, logs), enforce schema validation with Confluent Schema Registry, and guarantee end-to-end delivery.Performance tuning & optimization: Profile Spark jobs, tune cluster configurations, and refine SQL queries to reduce pipeline latency and resource consumption.Data quality & governance: Build automated checks (row counts, checksum validation, schema drift detection) and integrate with data catalog solutions to ensure data accuracy and lineage.Collaboration & documentation: Partner with data scientists, analysts, and product teams to translate business requirements into reliable data solutions; maintain clear technical documentation and runbooks.I thrive on solving complex data challenges, enabling teams to unlock insights and drive better business outcomes through robust, production-grade data platforms.