Vijay Gupta
Lead Data Engineer/Lead Solution Engineer/Data Architect @Nielsen
- Role
- Solution Engineer Data Architect at Nielsen
- Location
- Charlotte, NC, US
- LinkedIn followers
- 500 followers
About Vijay Gupta
o Over 16 years of total IT experience with 13 years in United States working on ETL…
Experience
Solution Engineer Data Architect
Dec 2021 — Present · US
o Designed and developed Airflow pipelines to process audience demographics data, Nielsen panel data using python, spark, SQL.o Built an ML classification model to predict potential solutions to upcoming airflow Dags failures based on patterns in the log. Transforming this model in to Generative AI model using Amazon bedrocks models, fine tuned them.o Designed and developed pipeline using time-series data, captured AWS resource utilization such as CPU & memory utilization and utilization ratio, reconfigure underutilized clusters for numerous pipelines across organization, saved AWS cost by 50%.o Automated solution to extract ticket information from GitLab & generate tracking reports, improving productivity and visibility.o Developed an automated solution to generate weekly aggregate cost reports by stack, process, run ID, and leader etc, providing leadership with greater cost visibility and accountability.o Analyze daily, weekly, and monthly AWS service costs using Cost Explorer to identify opportunities for cost reduction.o Optimize multiple data pipelines (Identity, Digital ads rating, Dag rest of web and Nielsen One Alpha products) that process huge amounts of data, provided many dynamic configurations to reduce AWS resources.o Created different cluster configurations such as child inheriting parent cluster, distributing clusters for parallel tasks, spawning clusters according to data size or specific parameter/’s in job requests and split loads into multiple runs. Created well tested T-shirt size spark templates. Additionally, created wiki pages to leverage these configurations.o Created Python package to interact with Google Sheets, Postgres, and presto/Trino, simplified data ingestion into MDL.o Migrated pipelines from Spark 2.4 to Spark 3.0/3.2, documenting issues & solutions in detailed wiki pages for future reference.
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.