Vaibhav Rai
Data Engineer at IBM | Ex-Infosys | Big Data & ETL | Meltano ELT tool | Python | PySpark | Hive | Hadoop | AWS | GCP | PostgreSQL | SQL | Docker | Kubernetes | Pub/Sub | Dataform
- Role
- Senior Data Engineer at IBM
- Location
- Gurugram, HR, IN
- LinkedIn followers
- 500 followers
About Vaibhav Rai
SKILLS Programming Languages: Python, Bash, SQL, Cobol Databases: MySQL, Oracle, PostgreSQL, Redshift, BigQuery Libraries: PySpark, pandas, NumPy, matplotlib Project Management Tools: Confluence, Git, Jira Cloud: AWS, GCP ETL Tools: Meltano Additional Technologies: Docker Databricks (Delta Live Tables ETL Framework, Databricks Utilities (Widgets, File system, Mounts)) AWS Services: AWS EC2, AWS S3, AWS Lambda GCP Services: Cloud Run, BigQuery, Pub/Sub, Storage, Composer, DataformEXPERIENCEBig Data EngineerOct 2022 - PresentInfosys, Gurgaon Automated ETL processes using Meltano ELT tool for data extraction from MSSQL and loading into BigQuery, ensuring efficient and accurate data transfer. Developed and containerized the ETL pipeline using Docker, storing images in Artifact Registry and deploying via Cloud Run for serverless execution. Created Python scripts to generate Apache Airflow DAG files, managed via Google Composer, facilitating automated scheduling and orchestration of ETL tasks. Uploaded DAG files to Google Cloud Storage and integrated with Apache Airflow, enhancing workflow automation and task management. Led migration initiative, orchestrating the transition from legacy Mainframe Cobol technologies to cutting-edge ETL solutions within Databricks Medallion Architecture. Developed robust PySpark scripts to automate the execution of the Databricks DLT pipeline, facilitating the seamless transformation of Cobol and Easytrieve files into PySpark and DLT SQL formats.Big Data EngineerAug 2021 - Oct 2022NITS Solutions Spearheaded migration projects, transitioning ETL pipelines from AWS S3-Oracle-React to AWS S3-Python with PostgreSQL-React. Analyzed summary tables using PySpark for enhanced insights. Developed an API in Java and Apache Spark technologies to accumulate data from different sources provided by the client. Created ETL processes in Python (Pandas), PostgreSQL, and Shell Script to process records on a scheduled basis through cron job. Worked on Agile Methodology.Software DeveloperDec 2018 - Aug 2021Indian Agriculture Statistics Research Institute (IASRI) Created Python pandas scripts to analyze a huge agricultural dataset. Worked on ETL pipeline from ingesting data into MySQL database and analyzing data with the help of Python pandas and matplotlib library. Worked on Agile Methodology. Used Git for version control.
Experience
Senior Data Engineer
Nov 2024 — Present · Gurugram, IN
Education
Bhai Parmanand Institute of Business Studies
Master of Computer Applications, Computer Software Engineering
2014 — 2017
Ivy Professional School
Certification, Big Data and Analytics
2020 — 2021
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.