Abhi

About Abhi

With over 9 years of dedicated experience in Data Engineering, I possess a comprehensive skill set spanning various cloud platforms including GCP, AWS, and Azure. My expertise lies in the Big Data Ecosystem, adeptly utilizing technologies such as Hadoop, Map Reduce, Pig, Hive, and Spark. Throughout my career, I\'ve been involved in all phases of project development, from initiation to implementation, ensuring robust solutions at every stage. My proficiency extends to AWS Cloud Computing services, encompassing EC2, S3, EBS, VPC, ELB, Route53, CloudWatch, Security Groups, EKS, IAM, CloudFront, RDS, and Glacier. Additionally, I have extensive experience in Google Cloud Platform (GCP) and its suite of big data products including BigQuery, Cloud DataProc, Google Cloud Storage, and Composer. I excel in SQL concepts, Presto SQL, Hive SQL, Python (Pandas, NumPy, SciPy), Scala, and Spark, enabling me to handle large volumes of data efficiently. I\'m skilled in Shell/Bash scripting and proficient in building data pipelines on Unix/Linux systems. My experience with Hortonworks on-premises Hadoop distribution and GCP Dataproc further strengthens my capabilities in big data management. Throughout my career, I\'ve demonstrated expertise in continuous integration and deployment using Terraform, ensuring scalable infrastructure deployments. I\'ve also implemented Big Data integration solutions with Talend, catering to diverse data sources and formats. Furthermore, I bring hands-on experience in data visualization using tools like Tableau, complementing my proficiency in SQL for dashboard creation and insightful recommendations. With a keen understanding of Quality Assurance processes and SDLC, I have consistently delivered high-quality solutions aligned with business objectives.

Experience

1. Gcp Data Engineer

   Mastercard

   Mar 2022 — Present

   Developed ETL pipelines on GCP using Apache Beam and Dataflow to process large-scale data in real-time, resulting in a 20% improvement in data processing time.* Built and deployed data pipelines using Cloud Composer and Cloud Functions, enabling seamless integration with other GCP services such as BigQuery, Pub/Sub, and Cloud Storage.* Developed and maintained JIL scripts that triggered and monitored file transfer tasks, reducing manual intervention by over 40%.* Designed and implemented real-time monitoring dashboards in Grafana to track key performance indicators (KPIs) for data pipelines and cloud infrastructure.* Experience in using Stackdriver service/Dataproc clusters in GCP for accessing logs for debugging.* Designed and executed end-to-end testing strategies for GCP data pipelines, ensuring the accuracy and completeness of data from ingestion to analysis.* Utilized DevOps practices and tools such as Jenkins, Terraform, and Ansible to automate GCP infrastructure deployment and configuration, resulting in a 50% reduction in deployment time.* Worked with Python, SQL, and Bash scripts to develop custom data transformations and data quality rules, resulting in a 25% reduction in data processing errors.* Integrated Grafana with Prometheus, Cloud Monitoring (GCP) to visualize system performance and optimize resource utilization.* Experience in GCP Dataproc, GCS, Cloud functions, BigQuery, Azure Data Factory, and DataBricks.* Automated file transfer processes with custom Python/Perl scripts to interface between Autosys and Connect:Direct, resulting in improved scheduling accuracy and reduced error rates.* Developed custom Grafana dashboards to monitor ETL jobs, data pipeline health, and application logs, ensuring proactive issue resolution.* Configured alerting mechanisms in Grafana using email, Slack, and webhook notifications to enable real-time anomaly detection.

Education

• Vel Tech Rangarajan Dr. Sagunthala R & D Institute of Science & Technology, Chennai

  Bachelor's degree

  2010 — 2014