Murali M
Senior Data Engineer at IBM | 8x GCP Certified | 4x AWS | 4x Azure | GCP, Python, SQL, Shell Scripting, AWS, Azure | Ex-Capgemini Consultant | Cloud & Data Engineering Professional
- Role
- Senior Data Engineer at IBM
- Location
- Bengaluru, KA, IN
- LinkedIn followers
- 500 followers
About Murali M
With over 6 years of dedicated experience as a GCP Data Engineer, I possess a strong…
Experience
Senior Data Engineer
Nov 2024 — Present · Bengaluru, IN
Client: American Express | Dec 2024 – PresentBuilt and deployed a PySpark-based data migration framework on GCP Dataproc to replace legacy Informatica pipelines, enabling scalable, distributed processing of critical datasets.Developed complex data transformation logic using PySpark and Spark SQL, including multi-table joins, filters, and aggregations, to replicate Informatica ETL logic with improved performance.Designed a robust data reconciliation engine using PySpark to ensure accurate validation between Informatica output and new Spark pipelines, ensuring high data quality and integrity.Performed schema validations, referential checks, and GDPR/PCI-DSS compliance audits before loading data into downstream systems.Automated end-to-end ETL orchestration using Cloud Composer (Apache Airflow) and Event Engine, enabling dynamic workflow triggering and monitoring based on data and system events.Ingested processed data into BigQuery, leveraging partitioning, clustering, and table optimization techniques for high-performance analytics and cost control.Achieved a 30% improvement in processing performance by tuning Spark configurations and optimizing partition strategies within the Dataproc environment.Collaborated with BI and data science teams to provide validated, analytics-ready data for reporting and dashboarding in BigQuery.Maintained clean, version-controlled codebases using GitHub, enforcing CI/CD practices, code reviews, and Git flow branching strategies.Managed and tracked user stories, sprint planning, and task progress using Rally (Agile Central) to ensure on-time delivery of key deliverables.Participated in unit testing, UAT cycles, and production deployments, working closely with QA and release management teams to ensure stable releases.Contributed to decommissioning legacy ETL systems and aligning engineering efforts with modern, cloud-native architectures using GCP services.
Education
SJB Institute of Technology
Bachelor of Engineering - BE, Electrical, Electronics and Communications Engineering
2015 — 2019
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.