Sudhir Saxena
Senior Data Engineer @Reveal Global Consulting
Signup · Get unlimited contacts
WORK HISTORY
Senior Data Engineer @Reveal Global Consulting
Working on Data modernization from Legacy system to AWS cloud project to process huge amount of data which was loading into AWS redshift cloud Datawarehouse. Carried out the following activities: • Understand business requirements from Client, created source to target mapping to design Data pipeline through PySpark to fetch the data from S3 bucket where Upstream places the API joson file on daily basis as batch process. • Designed and implemented end-to-end data pipelines using AWS Glue, Lambda, S3, Redshift, and Athena, enabling real-time analytics for enterprise applications, reducing ETL processing time by 40%.• Led a team of 5 data engineers in developing scalable data lake architectures with AWS Lake Formation, improving data governance and security compliance.• Migrated on-premises data warehouse (Teradata) to AWS Redshift, handling over 30TB of data with zero downtime and achieving 60% cost savings on infrastructure.• Developed CI/CD workflows using AWS CodePipeline and Terraform for infrastructure as code (IaC), reducing manual deployment errors by 90%.• Implemented enterprise-wide data cataloging using AWS Glue Data Catalog, improving data discoverability and access for analytics teams.• Integrated third-party data sources via REST APIs and automated ingestion with Python and AWS Lambda.• Leading the design and implementation of cloud-native architectures using core AWS services such as EC2, EMR, Glue, S3, Glue, Lambda, RDS, ECS, Step Function, Redshift, CloudFormation, and IAM. End-to-end design.
EDUCATION
College of Engineering , Guindy, Chennai
MCA, Computer Application
BCA from IGNOU, Delhi
Bachelor of Computer Application, Computer Programming, Specific Applications
SKILLS
ABOUT SUDHIR SAXENA
Seeking a position to utilize my skills and abilities in the Information Technology Industry that offers professional growth while being resourceful, innovative and flexible.• Having 15+ years of extensive experience in Hadoop ecosystem, Bigdata, Pyspark, Spark, AWS, GCP, Big Query, Dataproc, Data Warehousing, ETL tool, Business Intelligence, Data Analytics, Data Integration, Implementation and Maintenance of BI and Database Platforms.• Leading the design and implementation of cloud-native architectures using core AWS services such as EC2, EMR, Glue, S3, Glue, Lambda, RDS, ECS, Step Function, Redshift, CloudFormation, and IAM. End-to-end design, development, and implementation of software systems, ensuring high performance, scalability, and security.•Worked extensively in Hadoop ecosystem, Bigdata, Pyspark, Python, Spark, AWS, Redshift, Glue, Lambda, Warehousing, ETL tool, Business Intelligence, Data Analytics, Data Integration, Implementation and Maintenance and the related Database Platforms. •Worked as technical lead to play a key role in software development team, make high-level design and architecture decisions, review code for quality, defect fixing during QA and prod phase, performance, and maintainability. •Worked closely with project managers and product owners to break down features and tasks, estimated timelines and help prioritize work, worked on end-to-end project completion, review all team member work, track progress and remove technical blockers.• Expertise in data Ingestion, data loading, data processing, data transformation, Batch Processing and Stream Processing•Implemented best practices for cloud security, cost optimization, high availability, and disaster recovery. •Collaborated with cross-functional teams including DevOps, Security, and Application Development to design end-to-end solutions aligned with business goals.•Setting up and managing CI/CD pipelines using AWS CodePipeline, CodeBuild, and third-party tools like Jenkins and GitLab CI/CD.•Created framework which pulls data from AWS S3 bucket, implement all the transformation using pyspark, CDC logic using apache hudi, and load the data into Redshift. •Leading and mentoring a team of developers and cloud engineers in delivering cloud-based projects, ensuring adherence to AWS well-architected frameworks. •Ensuring compliance with governance and regulatory requirements through proper implementation of identity and access management, encryption, and data protection strategies.
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.