Abhishek Tiwari

About Abhishek Tiwari

I am an AWS Data Engineer with hands-on expertise in designing and automating scalable data platforms that power analytics, reporting, and business decision-making. I specialize in building and optimizing data lakes, ETL pipelines, and data warehouse environments using both AWS-native services and Cloudera Data Platform (CDP), bringing a unique blend of cloud and big data skills.In my current role, I have: • Implemented and configured CDP DataLake and DataHub using CloudFormation templates, enabling faster provisioning and standardized deployments. • Optimized Cloudera Data Warehouse (VDW) workloads and reduced costs by tuning EC2 and RDS instances and introducing Grafana-based load monitoring. • Automated daily snapshot creation with AWS Lambda, strengthening data backup and recovery processes. • Developed and enforced Ranger security policies, renewed TLS/SSL certificates, and conducted Ranger audits to ensure compliance and secure data access. • Configured production Spark and Airflow clusters to support large-scale data pipelines, while proactively troubleshooting job failures and improving reliability. • Partnered with vendors to resolve Kubernetes vulnerabilities and streamlined cluster upgrades and testing, ensuring stable production performance.Beyond infrastructure, I am passionate about data engineering best practices—from building reliable ETL/ELT pipelines in Python, PySpark, and SQL to managing bucket-level access policies and JDBC/ODBC connectivity for analytics teams.

Experience

1. Data Eng,mgmt & Governance Analyst

   Accenture

   Sep 2023 — Present · Bengaluru, IN

   Designed and optimized AWS Glue jobs to automate ETL pipelines for structured/semi-structured data, reducing pipeline runtime by 30%. • Automated daily backups and data protection with AWS Lambda, ensuring 99.9% data availability and faster recovery. • Optimized AWS EC2 and RDS instances, improving resource utilization by 20% and lowering monthly infrastructure costs. • Implemented CDP DataLake and DataHub using CloudFormation templates, accelerating provisioning and standardizing deployments. • Managed Cloudera VDW with cost monitoring tags, achieving 10–15% cloud cost savings. • Configured and scaled CDP clusters to support Spark and Airflow pipelines, improving reliability and throughput. • Troubleshot and resolved workflow issues, boosting cluster uptime to 99%. • Developed Ranger policies for databases and S3, strengthening security and compliance. • Conducted Ranger audits to resolve user access issues, reducing escalations by 25%. • Renewed TLS/SSL certificates, mitigating security risks and avoiding downtime. • Established JDBC/ODBC connectivity for BI and visualization teams, enabling seamless reporting. • Applied vendor hotfixes and patches, collaborating on vulnerability mitigation to ensure compliance. • Troubleshot Kubernetes pod failures and created Grafana dashboards for VDW monitoring, enhancing system stability. • Configured termination protection and NLBs for CDP nodes, improving resilience and preventing disruptions. • Automated project-specific S3 bucket access requests to ensure secure, auditable operations. • Led CDP cluster testing and upgrades, minimizing downtime and ensuring smooth transitions.

Education

• Kalinga Institute of Industrial Technology, Bhubaneswar

  Bachelor's degree

  2016 — 2019

• KIIT - Kalinga Institute of Industrial Technology

  Master of Computer Applications - MCA

  2019 — 2021