Veera Harish Pedapudi

About Veera Harish Pedapudi

I turn complex datasets into powerful, cloud-driven systems and actionable insights that…

Experience

1. Senior Data Engineer

   Onengine

   Jan 2022 — Present

   Designed and Maintained Scalable AWS Infrastructures: Ensured secure, efficient, and reliable data operations across managed services. Built Robust ETL Pipelines: Leveraged AWS Glue, Python, and Spark to integrate data from various sources into centralized data warehouses. Developed Advanced Applications: Used PySpark and Spark-SQL for data extraction, transformation, aggregation, and handling complex file formats. Implemented Data Engineering Best Practices: Expertise in data modeling, pipeline architecture, and warehousing. Managed Large-Scale Datasets: Proficient in big data tools like Hadoop, Hive, Pig, and HBase for structured and unstructured data. Built End-to-End ELT/ETL Pipelines: Extracted data from MySQL and AWS S3 and loaded into Amazon Redshift for actionable insights.🧩 Developed Reusable Data Models: Used DBT for automated transformations and integrated Glue metadata catalogs with Athena for simplified querying. Created Spark Applications: Designed real-time workflows and collaborated with data scientists to deploy machine learning models using scikit-learn and TensorFlow on AWS. Merged Datasets Using SQL and SAS: Executed complex joins, parsed XML data with Python, and automated workflows with AWS Step Functions and Lambda. Facilitated Cross-Cloud Operations: Integrated Azure Data Factory for seamless data transfer between AWS S3 and Azure Blob Storage. Built Interactive Dashboards: Utilized Tableau and Power BI to streamline decision-making and present actionable insights. Optimized Operations: Automated SAS executions on UNIX systems and deployed aggregated data through EC2 clusters. Focused on Scalability and Efficiency: Ensured seamless data pipeline integration, enabling organizations to unlock their full data potential.