Venkata Manideep T

About Venkata Manideep T

Experienced IT professional with a decade of expertise in Data Engineering, Big Data, and Cloud Solutions across AWS, Azure, and GCP. I specialize in building robust, scalable data pipelines and cloud-based solutions across AWS, Azure, and Google Cloud Platform (GCP). My expertise lies in developing end-to-end data engineering solutions, with a strong focus on real-time and batch data processing using tools like Hadoop, Spark, Kafka, and PySpark. I have extensive hands-on experience with cloud services such as AWS S3, EMR, ECS, Lambda, and Databricks, enabling seamless data extraction, transformation, and loading (ETL) across a variety of data sources.Proficient in building real-time data streaming pipelines with Google Cloud Pub/Sub and AWS Kinesis, ensuring timely and efficient data processing. My strong DevOps skills include implementing CI/CD pipelines with Jenkins, using Docker for containerization, and leveraging Terraform for infrastructure automation.Adept at optimizing cloud costs and enhancing data models within Snowflake, as well as working with NoSQL databases like MongoDB, Cassandra, and HBase. My deep understanding of data warehouse design, logical and physical database models, and entity-relationship modeling allows me to drive impactful data solutions.Collaborating within Agile teams, prioritize innovation and performance optimization, using tools like Tableau, Power BI, and AWS QuickSight for data visualization and insights. Passionate about leveraging cutting-edge technologies, continuously strive to deliver high-quality, data-driven solutions that enhance business processes and decision-making.

Experience

1. Data Engineer

   Optum

   Sep 2022 — Present · Minneapolis, MN, US

   Worked with Hadoop 2.x and Spark 2.x (Python and Scala) for interactive queries and streaming data processing, integrating with NoSQL databases to handle large volumes of data- Developed custom ETL solutions and real-time data ingestion pipelines using Python and shell scripting, efficiently processing large datasets through Spark\'s partitioning, in-memory capabilities, and effective joins- Utilized Azure Databricks, Data Bricks automation tools, and APIs for orchestrating data pipelines, along with Google Cloud Dataflow and Pub/Sub for real-time analytics solutions- Leveraged Spark SQL for ETL tasks, extracting and transforming data from various sources to analyze customer usage patterns- Enhanced performance by optimizing algorithms in Hadoop using Spark Context, DataFrames, and Pair RDDs- Managed data transfer between Oracle databases and HDFS using Sqoop, and developed Spark jobs to clean data for Hive ingestion- Configured Oozie workflows for independent Hive jobs, utilizing HiveQL for reporting and trend analysis- Streamlined processes for a regional bank by configuring Hadoop ecosystem components and moving data to HDFS- Imported data from AWS S3 into Spark RDDs, performing transformations, and worked with Hortonworks Hadoop distribution- Created HBase tables for structured, semi-structured, and unstructured data, supporting project planning and reporting for BI teams- Assisted in data capacity planning, node forecasting, and migrated databases to SQL Azure, optimizing performance- Collaborated with infrastructure, network, and BI teams to ensure data quality, and designed ETL processes using Informatica to load data into Oracle Data Warehouse.