Reddy Jithendhar

Experience

1. Senior Data Engineer

   TD

   Aug 2023 — Present · Cherry Hill, NJ, US

   Developed real-time data ingestion solutions from AWS S3 using Spark Streaming with data transformation, aggregation, and storage in HDFS. Used AWS Glue crawlers to extract metadata from S3 and perform SQL query operations. Engineered data integration pipelines to transfer data from S3 to Redshift using Python and AWS Glue. Maintained a Hadoop cluster on AWS EMR, leveraging EC2 and S3 for processing and storing structured data. Automated workflow management using Python and Apache Airflow, including task dependencies and time-based execution. Built Oracle APEX applications to provide secure, web-based access to Oracle data warehouses and automate internal ETL workflows. Developed Python-based ETL pipelines using pandas, openpyxl, paramiko, pyyaml and boto3 for data ingestion, transformation and cloud integration. Designed dimensional data models and star schemas in Oracle for enterprise reporting and analytics. Deployed and managed data solutions on AWS including S3, RDS, Lambda, Glue and Redshift. Built and optimized data pipelines on Google Cloud Platform (GCP) using BigQuery and Google Cloud Storage. Implemented Git-based CI/CD pipelines and followed Agile SDLC practices for data engineering projects. Created interactive executive dashboards using AWS QuickSight connected to Redshift. Integrated React dashboards with AWS Lambda APIs and Chart.js for real-time data visualization. Stored transformed time-series data from Spark (via Hive) to Amazon S3 and Redshift. Imported/exported data between DB2, SQL Server, Teradata, and HDFS using Apache Sqoop. Managed Kafka cluster configurations including Brokers, Topics, and implemented Kafka Consumers, Producers, and Streams API. Created branching strategies while collaborating with peer groups and other teams on shared repositories. Developed various interactive reports using Power BI based on client specifications with row-level security features.