Durga Bhavani

Data Engineer @Coreits Llc

South Plainfield, NJ, US

Experience in building and architecting multiple Data pipelines, end to end ETL and ELT process for Data ingestion and transformation. • Worked extensively with AWS services like EC2, S3, VPC, App flow, ELB, Auto Scaling Groups, Route 53, IAM, CloudTrail, CloudWatch, CloudFormation, CloudFront, SNS, and RDS. • Design, Development, Testing, and implementation of ETL processes using Informatica Cloud.• Worked on managing and reviewing Hadoop log files.• Developed BI data lake POCs using AWS services including Athena, s3, EC2 and AWS Quick sight.• Integrating data from source system which include nested JSON formatted data into snowflake table.• Build the logical and physical data model for snowflake as per the changes required.• Experience in developing and deploying a Microservice application utilizing almost of the AWS Stack including s3, EC2, DynamoDB.• Used Data Bricks to perform complex mathematical operations, manipulate large datasets, and optimize their code for performance.• Deploying, and managing AWS Database Services like Amazon RDS, Aurora, DynamoDB, and others for storing and managing data using Django.• Experience with DBT (Data Build Tool), a tool for transforming and aggregating data in a data warehouse.• Collaborated with data engineers and analysts to design and implement end-to-end data solutions using DBT • Building data pipeline ETLs for data movement to S3, then to Redshift. • Experience in Data modeling (Dimensional and Relational) concepts like star-schema data modeling, Snowflake schema data modeling and Fact-Dimension tables.• Worked on Data Extraction, aggregations, and consolidation of Adobe data within AWS Glue using PySpark, Data Bricks• Developed spark programs and created the data frames and worked on transformations. • Extensively used spark SQL and Data frames API in building spark applications. • Developed the Spark code for AWS Glue jobs and for EMR using Pandas.