Hari M

Data Engineer @Core Specialty Insurance Holdings, Inc.

NJ, US

Created Sqoop scripts to import/export user profile data from RDBMS to S3 data lake.• Developed various spark applications using PySpark to perform various enrichments of user behavioral data (click stream data) merged with user profile data.• Fine-tuning spark applications/jobs to improve efficiency and overall processing time for the pipelines.• Designed, implemented, and maintained data pipelines used for efficient transformation and orchestration of data within AWS cloud environments.• Developed Spark-Streaming applications to consume the data from Kafka topics and to insert the processed streams into Cassandra.• Integrated DBT seamlessly with AWS services such as Amazon Redshift, Amazon S3, and AWS Glue, optimizing data processing tasks for scalability, reliability, and cost-effectiveness.• Proficient in Apache Spark for distributed data processing and transformation, with hands-on experience using PySpark within AWS Glue.• Developed data pipelines to transform raw data and dumb data into s3 buckets based on business requirements.• Monitor Apache Airflow scheduler, executor, and worker performance metrics to identify bottlenecks and optimize resource utilization.• Utilized Spark in Memory capabilities, to handle large datasets. • Experienced in working with EMR cluster and S3 in AWS cloud. • Integrated AWS Lambda with other AWS services such as S3 for data storage, DynamoDB for NoSQL database storage, and Kinesis for streaming data processing.• Creating Hive tables, loading and analyzing data using hive scripts. Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.• Integrated AWS Glue with other AWS services such as Amazon S3, Amazon Redshift, Amazon RDS, and Amazon Athena to enable seamless data movement and analytics workflows.• Extensive experience with AWS Athena for ad-hoc querying and analysis of data stored in Amazon S3, using standard SQL syntax.