Kesav R.

Experience

1. Data Engineer Ii

   NextEra Energy Resources

   Oct 2022 — Present · Phoenix, AZ, US

   Designed and implemented batch data ingestion pipelines to efficiently process files received from upstream systems, ensuring timely and accurate data availability.• Developed robust PySpark applications to clean, transform, and process large datasets, handling a volume of over 1 million transactions per day ensuring high performance and scalability for complex transformations.• Automated the scheduling and orchestration of Spark jobs using Apache Airflow, ensuring seamless data pipeline workflows.• Stored processed data in distributed systems like Apache Cassandra, Elastic search, and HDFS, optimizing for both high availability and low latency access.• Optimized Spark jobs, reducing runtime by 43%, improving processing efficiency and enabling faster insights• Designed and implemented changes to the Cassandra schema and PySpark application to handle transaction loss, ensuring accurate transaction counts and amounts. Developed robust data validation mechanisms to track and reconcile transactions, significantly improving data integrity and reliability in transaction processing.• Generated reporting files from processed data and automated delivery to downstream systems via SFTP, ensuring accuracy and compliance with service-level agreements (SLAs).• Integrated pipeline metrics and data outputs into Prometheus, enabling real-time monitoring and alerting to track performance and identify issues proactively.• Enabled visualization of pipeline metrics and processed data in Grafana dashboards, enhancing system observability and facilitating data-driven decisions.