Sudheer S

Senior Python Developer @JPMorganChase

CA, US

Designed and orchestrated complex data pipelines using Airflow DAGs (Directed Acyclic Graphs) to automate data ingestion, transformation, and loading (ETL) processes.• Utilized Spark SQL for data querying and analysis on distributed datasets stored in HDFS or data warehouses.• Developed a Spark streaming pipeline to batch real-time data, detect anomalies by applying business logic with 95% accuracy, and write the detected anomalies to an HBase table with a latency of under 5 seconds.• Establish Apache Flink streaming jobs to continuously ingest and analyze application logs, enabling faster detection of potential system issues and enhancing operational efficiency.• Execute data pipelines using Scala and PySpark (including libraries like NumPy and Pandas) to efficiently ingest, clean and transform large-scale datasets (structured, semi-structured, and unstructured) from various sources.• Streamlined data acquisition and integration from multiple sources, improving accuracy and completeness by 35%.

Andhra University

Bachelor's degree, Computer Science

Trine University

Master's degree, Information Science/Studies