Rakesh y

Senior Data Engineer @Aflac

Columbia County, GA, US

Proficient in using Pyspark for Big Data processing and analysis on large-scale datasets.• Experience in building ETL pipelines using Spark to extract, transform and load data from various sources such as Hadoop, Hive, HBase, etc.• Strong understanding of distributed computing and parallel processing concepts using Spark.• Experience in working with various Spark components such as Spark SQL, Spark Streaming, and Spark MLlib.• Hands-on experience in building machine learning models using Pyspark MLlib and implementing them in production systems.• Familiar with monitoring and profiling tools for Spark Streaming jobs, such as Spark UI, Ganglia, and JConsole, to identify performance issues and optimize resource allocation.• Collaborated with cross-functional teams to design and implement high-performance Spark Streaming pipelines for real-time data processing, ensuring data accuracy, reliability, and scalability.• Knowledgeable in SQL query optimization techniques, such as indexing, partitioning, and join optimizations, to improve Spark SQL performance and reduce query execution time.• Knowledgeable in using Kafka\'s messaging system to efficiently process and transport data in distributed environments.• Skilled in configuring and tuning Kafka and Spark to achieve optimal performance and scalability.• Familiar with various components of Kafka and Spark ecosystems, such as Kafka Connect, Kafka Streams, and Spark MLlib• Designed and developed a Snowflake data warehouse architecture for high-performance ETL and data integration.• Created complex ETL pipelines using Snowflake\'s Snow pipe feature to automatically load data into the data warehouse from various sources.• Developed bulk loading strategies using Snowflake\'s COPY command to efficiently load large amounts of data into the data warehouse.• Proficient in writing custom Airflow plugins to extend its functionality and provide custom integrations.