Nagalakshmi J

Aws Data Engineer @Amway

Ada, MI, US

Developed and optimized Big Data solutions using Spark, Hive, and Pig to handle large-scale data processing, improving data ingestion and transformation workflows.Implemented Spark SQL and Spark Streaming for real-time data processing, enhancing the decision-making process by providing timely insights into sales and market trends.• Managed HBase databases to ensure efficient data storage and retrieval, which supported high-volume, real-time applications critical to business operations.Utilized Sqoop to efficiently transfer data between Hadoop and structured datastores, significantly reducing the time required for data synchronization.Designed and maintained scalable and reliable data pipelines using Kafka, AWS Data Pipe Lines, and AWS Glue, facilitating seamless data flow across diverse systems.Configured and managed AWS EC2 instances for deploying and scaling applications, optimizing resource allocation based on computational needs.Leveraged AWS S3 for secure and scalable storage solutions, implementing best practices for data lifecycle management and cost optimization.Employed Cloudera management tools to monitor and manage Hadoop clusters, ensuring high availability and performance of data services.Developed applications in Scala using Scala IDE (Eclipse), enhancing functionality and improving performance of data processing tasks.Wrote and maintained scripts using Linux Shell Scripting to automate routine data management tasks, increasing operational efficiency.Designed and implemented data models in HDFS to support scalable and distributed data architecture requirements.Utilized Python for scripting and automation of data processes, improving accuracy and efficiency in data transformation and analysis.Integrated Snowflake cloud data platform to consolidate different data sources for analytical reporting, enhancing business intelligence capabilities.