Meghana A

About Meghana A

Senior Big Data Engineer with over a decade of experience in the IT industry, I have honed my skills in leveraging Big Data technologies to drive data-driven solutions across various sectors. My expertise spans Cloudera and Hortonworks distributions, including Hadoop, Spark, MapReduce, Kafka, Hive, Ambari, Sqoop, HBase, and Impala. Proficient in Python, Scala, and Java, I excel in designing, developing, and supporting large-scale distributed systems.Throughout my career, I have successfully implemented scalable cloud-based web applications using AWS and Azure, developed real-time data processing solutions, and optimized data pipelines for performance and reliability. My experience includes working with NoSQL databases like Cassandra, HBase, and MongoDB, as well as creating custom UDFs in Hive and MapReduce jobs.I have a strong track record in data migration, data lake management, and implementing end-to-end data security and governance within Hadoop platforms. My ability to translate complex business requirements into effective Big Data strategies and solutions has enabled me to contribute significantly to the success of my clients and organizations.I am passionate about utilizing cutting-edge technologies to solve complex data challenges and drive innovation in data engineering. Let\'s connect and explore how we can collaborate to leverage Big Data for impactful results.

Experience

1. Sr Big Data Engineer

   Experian

   Oct 2023 — Present · Costa Mesa, CA, US

   As a senior data engineer, I implemented Kafka brokers for initiating Spark context and processing live data streams. I developed Scala scripts and UDFs for data aggregation and querying in Spark, and wrote data back to OLTP systems using Sqoop. I managed HBase for data storage and analysis and collaborated with the Kafka Admin team to set up Kafka clusters, implementing producer and consumer applications with Zookeeper.I extracted data from relational databases using Sqoop and employed PySpark and Pandas for stock analysis. I developed Hadoop MapReduce jobs in Java and Scala, supported applications with HBase/Phoenix, and designed Oozie workflows for Sqoop and Hive job scheduling. I optimized Hadoop algorithms using Spark, and developed Talend ETL jobs for data migration and integration.I architected serverless CI/CD pipelines using AWS Lambda, created stored procedures in Snowflake, and structured data into AWS S3 and Redshift. I migrated Hive QL to Impala, conducted various testing phases for Hadoop applications, and developed custom Kafka producers and consumers. I used Spark Streaming for real-time data processing, performed data preprocessing for predictive analytics, and generated reports using Python and Tableau.Throughout my role, I applied Agile methodologies, utilized Docker for deployment, and leveraged AWS services for serverless computing.

Education

• Osmania University

  Bachelor's degree, Computer Science

  2009 — 2013