Veena Rao

About Veena Rao

Over 8 years of experience in software analysis, datasets, design, development, testing, implementation of Cloud, Big Data, Big Query, Spark, Scala, and Hadoop. Expertise in Big Data technologies, Data Pipelines, SQL, and Cloud based RDS, Distributed Database, Server less Architecture, Data Mining, Web Scrapping, Cloud technologies like AWS EMR, Cloud Watch. Experience in implementing ML Algorithms using distributed paradigms of Spark/Flink, in production, on Azure Databricks/AWS Sage maker Hands on experience in designing and implementing data engineering pipelines and analyzing data using Hadoop ecosystem tools like HDFS, Spark, Sqoop, Hive, Flume, Kafka, Impala, PySpark, Oozie and HBase. Reliable Data Engineer keen to help companies collect, collate and exploit digital assets. Skilled administrator of information for Azure services ranging from Azure data bricks, Azure relational database and non-relational database, and Azure data factory and cloud services. Practiced at cleansing and organizing data into new, more functional formats to drive increased efficiency and enhanced returns on investment. Experience in implementing E2E solutions on Big Data using Hadoop framework, executed, and designed big data solutions on multiple distribution systems like Cloudera (CDH3 & CDH4), Hortonworks. Strong knowledge in writing PySpark UDF, Generic UDF\'s to incorporate complex business logic into data pipelines.

Experience

1. Senior Data Engineer

   Vanguard

   Apr 2021 — Present · Dallas, TX, US

   Developed Spark Applications by using Scala and Implemented Apache Spark data processing project to handle data from various RDBMS and Streaming sources.• Worked with the Spark for improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Spark MLib, Data Frame, Pair RDD\'s, Spark YARN.• Used Spark streaming to receive real time data from the Kafka and store the stream data to HDFS using Scala and NoSQL databases such as HBase and Cassandra.• Imported data from AWS S3 into Spark RDD, Performed transformations and actions on RDD\'s.• Used Kafka for live streaming data and performed analytics on it. • Loaded data from Web servers and Teradata using Sqoop, Flume and Spark Streaming API. • Utilized DBT to build scalable and modular data models for analysis and reporting. Developed SQL transformations and defined business logic using DBT\'s templating capabilities, ensuring data consistency and maintainability.• Used various AWS services including S3, EC2, AWS Glue, Athena, RedShift, EMR, SNS, SQS, DMS, Kinesis. • Leveraged Kinesis Data Analytics for real-time data processing and analysis.• Developed SQL-based applications within Kinesis Data Analytics to gain insights from streaming data.• Installed and configured Apache Airflow for S3 bucket and Snowflake data warehouse and created Dags to run the Airflow.• Worked on monitoring, scheduling, and authoring Data Pipelines using Apache Airflow.• Created interactive and visually appealing dashboards using Tableau, presenting key insights and trends to stakeholders, and driving data-informed decision-making. • Created AWS Data pipelines using various resources in AWS including AWS API Gateway to receive response from AWS Lambda and retrieve data from Snowflake using Lambda function and convert the response into JSON format using database as Snowflake, DynamoDB, AWS Lambda function and AWS S3.

Education

• Jawaharlal Nehru Technological University

  Bachelor's degree, Information Technology