Teja R.

About Teja R.

Around 9+ years of Professional experience as a Data Engineer and Big Data Developer with expertise in, Python, Hadoop, Spark etc. Development, Implementation, Deployment and Maintenance using Bigdata technologies in designing and implementing complete end-to-end Hadoop based data analytics solutions using HDFS, MapReduce, Spark, Scala, Yarn, Kafka, PIG, HIVE, Sqoop, Flume, Oozie, Impala, HBase. Experience in using Azure Cloud, Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Azure Analytical services, Azure Cosmos DB, NOSQL DB, Azure HDInsight, Big Data Technologies (Hadoop and Apache Spark) and Databricks. Experience on Migrating SQL database to Azure data Lake, Azure SQL Database, Data Bricks and Azure SQL Data warehouse and controlling and granting database access and Migrating On premise databases to Azure Data Lake storage using Azure Data factory. Strong Experience in Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory and T-SQL. Eexperience with Amazon EC2, Amazon S3, Amazon RDS, VPC, IAM, Amazon Elastic Load Balancing, Auto Scaling, CloudWatch, SNS, SES, SQS, Lambda, EMR and other services of the AWS family. Experience of designing, building, and deploying data pipelines using tools using BigQuery, Cloud Dataflow, Cloud Proc, Cloud Pub/Sub, Cloud Composer, Google Data Studio, Google Cloud Storage Experienced in handling various types of data (Structured, Semi-Structured, Unstructured) coming from various sources with both batch and real time data pipelines. Experienced in handling large datasets using partitions, Spark in-memory capabilities, Broadcasts in Spark, effective & efficient Joins, Transformation and other during ingestion process itself. Expertise developing iterative algorithms using Spark Streaming in Scala and Python to build near real-time dashboards. Developed and designed a system to collect data from multiple portals using Kafka and process it using Spark. Designed and implemented Kafka by configuring Topics in new Kafka cluster in all environments. Experience working on various file formats including delimited text files, clickstream log files, Apache log files, Parquet files, Avro files, JSON files, XML files and others. Hands on Experience in implementing and orchestrating data pipelines using Oozie and Airflow.

Experience

1. Data Engineer

   Ford Motor Company

   Apr 2022 — Present · Dearborn, MI, US

   Designed and developed data pipelines using Swagger, incorporating metadata such as source connection info,source columns, target table info, target columns, app info, and domain info to ensure efficient and accuratedata transfer. Created YAML files for each pipeline, stored in S3 and loaded using AWS Glue, to load data based on the pipelinemetadata. Developed an ingestion framework to load batch data from various sources, including CSV files, S3 buckets, andKafka topics, using functions to read, write, join, and flatten tables and to handle nested JSON data. Utilized AWS services such as S3, DynamoDB, Athena, and Secrets Manager to store and manage data, pipelinerecords, execution details, and database credentials. Developed transform code to process data loaded into staging tables using AWS Glue, and incorporated bothstaging and transform jobs into Step Functions for scheduling and archival. Utilized tools such as Dbeaver and pgAdmin to query source and target tables. Followed agile methodology for user stories, tasks, bugs, and spikes to ensure efficient and effectivedevelopment and delivery of data pipelines.

Education

• Osmania University, Hyderabad

  Bachelor's degree

  2009 — 2013