Dash T

About Dash T

As a seasoned Data Scientist with over 9 years in IT, I specialize in ETL development, leveraging tools like Hadoop, Spark, and RDBMS. My expertise extends to cloud platforms (AWS, Azure, GCP), programming in Python, Unix, and Java, and enhancing Hive and Pig with custom functions. I’ve crafted data pipelines with Nifi, Kafka, Pubsub, Oozie, and Airflow, and managed cloud-based data workflows and migrations.I’m adept with Informatica, Cloudera, Hortonworks, and NoSQL databases (Cassandra, MongoDB), and have facilitated data analytics using Apache Solr. My role often involves data import/export tasks using Sqoop, crafting queries in Spark SQL, Hive, and Pig Latin, and working with Hadoop ecosystem components and AWS services to monitor and schedule jobs.My strong communication skills aid in liaising with both technical and business teams, and I’m well-versed in SDLC projects, using tools like JIRA, Play, and GIT for effective coordination and project management.

Experience

1. Senior Big Data Engineer

   Broadridge

   Jul 2021 — Present · TX, US

   Worked on Ingesting data by going through cleansing and transformations and leveraging AWS Lambda, AWS Glue and Step Functions• Created monitors, alarms, notifications and logs for Lambda functions, Glue Jobs, EC2 hosts using CloudWatch.• Worked on EMR clusters of AWS for processing Big Data across a Hadoop Cluster of virtual servers.• Developed various Mappings with the collection of all Sources, Targets, and Transformations using Informatica Designer• Created and maintained the tables/views in Snowflake data warehouse for downstream consumers.• Worked on Docker containers snapshots, attaching to a running container, removing images, managing Directory structures, and managing containers.• Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary Transformations and Aggregation on the fly to build the common learner data model and persists the data in HDFS. Data engineering using Spark, Python and Pyspark.• Designed both 3NF Data models and dimensional Data models using Star and Snowflake Schemas Handling message streaming data through Kafka to s3.• Implementing python script for creating the AWS Cloud Formation template to build EMR cluster with instance types. Used Apache NiFi to copy data from local file system to HDP.• Developed highly complex Python and PySpark code, which is maintainable, easy to use, and satisfies application requirements, data processing and analytics using inbuilt libraries.• Experience with deploying Hadoop in a VM and AWS Cloud as well as physical server environment.• Monitor Hadoop cluster connectivity and security and File system management.• Developed Mappings using Transformations like Expression, Filter, Joiner, and Lookups for better data messaging and to migrate clean and consistent data.• Data replacement and appending to the Hive database via pulling it using Sqoop processing tool to HDFS database from multiple data marts as source.

Education

• Rajiv Gandhi Prodyogiki Vishwavidyalaya

  Bachelor's degree

  2010 — 2014