Sohin Raj

About Sohin Raj

Turning complex data into powerful decisions — that’s where I thrive.With 5+ years shaping cloud-first data platforms across Azure, AWS, and GCP, I build scalable pipelines that don’t just move data — they move businesses forward. Whether it\'s architecting real-time streams with Spark and Kafka, orchestrating workflows with Airflow, or migrating massive datasets into Azure Data Lake, Redshift, and BigQuery — I engineer solutions built for velocity, precision, and impact.Python, Scala, SQL, and machine learning frameworks like TensorFlow are part of my arsenal — but strategy is where the real craft lies. I align data engineering with business outcomes, delivering systems that fuel insights, innovation, and competitive edge.Data doesn’t just tell a story. It drives the future. I build the pipelines that make sure it’s a future worth betting on.

Experience

1. Data Engineer

   Visa

   Nov 2022 — Present · Austin, TX, US

   Configured high-availability EC2 instances using Terraform and CloudFormation, developing additional Terraform plugins for new feature support.• Designed and implemented Snowflake data warehousing solutions on AWS to enhance data processing speed.• Worked with business analysts and stakeholders to translate requirements into Snowflake data models and ETL pipelines.• Built and maintained data and ML pipelines using best practices for reliable processing and analysis.• Imported data from Hadoop HDFS, Apache HBase, AWS S3, and Amazon RDS into Spark RDD, using PySpark for computations.• Used Scala’s Spark SQL interface to optimize performance with custom aggregators, higher-order functions, and implicit conversions.• Migrated Java-based data processing code to Scala, improving performance and maintainability.• Used DBT for data quality testing (schema, referential integrity, and custom tests) and debugging query chains through modular models and macros.• Developed Scala-based ingestion modules with Apache Kafka for real-time data streaming and high-throughput Spark processing.• Designed a security framework for fine-grained access to AWS S3 objects and DynamoDB tables using AWS Lambda.• Ensured data integrity by cleaning and transforming data per business requirements.• Integrated AWS Step Functions with Lambda for iterative processing and DynamoDB status updates.• Developed RESTful APIs for seamless external system integration.• Designed and deployed an Enterprise Data Lake (EDL) for data storage, processing, analysis, and reporting using Amazon Quicksight• Conducted logical and physical data modeling for Amazon Redshift• Managed database loading, transformation, and exporting using SQL Server Integration Services (SSIS)• Configured Kerberos authentication for secure network communication, enabling HDFS, Hive, Pig, and MapReduce access in Hadoop clusters• Assessed and optimized AWS services, including Apache Airflow, Amazon EMR, Redshift, and AWS S3.

Education

• SYMBIOSIS INTERNATIONAL UNIVERSITY

  BBA.LLB

  2014 — 2019

• Indiana Wesleyan University

  Master's in Data Science

  2023