Rishabh Vaidya

Data Engineer @AT&T

Dallas, TX, US

Collaborated with cross-functional teams to architect and implement scalable data warehouses, emphasizing data modeling and ETL pipelines.• Engineered efficient Python-based ETL pipelines, managing vast and diverse data sources with Pandas and Numpy.• Applied SQL expertise to ensure data quality, performing filtering, grouping, joining, and aggregation.• Leveraged Hadoop and Hive for distributed processing, optimizing data analysis performance.• Designed cloud-based data processing pipelines on AWS and Azure, spanning ingestion, processing, transformation, and loading.• Spearheaded real-time data processing using Apache Kafka and Spark Streaming for swift data ingestion and processing.• Employed Snowflake for high-performance data warehousing, facilitating real-time analytics.• Pioneered a real-time analytics platform with Kafka, Spark Streaming, and Snowflake integration.• Established robust security and access controls to ensure data privacy and compliance.• Collaborated with software engineers to integrate data pipelines, enabling real-time insights.• Proficiently utilized AWS services (S3, Redshift, EMR, Glue) and either GCP or Azure for scalable data solutions.

Minnesota State University, Mankato

Bachelor's degree