Vidya P.
Data Engineer @Assurant | AWS | Azure | Python | SQL
- Role
- Data Engineer at Assurant
- Location
- Boston, MA, US
- LinkedIn followers
- 500 followers
About Vidya P.
4+ years of experience in developing spark applications,ETL and ELT pipelines in Azure,AWS,SSIS,Airflow.• Implemented and maintained 5+ RDBMS databases using MySQL, SQL Server, Postgresql, Oracle, and NOSQL like MongoDB,Cassandra, resulting in a 99% Uptime and minimal Downtime.• Experience in performing complex data Transformations,Validations,Cleansing,Modeling techniques.• Created Real-time streaming applications utilizing Kinesis,EventHubs,Kafka and Spark Streaming.• Utilized docker, kubernetes and Azure Devops services to build CICD Pipelines.
Experience
Data Engineer
May 2023 — Present · US
Orchestrated Azure Data Factory Pipelines for data ingestion, effectively migrated all tables from an on-premises SQL Server database to Azure Data Lake Storage Gen2.• Orchestrated workflows in Databricks and migrated data to delta tables and utilized delta live tables for real time data processing and loading.• Integrated Power BI with Azure SQL database to extract views, creating an engaging and interactive dashboard that effectively displayed and analyzed the data.• Integrated Azure Logic Apps and Azure Functions with Azure Monitor for real-time data pipeline monitoring and alerting, resulting in a 30% improvement in issue resolution time.• Scheduled workflows in Apache Airflow, ensuring seamless transformation and resolving 100% issues checking Airflow tools, logs, and corrective actions.• Designed snowflake modeling in data warehousing, improving data cleansing by 25%, optimizing slowly changing dimension, enhancing change data capture and data validation.
Education
UMass Boston
Master's degree, Computer Science
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.