Anand Kushwaha
Data Engineer | Transforming Data with SQL, SQL SERVER, Azure, ADF, SSIS, PySpark
- Role
- Data Engineer at ZingMind Technologies
- Location
- Indore, MP, IN
- LinkedIn followers
- 500 followers
About Anand Kushwaha
I’m a Data Engineer with 5+ years of experience working in data engineering and…
Experience
Data Engineer
May 2021 — Present · Indore, IN
responsibility- • Develop data pipeline on data factory for migrate Tera byte of data. • Develop and schedule pipeline for dump data into data lake and migrate CDC data to data warehouse. • Create SSIS packages and deployed them. • Preform data validation from Source system to destination system manually and using scripts. • Deployed customize python/spark script and automate execution of use cases on data bricks. • Automate deployment and scaling of, on time delete and schedule clusters for cost saving. • Deploy Cozyroc ssis in IR for data migrationChallenges: 1. Loading fixed-width files with proper field alignment. Handling complexity when loading MongoDB JSON data into SQL tables.3.Converting and transforming MongoDB data to a relational schema. Ensuring data integrity and validation between source and destination. Optimizing performance when dealing with large datasets. Managing schema changes (schema drift) in MongoDB. Building error handling and logging in ADF pipelines. Automating and scheduling data loads efficiently. Optimizing cloud resource usage to control costs. Ensuring data security and compliance.
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.