Divyanshu Shekhar
Big Data Engineer | Hadoop, Spark, Azure Expert | Empowering Data Teams & Transforming Business Insights
- Role
- Big Data Developer at LTIMindtree
- Location
- Hyderabad, TG, IN
- LinkedIn followers
- 500 followers
About Divyanshu Shekhar
At LTIMindtree, I work as a Big Data Developer, focusing on designing and implementing…
Experience
Big Data Developer
Jan 2023 — Present · Hyderabad, IN
Designed and developed ETL pipelines using PySpark and Scala on Azure Databricks to process large-scale structured and semi-structured datasets from diversesources including Azure Data Lake Storage and Azure Blob.Implemented data ingestion pipelines from source systems to Azure Cosmos DB (NoSQL) using Azure Data Factory and PySpark for high-speed data processing andnear-real-time analytics.Worked on Azure Data Services such as Azure Data Factory, Azure Synapse Analytics, AzureData Lake, and Azure Key Vault for secure,automated, and scalable cloud data solutionsOptimized Spark applications by tuningpartitioning strategies, caching, and managing memory for better performance and cost efficiency.Developed and managed complex transformations and aggregations on streaming and batch data using Databricks Notebooks, integrating with Power BI for end-to-end analytics solutions.Created CI/CD pipelines using Azure DevOps for continuous deployment of notebooks and data pipelines in production environments.Maintained data quality and governance using Delta Lake and Azure Purview, ensuring compliance and lineage across the data lifecycle.
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.