Charul Gupta
Lead Data Engineer @Smbc Group - Sumitomo Mitsui Finance & Leasing
Signup · Get unlimited contacts
WORK HISTORY
Lead Data Engineer @Smbc Group - Sumitomo Mitsui Finance & Leasing
US
As a Lead Data Engineer at SMBC, I spearhead the architecture and implementation of our next-generation real-time data platform on Databricks. I am building low-latency, scalable streaming pipelines to power critical business intelligence, drive real-time to enhance customer experiencesMy core focus includes:Lead the design and implementation of a company-wide real-time data streaming platform on the Databricks Lakehouse, enabling sub-second analytics for key business functions.Architected and manage high-throughput, fault-tolerant data ingestion pipelines from diverse sources using Apache Kafka and Azure Event Hubs, ensuring 99.9% data availability.Engineered and optimized Spark Structured Streaming jobs within Databricks to process terabytes of real-time data, reducing insight latency from hours to seconds.Implemented a robust DataOps framework using GitHub Actions for CI/CD, automating the testing, validation, and deployment of data assets (DBT models, Spark jobs) across environments.Championed a Git-based workflow for all data engineering code, improving collaboration, version control, and rollback capabilities for the entire data team.Manage and mentor a team of 5 data engineers, conducting code reviews, setting technical roadmaps, and driving best practices in data architecture, coding standards, and performance tuning.
EDUCATION
Hindusthan College of Engineering and Technology
MCA, Computer Science
Hindustan Institute of Technology & Management
Master of Computer Applications - MCA, Computer Science
SKILLS
ABOUT CHARUL GUPTA
A versatile professional holding 15 years of IT experience in playing various roles - Solution Technical Architect and Consultant. Excellence in design and implementation of solutions related to – BI, Data Engineering over Cloud and On-Premises Platform. Implemented enterprise level robust and scalable data warehousing solutions including data migration, end to end data integration and data mart creation. Domain Knowledge of Logistics, Manufacturing and Retail and Finance Banking. Ability to build strong relationships with business partners and team members. Considerable onsite US experience of 8 weeks of requirement gathering and defining the project scope. Hold H1B and B1 visa.ProgrammingLanguages PySpark, SQLCloud Services : Azure Data Factory, Azure Databricks, Snowflake, Synapse, ADLS GEN2, Storage accountsBig Data Technologies.ETL Tools SSIS, Azure Data Factory, Matillion Database MS SQL Server, Azure SQL Database, Snowflake, Netezza, Cosmos DB.BI Tools Power BI, TableauTools & Platform JIRA, Visual Studio, GIT
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.