Prem Kuwar
Senior Lead Data Engineer @American Express
Signup · Get unlimited contacts
WORK HISTORY
Senior Lead Data Engineer @American Express
NY, US
As a Data Engineer, I manage a project in which we manage data processes and create efficient workflows utilizing Amazon Web Services (AWS) tools such as S3, Redshift, and Lambda. I optimize data storage, use PySpark code to seamlessly transfer data, and use Kafka Connect and Spark Streaming to create real-time data pipelines.A major accomplishment is successfully migrating data between platforms, such as Oracle to AWS Redshift and Teradata to AWS Snowflake, with Python and BI tools. The project uses a variety of technologies, including Apache Airflow for scheduling, and I actively engage in daily meetings and planning sessions that follow SCRUM techniques.I focus on data warehousing technologies such as Star and Snowflake schemas to improve organization and use Snowflake capabilities to gain precise insights into investment performance. Advanced analytics in Databricks use PySpark scripts to process data and put it into DynamoDB.Working with AWS databases such as RDS (Aurora), Redshift, DynamoDB, and Elastic Cache, I use GitLab to manage version control and Amazon EC2 for infrastructure. The project uses Agile principles, specifically SCRUM, to ensure adaptability and quality results.To meet different data processing requirements, i manage both batch and stream processing utilizing AWS Batch, AWS Kinesis, and AWS Data Pipeline. I use Hive to implement MapReduce tasks and design effective data processes. Tools like as Tableau and Amazon QuickSight are used to analyze and visualize data from Hadoop Hive tables. Improving data storage entails constructing efficient Hive tables, while real-time data techniques include using Kafka and Spark Streaming for rapid transformations and storing data in Dynamo.Addition to this project entails efficiently processing data with AWS technologies, varied approaches, and successful data migrations, all while adhering to Agile norms for good project management.
EDUCATION
Osmania University
Master of Business Administration - MBA
Guru Nanak Institute of Technology
Bachelor of Technology - BTech
ABOUT PREM KUWAR
Certified Azure Data Engineer with over 11+ years of expertise in architecting, designing, and maintaining intricate data pipelines. Proficient in Hadoop, Spark, PySpark, SQL, Python, and more, with hands-on experience in diverse tools and technologies such as Airflow, Databricks, Snowflake, and both AWS and Azure services. Accomplished in developing PySpark scripts for Spark Streaming, optimizing Spark transformations, and enhancing Spark applications for improved efficiency. Extensive work in Spark using Scala on clusters, transforming Hive queries into Spark transformations, and developing Spark applications in Databricks for data extraction and aggregation. Evaluated and improved Databricks performance by migrating complex Redshift scripts to Spark SQL. Proficient in managing Data Warehouses, implementing Type II Slowly Changing Dimensions (SCD), and creating OLAP applications with SQL Server. Responsible for building AWS and Azure data pipelines that use a variety of cloud services and API gateways. S3, EC2, EMR, Kinesis, Lambda, Data Lake, Synapse Analytics, and other technologies are among the expertise. Capable of constructing scalable ETL pipelines with Airflow and implementing robust ETL operations in Azure Databricks. Experienced in web application development using Python, Django, C#, and various front-end technologies. Proficient in data analysis using Python, R, SQL, Hive, PySpark, and Spark SQL for Data Mining, Data Cleansing, and Machine Learning. Strong database skills encompass Oracle, MySQL, Postgres, DB2, SQL Server, and Teradata. Well-versed in CI/CD processes, utilizing Jenkins on Kubernetes and GitLab, and skilled in orchestrating data pipelines with Oozie and Airflow. Hands-on experience in designing and implementing efficient data models and ETL processes for data warehousing and analytics. Proficient in developing visualizations and dashboards using Tableau and Power BI. Active involvement in project management using JIRA, Git for source code management, Jenkins for continuous integration, and Crucible for code reviews. A dedicated team player with a proven track record of delivering data solutions aligned with business objectives. Microsoft Certified: Azure Data Engineer Associate.
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.