Harsh Patel

Big Data Engineer @Cognizant

Mississauga, ON, CA

Designing and building data pipelines to ingest, process and analyze data from various sources such as Databases, External APIs, Streaming Platforms and Files, and integrating them into a Unified System.* Developing and implementing BigData processing solutions using technologies like Databricks, Hadoop, Spark, PySpark, Kafka, or similar frameworks to handle large volumes of data(E.g. Billions/Millions of records) efficiently.* Designing and implementing Extract, Transform, Load (ETL) processes to move data between systems, perform transformations, cleansing, and enrichment of data then load data into target databases or Data Lake/Data Warehouses.* Designing, implementing, and maintaining database to store and organize structured and semi-structured data efficiently. This involves using database management systems (DBMS) like SQL Server, MySQL, HDFS, Cloud Storage Systems (e.g, Amazon S3)* Optimizing BigData applications and infrastructure for Performance, Scalability, and Reliability. Tuning SQL Queries, and data processing codes(Python/Scala)/Workflows for efficient resource utilization, leveraging techniques such as parallel processing, partitioning, and caching.* Ensuring data quality by implementing data validation checks, monitoring data integrity, and addressing data quality issues as they arise.* developing data models to represent business processes and relationships. Which can be used in data analysis and data visualization tools to derive insights and support decision-making.* Collaborating with cross-functional teams including analysts and business stakeholders to understand data requirements, develop data solutions and communicate insights effectively.* Take the ownership of the project and lead offshore team.* Provide oncall production support and ensure stable job execution to provide data to upstream on time for Business Analysis.