Piu Mallick
Senior Data Engineer | 2x Databricks Certified | Data Modeler | PySpark | SQL | Python | Databricks | Azure
- Role
- Senior Data Engineer at 84.51˚
- Location
- Pittsburgh, PA, US
- LinkedIn followers
- 500 followers
About Piu Mallick
Professional Summary: Around 10 years of overall experience as software developer in Analysis, Design, Development, Deploying and supporting large scale distributed systems. Well-versed in various Big Data ecosystem tech stack. Expert in working with Hive data warehouse tools – creating tables, data distribution by implementing partitioning and bucketing, writing, and optimizing the HiveQL queries. Experience in querying and analysis and transformation of large datasets stored on HDFS using Hive Query Language (HiveQL) and Spark. Strong knowledge on implementing of data processing on Spark-Core using RDD, Spark Dataframes and Spark SQL. Hands on experience in working on spark - SQL queries, import data from Data sources, perform transformations, and perform read/write operations, save the results to output directory into HDFS. Developed Spark Applications that can handle data from various RDBMS (MySQL, Oracle Database) and Streaming sources. Proficient SQL experience in querying, data extraction/transformations and developing queries for a wide range of applications. Expert in providing ETL solutions for various business models. Experience in using ETL tools such as Informatica and Talend. Provided and constructed solutions for complex data issues. Performed statistical analysis to inform business decisions. Good exposure to Development, Testing, Implementation, Documentation and Production support. Develop effective working relationships with client teams to understand and support requirements, develop tactical and strategic plans to implement technology solutions, and effectively manage client expectations. An excellent team member with an ability to perform individually, good interpersonal relations, strong communication skills, hardworking and high level of motivation. Used IDEs like Eclipse, IntelliJ IDE, PyCharm IDE, Notepad ++, and Visual Studio for development.Technical Skills:Programming Languages: Python, ScalaRDBMS (Relational Data Models): SQL, MySQL, Oracle, PostgreSQLNoSQL Databases: HBase, MongoDBBig Data Ecosystem: HDFS, YARN, Hadoop, Sqoop, Hive, HBase, Spark, Kafka (Intermediate)File Formats – AVRO, ORC, Parquet, CSV, JSONETL Framework – InformaticaLibraries/Framework: Numpy, Pandas, scikit learn (sklearn), Seaborn, matplotlib, Jupyter NotebookStatistical Analysis: R (R Studio)Cloud Services: Microsoft Azure (Data Lake, Customer Insights, Databricks, Data Factory) Scripting: Unix Shell ScriptingData Visualization Tool: TableauWorkflow/Pipelines: CI/CD Pipelines, Airflow
Experience
Senior Data Engineer
May 2024 — Present · US
Education
University of Pittsburgh
Master of Science - Information Sciences, Big Data Analytics
2019 — 2020
West Bengal University of Technology, Kolkata
Bachelor of Technology (B.Tech.), Computer Science
2003 — 2007
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.