Abhijit Kumar

Director @Harekrishna Consulting Ltd

Client: Office for National Statistics\\\\nData Engineer/Scientist\\\\n• Have got experience in applying Azure DevOps and management of infrastructure, database and schema.\\\\n• Created Data Quality Scripts using Spark SQL and Hive to validate successful data load and quality of the data. Created various types of data visualizations using Python and PowerBI.\\\\n• Data sources are extracted, transformed and loaded to generate CSV data files with Python programming and Spark SQL queries.\\\\n• Used Nifi for moving around Data.\\\\n• Wrote Data pipelines in PySpark to cleanse, slice & dice the data. Saving the final output in hive/csv for downstream data pipelines to pick up the data.\\\\n• Designed and developed complex data pipelines and maintained the data quality to support a rapidly growing business\\\\n• I have built a Data Quality Framework in Scala & Azure SQL Server. Whenever data is loaded into the staging environment, Data Quality Framework generates the report about the data. This helps data stewards to get an idea about the fitness of the data.\\\\n• Data Quality Framework used to generate the report about the staging data but now we have augmented it. We are storing the results in a fact table & making the PowerBI to investigate it. Data stewards & business could see validity, completeness, consistency matrices for the received data.\\\\n• Build a configurable module so that Data Quality rules could be integrated with the main pipeline.\\\\n• Automated ETL processes, making it easier to wrangle data and reducing time by as much as 40%.\\\\n\\\\n