Daniel González Pérez

Senior Data Engineer @Version 1

London, GB

I am responsible for total Data Science and Big Data Analytics solutions, from architecture and design, through development to productionisation and integration around different technologies such as Databricks, Spark, Azure Data Factory, Azure Data Lake, Microsoft Azure Cloud, CosmosDB with geospatial data.\\\\n\\\\nOn my last project:\\\\n\\\\nI\'ve been working for a customer in a project designing and building a data platform that consists of a vast data lake that stores all the company\'s data assets and the analytics capabilities to analyze them. The Platform consists of a dynamic infrastructure deployed on Azure using Kubernetes and Terraform on-demand and Spark as the main processor engine.\\\\n\\\\nI have worked defining the data pipelines using Spark (Databricks), Kubernetes, and Python for ingesting and analyzing hundreds of data sources of different types, including real-time data, geospatial data, satellite images. I have been involved in various challenges that include data propagation and data consistency of massive data sources in a distributed environment, with data arriving at different times and in entirely different formats.\\\\n\\\\nUsing Spark for cutting edge use cases like processing Satellite Imagery, doing geospatial analytics using existing libraries like GeoSpark, and building our implementation based on JTS. Using Spark for training Machine Learning models in a distributed way.\\\\n\\\\nDevOps, CI/CD, and automation using Kubernetes and Azure DevOps.\\\\n\\\\nDefinition of best practices for working with Databricks and doing testing on data pipelines (Unit Testing, Integration Testing, Systemwide testing, data quality testing)\\\\n \\\\n\\\\n \\\\n \\\\n\\\\n \\\\n

Universitat Oberta de Catalunya

Master of Science (MSc), Computer Science

ESIC: Business & Marketing School

Master's degree, Logistics and Supply Chain Management

Universidad Politécnica de Madrid

Bachelor's degree, Aerospace Engineering