Joe Pallister

Junior Data Engineer @Cervus

City Of Bristol, GB

Implemented updates to one of our APIs, schema updates to manage new data, SQL queries to display historical data for users as well as custom metrics. Added DB triggers to notify when new data arrived, which was picked up by a notifier that forwarded messages to a frontend SSE service. Designed these messages to be cumulative so that any data that was dropped would be picked up in the next. Involved with project planning to assess the old system and decide how we would make changes to meet requirements. Created a Flask API for ingesting data from a client. Deployed the API on EC2 using images built and stored in ECR and set up an RDS instance using Alembic to apply our schema. Initially deployed with LightSail but due to the inability to whitelist IPs switched to EC2. Expanded data pipelines (Kafka) to allow for data ingestion from a new source. Used Python to manipulate the source data to match our schema and to function with existing metrics for dashboarding. Wrote extensive unit tests and integration tests to maintain core functionality. With this in place was able to safely refactor, reducing tech debt and allowing our APIs to have better separation of concerns.