Ilias Sotiriadis

Data Engineer @Aker Systems

London, GB

Built end-to-end data pipelines, using Terraform to provision S3 buckets and IAM policies for source and archive storage; configured feed metadata (frequency, format, source/destination locations) via Liquibase; automated MSK Kafka topic creation and certificate generation for data messaging using Bash; enabled secure access with dynamic HashiCorp Vault credentials and AWS Secrets Manager to RDS and S3 for applications; and deployed Kafka Connect for Avro to Parquet conversion (enabling efficient Athena querying of data in S3) via Helm charts to Kubernetes namespaces.• Developed Python scripts to deserialise data from Kafka topics using kafka-python and confluent-kafka libraries, publishing data to multiple storage solutions including S3 (boto3), email inboxes (smtplib), other Kafka topics, DocumentDB (pymongo), and Snowflake (snowflake-connector-python); deployed these scripts as AWS Lambda functions using Terraform with integrated HashiCorp Vault dynamic credentials and AWS Secrets Manager for secure access management.• Deployed AWS VPC endpoints using Terraform for services including CloudWatch, S3, and DynamoDB, configured within the same subnet as Lambda functions to enable metadata extraction such as tags; configured Lambda egress rules to allow secure access to these services.• Optimised S3 data storage by implementing partitioning strategies, batching writes to create fewer large files rather than many small files (configuring Lambda batch size > 1), and storing data in columnar Parquet format for improved query performance and reduced storage costs.• Reduced CloudTrail and CloudWatch costs by leveraging OpenSearch Service SQL to analyse logging patterns; identified unnecessary logs stored in S3, optimised retention policies, and eliminated redundant data via boto3 and advanced event selectors using Terraform, resulting in cost savings of over £40k per month.

University of Greenwich

Bachelor of Science (BSc), Mathematics

General Assembly

Data Science

Central Foundation Boys School

GCSE

iC6 Sixth Form Consortium

A-Level