You will own the data layer of the platform: the streaming and batch pipelines that move data from our applications, operational databases, and AI models into the analytical database behind our dashboards, APIs, and analyst tools. You make sure data arrives reliably, is modelled well, and can be queried fast. This is a hands-on role on a self-hosted stack running on bare-metal Kubernetes, not a managed cloud data platform. There is no BigQuery, Snowflake, or Databricks here. The DevOps engineer runs the Kafka and Kubernetes infrastructure. You own what flows through it and how it is modelled. You will be building the data foundation of the platform, and the data team will grow as the platform does. You'll have a direct hand in shaping what comes next. If your experience is mostly writing dbt models on a cloud warehouse someone else operates, this is not the right fit.
- Design and build streaming pipelines on Kafka: topic design, partitioning, schema management, and processing services in Python (or Go)
- Build batch ingestion and transformation pipelines using an orchestrator (Airflow, Dagster, or equivalent) running on Kubernetes
- Design and optimize data models in ClickHouse, including time-series and location data: partitioning, sort keys, materialized views, and query tuning for high-volume dashboard queries
- Replicate operational data from PostgreSQL into the analytical layer using change data capture (CDC)
- Ingest results from AI models (such as object detections, OCR extractions, and match results) as structured, queryable data linked to their sources
- Define data contracts and query patterns with the backend and fullstack engineers who build on top of your data
- Implement data quality checks and pipeline monitoring (metrics, dashboards, alerting). You own the health of the pipelines you ship
- Implement access control at the data layer, aligned with the platform's Keycloak roles, together with the backend and fullstack engineers
- Containerize and deploy pipelines with the DevOps engineer, using Docker, Helm, GitLab CI/CD, and ArgoCD
- 5+ years in data engineering, with at least 2 years building streaming pipelines in production
- Production experience with Kafka as a builder, not just a consumer: topic design, schema evolution, and delivery guarantees
- Has designed and tuned at least one analytical database in production (ClickHouse, Trino/Presto, or equivalent)
- Strong Python and SQL
- Has run data pipelines on self-hosted or on-premise infrastructure, not only on managed cloud data services
- Has built and operated pipelines with an orchestrator (Airflow, Dagster, or equivalent) in production
Preferred:
- ClickHouse specifically
- CDC pipelines (Debezium or equivalent) from PostgreSQL
- Deploying services on Kubernetes with Helm and GitLab CI
- Experience with geospatial or time-series data
- Exposure to handling outputs from AI/ML models