What a Data Engineer does
Data engineers build the pipelines that move data from products into warehouses, dashboards and models. The best JDs describe the data itself: its sources, its volume and who uses it.
Hiring tip: Describe the data, not just the tools: "about 50 million events a day from our app into BigQuery" tells a candidate far more than a list of technologies.
Data Engineer responsibilities
- Build and maintain batch and streaming data pipelines
- Model data in the warehouse so analysts and data scientists can query it reliably
- Own data quality: validation, monitoring and fixing broken pipelines quickly
- Optimise pipeline cost and performance as data volume grows
- Work with analytics, product and ML teams on what data they need and when
For senior and lead roles, add:
- Design the data platform architecture and choose tools for ingestion, storage and orchestration
- Set data modelling and governance standards and mentor the team
Skills to list
Must-have: SQL, Python, Spark, Airflow, ETL, AWS.
Nice to have: Kafka, Scala, dbt, GCP.
Swap in your real stack. Keep must-haves to what the hire will use every week; everything else belongs under nice-to-have.
Experience by seniority
| Title | Typical experience |
|---|---|
| Junior Data Engineer | 0–2 years |
| Data Engineer | 2–5 years |
| Senior Data Engineer | 5–8 years |
| Lead Data Engineer | 8+ years |