Dari Web Developer ke Data Engineer Banyak data engineer berasal dari latar belakang web development. Skill yang sudah kamu miliki sangat relevan — transisi…
Dari Web Developer ke Data Engineer
Banyak data engineer berasal dari latar belakang web development. Skill yang sudah kamu miliki sangat relevan — transisi ini lebih mulus dari yang kamu kira.
Skill yang Sudah Kamu Punya
| Web Dev Skill | Relevansi di Data Engineering |
|---|
| SQL (CRUD, JOINs) | Fondasi utama — tinggal tambah window functions, CTEs |
| API development | Membangun dan consume data APIs, webhooks |
| Git & version control | Sama persis — dbt pakai Git, pipeline code di repo |
| Docker | Container untuk pipeline, Airflow, database |
| CI/CD | Deploy pipeline, run dbt tests di CI |
| Database design | Basis untuk dimensional modeling |
| Python/JavaScript | Python adalah bahasa utama data engineering |
| Cloud (AWS/GCP) | Storage, compute, managed services |
Skill yang Perlu Dipelajari
// Learning path untuk web dev → data engineer
// Estimasi timeline: 3-6 bulan part-time
// Month 1-2: Fondasi
1. SQL advanced → Window functions, CTEs, query optimization
2. Python data stack → Pandas, data manipulation
3. Data formats → Parquet, JSON Lines, Avro
// Month 2-3: Pipeline
4. ETL/ELT concepts → Build pipeline Python sederhana
5. dbt → Transform di warehouse
6. Airflow basics → Scheduling & orchestration
// Month 3-4: Infrastructure
7. Data warehouse → BigQuery/Snowflake, dimensional modeling
8. Streaming basics → Kafka concepts, when to use
9. Data lake → Object storage, partitioning
// Month 5-6: Advanced & Portfolio
10. Data quality → Testing, monitoring, alerting
11. CDC → Debezium, real-time sync
12. Portfolio project → End-to-end pipeline
Portfolio Project Ideas
- E-commerce analytics pipeline — Scrape/API data → transform → dashboard (dbt + BigQuery + Looker/Metabase)
- Real-time social media tracker — Twitter/Reddit API → Kafka → ClickHouse → live dashboard
- Web app analytics — Instrument app kamu sendiri → event tracking → data warehouse → insights
- Public dataset analysis — Government/open data → clean → model → publish insight
Career Path
// Typical progression
Web Developer (backend)
↓ (learn SQL advanced + Python data stack)
Junior Data Engineer
↓ (build production pipelines + dbt)
Mid Data Engineer
↓ (architect systems + lead projects)
Senior Data Engineer
↓
Staff/Principal DE or Engineering Manager or Data Architect
// Salary range (Indonesia, 2024):
// Junior DE: Rp 8-15 juta/bulan
// Mid DE: Rp 15-30 juta/bulan
// Senior DE: Rp 30-60 juta/bulan
// Lead/Staff: Rp 50-100+ juta/bulan
// Remote/global companies biasanya 2-5x rates di atas
Day-to-Day Perbedaan
| Aspek | Web Developer | Data Engineer |
|---|
| Produk | Web app untuk end users | Data pipeline & warehouse untuk internal |
| Stakeholder | Product manager, designer | Data analyst, data scientist, business |
| On-call | App down → fix | Pipeline gagal → data terlambat → fix |
| Testing | Unit test, integration test | Data quality test, reconciliation |
| Deploy | CI/CD → production server | CI/CD → Airflow/dbt Cloud |
| Monitoring | APM, error tracking | Pipeline success rate, data freshness, row counts |
