Streaming
spark/jobs/streaming_ingest.py
Reads Kafka-compatible events and writes checkpointed Parquet outputs.
StreamFlow Phase 1
Containerized stream processing platform
A Kafka-compatible broker, Spark Structured Streaming ingest job, bounded Spark summary job, Airflow DAG, and data quality contract for event-driven data engineering practice.
01 Producer publishes synthetic events
02 Spark consumes topic incrementally
03 Valid and rejected records are persisted
04 Airflow coordinates bounded summary work
05 Project record links architecture to tradeoffs
Why it matters
Data quality before AI quality
The project demonstrates how event data becomes trustworthy enough for analytics, RAG corpora, and model evaluation workflows.
Data Quality
src/streamflow/quality.py
Separates valid events from rejected records with explicit reasons.
Orchestration
airflow/dags/streamflow_daily_summary.py
Coordinates bounded batch summary work instead of owning the streaming loop.