GOV-DENGUE-ETL: AUTOMATED DENGUE DATA PIPELINE

GOV-DENGUE-ETL is a production-grade data engineering solution designed to automate the collection and processing of daily Dengue outbreak data from Sri Lanka's National Dengue Control Unit. The system addresses the challenge of manually processing PDF-based health bulletins by implementing a fully automated, containerized ETL pipeline.

The pipeline leverages Apache Airflow for orchestration, scheduling daily extractions with built-in retry logic and failure handling. Using pdfplumber, the system intelligently parses tabular data from government-issued PDF documents, transforms it into structured formats, and loads it into a PostgreSQL data warehouse for analytical querying.

Beyond the ETL engine, the project delivers an interactive Streamlit dashboard that visualizes disease trends and geographical distribution patterns. The dashboard integrates geospatial data (GeoJSON) to provide district-level outbreak mapping, enabling public health officials and researchers to make data-driven decisions.

TECH STACK

KEY FEATURES

• Automated PDF scraping and parsing from government health portals
• Scheduled daily ETL workflows with Apache Airflow orchestration
• Idempotent data processing with raw and processed data lake tiers
• Interactive Streamlit dashboard with geospatial outbreak visualizations
• Containerized deployment using Docker Compose for reproducibility
• Comprehensive logging and monitoring for pipeline observability
• SQLAlchemy ORM for clean database operations and schema management
• GeoJSON integration for district-level disease mapping