Data Management API

The Data Management component provides the backend APIs and data management layer for the IDS-DRR platform.

Repository: IDS-DRR-Data-Management

Overview

The Data Management layer handles:

• Data Storage: PostGIS-enabled PostgreSQL database for storing processed data, risk scores, and geographical boundaries

• Caching: Redis for caching query results (maps, tables, time trends, indicators)

• API Endpoints: GraphQL APIs (via Strawberry) for frontend data access

• Data Ingestion: Django management commands for importing geographical and indicator data

• Geographical Data: Support for states, districts, revenue circles, sub-districts, blocks, and tehsils

Tech Stack

• Framework: Django 4.2

• Database: PostgreSQL with PostGIS extension

• Cache: Redis

• API: Strawberry GraphQL

• Server: Uvicorn (ASGI)

• Containerization: Docker & Docker Compose

Prerequisites

• Python 3.12+

• PostgreSQL with PostGIS extension

• Redis

• GDAL, GEOS, and PROJ libraries (for geospatial operations)

• Docker & Docker Compose (recommended)

Local Development

Option 1: Using Docker (Recommended)

# Clone the repository
git clone https://github.com/CivicDataLab/IDS-DRR-Data-Management.git
cd IDS-DRR-Data-Management

# Create report configuration (required for state listing and PDF reports)
cp report_config.local.json report_config.json

# Start services
docker compose up -d --build

# Run migrations
docker exec context_layer_Backend python manage.py makemigrations
docker exec context_layer_Backend python manage.py migrate

# Import data (see Data Import section below)
docker exec context_layer_Backend python manage.py import_data

# The API will be available at http://localhost:8000

To override the defaults (e.g. credentials), create a .env file in the project root; Docker Compose picks it up automatically. See Environment Variables below.

See report_config.example.json for the full configuration format, including report sections and chart definitions.

Option 2: Manual Setup

# Clone the repository
git clone https://github.com/CivicDataLab/IDS-DRR-Data-Management.git
cd IDS-DRR-Data-Management

# Create and activate virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Create environment file and configure (see Environment Variables below)
# Set POSTGRES_HOST=localhost and REDIS_URL=redis://127.0.0.1:6379/1
# for a manual setup where Postgres and Redis run on the host

# Run migrations
python manage.py makemigrations
python manage.py migrate

# Import data (see Data Import section below)
python manage.py import_data

# Start development server
python manage.py runserver

Environment Variables

When using Docker, all variables have sensible defaults; no .env file is needed. To override any value, create a .env file in the project root; Docker Compose picks it up automatically for variable substitution.

For manual (non-Docker) setup, a .env file is required since there are no defaults for database connection details.

Variable	Description	Docker Compose default	Manual setup value
POSTGRES_NAME	Database name	postgres	your database name
POSTGRES_USER	Database user	postgres	your database user
POSTGRES_PASSWORD	Database password	postgres	your database password
POSTGRES_HOST	Database host	db	localhost
POSTGRES_PORT	Database port	5432	5432
REDIS_URL	Redis connection URL	redis://redis:6379/1	redis://127.0.0.1:6379/1
CHART_API_BASE_URL	DataSpace chart generation endpoint, used to render charts as images in PDF reports (e.g. http://localhost:8001/api/generate-dynamic-chart/)
WHITELIST_INDICATORS	Comma-separated indicator slugs to import in addition to visible indicators	topsis-score	topsis-score

Production

Variable	Description
DJANGO_ENV	Set to production to disable debug mode by default.
SECRET_KEY	Django secret key, used for cryptographic signing. Must be set to a unique, unpredictable value in production. The default (!!!SECRET_KEY!!!) is not safe for production.
ALLOWED_HOSTS	Comma-separated list of hostnames the server will accept requests for, appended to the built-in defaults (localhost, 127.0.0.1, 0.0.0.0). Must include the domain(s) serving the API.
DEBUG	Enable Django debug mode. Defaults to True unless DJANGO_ENV=production. Set to True or False explicitly to override.

Generate a secret key with:

python -c "from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())"

Data Import

The data management layer uses the import_data management command for importing and updating data.

Import all data (geojson, indicators, and data for all states):

python manage.py import_data

Or, import data for a specific state:

python manage.py import_data --state assam
python manage.py import_data --state himachal_pradesh
python manage.py import_data --state odisha

Or, import data for a specific district within a state:

python manage.py import_data --state assam --district 201

What the Import Command Does

1. Migrates GeoJSON Data: Imports geographical boundaries from layer/assets/geojson/ directory

   • Creates state, district, revenue circle, sub-district, block, and tehsil records

   • Stores MultiPolygon geometries for each geographical unit

2. Updates Indicators: Imports indicator definitions from layer/assets/indicators/*_indicators.csv

   • Creates/updates indicator metadata (name, description, category, unit, data source)

   • Links indicators to their parent indicators and geographical units

3. Imports State Data: Imports actual data values from layer/assets/data/*_data.csv

   • Associates data values with indicators and geographical units

   • Supports time-period based data

Indicator CSV Format

The indicator CSV files should have the following columns:

Column	Description
indicatorSlug	Unique identifier for the indicator
indicatorTitle	Display name of the indicator
indicatorDescription	Detailed description
indicatorCategory	Category grouping
unit	Unit of measurement
datasource	Data source reference
parent	Parent indicator name (for hierarchical indicators)
visible_on_platform	y or n to control visibility

Data CSV Format

The data CSV files should have:

• object-id: Geography code (used as index)

• timeperiod: Time period for the data (e.g., 2024_08)

• Columns for each indicator slug with their values

Database Schema

Core Models

• Geography: Stores geographical boundaries (states, districts, etc.) with PostGIS geometry

• Indicators: Stores indicator metadata and hierarchical relationships

• Data: Stores actual data values linked to indicators and geographies

• Unit: Stores units of measurement

Docker Services

The docker-compose.yml provides three services:

Service	Container Name	Port	Description
db	context_layer_PG	54321:5432	PostGIS database
redis	context_layer_Redis	6380:6379	Redis cache
web	context_layer_Backend	8000:8000	Django application

Troubleshooting

Common Issues

1. PostGIS Extension Missing: Ensure PostgreSQL has PostGIS extension installed

   CREATE EXTENSION postgis;
   

2. GDAL/GEOS Not Found: Install geospatial libraries

   # Ubuntu/Debian
   apt-get install gdal-bin python3-gdal libgeos-dev libproj-dev
   
   # macOS
   brew install gdal geos proj
   

3. State Data File Missing: Ensure data files exist in layer/assets/data/ with correct naming convention ({state}_data.csv)