This repository contains the core components and automation scripts for a Drought-map Hub, a platform designed to provide streamlined, user-friendly, and validated Combined Drought Indicator (CDI) products. It enhances drought monitoring capabilities by automating data management and promoting public access to critical information.

The Drought-map Hub is a collaborative platform focused on validating, publishing, and exploring Combined Drought Indicator (CDI) products. It aims to enhance a country's drought monitoring capabilities through efficient data management and public accessibility.

The platform is structured into two main areas:

• Admin Area: Dedicated to authorized users (administrators and reviewers) for reviewing, validating, discussing, and approving CDI outputs before public release. This ensures high-quality, reliable information is made available.
• Public Area: Provides open and intuitive access to comprehensive drought-related datasets, allowing users to explore trends and patterns for research, planning, and decision-making.

The Combined Drought Indicator (CDI) is a scientifically robust indicator that combines multiple drought-relevant datasets into a single map reflecting overall drought severity. These datasets typically include:

• Land Surface Temperature (LST): Measures heat stress and temperature anomalies.
• Normalized Difference Vegetation Index (NDVI): Captures vegetation health and greenness.
• Precipitation (e.g., Standardized Precipitation Index or CHIRPS): Assesses rainfall deficits.
• Soil Moisture (SM): Reflects water availability for crops and plants.

The CDI is computed using a weighted average system, which can be adjusted based on regional characteristics or data quality.

The Drought-map Hub offers a range of functionalities for efficient drought monitoring:

• Automated CDI Generation: The platform automatically computes CDI monthly using external datasets and scripts from the National Drought Mitigation Center (NDMC).
• Structured Data Processing and Storage: CDI data, including GeoTIFF raster files and aggregated GeoJSON vector files for administrative regions, are processed and stored on a GeoNode platform.
• Web Application for Collaboration and Publication: A web interface allows authorized users to review, validate, and discuss CDI maps before their public release.
• Role-Based User Access Control: The platform supports three user types:
  • Visitors: Read-only access to published CDI maps and public features.
  • Reviewers: Access to review and provide feedback on computed CDI maps prior to publication.
  • Administrators: Full access to manage the platform, users, and data publication processes.
• Interactive Data Visualization: Users can interact with published CDI maps, zoom, pan, and view CDI values for different administrative regions.
• Historical Map Comparison: Allows visitors to compare two published maps from different time periods to observe changes in CDI values.
• Map Download Options: Published maps are available for download in formats like GeoJSON, shapefile, PNG, and SVG. Reviewers can also access non-aggregated CDI Raster Maps for context.
• Feedback Mechanism: A public feedback form enables users to submit comments or report bugs directly to administrators.

The Drought-map Hub's high-level architecture integrates several components, external services, and software dependencies to automate and manage CDI products.

• Front-end (Next.js):

  • Role: User-facing interface with dynamic routing and server-side rendering (SSR) for performance and SEO.
  • Interdependencies: Communicates with the Back-end via RESTful APIs, exchanging JSON data. It consumes TopoJSON data from a config.js file generated by the backend. Styling is handled by Tailwind CSS and Ant Design, and interactive maps use Leaflet.

• Back-end (Django):

  • Role: Manages user authentication, data validation workflows, and publishing processes. It acts as an intermediary between the front-end and the GeoNode API for dataset management.
  • Interdependencies:
    • Interacts with the Database (PostgreSQL) using Django’s ORM.
    • Utilizes Django Rest Framework (DRF) for Web APIs and PyJWT for secure user authentication.
    • Integrates with Mailjet for email notifications (e.g., user invitations, password resets, review requests, feedback).
    • Uses Pandas and GeoPandas for data processing and analysis of geospatial data.
    • Communicates with Rundeck to trigger and manage CDI automation jobs.
    • Employs Worker/DjangoQ for background processes like monitoring automation and downloading maps.

• Database (PostgreSQL):

  • Role: Stores application data, including user details, administrative regions, publication statuses, reviews, and job logs. GeoNode maintains its own external database accessed via its API.

• CDI Automation with Rundeck:

  • Role: Orchestrates the automated monthly process for generating and updating drought data. This includes checking external datasets (CHIRPS/Precipitation, NDVI, LST, Soil Moisture), running NDMC scripts, and uploading outputs to GeoNode. It also manages the VM hosting the automation server (turning it on/off).
  • Interdependencies: Triggers an endpoint in the DroughtMap Hub to create a new Publication Process object upon success. Sends email notifications via Mailjet for job status (success or error). Requires NASA Earthdata credentials for some dataset downloads (LST, NDVI, SM).

• GeoNode (Spatial Data Infrastructure):

  • Role: Central repository for spatial data, storing CDI raster data and aggregation results (GeoJSON files for administrative regions). It manages categories (e.g., cdi-raster-map) for uploaded datasets.
  • Interdependencies: The DroughtMap Hub's back-end interacts with GeoNode via its API for data retrieval and upload.

• Container Orchestration (Docker Compose):

  • Role: Manages and orchestrates containers for consistent deployment across development and production environments.
  • Interdependencies: Defines how backend, frontend, and database services are deployed and interact within their containers.

• CI/CD Pipeline (GitHub Workflows):

  • Role: Automates continuous integration and deployment, ensuring code quality through automated tests and triggering builds and deployments to appropriate environments upon success.

• Jobs (DjangoQ):

  • Role: Manages job queues and asynchronous task execution within the Drought-map Hub, handling background job processing and schedule management.
  • Interdependencies: Schedules tasks like checking automation status and downloading maps, monitors their progress, and notifies users of outcomes.

The CDI automation script (job.sh) is designed to run periodically (ideally monthly) to keep drought-related data up-to-date. This automated process, orchestrated by Rundeck, involves several key steps:

1. Check External Datasets: The system verifies the availability of the latest datasets from global sources, including CHIRPS/Precipitation, LST, NDVI, and Soil Moisture. Some datasets require NASA Earthdata login credentials.
2. Check Data is Up-to-Date: It compares the number of available external dataset entries with the number of downloaded input data. If discrepancies are found, missing data is identified and downloaded.
3. Run NDMC Scripts (CDI Scripts): Once all necessary datasets for the previous month are updated, the CDI script is executed to produce the CDI Map output, typically as GeoTIFF raster files.
4. Run Aggregation Script: The raw CDI raster output is processed by an aggregation script. This script filters out no-data pixels, creates percentile values, and generates a median percentile value and a drought category for each polygon/administrative boundary as defined in a GeoJSON boundary file.
5. Upload Outputs to GeoNode: All CDI outputs (both raster and GeoJSON) are uploaded to the GeoNode instance via its authenticated API. The system tracks the asynchronous upload progress and updates metadata to set specific categories (e.g., cdi-raster-map, spi-raster-map) for each output.
6. Send Success Message: Upon successful upload, a notification is sent to Drought-map Hub administrators, and an endpoint in the Drought-map Hub is triggered to create a new Publication Process object, initiating the review and validation workflow.

To deploy a new instance of the DMH for a client, follow these steps:

1. Clone the Repository: Obtain the DMH repository and navigate into its directory.
2. Run Installation Script: Execute the configure.sh script provided within the repository.
3. Configure Domain Names:
   • Set the GeoNode web domain by configuring WEBDOMAIN_GEONODE (default: http://localhost).
   • Set the DMH web domain by configuring WEBDOMAIN (default: http://localhost:3000).
4. Configure Client-Specifics:
   • Configure the DMH for the Technical Working Group (TWG) list.
   • Configure CDI by setting the Country Name and Boundaries.
5. Set Earthdata Credentials: Configure the NASA Earthdata credentials (username, password) required for downloading certain datasets. These variables are typically defined in the .env file for the CDI Automation repository.
6. Confirm Deployment Environment: Confirm the deployment environment (e.g., --mode=development and --service=all are defaults).

The configure.sh script will automatically perform the following:

• Save all configuration data to their respective environment services.
• Start the GeoNode and DMH containers using Docker Compose.
• Seed default administrator users for both GeoNode and DMH.
• Generate a RUNDECK_TOKEN using the CLI.
• Create a new Rundeck project and import the Job template via the API.
• Seed the Rundeck integration into the DMH.
• Wait until all containers are ready and accessible.

After the initial script execution, some manual configurations are required:

1. Rundeck Configuration

   Follow these steps to register the remote VM that will run CDI automation scripts and to wire Rundeck into DMH.

   Checklist

   • Ensure you can log in to Rundeck Admin (credentials are in app/rundeck.env).
   • Have the CDI VM hostname or IP and a login user ready (example users: akvo-app).
   • Decide whether to use password or SSH key authentication.

   Step-by-step

   1. Log in to the Rundeck Admin panel using the credentials in app/rundeck.env.
   2. Open the DroughtMapHub project and click Project Settings → Key Storage.
      • Add a new key entry for the VM:
        • For password auth: create a password entry and give it an ID (for example cdi/vm-password).
        • For SSH key auth: add a key pair (private key) and optionally a passphrase entry.
   3. Still in Project Settings, open Nodes → Edit Nodes → Edit (switch to XML view).
      • Replace or append a single node entry for your VM. Use one of the templates below and update the placeholders.

   Template: password authentication

   <node name="vm-cdi-scripts" description="Remote VM for executing CDI commands" tags="linux" hostname="<<your-cdi-host>>" osArch="amd64" osFamily="unix" osName="Linux" osVersion="" username="<<username-on-vm>>" ssh-password-storage-path="keys/cdi/vm-password" ssh-authentication="password"/>

   Template: private-key authentication

   <node name="vm-cdi-scripts" description="Remote VM for executing CDI commands" tags="linux" hostname="<<your-cdi-host>>" osArch="amd64" osFamily="unix" osName="Linux" osVersion="" username="<<username-on-vm>>" ssh-key-storage-path="keys/cdi/vm-ssh-key" ssh-key-passphrase-storage-path="keys/cdi/vm-passphrase" ssh-authentication="privateKey"/>

   Notes on placeholders

   • <<your-cdi-host>>: hostname or IP reachable from the Rundeck server.
   • <<username-on-vm>>: the user that runs the CDI scripts (e.g., akvo-app).
   • keys/cdi/vm-password, keys/cdi/vm-ssh-key, keys/cdi/vm-passphrase: example storage paths in Rundeck Key Storage. Use the IDs you created in Key Storage.

   Validate the node

   • After saving the node XML, go to the Nodes view and click the node to test a simple command (for example echo hello).
   • If the test fails, check the Rundeck service logs and the Key Storage entries.

   DMH integration

   • In the DMH admin UI (log in with the DMH admin user), open Settings → Rundeck integration.
   • Provide the Rundeck URL, API token (the RUNDECK_TOKEN generated during configuration), and select the DroughtMapHub project and job names.
   • Save and run a test trigger from DMH to ensure the job starts and the webhook/callback works.

   Troubleshooting tips

   • "Permission denied" when using key auth: verify the private key is correct and has the right permissions; try connecting from the Rundeck host via SSH.
   • "Connection timed out": confirm Rundeck server can reach the VM hostname/IP on port 22 and no firewall blocks the traffic.
   • If jobs run but scripts fail on the VM: inspect the remote user's environment (PATH, Python, virtualenv). Rundeck runs non-interactive shells.

   Quick verification commands (run from Rundeck host or locally to confirm reachability)

   # from Rundeck host: test ssh (replace placeholders)
   ssh -i /path/to/private_key <<username-on-vm>>@<<your-cdi-host>> 'echo ok'

   Summary

   • Use Rundeck Key Storage to securely store credentials.
   • Add the VM as a node with the correct SSH auth method.
   • Wire DMH to Rundeck using the generated API token and test a job trigger.

   Environment template

   • A template file with common Rundeck environment variables is available at app/rundeck.env.template — copy it to app/rundeck.env and fill in real values (RUNDECK_URL, RUNDECK_TOKEN, RD_USER, RD_PASSWORD, etc.).

   Troubleshooting checklist

   • Rundeck unreachable from DMH: confirm RUNDECK_URL is reachable from the DMH container and that the RUNDECK_TOKEN is valid.
   • Invalid token / 401 responses: regenerate a token using the Rundeck CLI or Admin UI and update app/rundeck.env.
   • Node connection problems: test SSH from the Rundeck host to the VM and verify Key Storage entries point to the correct IDs.
   • Job fails only in Rundeck but works manually: inspect the job log for environment differences (PATH, virtualenv activation). Consider wrapping commands in a shell script that sets up the environment.

2. DMH Configuration:

   • Log in to the DMH using the default administrator credentials.
   • Navigate to the settings page.
   • If Rundeck integration is successful, configure the email for CDI notifications.
   • If Rundeck integration fails, create a new Rundeck connection with the available projects and jobs, then configure email notifications.

Once all services are available and running properly, execute the post-installation.sh script to complete the GeoServer configuration:

./post-installation.sh

This script performs the following essential tasks:

• Updates GeoServer admin password: Changes the default GeoServer admin password from "geoserver" to the password specified in the GeoNode environment configuration.
• Enables XML External Entities: Configures GeoServer to allow XML external entities (xmlExternalEntitiesEnabled=true), which is required for proper functioning of the spatial data infrastructure.

Prerequisites for running the post-installation script:

• All Docker containers must be running and accessible
• GeoNode service must be fully initialized and responsive
• The ./geonode/.env file must contain the required environment variables:
  • GEOSERVER_PUBLIC_LOCATION: The public URL of the GeoServer instance
  • GEOSERVER_ADMIN_USER: The GeoServer admin username
  • GEOSERVER_ADMIN_PASSWORD: The desired admin password

Important Notes:

• The script will wait 15 seconds after updating the password to allow GeoServer to process the change
• If the password update fails, the XML entities configuration may also fail
• Verify that GeoServer is accessible at the configured public location before running the script

The DMH configuration is now complete and ready for use.