Skip to content
BranchesTags
 
 

Repository files navigation

OpenNSIS — Open National Soil Information System

A spatial data infrastructure for soils, designed to be deployed as a node of the GloSIS Federation. Each instance hosts the soil data of one country or organisation and exposes:

  • a public Web Mapping application (OpenLayers) for viewing rasters, soil profiles and administrative division boundaries on a map;
  • an Admin Panel for settings, users, projects, soil-profile ETL (CSV upload → validate → ingest → prune), raster upload, a raster calculator (DST) and administrative division boundary layers (GeoJSON / zipped Shapefile / GeoPackage, reprojected to WGS 84 on the fly);
  • a REST API (FastAPI) for soil profiles, observations, layers, settings and admin operations;
  • an OGC API Records / CSW metadata catalogue (pyCSW) for the rasters;
  • a WMS / WCS server (MapServer) for rasters and profile points;
  • an opt-in Federation API (sis-api-glosis) that lets the GloSIS Discovery Hub query this node for soil-profile data.

The data model is based on ISO 28258 and is derived from ISRIC's open-source implementation, iso-28258 (MIT licence). See sis-database/NOTICE for attribution and citation.

Components

Component Tech Host port (dev) Role
sis-nginx nginx 80, 443 Front door — TLS, routing, rate limiting, headers
sis-web-mapping OpenLayers + nginx 8001 Static single-page application (SPA): map view + admin dashboard
sis-api FastAPI 8002 JWT auth, ETL, layers, settings, users, admin
sis-metadata pyCSW 8003 OGC API Records — raster metadata catalogue
sis-web-services MapServer 8004 WMS / WCS for raster layers and soil profile points
sis-database PostgreSQL+PostGIS (internal) Single shared database (multiple schemas)
sis-api-glosis FastAPI 8006 Read-only federation endpoints (manifest/profile/observation)

In production only sis-nginx (80/443) should be reachable from the network; the other ports are dev conveniences. The single docker-compose.yml currently publishes all of them — operators wanting a hardened setup should override those ports: mappings via a docker-compose.override.yml.

Project structure

sis-dev/
├── .env                                    # Per-deployment secrets (gitignored — generated by deploy.sh)
├── .env.example                            # Template
├── deploy.sh                               # End-to-end deployment script (single node)
├── update.sh                               # In-place update: git pull + migrations + rebuild
├── docker-compose.yml
├── LICENSE                                 # Apache-2.0 (code)
├── NOTICE                                  # Attribution + third-party content licences
├── README.md
│
├── ops/
│   ├── single/                             # One-shot remote deploy of a dedicated server
│   └── workshop/                           # Multi-country shared-host deploy layer
│
├── sis-api/                                # FastAPI admin/data API
│   ├── Dockerfile
│   ├── main.py
│   ├── shared.py                           # Auth, DB, models — shared with sis-api-glosis
│   ├── dst/                                # Raster calculator (DST) engine
│   ├── raster_registry/                    # Raster registration, COG conversion, ISO XML, pyCSW load
│   └── requirements.txt
│
├── sis-api-glosis/                         # Federation API (read-only)
│   ├── Dockerfile
│   ├── main.py                             # /manifest /profile /observation
│   └── requirements.txt
│
├── sis-database/                           # PostgreSQL + PostGIS
│   ├── init.sql                            # Roles + extensions
│   ├── sis-database_latest_with_codelist.sql  # Seed dump: schema + GloSIS codelists
│   ├── migrations/                         # Numbered idempotent migrations (see its README)
│   ├── owl2sql/                            # GloSIS ontology → SQL codelist conversion
│   └── volume/                             # Postgres data dir (gitignored)
│
├── sis-metadata/                           # pyCSW
│   ├── pycsw_default.yml                   # Template (substituted at deploy time)
│   └── volume/                             # Metadata XML records
│
├── sis-nginx/
│   └── nginx.conf                          # Front-door config
│
├── sis-web-mapping/                        # SPA (built with Parcel, served by nginx)
│   ├── Dockerfile                          # Multi-stage: builder → nginx-unprivileged
│   ├── nginx-static.conf
│   ├── package.json
│   ├── public/
│   └── src/
│
├── sis-web-services/                       # MapServer
│   └── volume/                             # *.tif and *.map files
│
└── sis-data-cube/                          # Raster preprocessing helpers (GDAL)
    └── ...

Deploying a node

Prerequisites

  • Docker Engine + Docker Compose plugin
  • openssl (for secret generation in deploy.sh)
  • Country/project rasters and metadata in ~/Downloads/FAO/AFACI/<COUNTRY>/output/ (*.tif, *.map, *.xml). Adjust the path in deploy.sh if your data lives elsewhere.

Deployment layers

  • ./deploy.sh — run on the server; one country per server, nginx on 80/443.
  • ops/single/first-deploy.sh user@SERVER_IP <CC> — same result, run from your laptop: installs Docker, clones the repo to /opt/sis and runs deploy.sh end to end.
  • ops/workshop/deploy-workshop.sh <CC> <HOST_PORT> — many isolated country instances side by side on one shared host (see ops/workshop/README.md).

First deploy

cd sis-dev
./deploy.sh

On the first run deploy.sh will:

  1. Generate .env with random per-deployment secrets (POSTGRES_PASSWORD, POSTGRES_GLOSIS_PASSWORD, SECRET_KEY, WEB_MAPPING_API_KEY) and chmod 600 it. .env is gitignored — never commit it.
  2. Tear down any prior SIS containers (docker compose down -v) and wipe the local Postgres volume. It does not touch other containers on the host.
  3. Bring up sis-database and apply init.sql + the schema dump.
  4. Rotate the sis_glosis Postgres role password to the value in .env.
  5. Insert dummy data (api.insert_dummy_data) and seed api.setting.
  6. Seed the web-mapping API client with the value of WEB_MAPPING_API_KEY.
  7. Start sis-api, then provision the admin user with a fresh random password and print it once at the very end of the run.
  8. Start sis-api-glosis, sis-nginx, sis-web-services, sis-metadata, sis-web-mapping. Copies the country's .tif / .map / .xml files into the right volumes and runs pycsw-admin.py load-records.

When the script finishes you'll see a banner like:

============================================================
 SIS deployment complete.
------------------------------------------------------------
 Admin login:    admin
 Admin password: <random 16-char string>

 Save this password now — it will not be shown again.
 Change it via the Administration tab after first login.
============================================================

Visit http://localhost/ to see the SPA.

Subsequent deploys (same node)

Re-running deploy.sh is destructive — it wipes the database volume and re-applies the schema. Use it for rebuild-from-scratch flows. To rebuild only one or two services without touching the data, prefer:

docker compose up -d --build --no-deps sis-api sis-web-mapping

Enabling HTTPS (automatic certificates)

An optional Caddy front (sis-caddy, compose profile tls) terminates TLS with automatic Let's Encrypt certificates and proxies everything to sis-nginx unchanged. It is dormant by default — instances without a domain run plain HTTP exactly as before.

Prerequisites — check all three first:

  1. A DNS A record for your domain points at this server's public IP (no CDN or institutional proxy intercepting /.well-known/acme-challenge).
  2. Host ports 80 and 443 are free (ss -tlnp | grep -E ':80|:443' shows only the SIS's own nginx, or nothing) and open in the firewall.
  3. No other ACME client (certbot, Traefik, another Caddy) runs on this host. If another web server or ACME stack already owns 80/443, do not enable this — register the SIS as a virtual host in that existing proxy instead.

Never enable this on a shared multi-instance host (e.g. the workshop layout).

New install: export DOMAIN=soil.example.org before running deploy.sh — that is all; the settings are persisted into .env.

Existing install: after ./update.sh, add four lines to .env:

DOMAIN=soil.example.org
COMPOSE_PROFILES=tls
NGINX_HTTP_BIND=127.0.0.1:8080
NGINX_HTTPS_BIND=127.0.0.1:8443

then docker compose up -d. Caddy takes ports 80/443 (nginx retreats to loopback), obtains the certificate on first request and renews it automatically; certificates persist in the caddy-data volume. To disable, remove the four lines and run docker compose up -d --remove-orphans.

Updating to the latest release

To pull the latest code and apply it without touching your data, run from the install dir:

./update.sh          # add -y to skip the confirmation prompt

It git pulls, applies any pending DB migrations (sis-database/migrations/*.sql, tracked in api.schema_migration), rebuilds only the application containers (sis-api, sis-api-glosis, sis-web-mapping) and reloads nginx. It never runs down, never passes -v, and never reloads the seed dump — so the Postgres / pyCSW / raster volumes (everything you uploaded) are preserved.

DB schema/function/view changes ship as numbered, idempotent migrations under sis-database/migrations/ (see sis-database/migrations/README.md). Base-image upgrades of the data containers (Postgres / pyCSW / MapServer) are a deliberate manual step and are out of update.sh's scope.

Per-country settings

deploy.sh reads its configuration from the environment:

export PROJECT_DIR=/opt/sis    # install dir (defaults to the repo checkout)
export COUNTRY=KH              # ISO 3166-1 alpha-2; full name and map centroid
                               # are looked up from soil_data.country
export ORG_LOGO_URL=https://example.org/logo.png
./deploy.sh

Everything else (title, map position, zoom, base map, logo) can be changed later from Administration → Settings.

Configuration: .env

Generated automatically on first ./deploy.sh. Reference template is .env.example. Keys:

Key Used by
POSTGRES_HOST/PORT/DB/USER sis-api, sis-metadata
POSTGRES_PASSWORD sis-api, sis-metadata, deploy.sh
POSTGRES_GLOSIS_USER sis-api-glosis (read-only role)
POSTGRES_GLOSIS_PASSWORD sis-api-glosis
SECRET_KEY sis-api JWT signing
ACCESS_TOKEN_EXPIRE_MINUTES sis-api
WEB_MAPPING_API_KEY sis-web-mapping (baked into bundle), seeded into api.api_client

All four secret values are unique per node. Sharing them across deployments defeats the per-node JWT and key separation.

GloSIS Federation

sis-api-glosis is shipped but disabled by default. To join the GloSIS Federation:

  1. Log into the SPA as admin.
  2. Open Administration → GloSIS Federation.
  3. Click Enable. The node generates a singleton federation token (shown on the same panel) and exposes
    • GET /manifest
    • GET /profile
    • GET /observation on host port 8006 (or via your nginx, behind /glosis/).
  4. Copy the token plus the endpoint URLs to the GloSIS Discovery Hub admin.

The data exposed is exactly the soil profiles currently visible on this node (controlled under Layers → Soil profiles). Rasters travel separately via the public metadata catalogue and the WMS endpoint.

sis-api-glosis connects to Postgres as the read-only sis_glosis role, which has SELECT only on three views (vw_api_manifest, vw_api_profile, vw_api_observation) plus the federation token view. Even with a valid token, callers cannot read or write anything else.

Click Disable to revoke access without losing the token, or Disable & Delete token to revoke and force re-issuance on next Enable.

Security notes

The deployment pattern enforces:

  • per-deployment secrets generated at install time, never committed
  • no default admin password — the password is randomized and printed once
  • Postgres is not published to the host (use docker exec for psql)
  • read-only Postgres role for federation traffic
  • JWT revocation on password change (via password_changed_at)
  • account lockout after 5 failed logins (15-minute window)
  • nginx-level rate limiting on /api/auth/login
  • append-only api.audit log enforced by SQL trigger
  • baseline hardening headers (X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy)
  • /docs, /redoc and /openapi.json disabled in production (set ENABLE_DOCS=true in the env to re-enable for local dev)
  • CSV upload caps (50 MB / 200 000 rows) and table-name length safety in the ETL path; administrative-division boundary uploads capped at 1 GB
  • public metadata popup escapes every interpolation (no XSS via pyCSW records)

For TLS, the recommended path is the automatic-HTTPS front (see Enabling HTTPS above). Manual certificates remain possible: drop a cert into sis-nginx/ssl/, uncomment the HTTPS server block + the HSTS header in sis-nginx/nginx.conf, and uncomment the ./sis-nginx/ssl:/etc/nginx/ssl:ro mount in docker-compose.yml.

Licensing

  • Code — Apache License 2.0. Attribution details in NOTICE.
  • Soil property & lab method codelists — derived from the GloSIS ontology and remain under its CC BY-NC-SA 3.0 IGO licence (attribution, non-commercial, share-alike). They are not relicensed by this repository. This covers sis-database/owl2sql/ and the codelist rows in the seed dumps.
  • Documentation and diagrams — CC BY 4.0.

About

Open-source national soil information system — soil property maps, profile data, metadata catalogue and GloSIS federation, deployed with one command.

un-fao.github.io/OpenNSIS/

Resources

Readme
License
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages