Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,10 @@ jobs:
- run: uv sync --group dev
- run: uv run ruff check src tests
- run: uv run ruff format src tests --check
- run: uv run pytest --cov --cov-report=term-missing
- run: uv run pyright --level=warning --warnings src tests
- run: uv run pytest --cov --cov-report=term-missing --cov-report=xml
- run: uv run pyright --level=warning --warnings src tests
- uses: codecov/codecov-action@v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./coverage.xml
fail_ci_if_error: true
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 MXmap project

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
50 changes: 21 additions & 29 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
# Email Provider Dependencies and Email Security in Municipalities Across Germany, Austria, and Switzerland -- ESORICS 2026

![CI](https://github.com/davidhuser/paper-municipality-emails/actions/workflows/ci.yml/badge.svg)
[![CI](https://github.com/mxmap/esorics2026/actions/workflows/ci.yml/badge.svg)](https://github.com/mxmap/esorics2026/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/mxmap/esorics2026/branch/main/graph/badge.svg)](https://codecov.io/gh/mxmap/esorics2026)
[![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![Checked with pyright](https://img.shields.io/badge/type%20checker-pyright-yellow.svg)](https://github.com/microsoft/pyright)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)

Automated pipeline to collect email domains, classify email providers, and analyze email security for municipalities in Switzerland (~2,100), Germany (~9,000), and Austria (~2,350).

Expand All @@ -12,39 +17,24 @@ Automated pipeline to collect email domains, classify email providers, and analy

## How to run

Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/) then:
Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/), then:

```bash
uv sync # install dependencies

# Stage 1: Resolve email domains
uv run resolve ch # resolve Swiss municipalities
uv run resolve de # resolve German municipalities
uv run resolve at # resolve Austrian municipalities
uv run resolve --all # resolve all three countries
uv run resolve ch --dry-run # statistics only, no scraping
uv run resolve ch -v # verbose logging
uv run resolve ch --no-cache # ignore cached network results

# Stage 2: Classify email providers
uv run classify ch # classify Swiss municipality providers
uv run classify de # classify German municipality providers
uv run classify at # classify Austrian municipality providers

# Stage 3: Security scan (requires Docker)
uv run scan ch # scan Swiss municipalities
uv run scan de # scan German municipalities
uv run scan at # scan Austrian municipalities
uv run scan ch -v # verbose (streams Docker output)
uv sync # install deps
uv run resolve <ch|de|at|--all> # Stage 1: resolve email domains
uv run classify <ch|de|at> # Stage 2: classify providers
uv run scan <ch|de|at> # Stage 3: DANE/SPF/DMARC (requires Docker)
```

Common flags: `--dry-run`, `-v`, `--no-cache`. See `uv run <cmd> --help`.

> [!IMPORTANT]
> This tool requires unrestricted outbound port 25 (SMTP). Most residential ISPs and laptops block this.
> For best results, run from a cloud VM with port 25 access opened.

## Validation

The classifier is validated via bounce probing — a separate, manual process (not part of the pipeline above) that sends probe emails to a stratified sample of municipalities and parses the resulting NDRs to identify the actual backend MTA. The validation tooling is maintained in a separate repository (available on request).
The classifier is validated via bounce probing — a separate, manual process (not part of the pipeline above) that sends probe emails to a stratified sample of municipalities and parses the resulting NDRs to identify the actual backend MTA. The validation tooling and data is maintained in a separate repository.

## Output files

Expand All @@ -62,15 +52,17 @@ Results are written to `output/`:

## Maps

Interactive Leaflet maps visualize provider classification per country:
Interactive Leaflet maps visualizing email provider distribution and security posture are available in the `maps/` directory.

To view them:

```bash
python3 -m http.server # from the project root
# then open http://localhost:8000/maps/
```

- [Switzerland](maps/ch.html) — ~2,100 municipalities
- [Austria](maps/at.html) — ~2,100 municipalities
- [Germany](maps/de.html) — ~11,100 municipalities
## License

First-party code in this repository is released under the **MIT License** — see [LICENSE](LICENSE).

Each map colors municipalities by email jurisdiction (domestic, US cloud, foreign) with confidence-level shading and interactive popups showing MX/SPF records and classification signals.
Third-party subtrees retain their own licenses. Notably, `src/security_test/scanner/testssl.sh/` is vendored under **GPL-2.0** (see the `LICENSE` file in that directory). Redistribution of the repository as a combined binary artifact (e.g., a single Docker image) is therefore constrained by GPL-2.0 terms; the MIT grant applies to the first-party Python code.
Loading