ansible-dingofs

Ansible playbooks and roles for deploying and managing DingoFS meta server clusters, DingoFS Cache nodes, and DingoFS Client (FUSE mount) nodes on CentOS/RHEL/Rocky and Ubuntu/Debian Linux.

Prerequisites

Control node: Linux with Ansible >= 2.12 installed
Target nodes: CentOS/RHEL/Rocky Linux (3 nodes minimum for HA)
Network: All target nodes reachable via SSH from the control node
Privileges: Root or sudo access on target nodes

Repository Structure

ansible-dingofs/
├── ansible.cfg                  # Ansible configuration
├── inventory/
│   ├── hosts.yml                # Inventory (edit with your IPs)
│   └── group_vars/all.yml       # Global variables (edit to customize)
├── playbooks/
│   │                            # --- meta server ---
│   ├── meta_site.yml            # Full meta deployment (01 -> 05, NOT self-contained:
│   │                            #   see the Meta Deployment section for the two
│   │                            #   steps it leaves out)
│   ├── 01_prepare.yml           # System prep: user, /etc/hosts, SSH, tools
│   ├── 01b_lvm_data.yml         # NVMe LVM data disks at /mnt/diskN
│   ├── 02_tune.yml              # Performance tuning
│   ├── 03_rqlite.yml            # RQLite database cluster
│   ├── 04_dingo_cli.yml         # Podman + dingo CLI installation
│   ├── 05_deploy_cluster.yml    # DingoFS cluster deployment
│   ├── 99_status.yml            # Cluster health check
│   │                            # --- cache ---
│   ├── cache_site.yml           # Full cache deployment (cache_01 -> 05)
│   ├── cache_01_prepare.yml     # Cache: basic tools, cpupower
│   ├── cache_02_lvm.yml         # Cache: NVMe LVM setup
│   ├── cache_02b_remap.yml      # Cache: normalize mount layout by disk size
│   ├── cache_03_tune.yml        # Cache: performance tuning
│   ├── cache_04_jemalloc.yml    # Cache: compile & install jemalloc
│   ├── cache_05_deploy.yml      # Cache: dingo-cache service
│   ├── cache_06_upgrade.yml     # Cache: upgrade binaries
│   ├── cache_99_status.yml      # Cache: health check
│   │                            # --- client ---
│   ├── client_site.yml          # Full client deployment (client_01 -> 06)
│   ├── client_01_prepare.yml    # Client: basic tools, cpupower
│   ├── client_02_tune.yml       # Client: performance tuning
│   ├── client_03_jemalloc.yml   # Client: compile & install jemalloc
│   ├── client_04_deploy.yml     # Client: dingo-client FUSE service
│   ├── client_05_label.yml      # Client: node labels
│   ├── client_06_dingocli.yml   # Client: ~/.dingo/dingo.yaml + dingo binary
│   ├── client_07_upgrade.yml    # Client: upgrade binaries, restart mounts
│   ├── client_99_status.yml     # Client: health check
│   │                            # --- cross-cutting / standalone ---
│   ├── config_podman_rootless.yml  # Rootless podman + systemd linger
│   └── config_cron_cleanup.yml     # Log cleanup script + daily cron job
└── roles/
    ├── common/                  # User, /etc/hosts, SSH keys, packages
    ├── tune/                    # CPU, ulimits, sysctl, hugepages
    ├── rqlite/                  # RQLite install + systemd cluster
    ├── dingo_cli/               # dingo CLI binary + config
    ├── podman/                  # Podman container engine (rootful)
    ├── podman_rootless/         # Rootless podman: linger, storage.conf, XDG
    ├── dingofs_cluster/         # Topology, image pull, cluster deploy
    ├── lvm_data/                # Meta: NVMe LVM data disks (/mnt/diskN)
    ├── dingo_env/               # ~/.dingo/dingo.yaml (mdsaddr and friends)
    ├── cache_prepare/           # Cache node system preparation
    ├── lvm_cache/               # Cache: NVMe LVM detection + setup
    ├── jemalloc/                # Compile jemalloc from source
    ├── dingo_cache/             # Dingo-cache service deployment
    ├── dingo_client/            # Dingo-client FUSE mount service
    ├── dingo_client_upgrade/    # Dingo-client upgrade
    └── cron_cleanup/            # Log cleanup script + cron job

Some playbooks are deliberately not part of any *_site.yml and must be run explicitly. The ones that matter for a working meta deployment: 01b_lvm_data.yml and client_06_dingocli.yml -- see the Meta Deployment section below.

client_06_dingocli.yml defaults to target_hosts: client_servers, so against a meta-only inventory it matches zero hosts and silently does nothing. Pass -e target_hosts=admin when running it for a meta deployment. (The same trap applied to config_podman_rootless.yml; it no longer matters for meta, because 04_dingo_cli.yml now runs the role directly.)

DingoFS Meta Deployment

1. Configure inventory

Edit inventory/hosts.yml with your server IPs and hostnames:

all:
  children:
    meta_servers:
      hosts:
        node-1:
          ansible_host: 10.0.0.1
          rqlite_node_id: 1
        node-2:
          ansible_host: 10.0.0.2
          rqlite_node_id: 2
        node-3:
          ansible_host: 10.0.0.3
          rqlite_node_id: 3
    rqlite_master:
      hosts:
        node-1:
    rqlite_slaves:
      hosts:
        node-2:
        node-3:
    admin:
      hosts:
        node-1:

2. Customize variables

Edit inventory/group_vars/all.yml to adjust:

Container images and versions
Service ports
Performance tuning parameters
Dingo CLI settings

3. Run full deployment

meta_site.yml runs phases 01-05. Two steps live outside it and have to be run around it, in this order:

INV=inventory/hosts.yml            # or -i <region>/hosts.yml

# (a) Data disks FIRST -- meta_site.yml does not create them, and
#     dingofs_cluster does not fail when they are missing: it writes the data
#     onto the system disk instead.
ansible-playbook -i $INV playbooks/01b_lvm_data.yml

# (b) Main deployment. This now also configures rootless podman for the service
#     user (04_dingo_cli.yml runs the podman_rootless role), so no separate
#     step is needed for the containers to start.
ansible-playbook -i $INV playbooks/meta_site.yml

# (c) Write ~/.dingo/dingo.yaml, which carries the MDS address used by every
#     `dingo fs ...` command. Without it the CLI falls back to 127.0.0.1:7400 and
#     each command dies with
#         Error-Code: 660000 / rpc request to mds cluster failed / context deadline exceeded
ansible-playbook -i $INV playbooks/client_06_dingocli.yml -e target_hosts=admin

# (d) Verify
ansible-playbook -i $INV playbooks/99_status.yml

Notes:

Run 01b_lvm_data.yml before meta_site.yml. The common role creates /mnt/disk1/corefiles; doing that before the disks are mounted leaves the directory on the system disk, where it is then hidden by the mount.
01b_lvm_data.yml cannot be previewed with --check: the work is done by a shell script, which check mode skips. Its closing mount assertion still runs (deliberately) and fails with "not mounted" -- that is the expected result, not a broken playbook.
Step (c) is idempotent; re-run it freely.

4. Run individual phases

# Data disks (must precede 01_prepare)
ansible-playbook playbooks/01b_lvm_data.yml

# System preparation only
ansible-playbook playbooks/01_prepare.yml

# Performance tuning only
ansible-playbook playbooks/02_tune.yml

# RQLite database only
ansible-playbook playbooks/03_rqlite.yml

# Podman + Dingo CLI only
ansible-playbook playbooks/04_dingo_cli.yml

# DingoFS cluster deployment only
ansible-playbook playbooks/05_deploy_cluster.yml

# Check cluster status
ansible-playbook playbooks/99_status.yml

Twenty of the playbooks set remote_user: root explicitly. The rest inherit remote_user from ansible.cfg (which defaults to dingofs, with become: true), and most region inventories override that with ansible_user: root in group_vars/all.yml anyway -- so in practice everything runs as root.

DingoFS Cache Deployment

Dingo Cache is deployed independently from the meta server cluster, on a separate set of cache_servers nodes.

Prerequisites

Cache nodes with unused NVMe devices (for LVM setup)
jemalloc source tarball on the control node (default: /root/dingofs/pkg/jemalloc-5.3.0.tar.bz2)
dingo-cache binary on the control node (default: /root/dingofs/pkg/dingo-cache)

1. Configure cache inventory

Add your cache server IPs to inventory/hosts.yml:

    cache_servers:
      hosts:
        cache-node-1:
          ansible_host: 10.0.0.10
        cache-node-2:
          ansible_host: 10.0.0.11

2. Customize cache variables

Edit inventory/group_vars/all.yml to set:

dingo_cache_mds_addrs — MDS addresses from the meta server cluster
dingo_cache_cache_dirs / dingo_cache_log_dir — must match your LVM mount layout
dingo_cache_ip_pattern — network prefix for auto-detecting the listen IP
dingo_cache_cache_size_mb — total cache size in MB
jemalloc_local_src — path to jemalloc tarball on the control node

3. Run full cache deployment

# Dry run
ansible-playbook playbooks/cache_site.yml --check

# Full deployment
ansible-playbook playbooks/cache_site.yml

4. Run individual cache phases

# System preparation (basic tools, cpupower)
ansible-playbook playbooks/cache_01_prepare.yml

# LVM setup (detect NVMe, create VGs, format, mount)
ansible-playbook playbooks/cache_02_lvm.yml

# Performance tuning
ansible-playbook playbooks/cache_03_tune.yml

# Compile and install jemalloc
ansible-playbook playbooks/cache_04_jemalloc.yml

# Deploy dingo-cache service
ansible-playbook playbooks/cache_05_deploy.yml

# Check cache status
ansible-playbook playbooks/cache_99_status.yml

All cache playbooks connect with remote_user: root.

DingoFS Client Deployment

Dingo Client (FUSE mount) is deployed independently from both meta server and cache deployments, on a separate set of client_servers nodes.

Prerequisites

jemalloc source tarball on the control node (default: /root/dingofs/pkg/jemalloc-5.3.0.tar.bz2)
dingo-client binary on the control node (default: /root/dingofs/pkg/dingo-client)
dingo CLI binary on the control node (default: /root/dingofs/pkg/dingo)

1. Configure client inventory

Add your client server IPs to inventory/hosts.yml:

    client_servers:
      hosts:
        client-node-1:
          ansible_host: 10.0.0.20
        client-node-2:
          ansible_host: 10.0.0.21

2. Customize client variables

Edit inventory/group_vars/all.yml to set:

dingo_client_mds_addrs — MDS addresses from the meta server cluster
dingo_client_fsname — filesystem name to mount
dingo_client_mount_point — where to mount the filesystem
dingo_client_cache_group — cache group name (must match dingo_cache_group_name)
FUSE/VFS tuning parameters as needed

3. Run full client deployment

# Dry run
ansible-playbook playbooks/client_site.yml --check

# Full deployment
ansible-playbook playbooks/client_site.yml

4. Run individual client phases

# System preparation (basic tools, cpupower)
ansible-playbook playbooks/client_01_prepare.yml

# Performance tuning
ansible-playbook playbooks/client_02_tune.yml

# Compile and install jemalloc
ansible-playbook playbooks/client_03_jemalloc.yml

# Deploy dingo-client FUSE mount service
ansible-playbook playbooks/client_04_deploy.yml

# Check client status
ansible-playbook playbooks/client_99_status.yml

All client playbooks connect with remote_user: root.

Meta Server Deployment Phases

Phase	Playbook	Target	Description
1	`01_prepare.yml`	All nodes	Create user, /etc/hosts, SSH keys, install packages
2	`02_tune.yml`	All nodes	CPU governor, ulimits, sysctl, hugepages
3	`03_rqlite.yml`	All nodes	RQLite database cluster (master first, then slaves)
4	`04_dingo_cli.yml`	All + admin	Podman on all nodes, Dingo CLI on admin node
5	`05_deploy_cluster.yml`	Admin node	Pull images, configure and deploy DingoFS cluster

Key Variables

Variable	Default	Description
`dingofs_user`	`dingofs`	System user for DingoFS
`dingofs_cluster_name`	`v5.1`	Cluster name identifier
`dingofs_container_image`	`harbor.zetyun.cn/dingofs/dingofs:v5.1-b484ef2`	DingoFS container image
`rqlite_version`	`v9.4.5`	RQLite version
`container_engine`	`podman`	Container engine
`tune_hugepages`	`1024`	Number of huge pages
`tune_swappiness`	`10`	VM swappiness

See inventory/group_vars/all.yml for all configurable variables.

Cluster Architecture

┌─────────────────────────────────────────────────────┐
│                  DingoFS Cluster                    │
├──────────────┬──────────────┬───────────────────────┤
│   Node 1     │   Node 2     │   Node 3              │
│  (admin)     │              │                       │
├──────────────┼──────────────┼───────────────────────┤
│ RQLite       │ RQLite       │ RQLite                │
│ (master)     │ (slave)      │ (slave)               │
├──────────────┼──────────────┼───────────────────────┤
│ Coordinator  │ Coordinator  │ Coordinator           │
│ Store        │ Store        │ Store                 │
│ MDS x3       │ MDS x3       │ MDS x3                │
│ Executor     │              │                       │
├──────────────┼──────────────┼───────────────────────┤
│ Dingo CLI    │              │                       │
│ Podman       │ Podman       │ Podman                │
└──────────────┴──────────────┴───────────────────────┘

Cache Deployment Phases

Phase	Playbook	Description
1	`cache_01_prepare.yml`	Install basic packages, cpupower, development tools
2	`cache_02_lvm.yml`	Detect unused NVMe devices, create LVM VGs, format XFS, mount
3	`cache_03_tune.yml`	CPU governor, ulimits, sysctl, hugepages (reuses tune role)
4	`cache_04_jemalloc.yml`	Copy, compile, and install jemalloc with profiling support
5	`cache_05_deploy.yml`	Configure and start dingo-cache systemd service

Cache Key Variables

Variable	Default	Description
`dingo_cache_listen_port`	`11000`	Service listen port
`dingo_cache_group_name`	`dingofs-group`	Cache group name
`dingo_cache_cache_size_mb`	`3145728`	Cache size in MB (3TB)
`dingo_cache_mds_addrs`	(must set)	Comma-separated MDS addresses
`dingo_cache_ip_pattern`	`192.168.`	Network prefix for listen IP detection
`jemalloc_version`	`5.3.0`	Jemalloc version to compile

Client Deployment Phases

Phase	Playbook	Description
1	`client_01_prepare.yml`	Install basic packages, cpupower, development tools
2	`client_02_tune.yml`	CPU governor, ulimits, sysctl, hugepages (reuses tune role)
3	`client_03_jemalloc.yml`	Copy, compile, and install jemalloc with profiling support
4	`client_04_deploy.yml`	Deploy dingo-client FUSE mount systemd service

Client Key Variables

Variable	Default	Description
`dingo_client_fsname`	`alayanew`	Filesystem name to mount
`dingo_client_mount_point`	`/dingofs/data`	FUSE mount point
`dingo_client_port`	`12000`	VFS dummy server port
`dingo_client_mds_addrs`	(must set)	Comma-separated MDS addresses
`dingo_client_cache_group`	`dingofs-group`	Cache group name
`dingo_client_fuse_max_threads`	`512`	FUSE max threads

Standalone Playbooks

Not referenced by any *_site.yml; they only run when invoked explicitly.

Playbook	Default target	Purpose
`config_podman_rootless.yml`	`client_servers`	Standalone entry point for the `podman_rootless` role: enables systemd linger for the service user, writes `storage.conf`, sets `XDG_RUNTIME_DIR`. The meta path does not need it -- `04_dingo_cli.yml` runs the same role for `meta_servers`.
`config_cron_cleanup.yml`	`all`	Deploys `cleanup_and_compression.sh` plus a daily cron job. Defaults to cleaning `dingo_cache_log_dir`; override `cron_cleanup_directories` for other layouts. The script requires at least one directory argument, so an empty list will not work.
`cache_02b_remap.yml`	`cache_servers`	Renumbers cache mounts by disk size (smallest -> `/mnt/disk1`). Check-only unless `-e remap_apply=true`.
`cache_06_upgrade.yml`	`cache_servers`	Replaces cache binaries and restarts the service.
`client_05_label.yml`	`client_servers`	Applies node labels.
`client_06_dingocli.yml`	`client_servers`	Installs the dingo binary into `~/.dingo/bin` and writes `~/.dingo/dingo.yaml`. Also needed on the meta admin node.
`client_07_upgrade.yml`	`client_servers`	Replaces client binaries and restarts mounts. Does not create missing mounts -- that is `client_04_deploy.yml`.

Override the target with -e target_hosts=<group>.

Troubleshooting

# Test connectivity
ansible meta_servers -m ping

# Check a specific node. Use the absolute path: ~/.dingo/bin is only on PATH for
# login shells, and `become` runs a non-login shell.
ansible node-1 -m shell -a "/home/dingofs/.dingo/bin/dingo cluster status" --become-user dingofs

# Rqlite health (rqlite binds to ansible_host, not localhost)
ansible admin -m uri -a "url=http://{{ rqlite_master_host }}:4001/status"

Failures that are easy to misread:

crun: sd-bus call: Permission denied while a container starts. Rootless podman has no systemd user session. 04_dingo_cli.yml runs the podman_rootless role, which calls loginctl enable-linger; if this appears, check it actually ran (it is skipped when podman_rootless_user resolves to the wrong user). To fix an existing host without a full re-run, use config_podman_rootless.yml -e target_hosts=<group>.
dingo fs ... hangs for ~30s, then Error-Code: 660000 / context deadline exceeded. ~/.dingo/dingo.yaml is missing, so the CLI fell back to the default mdsaddr=127.0.0.1:7400. Run client_06_dingocli.yml against the admin node.
Error-Code: 620002 / mkdir ... Permission denied for /mnt/diskN/dingofs-vX.Y. The version directory does not exist, or is not owned by the service user -- a freshly created XFS mount is root:root 0755. Set lvm_data_subdir so 01b_lvm_data.yml creates and chowns it.
Data ends up on the system disk. /mnt/diskN was never mounted; dingofs_cluster does not check.
Error-Code: 410009 / cluster already exist. A previous run left the cluster registered. Re-running is safe -- the role treats this as success.
Error-Code: 100000 / init SQLite database failed. ~/.dingo/dingocli.cfg still points at an old rqlite address. The role writes the config before invoking the CLI, so re-running 04_dingo_cli.yml fixes it.
Output full of … and [1A[J. The dingo CLI draws a progress spinner even when stdout is not a TTY. The deploy and status tasks strip it.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
inventory		inventory
playbooks		playbooks
roles		roles
.gitignore		.gitignore
README.md		README.md
ansible.cfg		ansible.cfg
requirements.yml		requirements.yml

Navigation Menu

Repository files navigation

ansible-dingofs

Prerequisites

Repository Structure

DingoFS Meta Deployment

1. Configure inventory

2. Customize variables

3. Run full deployment

4. Run individual phases

DingoFS Cache Deployment

Prerequisites

1. Configure cache inventory

2. Customize cache variables

3. Run full cache deployment

4. Run individual cache phases

DingoFS Client Deployment

Prerequisites

1. Configure client inventory

2. Customize client variables

3. Run full client deployment

4. Run individual client phases

Meta Server Deployment Phases

Key Variables

Cluster Architecture

Cache Deployment Phases

Cache Key Variables

Client Deployment Phases

Client Key Variables

Standalone Playbooks

Troubleshooting

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

ansible-dingofs

Prerequisites

Repository Structure

DingoFS Meta Deployment

1. Configure inventory

2. Customize variables

3. Run full deployment

4. Run individual phases

DingoFS Cache Deployment

Prerequisites

1. Configure cache inventory

2. Customize cache variables

3. Run full cache deployment

4. Run individual cache phases

DingoFS Client Deployment

Prerequisites

1. Configure client inventory

2. Customize client variables

3. Run full client deployment

4. Run individual client phases

Meta Server Deployment Phases

Key Variables

Cluster Architecture

Cache Deployment Phases

Cache Key Variables

Client Deployment Phases

Client Key Variables

Standalone Playbooks

Troubleshooting

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages