[fix][mds] Make crontab lifecycle safe; simplify scheduling

[rock-git]

Summary

Crontab tasks could outlive their owner during shutdown. The old CrontabManager::Stop() papered over this with polling: it flipped a pause flag, then looped bthread_timer_del with bthread_usleep(1000) until each timer disappeared, and separately waited on an in-flight async counter. Timer callbacks rescheduled themselves from a raw Crontab*, so any cancellation window that was missed became a use-after-free against freed memory.

Each Crontab now has an explicit Stop() / Join() lifecycle backed by reservation counting. A timer or a dispatched bthread holds a reservation until it finishes, and Join() blocks on a condition variable until the count drains. There is no pause flag, no sleep-poll, and no raw-pointer reschedule.

The MDS dev lifecycle scripts ship in the same branch: verifying this path meant redeploying MDS repeatedly, and the four-way split was getting in the way.

Changes

Scheduling model (src/mds/common/crontab.{h,cc})

• CrontabConfig gains max_times and immediately; callback is now std::function<void()> instead of void(void*). AddCrontab takes configs by value and owns them, so callers no longer keep a live reference for the task's lifetime.
• ArmLocked() reserves a pending op before publishing the timer. Only a successful bthread_timer_del or the timer callback itself returns the reservation, so a timer can never fire against a task that Join() already observed as drained.
• run_count_ is incremented at invocation admission, before the callback runs. Queued bthreads therefore can't overshoot max_times, and a throwing callback still counts.
• AddCrontab after Stop() is a no-op. Stop() is safe to call concurrently; it snapshots the task list, stops all tasks, then joins them.
• CrontabManager uses one bthread_mutex_t plus a vector of tasks; the id map and per-crontab pause/timer bookkeeping are gone.
• DescribeByJson reports stop instead of pause, and immediately now reflects configuration rather than mutable run state.

Callers

src/mds/server.cc and both client metasystems (local, mds) drop the unused void* parameter from their crontab callbacks.

Dev scripts (scripts/dev-mds/)

• clean_start.sh, deploy_mds.sh, start_mds.sh, stop_mds.sh collapse into operate_mds.sh {stop|deploy|start|restart}. restart aborts if stop or deploy fails rather than starting half-broken.
• mds_deploy_parameters becomes env.example. env.local is the local override and is gitignored; it holds host-specific values and credentials.
• mds.template.conf renders log level/verbosity from $MDS_LOG_LEVEL / $MDS_LOG_V; start_client.sh and start_cache.sh expose --log_level / --log_v flags instead of hardcoding them.

Tests

test/unit/mds/common/test_crontab.cc is rewritten around the new lifecycle: stop/join ordering, max_times under async overlap, throw handling, and add-after-stop.

Test plan

./build/bin/test/test_mds --gtest_filter='*Crontab*'

Plus a full build with -DBUILD_UNIT_TESTS=ON and an MDS restart via bash scripts/dev-mds/operate_mds.sh restart.

---

[Copilot]

Copilot review overview

🟡 Changes recommended

Critical deployment configuration issues and moderate workflow/test issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 · 2

Open (4)

• Fix mismatched logging variable names in config template · New
• Enable config rendering by default on restart · New
• Update create scripts to use the new parameter file · New
• Update assertion to verify the stop field · New

What changed in this PR

Refactors MDS crontab shutdown for safe lifecycle management and consolidates development deployment scripts.

Changes:

• Adds reservation-based Stop()/Join() scheduling lifecycle.
• Updates callbacks and lifecycle tests.
• Replaces split MDS scripts with operate_mds.sh.
• Adds configurable deployment and logging settings.

File	Summary
test/​unit/​mds/​common/​test_crontab.cc	Lifecycle tests; assertions still reference removed pause field.
src/​mds/​server.cc	Updates crontab callback signatures.
src/​mds/​common/​crontab.h	Adds lifecycle-safe scheduling API.
src/​mds/​common/​crontab.cc	Implements reservation-based shutdown and joining.
src/​client/​vfs/​metasystem/​mds/​metasystem.cc	Updates callback signatures.
src/​client/​vfs/​metasystem/​local/​metasystem.cc	Updates callback signatures.
scripts/​dev-mds/​stop_mds.sh	Superseded by consolidated operations script.
scripts/​dev-mds/​start_mds.sh	Superseded by consolidated operations script.
scripts/​dev-mds/​start_client.sh	Adds configurable logging flags.
scripts/​dev-mds/​start_cache.sh	Adds configurable logging flags.
scripts/​dev-mds/​operate_mds.sh	Consolidates operations; config rendering defaults and multi-process waiting need fixes.
scripts/​dev-mds/​mds.template.conf	Adds logging placeholders; variable names do not match substitutions.
scripts/​dev-mds/​env.example	Adds environment template; documented consumers still use the removed filename.
scripts/​dev-mds/​deploy_mds.sh	Superseded by consolidated operations script.
scripts/​dev-mds/​clean_start.sh	Superseded by consolidated operations script.
.gitignore	Ignores local environment configuration.
.agents/​skills/​dev-deploy/​SKILL.md	Updates deployment workflow documentation.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.