Skip to content

docs: add design proposal for EROFS rootfs support#283

Open
novahe wants to merge 1 commit into
TencentCloud:masterfrom
novahe:doc/erofs-rootfs
Open

docs: add design proposal for EROFS rootfs support#283
novahe wants to merge 1 commit into
TencentCloud:masterfrom
novahe:doc/erofs-rootfs

Conversation

@novahe
Copy link
Copy Markdown
Contributor

@novahe novahe commented May 16, 2026

Description

This PR introduces a comprehensive design proposal for integrating EROFS (Enhanced Read-Only File System) into the CubeSandbox runtime for container Rootfs artifacts. The goal is to optimize storage efficiency, distribution speed, and cold-start performance for read-only layers.

Key Highlights:

  • End-to-End Lifecycle: Defines the integration from CubeMaster build pipeline to Cubelet distribution and CubeShim runtime mount execution.
  • Engineering Hardening: Specifies mandatory erofs-utils versioning (>= 1.5.0) and xattr support to preserve OverlayFS metadata (Whiteouts).
  • Performance & Observability: Outlines a detailed benchmarking plan and critical metrics (compression ratios, cold-start latency) for production readiness.

Note: In this proposal, the Guest OS rootfs remains using the existing ext4 scheme to maintain stability, while workload Rootfs artifacts are extended to support EROFS.

Related Issues

Relates to #274

@novahe novahe requested a review from tinklone as a code owner May 16, 2026 02:00
@kinwin-ustc
Copy link
Copy Markdown
Collaborator

Thank you for your very detailed proposal. I think we can only support EROFS for container images, as the usage of guest images is extremely limited in our design. In typical scenarios, there should only be one guest image globally, which should be pre-deployed on each computing node and shared with all sandboxes through pmem. EROFS almost does not bring any benefits to the guest image, so we can focus on the container image scenario. In fact, we do use EROFS to support container images internally.

@charliecgxu @ls-ggg @hlz260 @zhuangel Do you have any suggestions?

@novahe novahe force-pushed the doc/erofs-rootfs branch 2 times, most recently from 11b7bfb to 22c8994 Compare May 16, 2026 14:17
This KEP proposes integrating EROFS (Enhanced Read-Only File System) as a
container Rootfs artifact support within CubeSandbox.

Key features:
- Support for EROFS artifacts with LZ4 compression.
- Explicit fs_type metadata propagation between CubeMaster, Cubelet, and CubeShim.
- Optimization for high-density and high-concurrency AI Agent sandbox scenarios.
- Strict scope limitation to container Rootfs (Guest OS remains ext4).

Signed-off-by: novahe <heqianfly@gmail.com>
@novahe novahe force-pushed the doc/erofs-rootfs branch from 22c8994 to dff8ccd Compare May 16, 2026 15:21
@novahe
Copy link
Copy Markdown
Contributor Author

novahe commented May 16, 2026

I agree with you. Since multiple sandboxes share a single Guest OS rootfs, the gains are pretty minimal, and we'd lose the DAX capability. Let's keep our proposal focused on the container rootfs instead.

I've updated the version, PTAL @kinwin-ustc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants