Skip to content

feat: add S3-compatible object storage backend - #131

Merged
carmark merged 1 commit into
masterfrom
feat/s3-backend-store
Mar 16, 2026
Merged

carmark merged 1 commit into
masterfrom
feat/s3-backend-store

Conversation

@carmark

@carmark carmark commented Mar 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add S3-compatible object storage as a new backend store for iSCSI targets
  • The virtual block device is divided into fixed-size chunks (default 4 MiB), each stored as an independent S3 object, enabling efficient random read/write on top of object storage
  • Compatible with AWS S3, MinIO, Ceph RGW, and other S3-compatible services
  • Add CI test step using in-process gofakes3 server (pure Go, no Docker dependency)
  • Update README with S3 backend documentation, configuration reference, and performance considerations

Changes

New files:

  • pkg/scsi/backingstore/s3store/s3store.go - Core S3 backend implementation
  • pkg/scsi/backingstore/s3store/s3store_test.go - 12 unit tests with in-memory fake S3 client
  • pkg/scsi/backingstore/s3store/integration_test.go - 2 integration tests using gofakes3
  • examples/config-s3.json - Example S3 backend configuration

Modified files:

  • pkg/api/types.go - Add BackendConfig field to SCSILu for backend-specific config
  • pkg/config/config.go - Add S3 configuration fields (endpoint, region, chunk size, etc.)
  • pkg/scsi/lun.go - Thread config to backend, add s3 to known backend types
  • pkg/scsi/backingstore/common.go - Add s3 to path parser
  • cmd/daemon.go - Register S3 backend store
  • .github/workflows/gotgt.yml - Add S3 backend test step
  • README.md - Document S3 backend features and configuration
  • go.mod / go.sum - Add AWS SDK v2 and gofakes3 dependencies

Test plan

  • 12 unit tests pass (fake S3 client: Open, Read, Write, Unmap, cross-chunk, sparse)
  • 2 integration tests pass (full lifecycle with gofakes3 in-process server)
  • go build ./... succeeds
  • gofmt verification passes
  • CI workflow passes with new S3 backend test step

🤖 Generated with Claude Code

Add a new backend store that enables iSCSI targets backed by
S3-compatible object storage (AWS S3, MinIO, Ceph RGW, etc.).

The implementation uses a chunked storage strategy where the virtual
block device is divided into fixed-size chunks (default 4 MiB), each
stored as an independent S3 object. This enables efficient random
read/write access on top of object storage.

Key features:
- Chunked storage with configurable chunk size
- Sparse device support (unwritten chunks treated as zeros)
- Concurrent multi-chunk I/O via errgroup
- Per-chunk locking for safe read-modify-write
- AWS SDK v2 with default credential chain
- In-process gofakes3 test server (no Docker needed)
- 12 unit tests + 2 integration tests

Also updates CI workflow to run S3 backend tests and updates
README with S3 backend documentation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@carmark
carmark merged commit a75e34e into master Mar 16, 2026
1 check passed
@carmark
carmark deleted the feat/s3-backend-store branch March 16, 2026 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant