diff --git a/README.md b/README.md index c3f2f850..2ebea70e 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ DevCloud is an **on-ramp to the cloud**, not a replacement for it. The goal is t |---------|----------|---------|------| | S3 | REST-XML | Filesystem + SQLite | [docs/services/s3.md](docs/services/s3.md) | | SQS | Query + JSON | In-memory | [docs/services/sqs.md](docs/services/sqs.md) | -| DynamoDB | JSON 1.0 | BadgerDB | [docs/services/dynamodb.md](docs/services/dynamodb.md) | +| DynamoDB | JSON 1.0 | SQLite | [docs/services/dynamodb.md](docs/services/dynamodb.md) | | Lambda | REST-JSON | SQLite + Filesystem | [docs/services/lambda.md](docs/services/lambda.md) | | IAM/STS | Query | SQLite | [docs/services/iam-sts.md](docs/services/iam-sts.md) | diff --git a/changes/unreleased/Documentation-20260913-212251.yaml b/changes/unreleased/Documentation-20260913-212251.yaml new file mode 100644 index 00000000..59bee880 --- /dev/null +++ b/changes/unreleased/Documentation-20260913-212251.yaml @@ -0,0 +1,5 @@ +kind: Documentation +body: 'The five per-service pages now list the operations DevCloud actually serves, derived from the fidelity manifest rather than from memory: S3 documented 8 of 37, DynamoDB 8 of 20, Lambda 6 of 25 and IAM 6 of 58, and each page''s limitations denied features that had since been implemented' +time: 2026-09-13T21:22:51.981235+09:00 +custom: + Issue: "167" diff --git a/docs/configuration.md b/docs/configuration.md index 30549722..5b7b0fd4 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -96,9 +96,6 @@ logging: | `server.port` | `4747` | HTTP server port | | `services..enabled` | `false` | **Required per entry.** Listing a service is not enough — `enabled: true` still has to be set. | | `services..data_dir` | `./data/` | Data directory for persistent storage | -| `services.lambda.runtime` | `""` | Lambda runtime configuration | -| `services.lambda.warm_containers` | `0` | Warm containers to keep | -| `services.iam.enforce_policies` | `false` | Enforce IAM policies (experimental) | | `admin.enabled` | `false` | Serve the admin REST API at `/devcloud/api/*` | | `logging.level` | `info` | `debug`, `info`, `warn`, `error` | | `logging.format` | `text` | `text` or `json` | @@ -114,6 +111,13 @@ when empty. > There is no `auth` key: SigV4 signature validation is not implemented and any > credentials are accepted. +> A service entry takes **only** `enabled` and `data_dir` — that is the whole of +> [`config.ServiceConfig`](https://github.com/skyoo2003/devcloud/blob/main/internal/config/config.go). +> Per-service tuning keys do not exist, and because parsing is non-strict an +> invented one (`services.lambda.warm_containers`) is dropped without a word +> rather than rejected. If a knob is not in the table above, writing it does +> nothing. + ## Provider namespacing DevCloud serves AWS today and is prepared to serve more ([roadmap](roadmap.md)), @@ -148,7 +152,7 @@ block** — one under its forward-compatible name, one under its historical one. | Service | Default `data_dir` | Backend | Contents | |---------|-------------------|---------|----------| | S3 | `./data/s3` | Filesystem + SQLite | Object files, `metadata.db` | -| DynamoDB | `./data/dynamodb` | BadgerDB | BadgerDB data files | +| DynamoDB | `./data/dynamodb` | SQLite | `dynamodb.db` (tables, items, TTL config, tags) | | IAM | `./data/iam` | SQLite | `iam.db` (users, roles, keys) | | STS | `./data/sts` | Shared with IAM | Uses IAM's database | | Lambda | `./data/lambda` | SQLite + Filesystem | `lambda.db`, `code/` | diff --git a/docs/contributing.md b/docs/contributing.md index 76397dbf..3c195241 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -130,8 +130,10 @@ DEVCLOUD_UPDATE_DOCS=1 go test ./cmd/devcloud/ -run TestUpdatePublishedFigures 3. **Implement the provider** in `internal/services//provider.go`. Start with the most commonly used operations; the generated base provider makes everything else return `NotImplementedError`. -4. **Implement the store** in `store.go` — SQLite for relational metadata, - BadgerDB for key-value, in-memory for ephemeral, filesystem for blobs. +4. **Implement the store** in `store.go` — SQLite + ([`internal/storage/sqlite`](../internal/storage/sqlite)) for anything + persistent, in-memory for ephemeral, filesystem for blobs. SQLite is the only + embedded database in the tree; adding a second one needs a reason in the PR. 5. **Register the plugin** from an `init()` in `register.go`, and blank-import the package in [`cmd/devcloud/imports.go`](../cmd/devcloud/imports.go). The interface contract, error convention and config keys are in diff --git a/docs/services/dynamodb.md b/docs/services/dynamodb.md index 847fc2da..87676890 100644 --- a/docs/services/dynamodb.md +++ b/docs/services/dynamodb.md @@ -2,22 +2,38 @@ ## Overview -DevCloud DynamoDB uses BadgerDB as its embedded key-value storage backend. Table metadata is kept in an in-memory index (protected by RWMutex), while items are persisted in BadgerDB with composite keys (`_item/{table}#{partitionKey}#{sortKey}`). +DevCloud DynamoDB persists tables and items in SQLite (`dynamodb.db` under the +service's `data_dir`), with table metadata also held in an in-memory index +guarded by an RWMutex. -Supported attribute types: S (String), N (Number), B (Binary), BOOL, NULL, L (List), M (Map). +All ten AttributeValue types are supported: S, N, B, BOOL, NULL, L (List), +M (Map), and the SS / NS / BS sets. ## Supported APIs +These 20 operations are `hand-verified` — implemented by the provider, not by +the [CRUD engine](../crud-engine.md). Everything else DynamoDB models is served +at a lower tier or not at all; [fidelity-manifest.md](../fidelity-manifest.md) +is the per-operation answer. + | Operation | Description | |-----------|-------------| -| CreateTable | Create table with partition key (HASH) and optional sort key (RANGE) | -| DeleteTable | Delete table and all its items | -| ListTables | List all table names | -| PutItem | Insert or overwrite an item | -| GetItem | Retrieve an item by primary key | -| DeleteItem | Delete an item by primary key | -| Query | Query items by partition key | -| Scan | Full table scan | +| CreateTable | Create table with partition key (HASH), optional sort key (RANGE), GSIs, LSIs and a StreamSpecification | +| UpdateTable / DescribeTable / DeleteTable / ListTables | Manage and inspect tables | +| PutItem | Insert or overwrite an item; honours `ConditionExpression` | +| GetItem | Retrieve an item by primary key; honours `ProjectionExpression` | +| UpdateItem | Apply an `UpdateExpression` to one item | +| DeleteItem | Delete an item by primary key; honours `ConditionExpression` | +| Query | Query by partition key, against the table or a named `IndexName`; honours `FilterExpression` | +| Scan | Full table scan; honours `FilterExpression` | +| BatchGetItem / BatchWriteItem | Multi-item reads and writes | +| TransactGetItems / TransactWriteItems | Transactional reads and writes | +| UpdateTimeToLive / DescribeTimeToLive | Store and read back a table's TTL configuration | +| TagResource / UntagResource / ListTagsOfResource | Manage table tags | + +Writes to a table created with `StreamEnabled` are published to the +`dynamodbstreams` service, which is what makes DynamoDB Streams → Lambda event +source mappings fire. ## boto3 Examples @@ -108,12 +124,21 @@ aws --endpoint-url http://localhost:4747 dynamodb get-item \ ## Known Limitations -- No UpdateItem (use PutItem to overwrite entire item) -- No batch operations (BatchGetItem, BatchWriteItem) -- No transactions (TransactGetItems, TransactWriteItems) -- No secondary indexes (GSI/LSI) -- No DynamoDB Streams -- No advanced filter expressions on Query/Scan -- No projection expressions -- No conditional writes (ConditionExpression) -- No TTL +- **TTL is configuration only.** `UpdateTimeToLive` stores the attribute name + and echoes it back; nothing sweeps expired items, so a row past its TTL is + still returned. +- **No pagination or capacity reporting.** Responses carry no + `LastEvaluatedKey` and no `ConsumedCapacity`, so `Query`/`Scan` return the + whole matching set in one page and code that loops on the cursor sees one + iteration. +- **No PartiQL** — `ExecuteStatement`, `ExecuteTransaction` and + `BatchExecuteStatement` are unimplemented and fail rather than answering. +- **Backups, global tables and exports answer from the CRUD engine.** + `CreateBackup`, `CreateGlobalTable`, `DescribeContinuousBackups` and their + neighbours return stored, plausible shapes with no behaviour behind them — + see [crud-engine.md](../crud-engine.md). +- No Kinesis streaming destination (`EnableKinesisStreamingDestination` is + unimplemented) +- No provisioned-throughput accounting or throttling; `BillingMode` is recorded, + never enforced +- Single account model (account ID: `000000000000`) diff --git a/docs/services/iam-sts.md b/docs/services/iam-sts.md index 2246fe85..43d81c12 100644 --- a/docs/services/iam-sts.md +++ b/docs/services/iam-sts.md @@ -8,14 +8,21 @@ Both services use the Query protocol (form-encoded requests, XML responses). ## Supported IAM APIs -| Operation | Description | -|-----------|-------------| -| CreateUser | Create an IAM user | -| ListUsers | List all IAM users | -| CreateRole | Create an IAM role with assume role policy document | -| ListRoles | List all IAM roles | -| AttachRolePolicy | Attach a managed policy ARN to a role | -| CreateAccessKey | Generate an access key pair for a user | +These 58 operations are `hand-verified` — implemented by the provider, not by +the [CRUD engine](../crud-engine.md). Grouped by the resource they act on; +[fidelity-manifest.md](../fidelity-manifest.md) is the per-operation answer. + +| Resource | Operations | +|----------|------------| +| Users | CreateUser, GetUser, UpdateUser, DeleteUser, ListUsers | +| Groups | CreateGroup, GetGroup, DeleteGroup, ListGroups, AddUserToGroup, RemoveUserFromGroup | +| Roles | CreateRole, GetRole, DeleteRole, ListRoles, UpdateAssumeRolePolicy | +| Instance profiles | CreateInstanceProfile, GetInstanceProfile, DeleteInstanceProfile, ListInstanceProfiles, AddRoleToInstanceProfile, RemoveRoleFromInstanceProfile | +| Managed policies | CreatePolicy, GetPolicy, DeletePolicy, CreatePolicyVersion, GetPolicyVersion, ListPolicyVersions | +| Policy attachment | Attach/Detach {User,Group,Role}Policy, ListAttached{User,Group,Role}Policies | +| Inline policies | Put/Get/Delete {User,Group,Role}Policy, List{User,Group,Role}Policies | +| Access keys | CreateAccessKey, UpdateAccessKey, DeleteAccessKey, ListAccessKeys | +| Tags | TagUser, UntagUser, ListUserTags, TagRole, UntagRole, ListRoleTags | ## Supported STS APIs @@ -23,7 +30,8 @@ Both services use the Query protocol (form-encoded requests, XML responses). |-----------|-------------| | GetCallerIdentity | Return account ID, ARN, and user ID | | AssumeRole | Generate temporary credentials (ASIA-prefixed keys, 1-hour expiry) | -| GetSessionToken | Generate MFA-backed session credentials | +| GetSessionToken | Generate session credentials | +| GetAccessKeyInfo | Return the account an access key ID belongs to | ## boto3 Examples @@ -110,18 +118,20 @@ aws --endpoint-url http://localhost:4747 sts assume-role \ ## Known Limitations **IAM:** -- No GetUser, DeleteUser, UpdateUser -- No DeleteRole, UpdateRole -- No inline policies (PutRolePolicy, PutUserPolicy) -- No groups +- **No policy evaluation.** Managed and inline policy documents are stored and + returned verbatim; nothing parses or enforces them, so attaching a `Deny` to a + user changes nothing about what that user can call. - No MFA device management - No login profiles / password management -- No tagging -- No policy enforcement — policies are stored but not evaluated +- No service-linked roles, SAML or OIDC identity providers +- No access advisor, credential reports, or policy simulation **STS:** -- Temporary credentials are generated but not tracked or validated -- Fixed 1-hour expiration (no custom duration) -- No external ID validation -- No policy enforcement on assumed roles +- Temporary credentials are generated but not tracked or validated — they are + never checked on a later request, and neither are long-lived ones +- Fixed 1-hour expiration; `DurationSeconds` is ignored +- `AssumeRole` does not evaluate the target role's trust policy, and no + `ExternalId` is required or checked +- `GetSessionToken` ignores `SerialNumber` / `TokenCode` — there is no MFA +- No `AssumeRoleWithWebIdentity` or `AssumeRoleWithSAML` - Single account model (account ID: `000000000000`) diff --git a/docs/services/lambda.md b/docs/services/lambda.md index 8967807b..187f7fe0 100644 --- a/docs/services/lambda.md +++ b/docs/services/lambda.md @@ -6,14 +6,27 @@ DevCloud Lambda stores function metadata in SQLite and function code (ZIP files) ## Supported APIs +These 25 operations are `hand-verified` — implemented by the provider, not by +the [CRUD engine](../crud-engine.md). Read the first row with the limitations +below: the control plane is real, the data plane is not. + | Operation | Description | |-----------|-------------| +| Invoke | Accepts the call and returns a placeholder — **your code never runs** | | CreateFunction | Create function with base64-encoded ZIP code | -| ListFunctions | List all functions in the account | -| GetFunction | Get function metadata and code location | -| DeleteFunction | Delete a function | -| UpdateFunctionCode | Update function code ZIP | -| Invoke | Invoke function (stub — returns placeholder response) | +| ListFunctions / GetFunction / DeleteFunction | Manage functions | +| UpdateFunctionCode / UpdateFunctionConfiguration | Update code ZIP or configuration | +| PublishVersion / ListVersionsByFunction | Immutable published versions | +| CreateAlias / GetAlias / UpdateAlias / DeleteAlias / ListAliases | Aliases onto versions | +| CreateEventSourceMapping / GetEventSourceMapping / UpdateEventSourceMapping | Wire an SQS queue or DynamoDB stream to a function | +| DeleteEventSourceMapping / ListEventSourceMappings | Manage those mappings | +| AddPermission / GetPolicy / RemovePermission | Resource-based policy (stored, not evaluated) | +| TagResource / UntagResource / ListTags | Function tags | + +Event source mappings are polled for real: the poller reads from the SQS queue +or DynamoDB stream, builds the AWS-shaped event, and POSTs it to the function's +invoke endpoint. That endpoint is the stub, so the wiring is observable end to +end while the handler body is not. ## boto3 Examples @@ -89,11 +102,20 @@ aws --endpoint-url http://localhost:4747 lambda invoke \ ## Known Limitations -- **No code execution** — Invoke returns a placeholder response. Docker runtime integration is planned but not yet implemented. +- **No code execution.** `Invoke` returns + `{"statusCode": 200, "body": "Lambda invoke requires Docker runtime"}` whatever + the function or payload. Nothing in the response distinguishes it from a real + result, so a test asserting only on the status code passes against a handler + that never ran. Docker runtime integration is planned, not implemented. +- **Event source mappings deliver to that stub.** Messages are read from the + source and the invoke is issued, so the plumbing is testable — but no handler + logic runs, and a delivery failure is logged rather than retried or sent to a + DLQ. +- **Resource-based policies are stored, never evaluated.** `AddPermission` + succeeds and `GetPolicy` reads it back; no invoke is ever denied by one. - No layers -- No aliases or versions -- No event source mappings -- No concurrency controls +- No concurrency controls (reserved or provisioned) - No function URLs -- No resource-based policies -- No environment variables support +- No environment variables — `Environment` is not part of the parsed + `CreateFunction` request, so it is dropped without a warning and `GetFunction` + will not return it diff --git a/docs/services/s3.md b/docs/services/s3.md index 2da25086..736c30f0 100644 --- a/docs/services/s3.md +++ b/docs/services/s3.md @@ -6,16 +6,36 @@ DevCloud S3 provides basic object storage using the filesystem for object data a ## Supported APIs +These 37 operations are `hand-verified` — implemented by the provider, not by +the [CRUD engine](../crud-engine.md). A sub-resource this provider does not +serve (`?lifecycle`, `?encryption`, …) returns a clean AWS error rather than +being quietly answered as a bucket listing; +[fidelity-manifest.md](../fidelity-manifest.md) is the per-operation answer. + | Operation | Description | |-----------|-------------| -| ListBuckets | List all buckets in the account | -| CreateBucket | Create a new bucket | -| DeleteBucket | Delete an empty bucket | -| ListObjects | List objects in a bucket (supports prefix filtering) | +| ListBuckets / CreateBucket / DeleteBucket / HeadBucket | Bucket lifecycle | +| GetBucketLocation | Returns the fixed region | +| ListObjects / ListObjectsV2 | List objects, with prefix, delimiter and continuation tokens | | PutObject | Upload an object (computes MD5 ETag) | -| GetObject | Download an object with metadata headers | -| HeadObject | Retrieve object metadata without body | -| DeleteObject | Delete an object from a bucket | +| GetObject / HeadObject | Download an object, or read its metadata alone | +| CopyObject | Server-side copy via `x-amz-copy-source` | +| DeleteObject / DeleteObjects | Delete one object, or a batch | +| CreateMultipartUpload / UploadPart / CompleteMultipartUpload | Multipart upload | +| AbortMultipartUpload / ListMultipartUploads / ListParts | Inspect and abandon multipart uploads | +| GetBucketTagging / PutBucketTagging / DeleteBucketTagging | Bucket tags | +| GetObjectTagging / PutObjectTagging / DeleteObjectTagging | Object tags | +| GetBucketCors / PutBucketCors / DeleteBucketCors | CORS configuration (stored, not applied) | +| GetBucketPolicy / PutBucketPolicy / DeleteBucketPolicy | Bucket policy (stored, not evaluated) | +| GetBucketAcl / PutBucketAcl | Bucket ACL (stored, not evaluated; a canned full-control ACL is returned when unset) | +| GetBucketVersioning / PutBucketVersioning | Versioning status (stored, not applied) | +| GetBucketNotificationConfiguration / PutBucketNotificationConfiguration | Event notifications — these **do** fire | + +Notification configuration is the one of those that has behaviour behind it: +`PutObject`, `CopyObject`, `CompleteMultipartUpload` and `DeleteObject` emit +`ObjectCreated:*` / `ObjectRemoved:Delete` events and deliver them to the +configured SQS queue or Lambda function, which is what makes the S3 → Lambda +integration work. ## boto3 Examples @@ -75,12 +95,19 @@ aws --endpoint-url http://localhost:4747 s3 cp s3://my-bucket/file.txt ./downloa ## Known Limitations -- No multipart upload support (CreateMultipartUpload, UploadPart, CompleteMultipartUpload) -- No object versioning or object lock -- No ACL or bucket policy management -- No encryption (SSE-S3, SSE-KMS) -- No CORS configuration -- No lifecycle policies -- No replication -- No presigned URLs +The pattern below is worth reading once: several configuration sub-resources +round-trip faithfully but change nothing about how requests are served. + +- **Versioning is a stored status, not versions.** `PutBucketVersioning` + records `Enabled`, and `GetBucketVersioning` reads it back, but no version IDs + are assigned and overwriting an object still destroys the previous one. +- **ACLs and bucket policies are stored, never evaluated.** No request is ever + denied by one. There is no object lock. +- **CORS configuration is stored, never applied.** DevCloud does not emit + `Access-Control-*` response headers or answer preflight requests from it. +- **Presigned URLs are not verified.** SigV4 signatures are not validated + anywhere, so a presigned request is served like any other and an expired or + forged one succeeds. +- No server-side encryption (SSE-S3, SSE-KMS) — `?encryption` is unimplemented +- No lifecycle policies, replication, or storage classes - Single account model (account ID: `000000000000`)