Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions apps/docs/environment-variables.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -263,6 +263,7 @@ as per-task auth tokens or workspace paths.
| `MODAL_ENDPOINT` | Optional | Modal endpoint override. |
| `MODAL_ENVIRONMENT` | Optional | Modal environment name. |
| `MODAL_APP_NAME` | Optional | Modal app name override. |
| `MODAL_BASE_IMAGE_REF` | Optional | Worker image Modal sandboxes start from. Defaults to the published image for the running release. Mutable tags such as `develop` are resolved to their current digest at launch; ECR refs are used as-is, so pin them. |
| `MODAL_REGIONS` | Optional | Comma-separated Modal sandbox placement regions (for example `us` or `us-west`). Unset keeps Modal default placement. |
| `MODAL_VM_MEMORY_MIB` | Optional | Memory allocated to Modal VM sandboxes used for nested Docker workloads. Defaults to `8192` MiB. |
| `E2B_API_KEY` | E2B | E2B API key. Can also be saved from **Settings > Sandboxes**. |
Expand Down
12 changes: 12 additions & 0 deletions apps/docs/providers/compute/modal.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,18 @@ the task. Roomote starts Docker as the VM sandbox's primary service and requests
standard sandbox runtime with 2 CPU cores and 4 GiB of memory.
Container builds and services share the task sandbox's CPU and memory.

## Worker image

Modal sandboxes start from the published Roomote worker image for the running
release. Override it with `MODAL_BASE_IMAGE_REF` to use a fork or registry
mirror. Modal caches images by their reference string, so a mutable tag such as
`develop` or `latest` would never be re-pulled after the first launch. Roomote
resolves mutable tags to their current digest at launch time and reuses the
last resolved digest if the registry is briefly unreachable. Immutable tags
(`develop-<sha>`, `main-<sha>`, `v*`) and `@sha256` references are used as-is.
ECR references are not resolved because the Roomote server holds no AWS
credentials of its own; pin a digest or enable ECR tag immutability there.

## Verify setup

1. save Modal credentials
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import type {
OpenCodeSession,
OpenCodeSessionMessage,
} from './types';
import { redactSecrets } from '@roomote/communication/redact-secrets';

// Bound every unary OpenCode HTTP call so a wedged server cannot leave the
// worker sitting forever on a bare `fetch` with only the task-wide cancel
Expand Down Expand Up @@ -484,7 +485,11 @@ export class OpenCodeServerClient {
)}`,
);
throw new Error(
`OpenCode request failed method=${method} path=${path} status=${response.status}`,
`OpenCode request failed method=${method} path=${path} status=${response.status}${
responseText
? ` body=${redactSecrets(responseText.slice(0, 300))}`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

responseText is redacted only in the thrown error, but the preceding this.logger.warn still writes its first 500 raw characters to harness.log. That log is exposed through the Logs sidebar, so a response body containing credentials can still leak despite this change's redaction goal. Redact the logger's body value as well (before logging it).

: ''
}`,
);
} catch (error) {
const elapsedMs = Date.now() - startedAt;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3602,9 +3602,12 @@ export class OpenCodeServerHarness
const message = error instanceof Error ? error.message : String(error);
const sessionId = 'opencode-session-create-failed';
const timeoutMs = this.client.sessionCreateTimeoutMsValue;
// The request error may carry a slice of the raw OpenCode response body,
// so redact it and render it as code before it reaches the transcript.
const safeMessage = redactSecrets(message).replace(/`/gu, "'");
const userText = message.includes('did not respond within')
? message
: `OpenCode session creation failed before the agent could start.\n\n${message}\n\nOpen the Logs sidebar and inspect harness.log for OpenCode lines (prefixed [opencode-server]).\n\n${formatOpenCodeSessionCreateTimeoutText(timeoutMs)}`;
: `OpenCode session creation failed before the agent could start.\n\n\`\`\`\n${safeMessage}\n\`\`\`\n\nOpen the Logs sidebar and inspect harness.log for OpenCode lines (prefixed [opencode-server]).\n\n${formatOpenCodeSessionCreateTimeoutText(timeoutMs)}`;

this.logger.error(
`OpenCode initial session create failed; failing the task terminally error=${message}`,
Expand Down
38 changes: 38 additions & 0 deletions packages/compute-providers/src/adapters/modal.test.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

62 changes: 56 additions & 6 deletions packages/compute-providers/src/adapters/modal.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,40 @@ import {
toAbortError,
throwIfAborted,
} from '../modal/abort';
import {
isImmutableImageTag,
parseImageRef,
pinModalBaseImageRef,
} from '../modal/registry-digest';
import { normalizeModalRpcError } from '../modal/rpc-diagnostics';

const DEFAULT_APP_NAME = 'roomote';

const warnedUnpinnedEcrRefs = new Set<string>();

/**
* ECR digests cannot be resolved from the controller: the OCI token flow does
* not apply, and the controller holds only the OIDC role Modal assumes, not
* AWS credentials of its own for `ecr:GetAuthorizationToken`. Modal keys its
* image cache on the ref string, so a mutable ECR tag will not be re-pulled
* after the first build. Warn once per process so operators know to pin.
*/
function warnUnpinnedEcrBaseImageRef(ref: string): void {
const parsed = parseImageRef(ref);
if (!parsed || parsed.digest || isImmutableImageTag(parsed.tag)) {
return;
}
if (warnedUnpinnedEcrRefs.has(ref)) {
return;
}
warnedUnpinnedEcrRefs.add(ref);
console.warn(
`[ModalClient] ECR base image uses a mutable tag; Modal will not re-pull it after the first build. Pin an @sha256 digest or an immutable tag, or enable ECR tag immutability ${JSON.stringify(
{ ref },
)}`,
);
}

const DEFAULT_MODAL_WORKDIR = '/sandbox';
const MODAL_VM_DOCKER_COMMAND = [
'/usr/bin/sudo',
Expand Down Expand Up @@ -300,18 +330,37 @@ export class ModalClient implements ComputeProviderClient {
return this.resolvedRegistrySecretPromise;
}

private async resolveImage(): Promise<Image> {
private async resolveImage(
signal?: AbortSignal,
): Promise<{ image: Image; imageRef: string }> {
if (this.imageMode === 'ecr-oidc') {
warnUnpinnedEcrBaseImageRef(this.baseImageRef);
const secret = await this.getEcrSecret();
return this.sdk.images.fromAwsEcr(this.baseImageRef, secret);
return {
image: this.sdk.images.fromAwsEcr(this.baseImageRef, secret),
imageRef: this.baseImageRef,
};
}

// Modal keys its image cache on the ref string, so a mutable tag such as
// `:develop` would never be re-pulled after the first build. Pin the tag to
// its current digest so a new push produces a new image definition.
const imageRef = await pinModalBaseImageRef({
ref: this.baseImageRef,
registryUsername: this.config.registryUsername,
registryPassword: this.config.registryPassword,
signal,
});

if (this.imageMode === 'registry-auth') {
const secret = await this.getRegistrySecret();
return this.sdk.images.fromRegistry(this.baseImageRef, secret);
return {
image: this.sdk.images.fromRegistry(imageRef, secret),
imageRef,
};
}

return this.sdk.images.fromRegistry(this.baseImageRef);
return { image: this.sdk.images.fromRegistry(imageRef), imageRef };
}

private normalizeSandboxTags(
Expand Down Expand Up @@ -450,8 +499,8 @@ export class ModalClient implements ComputeProviderClient {
throw error;
}

const image = await raceWithAbort({
promise: this.resolveImage(),
const { image, imageRef } = await raceWithAbort({
promise: this.resolveImage(input.signal),
signal: input.signal,
abortMessage: `Resolving Modal image "${this.baseImageRef}" was aborted`,
});
Expand All @@ -461,6 +510,7 @@ export class ModalClient implements ComputeProviderClient {
try {
console.log(
`[ModalClient] Creating sandbox... ${JSON.stringify({
imageRef,
encryptedPorts: input.ports,
regions: this.config.regions ?? '(default)',
cpu: this.config.cpu ?? '(default)',
Expand Down
43 changes: 43 additions & 0 deletions packages/compute-providers/src/adapters/roomote-broker.test.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 10 additions & 3 deletions packages/compute-providers/src/adapters/roomote-broker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import type {
} from '../types';
import { unsupported } from '../errors';
import { sleepWithSignal, throwIfAborted, toAbortError } from '../modal/abort';
import { pinModalBaseImageRef } from '../modal/registry-digest';
import { RoomoteBrokerExec } from './roomote-broker-exec';
import {
BrokerRequestError,
Expand Down Expand Up @@ -283,10 +284,16 @@ export class RoomoteBrokerClient implements ComputeProviderClient {
throwIfAborted(input.signal);

const idempotencyKey = input.idempotencyKey ?? randomUUID();
// Same mutable-tag pitfall as the direct Modal adapter: the broker keys
// its image cache on the ref string, so pin the tag to its digest first.
const imageRef = sourceSnapshotId
? undefined
: await pinModalBaseImageRef({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The broker client deliberately has no registry credentials (factory.ts keeps them broker-side), so this anonymous lookup cannot resolve a private base image. It falls back to the mutable tag and the broker path retains the stale Modal-image-cache behavior this change is meant to fix. Resolve the digest in the broker, where its registry credentials are available, or provide a trusted broker-side digest endpoint.

ref: this.config.baseImageRef,
signal: input.signal,
});
const body = JSON.stringify({
...(sourceSnapshotId
? { snapshotId: sourceSnapshotId }
: { imageRef: this.config.baseImageRef }),
...(sourceSnapshotId ? { snapshotId: sourceSnapshotId } : { imageRef }),
...(input.ports?.length ? { ports: input.ports } : {}),
...(input.tags && Object.keys(input.tags).length > 0
? { tags: input.tags }
Expand Down
Loading
Loading