diff --git a/.gitignore b/.gitignore index ec8b4a2..5593d06 100644 --- a/.gitignore +++ b/.gitignore @@ -362,3 +362,7 @@ MigrationBackup/ # Fody - auto-generated XML schema FodyWeavers.xsd /CoverageReport + +# Local environment variables +.env +.env.local diff --git a/Milvaion.slnx b/Milvaion.slnx index 3f778c2..ec5db20 100644 --- a/Milvaion.slnx +++ b/Milvaion.slnx @@ -85,6 +85,7 @@ + @@ -165,7 +166,13 @@ - + + + + + + + diff --git a/README.md b/README.md index 6a488ce..eb45d96 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,8 @@

A distributed job scheduling system built on .NET 10 +
+Already on Hangfire or Quartz.NET? Keep it — add Milvaion monitoring in two lines.

@@ -60,6 +62,53 @@ Milvaion solves these problems by **completely separating scheduling from execut --- +## Already Running Hangfire or Quartz.NET? + +**You don't have to migrate to adopt Milvaion.** + +Milvaion plugs into the scheduler you already run. Your scheduler keeps owning triggers, storage and cron expressions. Your job code doesn't change. Milvaion adds one real-time dashboard across every service, plus persisted execution history, metrics and alerting. + +```csharp +// Hangfire - your existing setup stays exactly as it is +builder.Services.AddMilvaionHangfireIntegration(builder.Configuration); +builder.Services.AddHangfire((sp, config) => config.UseMilvaion(sp)); + +// Quartz.NET - your existing setup stays exactly as it is +builder.Services.AddMilvaionQuartzIntegration(builder.Configuration); +builder.Services.AddQuartz(q => q.UseMilvaion(builder.Services)); +``` + +That's the whole integration. + +### The problem this solves + +A typical .NET estate ends up with background jobs scattered across a dozen services — each with its own dashboard behind its own URL and its own auth. Nobody can answer *"which jobs failed last night?"* without opening every one of them, and history vanishes when the retention window rolls over. + +| Before | After | +|--------|-------| +| One dashboard per service | One dashboard for the whole estate | +| History limited by Hangfire/Quartz retention | Full execution history in PostgreSQL | +| Logs only in each app's own sink | Real-time log streaming per execution | +| Failures noticed when someone complains | Multi-channel alerts on failure and timeout | +| No cross-service metrics | Success rate, duration and EPM per job | + +### The adoption path + +1. **Add monitoring** — two lines per application. No migration window, no change to job code. +2. **Get visibility** — every Hangfire and Quartz.NET job appears in the dashboard, flagged as external. +3. **Migrate selectively, or never** — when one job outgrows in-process execution, move just that job to a Milvaion worker. Everything else keeps running untouched. + +Plenty of teams stop at step 2. That's a perfectly good outcome. + +| Scheduler | Package | Status | +|-----------|---------|--------| +| **Hangfire** | `Milvasoft.Milvaion.Sdk.Worker.Hangfire` | ✅ Available | +| **Quartz.NET** | `Milvasoft.Milvaion.Sdk.Worker.Quartz` | ✅ Available | + +📖 **[Full Integration Guide →](https://portal.milvasoft.com/docs/1.0.1/open-source-libs/milvaion/external-schedulers)** + +--- + ## Features ![Milvaion Real Time](https://portal.milvasoft.com/assets/images/executions-4b5918b7fca1b603f54be133c7880397.gif) @@ -105,26 +154,28 @@ Milvaion solves these problems by **completely separating scheduling from execut - **Maintenance Worker** - Milvaion self data warehouse cleanup and archival ### External Scheduler Integration -Already using **Quartz.NET** or **Hangfire**? Keep your existing scheduler and gain Milvaion's monitoring capabilities: - -| Scheduler | Package | Status | -|-----------|---------|--------| -| **Quartz.NET** | `Milvasoft.Milvaion.Sdk.Worker.Quartz` | ✅ Available | -| **Hangfire** | `Milvasoft.Milvaion.Sdk.Worker.Hangfire` | ✅ Available | +Milvaion also monitors jobs running in **Quartz.NET** and **Hangfire** without replacing them — see [Already Running Hangfire or Quartz.NET?](#already-running-hangfire-or-quartznet) above. -```csharp -// Quartz.NET Integration -builder.Services.AddMilvaionQuartzIntegration(builder.Configuration); -builder.Services.AddQuartz(q => q.UseMilvaion(builder.Services)); +### MCP Server +Point Claude Code, Cursor or GitHub Copilot at Milvaion and ask about your jobs in plain language: -// Hangfire Integration -builder.Services.AddMilvaionHangfireIntegration(builder.Configuration); -builder.Services.AddHangfire((sp, config) => config.UseMilvaion(sp)); +```json +{ + "mcpServers": { + "milvaion": { + "type": "http", + "url": "https://milvaion.yourcompany.com/mcp", + "headers": { "X-ApiKey": "your-api-key" } + } + } +} ``` -External jobs appear in Milvaion dashboard with full monitoring, metrics, and execution history - without changing your existing scheduler setup. +> *"Which jobs failed last night and why?"* + +More than forty tools — reading, triggering, pausing, editing and deleting — each gated by the same permissions used everywhere else. A read-only api key gives an assistant full visibility and no ability to change anything. Milvaion is the data source here; it never calls a language model and stores no model provider keys. -[For more information...](https://portal.milvasoft.com/docs/1.0.1/open-source-libs/milvaion/external-schedulers) +📖 **[MCP Server Guide →](https://portal.milvasoft.com/docs/1.0.1/open-source-libs/milvaion/mcp-server)** --- @@ -378,6 +429,9 @@ Each workflow can also configure **Max Step Retries** and a **Timeout** (auto-ca | Document | Description | |----------|-------------| | [Introduction](https://portal.milvasoft.com/docs/1.0.1/open-source-libs/milvaion/introduction) | What is Milvaion, when to use it | +| [Hangfire & Quartz.NET Integration](https://portal.milvasoft.com/docs/1.0.1/open-source-libs/milvaion/external-schedulers) | Keep your existing scheduler, add Milvaion monitoring | +| [Api Keys](https://portal.milvasoft.com/docs/1.0.1/open-source-libs/milvaion/api-keys) | Credentials for CI pipelines, scripts and MCP clients | +| [MCP Server](https://portal.milvasoft.com/docs/1.0.1/open-source-libs/milvaion/mcp-server) | Connect Claude Code, Cursor or Copilot to Milvaion | | [Quick Start](https://portal.milvasoft.com/docs/1.0.1/open-source-libs/milvaion/quick-start) | Get running in under 10 minutes | | [Core Concepts](https://portal.milvasoft.com/docs/1.0.1/open-source-libs/milvaion/core-concepts) | Architecture and key terms | | [Your First Worker](https://portal.milvasoft.com/docs/1.0.1/open-source-libs/milvaion/your-first-worker) | Create a custom worker | @@ -397,6 +451,7 @@ Each workflow can also configure **Max Step Retries** and a **Timeout** (auto-ca | [Architecture](./docs/githubdocs/ARCHITECTURE.md) | Technical architecture deep-dive | | [Development](./docs/githubdocs/DEVELOPMENT.md) | Development environment setup | | [Worker SDK](./docs/githubdocs/WORKER-SDK.md) | Worker SDK reference | +| [MCP Server](./docs/githubdocs/MCP-SERVER.md) | MCP server and api key auth internals | | [Security](./SECURITY.md) | Security policies | --- diff --git a/docker-compose.yml b/docker-compose.yml index 1952ad0..9fc296e 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -226,7 +226,19 @@ labels: - "com.milvaion.service=grafana" - "com.milvaion.description=Grafana metrics dashboard" - + pgweb: + image: sosedoff/pgweb:latest + container_name: milvaion-pgweb + depends_on: + postgres: + condition: service_healthy + environment: + DATABASE_URL: "postgres://postgres:N4SQp.qW%3E6%3FxwWzg@postgres:5432/MilvaionDb?sslmode=disable" + ports: + - "8082:8081" + networks: + - milvaion-network + restart: unless-stopped volumes: postgres_data: driver: local diff --git a/docs/githubdocs/ARCHITECTURE.md b/docs/githubdocs/ARCHITECTURE.md index bb8db3b..ea55137 100644 --- a/docs/githubdocs/ARCHITECTURE.md +++ b/docs/githubdocs/ARCHITECTURE.md @@ -193,7 +193,13 @@ CREATE TABLE "ScheduledJobs" ( "JobType" VARCHAR(200) NOT NULL, "JobData" JSONB, "CronExpression" VARCHAR(100), + -- Configured start time, NOT the next run. The dispatcher advances a recurring + -- job's schedule in Redis and never writes it back here, so this goes stale + -- after the first run. Read the Redis sorted set for the live schedule. "ExecuteAt" TIMESTAMPTZ, + -- Set when a one-time job is dispatched, so it is never scheduled again even if + -- Redis is flushed. Separate from IsActive: "finished" is not "switched off". + "CompletedAt" TIMESTAMPTZ, "IsActive" BOOLEAN DEFAULT TRUE, "ConcurrentExecutionPolicy" INTEGER DEFAULT 0, "TimeoutMinutes" INTEGER, diff --git a/docs/githubdocs/MCP-SERVER.md b/docs/githubdocs/MCP-SERVER.md new file mode 100644 index 0000000..2fa3ed7 --- /dev/null +++ b/docs/githubdocs/MCP-SERVER.md @@ -0,0 +1,281 @@ +# MCP Server + +This document covers the internals of Milvaion's Model Context Protocol server and the api key authentication it depends on. For user-facing setup instructions see the [MCP Server](https://portal.milvasoft.com/docs/1.0.1/open-source-libs/milvaion/mcp-server) and [Api Keys](https://portal.milvasoft.com/docs/1.0.1/open-source-libs/milvaion/api-keys) portal docs. + +## Table of Contents + +- [Overview](#overview) +- [Api Key Authentication](#api-key-authentication) +- [MCP Server Wiring](#mcp-server-wiring) +- [Authorization Inside Tools](#authorization-inside-tools) +- [Adding a Tool](#adding-a-tool) +- [Design Decisions](#design-decisions) +- [Testing](#testing) + +--- + +## Overview + +Milvaion is an MCP **server**: it exposes its own data as tools that an MCP client (Claude Code, Cursor, Copilot) can call. Milvaion never talks to a language model and holds no model provider credentials. + +```text +MCP client Milvaion.Api + │ + │ POST /mcp (JSON-RPC, X-ApiKey header) + ▼ +ApiKeyAuthenticationHandler ──▶ ApiKeyStore (Redis cache ─▶ Postgres) + │ ClaimsPrincipal with permissions as role claims + ▼ +MapMcp endpoint ──▶ tool method ──▶ McpPermissionGuard.Require(...) + │ + ▼ + IMediator ──▶ existing CQRS query +``` + +### Source Layout + +| Path | Contains | +|------|----------| +| `src/Milvaion.Api/Mcp/MilvaionJobTools.cs` | Job and occurrence tools, including CRUD | +| `src/Milvaion.Api/Mcp/MilvaionOpsTools.cs` | Worker, workflow and dashboard tools | +| `src/Milvaion.Api/Mcp/MilvaionDiagnosticsTools.cs` | Dead letter failures and the activity log | +| `src/Milvaion.Api/Mcp/MilvaionInsightTools.cs` | Metric reports, infrastructure health, configuration | +| `src/Milvaion.Api/Mcp/MilvaionPrompts.cs` | Prompt templates for diagnosis workflows | +| `src/Milvaion.Api/Mcp/McpPermissionGuard.cs` | Per-tool permission enforcement | +| `src/Milvaion.Api/AppStartup/McpExtensions.cs` | Registration and endpoint mapping | +| `src/Milvaion.Api/Utils/ApiKeyAuthenticationHandler.cs` | Authentication scheme | +| `src/Milvaion.Api/Utils/KeyHelper.cs` | Key generation and signature validation | +| `src/Milvaion.Api/Services/ApiKeyStore.cs` | Cached key lookup and cache invalidation | +| `src/Milvaion.Api/Services/ApiKeyGenerator.cs` | `IApiKeyGenerator` implementation | +| `src/Milvaion.Application/Features/ApiKeys/` | Api key CQRS features | +| `src/Milvaion.Application/Features/Workflows/GetWorkflowRunAnalysis/` | Flattened workflow run, see [Design Decisions](#design-decisions) | +| `src/Milvaion.Application/Features/MetricReports/GetMetricReportSummaryList/` | Report metadata without payloads | + +Dependency: `ModelContextProtocol.AspNetCore`. + +--- + +## Api Key Authentication + +### Key Format + +A key is a JWT signed with `MilvaionConfig.ApiKey.Secret` (HMAC-SHA256), carrying: + +| Claim | Meaning | +|-------|---------| +| `jti` | Id of the `MilvaionApiKey` record | +| `kv` | Version of the signing secret | +| `exp` | Expiry, omitted for keys that never expire | + +The key itself is **never persisted**. Only `MaskedKey` — the trailing characters — is stored, purely so a user can match a listed key against the one in their configuration. + +### Why a Record Lookup + +Signature validation alone would be enough to prove a key is authentic, but a self-contained token cannot be revoked. Authentication therefore always resolves `jti` to a `MilvaionApiKey` row and checks `RevokedAt`, `ExpiresAt` and `KeyVersion` before accepting the request. + +That lookup is cached in **Redis**, not in process memory. With several API replicas an in-process cache would leave a revoked key working on every replica that did not happen to handle the revoke request. `RevokeApiKeyCommandHandler` and `DeleteApiKeyCommandHandler` call `IApiKeyCacheInvalidator.InvalidateAsync` so revocation takes effect immediately across the cluster. + +If Redis is unavailable, `ApiKeyStore` falls back to the database rather than failing authentication — a cache outage should not lock every integration out. + +### Permissions as Role Claims + +`AuthAttribute` derives from `AuthorizeAttribute` and matches on `Roles`. `ApiKeyAuthenticationHandler` therefore emits each granted permission as a `ClaimTypes.Role` claim, exactly as the login token does: + +```csharp +claims.AddRange(apiKey.Permissions.Select(p => new Claim(ClaimTypes.Role, p))); +claims.Add(new Claim(GlobalConstant.UserTypeClaimName, nameof(UserType.Manager))); +``` + +The consequence is that **every existing `[Auth(PermissionCatalog...)]` attribute already works for api key callers** with no change. Had this been implemented as an action filter instead of an authentication scheme, authorization would have had to be reimplemented separately for machine callers. + +The default authorization policy names both schemes: + +```csharp +options.DefaultPolicy = new AuthorizationPolicyBuilder( + JwtBearerDefaults.AuthenticationScheme, + ApiKeyAuthenticationDefaults.AuthenticationScheme) + .RequireAuthenticatedUser() + .Build(); +``` + +### LastUsedAt Throttling + +`LastUsedAt` would otherwise mean a database write on every request. `ApiKeyStore.ShouldWriteLastUsedAsync` uses a Redis `SET ... NX EX` as both throttle and lock: whoever sets the key owns that interval. Default interval is five minutes, configurable through `ApiKeyAuthenticationOptions.LastUsedWriteInterval`. + +Failures here are swallowed and logged. Bookkeeping must never fail an otherwise valid request. + +### `ApiAuthAttribute` + +`[ApiAuth]` restricts an endpoint to api key callers only, for endpoints that should not be reachable from a browser session. `[Auth]` accepts both. All parsing and validation lives in the authentication handler; the attribute only selects the scheme and applies the permission requirement. + +--- + +## MCP Server Wiring + +Registration is in `McpExtensions.AddMilvaionMcp`: + +```csharp +services.AddMcpServer(options => { /* ServerInfo, ServerInstructions */ }) + .WithHttpTransport(options => options.Stateless = true) + .WithToolsFromAssembly(); +``` + +`WithToolsFromAssembly` discovers every `[McpServerToolType]` class and registers each `[McpServerTool]` method. Tool classes are resolved from DI per request, so constructor injection of scoped services works. + +Mapping is in `McpExtensions.MapMilvaionMcp`: + +```csharp +app.MapMcp("/mcp").RequireAuthorization(); +``` + +> **Ordering matters.** `MapMilvaionMcp()` must be called before `MapFallbackToFile("index.html")` in `Program.cs`, or the SPA fallback swallows `/mcp`. + +### ServerInstructions + +`ServerInstructions` is sent to the client at initialize and shapes how the model uses the server. Milvaion's explains the job/occurrence distinction, steers diagnosis towards `list_failures` rather than `list_occurrences`, and tells the model to confirm before triggering anything. In practice this affects tool selection accuracy more than any individual tool description. + +--- + +## Authorization Inside Tools + +MCP tools are not MVC actions, so `[Auth]` never runs for them. Authentication is still enforced at the endpoint, but the per-permission check has to happen in the tool body: + +```csharp +_guard.Require(PermissionCatalog.ScheduledJobManagement.List); +``` + +`McpPermissionGuard` reads `HttpContext.User`, honours `App.SuperAdmin`, and throws `McpException` otherwise. The message names the missing permission on purpose — an agent told exactly what it lacks stops retrying and can report something actionable. + +`Has(permission)` is available for trimming optional detail out of a response instead of failing the whole call. + +--- + +## Adding a Tool + +1. Add a method to an existing `[McpServerToolType]` class, or create a new one — `WithToolsFromAssembly` finds it either way. +2. Call `_guard.Require(...)` **first**, before any work. +3. Delegate to the existing MediatR query or command. Do not reimplement data access; a change to filtering or projection should show up identically in the dashboard, the REST API and MCP. +4. Write a `[Description]` aimed at a model, not a developer: say when to reach for this tool rather than another one. +5. Add XML docs to match the rest of the codebase. + +```csharp +/// +/// Gets scheduled jobs. +/// +/// Free text search over job names and types. +/// +/// Paged job list with the total count. +[McpServerTool(Name = "list_jobs")] +[Description("Lists scheduled jobs in Milvaion. Use this to find a job's id before calling other tools.")] +public async Task ListJobsAsync( + [Description("Optional free text search over job names and types.")] string searchTerm = null, + CancellationToken cancellationToken = default) +{ + _guard.Require(PermissionCatalog.ScheduledJobManagement.List); + ... +} +``` + +### Conventions + +- **Tool names** are `snake_case` and verb-first: `list_jobs`, `get_occurrence`, `trigger_job`. +- **Annotations are mandatory.** Set `ReadOnly = true` on anything that only reads, `Destructive = true` on anything irreversible, `Idempotent = true` on state setters. Clients use these to decide what to auto-approve; an unannotated destructive tool looks identical to a list call. +- **Filter, do not paginate.** Every list tool should expose the filters a person would reach for - id, status, owner, date bounds - built through `BuildDateRangeCriterias`, `AddEqualityCriteria` and `ToFilterRequest` in `MilvaionJobTools`. Making the model page through thousands of rows to find ten is slow and expensive. +- **Sort explicitly.** The handlers do not all default their sort order, so a query without `Sorting` returns rows in whatever order the plan produces. Match whatever the dashboard sends for the same list, or the model and the user will be looking at different data. +- **Bound the response.** Anything that can grow without limit - logs especially - needs a cap and a note saying it was truncated. Blowing the context window is a silent failure: the answer just gets vague and expensive. +- **Page sizes** are clamped to `_maxPageSize` (100). A model asking for 10,000 rows should get 100, not an error. +- **Not-found** throws `McpException` with a message naming the id, rather than returning null. +- **Write tools** require their own permission and never bypass domain safeguards — `trigger_job` always dispatches with `Force = false`. +- **Attribution**: anything with a side effect stamps the calling key into the reason, so history distinguishes machine from human. + +--- + +## Design Decisions + +### Why stateless HTTP transport + +Milvaion is routinely deployed with several API replicas behind a load balancer. Stateful mode would require sticky sessions. Stateless rules out server-to-client requests (sampling, elicitation, roots), none of which these tools need. + +### Why `update_job` takes plain arguments + +`UpdateScheduledJobCommand` uses `UpdateProperty` to separate "not supplied" from "set to null". Exposing that shape to a model would mean asking it to emit `{"value": "...", "isUpdated": true}` per field, which is both awkward to describe and easy to get wrong. + +The tool takes nullable plain arguments instead and builds the wrappers in C#: null means unchanged. The cost is that a field cannot be cleared through MCP. That asymmetry is deliberate — a model accidentally blanking a cron expression is a worse outcome than not being able to blank one on purpose. + +### Why `set_job_active` exists separately + +Pausing a job is the most common intervention during an incident and it should not require a partial update payload. A dedicated tool also gives the description room to steer the model towards pausing rather than deleting, which is what users almost always mean by "stop this job". + +### Why workflow authoring is absent + +There is no `create_workflow` or `update_workflow`. A workflow is a directed graph with conditions, merge nodes and data mappings between steps; expressing one as tool arguments would be a large nested payload with plenty of ways to produce a graph that is syntactically valid and semantically wrong. Reading, triggering, cancelling and deleting workflows are all available — authoring stays in the visual builder. + +### Why the write tools are shaped around intent + +Rather than mirroring the REST surface one-to-one, the write tools are named after what a person wants: `set_job_active` rather than a generic update, `resolve_failures` rather than an update with six optional fields. Narrow tools are easier for a model to select correctly and give far less room for it to do something adjacent to what was asked. + +### Why tools delegate to MediatR + +Every tool sends an existing query. This keeps filtering, projection, permission semantics and localisation identical across all three entry points, and means new tools are cheap — most are a permission check plus a `Send`. + +### Why two features exist only for MCP + +Delegating to the dashboard's queries is the default and holds for 41 of the 43 tools. Two responses were shaped for a canvas rather than a reader, and reusing them made the tool worse in a way no wording in the description could fix. + +| Tool | Dashboard query | MCP query | Problem with reuse | +|------|-----------------|-----------|--------------------| +| `get_workflow_run` | `GetWorkflowRunDetailQuery` | `GetWorkflowRunAnalysisQuery` | Three parallel collections cross-referenced by GUID, plus `PositionX`/`PositionY`. `WorkflowStepRunDto.DependsOnStepIds` is never populated, so dependencies are only recoverable by joining `Edges` by id. | +| `list_reports` | `GetMetricReportListQuery` | `GetMetricReportSummaryListQuery` | Carries the full jsonb `Data` for every row. A page of twenty is hundreds of kilobytes of series data, discarded whenever the caller only wanted to know which reports exist. | + +`GetWorkflowRunAnalysisQuery` resolves the graph before returning it: steps come back in execution order, `dependsOn` and `blocks` hold step **names**, layout coordinates are dropped, and `failedSteps` / `skippedSteps` / `notReachedSteps` state the collapse directly. A step that never ran has a null `status` rather than `Pending` — conflating the two makes a halted run look like it is still going. + +`GetMetricReportSummaryListQuery` returns metadata plus `dataSizeBytes` and `ageMinutes`. Age matters more than it looks: the reporter worker runs on a schedule, so the newest report of a type is routinely hours old, and a caller given only `generatedAt` has to know the current time to notice. + +Both are ordinary features under `Milvaion.Application/Features` — nothing about them is MCP-specific beyond who currently calls them, and neither is wired to a controller. The REST endpoints still use the original queries. + +### Why `get_report` parses the payload + +`MetricReportDetailDto.Data` is a `string` over a jsonb column, so serializing the DTO produces a JSON string containing escaped JSON. `MilvaionInsightTools.Unwrap` parses it into a `JsonElement` so it arrives as real nested JSON. This is a serialization concern rather than a data-access one, which is why it lives in the tool and not in a third feature. A payload that fails to parse is passed through unchanged and flagged with `dataIsRawString`. + +--- + +## Testing + +`/mcp` speaks JSON-RPC over HTTP, so it can be exercised without an MCP client. + +```bash +# Expect 401 +curl -i -X POST http://localhost:5000/mcp \ + -H "Content-Type: application/json" \ + -H "Accept: application/json, text/event-stream" \ + -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' + +# Tool list +curl -X POST http://localhost:5000/mcp \ + -H "Content-Type: application/json" \ + -H "Accept: application/json, text/event-stream" \ + -H "X-ApiKey: $MILVAION_API_KEY" \ + -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' + +# Tool call +curl -X POST http://localhost:5000/mcp \ + -H "Content-Type: application/json" \ + -H "Accept: application/json, text/event-stream" \ + -H "X-ApiKey: $MILVAION_API_KEY" \ + -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"get_overview","arguments":{}}}' +``` + +Worth covering explicitly: + +- A key **without** `ScheduledJobManagement.Trigger` calling `trigger_job` must fail. This is the guarantee the read-only key story rests on. +- A **revoked** key must stop working immediately, not after the cache expires. +- A key issued under an older `ApiKey.Version` must be rejected with the retired-secret message. + +--- + +## Further Reading + +- [MCP C# SDK](https://csharp.sdk.modelcontextprotocol.io/) +- [Model Context Protocol specification](https://modelcontextprotocol.io/specification/) +- [Architecture](ARCHITECTURE.md) +- [Development](DEVELOPMENT.md) diff --git a/docs/githubdocs/TROUBLESHOOTING.md b/docs/githubdocs/TROUBLESHOOTING.md index 7446776..39b69f9 100644 --- a/docs/githubdocs/TROUBLESHOOTING.md +++ b/docs/githubdocs/TROUBLESHOOTING.md @@ -657,12 +657,26 @@ curl http://localhost:5000/api/v1/jobs/{jobId} ``` 2. **Verify next execution time:** + +Fastest route — the Upcoming Executions endpoint reads the live schedule and flags jobs +that are active and recurring but hold no run time at all. Such a job will never fire, and +because no occurrence is ever created, nothing else in the product reports it: + +```bash +curl -H "X-ApiKey: $MILVAION_API_KEY" \ + "http://localhost:5000/api/v1/jobs/upcoming?onlyProblems=true" +``` + +Or read the sorted set directly: + ```bash -# Check Redis docker exec -it milvaion-redis redis-cli ZRANGE Milvaion:JobScheduler:schedule 0 -1 WITHSCORES ``` +Do not use the job's `executeAt` field for this — it is the configured start time, not the +next run, and is in the past for every recurring job that has run at least once. + 3. **Check dispatcher is running:** ```bash docker logs milvaion-api | grep "Dispatcher" @@ -974,6 +988,66 @@ services: ## Deployment Issues +### Problem: Every job fires at once after a deploy + +**Cause:** + +Startup recovery used to write the `ExecuteAt` column back into the Redis sorted set. That +column is the *configured start time*, not the next run — the dispatcher advances a +recurring job's schedule in Redis and never persists it. For any job that had run at least +once, `ExecuteAt` was in the past, so recovery put the whole catalogue at a due time and +the next poll dispatched all of it. + +**Solution:** + +Upgrade. Recovery now treats Redis as authoritative and only fills gaps: + +| Situation | Behaviour | +|-----------|-----------| +| Redis holds a run time | Left untouched | +| Redis empty, recurring job | Next run derived from the cron expression | +| Redis empty, one-time job that never ran | Scheduled at `ExecuteAt` | +| Redis empty, one-time job with `CompletedAt` | Skipped — already ran | + +No migration or manual cleanup is needed beyond the `CompletedAt` column. + +**Verify after deploying:** + +```bash +curl -H "X-ApiKey: $MILVAION_API_KEY" \ + "http://localhost:5000/api/v1/jobs/upcoming?withinHours=24" +``` + +Every recurring job should report a future time. In the startup log, expect +`already scheduled` to account for nearly everything: + +``` +Startup recovery (Redis) completed. ZSET: 3 added, 128 already scheduled. ... +``` + +A large `added` count means Redis lost its data and the schedule was rebuilt from cron +expressions — which is correct, but worth knowing. + +### Problem: A one-time job ran twice + +**Cause:** + +Before the `CompletedAt` column existed, the only record that a one-time job had run was +its *absence* from the Redis sorted set. Redis is a cache: a flush or a restart erased that +fact, and recovery scheduled the job again. + +**Solution:** + +Upgrade and apply the migration adding `ScheduledJobs.CompletedAt`. The stamp is durable, +so a finished one-time job stays finished across restarts. + +```sql +-- Jobs that already ran and must never be rescheduled +SELECT "Id", "DisplayName", "CompletedAt" +FROM "ScheduledJobs" +WHERE "CronExpression" IS NULL AND "CompletedAt" IS NOT NULL; +``` + ### Problem: Containers failing health checks **Solutions:** diff --git a/docs/portaldocs/00-guide.md b/docs/portaldocs/00-guide.md index 0311e42..b960612 100644 --- a/docs/portaldocs/00-guide.md +++ b/docs/portaldocs/00-guide.md @@ -5,6 +5,9 @@ sidebar_position: 0 description: Documentation guide of Milvaion. --- +

+ Milvaion +

# Milvaion Documentation @@ -16,39 +19,46 @@ Welcome to the official Milvaion documentation. This documentation will help you | Document | Description | |----------|-------------| -| [01-introduction.md](01-introduction.md) | What is Milvaion, when to use it, comparison with alternatives | -| [02-quick-start.md](02-quick-start.md) | Get running locally in under 10 minutes | -| [03-core-concepts.md](03-core-concepts.md) | Understand the architecture and key terms | +| [Introduction](01-introduction.md) | What is Milvaion, when to use it, comparison with alternatives | +| [Quick Start](02-quick-start.md) | Get running locally in under 10 minutes | +| [Core Concepts](03-core-concepts.md) | Understand the architecture and key terms | ### Developer Guide | Document | Description | |----------|-------------| -| [04-your-first-worker.md](04-your-first-worker.md) | Create and deploy a custom worker | -| [05-implementing-jobs.md](05-implementing-jobs.md) | Write jobs with DI, error handling, testing | -| [06-configuration.md](06-configuration.md) | All configuration options for API and Workers | -| [14-built-in-workers.md](14-built-in-workers.md) | Pre-built workers (HTTP Worker, etc.) | -| [20-workflows.md](20-workflows.md) | Build multi-step job pipelines with DAG-based orchestration | -| [21-reporter-worker.md](21-reporter-worker.md) | Automated metric report generation worker | +| [Your First Worker](04-your-first-worker.md) | Create and deploy a custom worker | +| [Implementing Jobs](05-implementing-jobs.md) | Write jobs with DI, error handling, testing | +| [Configuration](06-configuration.md) | All configuration options for API and Workers | +| [Testing](23-testing.md) | Unit test jobs locally without RabbitMQ, Redis, or a live environment | ### Operations Guide | Document | Description | |----------|-------------| -| [07-deployment.md](07-deployment.md) | Production deployment with Docker and Kubernetes | -| [08-reliability.md](08-reliability.md) | Retry, DLQ, zombie detection, idempotency | -| [09-scaling.md](09-scaling.md) | Horizontal scaling strategies | -| [10-monitoring.md](10-monitoring.md) | Health checks, metrics, logging, alerting | -| [11-maintenance.md](11-maintenance.md) | Database cleanup and retention policies | -| [20-enterprise-features.md](20-enterprise-features.md) | User, role, permission management, activity tracking, auditing, and metric reports | +| [Deployment](07-deployment.md) | Production deployment with Docker and Kubernetes | +| [Reliability](08-reliability.md) | Retry, DLQ, zombie detection, idempotency | +| [Scaling](09-scaling.md) | Horizontal scaling strategies | +| [Monitoring](10-monitoring.md) | Health checks, metrics, logging, alerting | +| [Maintenance](11-maintenance.md) | Database cleanup and retention policies | +| [Built-in Workers](./built-in-workers/14-http-worker.md) | Pre-built workers (HTTP Worker, SQL Worker, Email Worker, Maintenance Worker) | +| [Workflows](20-workflows.md) | Build multi-step job pipelines with DAG-based orchestration | +| [Api Keys](24-api-keys.md) | Credentials for CI pipelines, scripts and MCP clients | +| [MCP Server](25-mcp-server.md) | Connect Claude Code, Cursor or Copilot and ask about your jobs | + ## Quick Links -- **First time→** Start with [Introduction](01-introduction.md) -- **Want to run it→** Jump to [Quick Start](02-quick-start.md) -- **Building a worker→** See [Your First Worker](04-your-first-worker.md) -- **Using built-in workers→** See [Built-in Workers](14-built-in-workers.md) -- **Going to production→** Read [Deployment](07-deployment.md) +- **First time →** Start with [Introduction](01-introduction.md) +- **Want to run it →** Jump to [Quick Start](02-quick-start.md) +- **Building a worker →** See [Your First Worker](04-your-first-worker.md) +- **Going to production →** Read [Deployment](07-deployment.md) +- **Security considerations →** Read [Security](12-security.md) +- **Milvaion UI Features →** See [UI Screenshots](13-dashboard-screenshots.md) +- **Built-in Workers →** See [Built-in Workers](./built-in-workers/14-http-worker.md) +- **Workflows →** See [Workflows](20-workflows.md) +- **Using it from an AI assistant →** See [MCP Server](25-mcp-server.md) +- **CI or script access →** See [Api Keys](24-api-keys.md) ## Reading Order @@ -60,7 +70,7 @@ For new users, we recommend reading in this order: 3. Core Concepts → Understand the architecture 4. Your First Worker → Build something real 5. Configuration → Reference as needed -6. (Optional) Reliability, Scaling, Monitoring for production +6. (Optional) Reliability, Scaling, Monitoring, Security for production, Built-in workers ``` ## Version @@ -73,4 +83,4 @@ This documentation covers **Milvaion 1.0.0** with: ## Feedback -Found an issue or want to suggest improvements? Open an issue on GitHub. +Found an issue or want to suggest improvements? Open an issue on [GitHub](https://github.com/Milvasoft/milvaion). diff --git a/docs/portaldocs/01-introduction.md b/docs/portaldocs/01-introduction.md index 5682c80..9a84cf4 100644 --- a/docs/portaldocs/01-introduction.md +++ b/docs/portaldocs/01-introduction.md @@ -1,17 +1,29 @@ --- id: introduction title: Introduction -sidebar_position: 1 +sidebar_position: 10 description: Overview of Milvaion, its purpose, and core philosophy. --- +

+ Milvaion +

+ # Introduction Milvaion is a **distributed job scheduling system** built on .NET 10. It separates the *scheduler* (API that decides when jobs run) from the *workers* (processes that execute jobs), connected via Redis and RabbitMQ. +:::tip Already using Hangfire or Quartz.NET? + +You do not have to migrate. Milvaion plugs into your existing scheduler in **two lines of code** and gives you a unified dashboard, execution history, metrics and alerting across every service — without touching your job code. + +**→ [Hangfire & Quartz.NET Integration](./18-external-scheduler.md)** + +::: + ## What Problem Does Milvaion Solve? -Most job schedulers run jobs **inside the same process** as the scheduling logic. This works fine until: +Most job schedulers run jobs **inside the same process** as the scheduling logic by default. This works fine until: - A long-running job blocks other jobs from executing - A crashing job takes down the entire scheduler @@ -43,16 +55,17 @@ Milvaion solves these problems by **completely separating scheduling from execut | **Multi-tenant systems** | Route jobs to specific worker groups | | **Jobs needing different resources** | GPU workers, high-memory workers, etc. | | **Compliance/audit requirements** | Full execution history with logs stored in PostgreSQL | +| **Multi-step pipelines with branching** | DAG-based [Workflows](./20-workflows.md) with conditions, merges and data mappings | +| **Existing Hangfire/Quartz.NET estates** | [Integrate without migrating](./18-external-scheduler.md) — monitor every scheduler in one dashboard | ### Not a Good Fit ❌ | Scenario | Better Alternative | |----------|-------------------| -| **Simple in-process background tasks** | Use `BackgroundService` or Hangfire | +| **Simple in-process background tasks** | Use `BackgroundService` or Hangfire — you can still [add Milvaion monitoring](./18-external-scheduler.md) on top | | **Real-time event processing** | Use dedicated event streaming (Kafka, Azure Event Hubs) | | **Sub-second job scheduling** | Milvaion polls every 1 second minimum | | **Single-server deployments** | Overhead of Redis + RabbitMQ not justified | -| **Workflow orchestration with branching** | Use Temporal, Elsa, or Azure Durable Functions | ## Core Concepts @@ -67,7 +80,7 @@ Milvaion solves these problems by **completely separating scheduling from execut ## How It Works -1. **Worker Auto Disvovery** service will auto discover your worker and containing worker jobs. +1. **Worker Auto Discovery** service will auto discover your worker and containing worker jobs. 2. **You create a job according to worker job** via REST API or Dashboard (e.g., "Send daily report at 9 AM") 3. **Scheduler stores it** in PostgreSQL and adds to Redis ZSET with next run time 4. **Dispatcher** checks Redis for due jobs @@ -97,6 +110,7 @@ Milvaion solves these problems by **completely separating scheduling from execut - Technical Logs -> Logs are sent to Seq. - **Worker health monitoring** via heartbeats - **OpenTelemetry support** for metrics and tracing +- **Alerting** - configurable notifications for job failures and system events via multiple channels (Google Chat, Microsoft Teams, Slack, Email and in-app) ### Developer Experience - **Simple `IAsyncJob` interface** - implement one method @@ -104,6 +118,18 @@ Milvaion solves these problems by **completely separating scheduling from execut - **Auto-discovery** - jobs registered automatically - **Cancellation support** - graceful shutdown +### AI Integration +- **[MCP server](./25-mcp-server.md)** - connect Claude Code, Cursor or GitHub Copilot and ask about your jobs in plain language +- **40+ tools** - reading, triggering, pausing, editing and deleting, each behind its own permission +- **Scoped by api key** - a read-only key lets an assistant investigate everything and change nothing +- **No model provider keys** - Milvaion is the data source; the model runs in the user's own editor + +### Enterprise Management +- **Role-based access control** - define roles with specific permissions +- **User management** - create and manage users with assigned roles +- **Permission granularity** - control access at job, worker, and dashboard level +- **[Api keys](./24-api-keys.md)** - non-interactive credentials for CI pipelines, scripts and MCP clients + ## Comparison with Alternatives | Feature | Milvaion | Hangfire | Quartz.NET | @@ -115,10 +141,21 @@ Milvaion solves these problems by **completely separating scheduling from execut | **Real-time Dashboard** | Built-in | Built-in | None | | **Log Streaming** | Real-time via RabbitMQ | Console plugin | None | | **Offline Resilience** | SQLite fallback | None | None | +| **Workflow Engine (DAG)** | Built-in, visual builder | Continuations only | None | +| **MCP server for AI assistants** | Built-in, 40+ tools | None | None | +| **Monitors the others** | ✅ Hangfire + Quartz.NET | N/A | N/A | | **Best For** | Distributed systems | Simple .NET apps | Embedded scheduling | +:::note Not an either/or + +The last row is the important one. Milvaion is not only a Hangfire or Quartz.NET replacement — it also **monitors them**. If you already run either, start with [the integration](./18-external-scheduler.md) and decide about migration later. + +::: + ## Next Steps +- **[Hangfire & Quartz.NET Integration](./18-external-scheduler.md)** - Already have a scheduler? Start here +- **[MCP Server](./25-mcp-server.md)** - Ask your scheduler questions from Claude Code, Cursor or Copilot - **[Quick Start](02-quick-start.md)** - Run Milvaion locally in 5 minutes - **[Core Concepts](03-core-concepts.md)** - Understand the architecture - **[Your First Worker](04-your-first-worker.md)** - Build and deploy a custom worker diff --git a/docs/portaldocs/03-core-concepts.md b/docs/portaldocs/03-core-concepts.md index dab2158..c1b8543 100644 --- a/docs/portaldocs/03-core-concepts.md +++ b/docs/portaldocs/03-core-concepts.md @@ -88,6 +88,45 @@ A **job** is a recurring or one-time task definition stored in PostgreSQL. } ``` +#### `executeAt` is the start time, not the next run + +This trips people up, so it is worth stating plainly: **`executeAt` is the time the job +was configured to start.** It is written when the job is created or updated and is never +touched again. + +For a recurring job it therefore goes stale the moment the job runs for the first time. +The dispatcher advances the schedule in Redis, not in the database, so a job that has been +running for months still shows an `executeAt` from months ago. That is correct and +expected — it is history, not a forecast. + +To see when a job will actually run next, use the **Upcoming Executions** screen or +`GET /jobs/upcoming`, both of which read the live schedule. Reading `executeAt` and +presenting it as "next run" will be wrong for every recurring job in the system. + +For a one-time job there is no such drift: it runs once, is never rescheduled, and +`executeAt` remains its whole schedule. + +#### One-time jobs and `completedAt` + +When a one-time job is dispatched, Milvaion stamps `completedAt`. From that point the job +is finished and will never be scheduled again, even if the API restarts or Redis is +flushed. + +`completedAt` is deliberately separate from `isActive`: + +| Field | Means | +|-------|-------| +| `isActive: false` | Somebody switched the job off | +| `completedAt` set | The job ran and has nothing left to do | + +Collapsing the two would make a finished job look disabled, and re-enabling it would fire +it a second time. In the dashboard a finished one-time job keeps its **Active** badge and +gains a **Completed** badge next to it. + +A one-time job whose time has passed but which has no `completedAt` has genuinely never +run — for example because the API was down when it was due — and will be scheduled on the +next startup. + ### Occurrence An **occurrence** is a single execution of a job. diff --git a/docs/portaldocs/06-configuration.md b/docs/portaldocs/06-configuration.md index a0da7fa..1eb3bac 100644 --- a/docs/portaldocs/06-configuration.md +++ b/docs/portaldocs/06-configuration.md @@ -54,7 +54,9 @@ This page documents all configuration options for Milvaion API and Workers. "AutomaticRecoveryEnabled": true, "NetworkRecoveryInterval": 10, "QueueDepthWarningThreshold": 100, - "QueueDepthCriticalThreshold": 500 + "QueueDepthCriticalThreshold": 500, + "ManagementEnabled": false, + "ManagementPort": 15672 }, "JobDispatcher": { "Enabled": true, @@ -141,8 +143,10 @@ Open telemetry configurations. Set null or empty via environment variables if yo | `Heartbeat` | `/` | Heartbeat interval in seconds (0 = disabled). | | `AutomaticRecoveryEnabled` | `/` | Automatic connection recovery enabled. | | `NetworkRecoveryInterval` | `/` | Network recovery interval in seconds. | -| `QueueDepthWarningThreshold` | `/` | Queue depth warning threshold. | -| `QueueDepthCriticalThreshold` | `/` | Queue depth critical threshold. | +| `QueueDepthWarningThreshold` | `5000` | Message count that triggers a Warning health status. | +| `QueueDepthCriticalThreshold` | `10000` | Message count that triggers a Critical health status. | +| `ManagementEnabled` | `false` | Enable RabbitMQ Management HTTP API integration for richer queue stats and dynamic queue discovery. | +| `ManagementPort` | `15672` | RabbitMQ Management plugin HTTP port. Used when `ManagementEnabled` is `true`. | ### Job Dispatcher Configuration @@ -208,6 +212,57 @@ Open telemetry configurations. Set null or empty via environment variables if yo | `ConsecutiveFailureThreshold` | `5` | Number of consecutive failures before a job is automatically disabled. Individual jobs can override this with their own AutoDisableThreshold setting. Default: 5 consecutive failures | | `FailureWindowMinutes` | `60` | Time window in minutes for counting consecutive failures. Failures older than this window don't count towards the threshold. This prevents jobs from being disabled due to old historical failures. Default: 60 minutes (1 hour) | +### BasePath Configuration + +Milvaion API supports hosting under a configurable sub-path (e.g. `/milvaion`) so that both the UI and the backend API can be scoped to a URL prefix. This is useful when deploying behind a reverse proxy alongside other services. + +| Setting | Default | Description | +|---------|---------|-------------| +| `BasePath` | `""` | URL prefix the application is mounted under. Leave empty to host at the root. Example: `/milvaion` | + +When `BasePath` is set: + +- The REST API is available at `{BasePath}/api/v1/...` (e.g. `/milvaion/api/v1/jobs`) +- The SignalR hub is available at `{BasePath}/hubs/jobs` +- The Prometheus metrics endpoint is available at `{BasePath}/metrics` +- The Scalar/OpenAPI documentation is available at `{BasePath}/scalar/v1` +- The SPA (UI) is served at `{BasePath}` and all sub-routes fall back to the SPA index + +When `BasePath` is empty or not set, the application is hosted at the root (`/`). + +#### Example Configuration + +```json +{ + "MilvaionConfig": { + "BasePath": "/milvaion" + } +} +``` + +#### Environment Variable Override + +```bash +MilvaionConfig__BasePath=/milvaion +``` + +#### Docker / docker-compose + +```yaml +services: + milvaion-api: + image: milvasoft/milvaion-api:latest # pull from Docker Hub, no rebuild needed + environment: + - MilvaionConfig__BasePath=/milvaion +``` + +To revert to root hosting, remove the override or set it to empty: + +```yaml +environment: + - MilvaionConfig__BasePath= +``` + --- ## Worker Configuration diff --git a/docs/portaldocs/08-reliability.md b/docs/portaldocs/08-reliability.md index d60c0e0..5ceee56 100644 --- a/docs/portaldocs/08-reliability.md +++ b/docs/portaldocs/08-reliability.md @@ -447,6 +447,64 @@ Clear synced records after 7 days --- +## Schedule Recovery on Restart + +### What Is It? + +The live schedule — when each job runs next — lives in a Redis sorted set that the +dispatcher polls. When the API starts, it reconciles that set with the jobs in the +database, so a deploy, a crash or a Redis flush does not lose the schedule. + +### The Rule: Redis Is Authoritative + +Recovery **fills gaps only**. It never overwrites a run time Redis already holds. + +| Situation | What recovery does | +|-----------|--------------------| +| Redis has a run time for the job | Leaves it alone | +| Redis is empty, job is recurring | Derives the next run from the cron expression | +| Redis is empty, job is one-time and never ran | Schedules it at `executeAt` | +| Redis is empty, job is one-time and `completedAt` is set | Skips it — it is finished | +| Cron expression cannot be parsed | Skips it, rather than scheduling in the past | + +The reason for the first row is the important one. The `executeAt` column is the +*configured start time*, not the next run — see +**[Core Concepts](03-core-concepts.md)**. For a recurring job that has been running for +a while it sits in the past. Writing it back into the sorted set would put every such job +at a due time, and the next dispatcher poll would fire the entire catalogue at once. + +:::note Upgrading +If you are coming from a version where a deploy caused all recurring jobs to run at +once, this is the change that fixes it. No migration or manual cleanup is needed — +recovery simply stops overwriting the live schedule. +::: + +### Verifying After a Deploy + +Open **Upcoming Executions** in the dashboard, or: + +```bash +curl -H "X-ApiKey: $MILVAION_API_KEY" \ + "https://milvaion.example.com/api/v1/jobs/upcoming?withinHours=24" +``` + +Every recurring job should show a future run time. The screen also flags jobs that are +active and recurring but hold no run time at all — those will never fire, and nothing +else in the product surfaces that. + +### Startup Log + +``` +Repopulating Redis with active jobs from database... +Startup recovery (Redis) completed. ZSET: 3 added, 128 already scheduled. Cache: 131 warmed. Total active jobs: 131. +``` + +On a normal restart with Redis still up, expect **`already scheduled` to account for +almost everything** and `added` to be near zero. A large `added` count means Redis lost +its data and the schedule was rebuilt from cron expressions. + +--- + ## Best Practices ### 1. Set Appropriate Timeouts diff --git a/docs/portaldocs/10-monitoring.md b/docs/portaldocs/10-monitoring.md index db3053e..198feaf 100644 --- a/docs/portaldocs/10-monitoring.md +++ b/docs/portaldocs/10-monitoring.md @@ -328,6 +328,64 @@ curl http://localhost:5000/api/v1/dashboard } ``` +### Resource Usage + +How much CPU, memory and disk the API process itself is using. The system configuration endpoint +returns these figures too, but only as one branch of a payload that also walks every configuration +section, parses the connection strings and enumerates the alerting channels - so it is the wrong +thing to poll. This endpoint reads the current process and nothing else. + +```bash +curl http://localhost:5000/api/v1/admin/resource-usage +``` + +```json +{ + "data": { + "sampledAt": "2026-07-20T13:41:02Z", + "hostName": "e5af1cd44f73", + "processorCount": 8, + "cpuUsagePercent": 3.4, + "totalMemoryMB": 4096, + "usedMemoryMB": 612, + "availableMemoryMB": 3484, + "memoryUsagePercent": 14.94, + "processMemoryMB": 1180, + "peakProcessMemoryMB": 1342, + "totalAllocatedMB": 91422, + "gen0Collections": 4821, + "gen1Collections": 913, + "gen2Collections": 26, + "threadCount": 54, + "uptime": "0.04:42:11", + "totalDiskGB": 200, + "availableDiskGB": 150, + "diskUsagePercent": 25 + } +} +``` + +Three of these are worth reading together rather than separately: + +- **`usedMemoryMB`** is the managed heap. **`processMemoryMB`** is what the operating system - and a + container memory limit - sees, and is always larger: it also holds the runtime, native allocations, + loaded assemblies and thread stacks. A gap that keeps widening while the heap stays flat points at + a native leak rather than a managed one. +- **`totalMemoryMB`** comes from the garbage collector, so under a container limit it reports the + limit rather than the host's physical memory. That is the number the process is actually judged + against, which is what makes `memoryUsagePercent` meaningful in a container. +- **`gen2Collections`** rising quickly means objects are surviving long enough to be promoted, which + is what memory pressure looks like before it becomes an out-of-memory failure. + +`cpuUsagePercent` is an average over the process lifetime, not an instantaneous reading. A process +that was busy an hour ago and is idle now still reports a high figure. + +If the counters cannot be read - which happens in a hardened container - every figure is zero and +`collectionError` explains why, rather than the response implying the process is idle. + +To break memory down per background service, use `/admin/diagnostics/services`, which reports each +service's current usage, its growth since start, and whether a leak is suspected. + ### Worker Status ```bash @@ -437,6 +495,115 @@ Or in Seq/ELK: CorrelationId = "corr-789" ``` +### Searching Execution Logs + +Worker execution logs are queryable without leaving Milvaion. Two endpoints, and they are +meant to be used in that order. + +**Summary first.** `GET /jobs/logs/summary` aggregates the logs into counts — by severity, +category, exception type, job, and repeated message — over a time window. The response +size does not grow with log volume, so this is safe on a busy installation where reading +the lines themselves would not be. + +```bash +curl -H "X-ApiKey: $MILVAION_API_KEY" \ + "http://localhost:5000/api/v1/jobs/logs/summary?withinHours=24&level=Error" +``` + +```json +{ + "since": "2026-07-18T12:00:00Z", + "until": "2026-07-19T12:00:00Z", + "totalCount": 18422, + "byLevel": [{ "value": "Information", "count": 17980 }, { "value": "Error", "count": 442 }], + "byExceptionType": [{ "value": "HttpRequestException", "count": 389 }], + "topJobs": [{ "jobName": "SyncOrdersJob", "count": 9120, "errorCount": 401 }], + "topMessages": [{ "message": "Upstream timed out", "level": "Error", "count": 389, "lastSeen": "..." }] +} +``` + +**Then search.** `GET /jobs/logs` returns individual lines, newest first, narrowed by what +the summary pointed at. + +| Parameter | Purpose | +|-----------|---------| +| `jobId`, `occurrenceId` | One job, or one execution | +| `level`, `category`, `exceptionType` | Narrow by classification | +| `since`, `until` | Time bounds, UTC | +| `searchTerm` | Substring match on the message | +| `includeData` | Return the values of structured fields, not just their names | +| `pageNumber`, `pageSize` | Paging, up to 200 lines per page | + +```bash +curl -H "X-ApiKey: $MILVAION_API_KEY" \ + "http://localhost:5000/api/v1/jobs/logs?level=Error&searchTerm=timeout&since=2026-07-19T00:00:00Z" +``` + +:::tip Pair `searchTerm` with `since` +Message search is a substring scan. The time bound is what keeps it fast — see +**[Scaling](09-scaling.md)** for the indexes involved. +::: + +Each line reports the **names** of its structured fields in `dataKeys`; the values come +back only when you ask for them with `includeData=true`. Those values are written by your +worker code and can contain business data, so request them deliberately. + +The same two capabilities are exposed to AI assistants as the `summarize_logs` and +`search_logs` tools — see **[MCP Server](25-mcp-server.md)**. + +--- + +## Upcoming Executions + +The **Upcoming Executions** screen answers "what runs next", for scheduled jobs and +workflows together. It reads the live schedule rather than the database, so it reflects +what will actually happen. + +```bash +curl -H "X-ApiKey: $MILVAION_API_KEY" \ + "http://localhost:5000/api/v1/jobs/upcoming?withinHours=24" +``` + +| Parameter | Purpose | +|-----------|---------| +| `withinHours` | How far ahead to look. Default 24, maximum 720 | +| `kind` | `0` jobs only, `1` workflows only. Omit for both | +| `searchTerm` | Filter by name | +| `onlyProblems` | Only entries that will not run | +| `limit` | Maximum rows, up to 500 | + +### Reading the Status Column + +Each row carries a health value, and the distinction matters: + +| Status | Meaning | +|--------|---------| +| **Scheduled** | The dispatcher holds this time in Redis. It will fire unless something changes. | +| **Projected** | Derived from the cron expression. Workflows only — the workflow engine works the time out on each poll rather than storing it. | +| **Not scheduled** | Active and recurring, but nothing holds a run time for it. **It will not run.** | +| **Invalid cron** | The expression could not be parsed, so no next run can be derived. | + +A job row is a promise; a workflow row is an estimate. + +:::warning "Not scheduled" is the one to act on +Nothing else in the product surfaces this. The job list still shows the job as active, and +because it never runs, no occurrence is ever created to fail. The count appears as a banner +at the top of the screen. + +External jobs are excluded from this check — Milvaion never dispatches them, so their +absence from the schedule is by design. +::: + +### When the Scheduler Is Unreachable + +The Redis client fails open through a circuit breaker: during an outage, reads return +empty rather than erroring. The response therefore carries `schedulerReachable`, and the +screen shows a banner when it is false. Without it, a Redis outage would look identical to +a healthy but idle system. + +Countdowns are measured against `asOfUtc` from the response, not the browser clock, so a +workstation with a skewed clock does not report runs as overdue. + --- ## OpenTelemetry Integration @@ -741,6 +908,82 @@ docker exec milvaion-rabbitmq rabbitmqctl list_connections --- +### Queue Depth Monitoring (`IQueueDepthMonitor`) + +Milvaion includes a built-in `IQueueDepthMonitor` service that tracks queue depths and health status. When the **RabbitMQ Management HTTP API** is enabled, it provides richer statistics — including unacknowledged message counts — and automatically discovers **dynamic consumer queues** (e.g. `scheduled_jobs_queue.SampleWorker`) that are created at runtime when workers bind to the exchange. + +#### Configuration + +Enable Management API integration in `appsettings.json`: + +```json +{ + "MilvaionConfig": { + "RabbitMQ": { + "ManagementEnabled": true, + "ManagementPort": 15672 + } + } +} +``` + +| Setting | Default | Description | +|---------|---------|-------------| +| `ManagementEnabled` | `false` | Enable RabbitMQ Management HTTP API integration | +| `ManagementPort` | `15672` | Management plugin HTTP port | +| `QueueDepthWarningThreshold` | `5000` | Message count that triggers a Warning health status | +| `QueueDepthCriticalThreshold` | `10000` | Message count that triggers a Critical health status | + +> **Note:** The Management API uses the same `Username` and `Password` credentials configured under `MilvaionConfig:RabbitMQ`. + +#### How it Works + +| Scenario | Behavior | +|----------|----------| +| `ManagementEnabled: true` | Uses `GET /api/queues/{vhost}/{queue}` for individual queue stats | +| `ManagementEnabled: true` | Uses `GET /api/queues/{vhost}` to **discover all queues**, including dynamic routing-key queues | +| `ManagementEnabled: false` or API unavailable | Falls back to AMQP passive queue declare (core queues only, `MessagesUnacknowledged` will be `0`) | + +#### Queue Health Statuses + +| Status | Condition | +|--------|-----------| +| `Healthy` | Message count below warning threshold | +| `Warning` | Message count ≥ `QueueDepthWarningThreshold` | +| `Critical` | Message count ≥ `QueueDepthCriticalThreshold` | +| `Unavailable` | Queue or broker unreachable | + +#### API Endpoints + +Retrieve queue stats via the Admin API: + +```bash +# All queues (includes dynamic consumer queues when Management API is enabled) +curl http://localhost:5000/api/v1/admin/queues + +# Single queue depth +curl http://localhost:5000/api/v1/admin/queues/{queueName} +``` + +**Example response for a single queue:** + +```json +{ + "queueName": "scheduled_jobs_queue", + "messageCount": 42, + "consumerCount": 3, + "messagesReady": 40, + "messagesUnacknowledged": 2, + "healthStatus": "Healthy", + "healthMessage": "Queue operating normally", + "timestamp": "2026-01-14T18:10:00.000Z" +} +``` + +> **Dynamic queues:** Without Management API enabled, only the 8 core queues (`jobs`, `worker_logs`, `worker_heartbeat`, `worker_registration`, `status_updates`, `failed_occurrences`, `external_job_registration`, `external_job_occurrence`) are monitored. With it enabled, all queues in the vhost — including per-worker routing-key queues like `scheduled_jobs_queue.SampleWorker` — are returned automatically. + +--- + ## Redis Monitoring ### Key Metrics diff --git a/docs/portaldocs/11-maintenance.md b/docs/portaldocs/11-maintenance.md index 4bca471..6b793be 100644 --- a/docs/portaldocs/11-maintenance.md +++ b/docs/portaldocs/11-maintenance.md @@ -273,6 +273,44 @@ CREATE INDEX IX_JobOccurrences_Status_EndTime ON "JobOccurrences" ("Status", "EndTime"); ``` +### Execution Log Indexes + +`JobOccurrenceLogs` is the largest table in the system — one row per log line per +execution. These support the occurrence detail view and the log search and summary +endpoints described in **[Monitoring](10-monitoring.md)**: + +```sql +-- All lines of one run; also the join path when filtering logs by job +CREATE INDEX IX_JobOccurrenceLogs_OccurrenceId_Timestamp +ON "JobOccurrenceLogs" ("OccurrenceId", "Timestamp"); + +-- Log search and summary both bound the scan by time first +CREATE INDEX IX_JobOccurrenceLogs_Timestamp +ON "JobOccurrenceLogs" ("Timestamp" DESC); + +-- "Only the errors in this window" - the narrowing that has to stay fast during an incident +CREATE INDEX IX_JobOccurrenceLogs_Level_Timestamp +ON "JobOccurrenceLogs" ("Level", "Timestamp" DESC); +``` + +### Optional: Trigram Index for Message Search + +Message search uses `ILIKE '%term%'`, which no B-tree can serve. Without a trigram index +the search still works, but only as fast as the time filter makes it — which is why +`searchTerm` should be paired with `since`. + +If you search logs often and can afford the write overhead and disk: + +```sql +CREATE EXTENSION IF NOT EXISTS pg_trgm; + +CREATE INDEX IX_JobOccurrenceLogs_Message_Trgm +ON "JobOccurrenceLogs" USING gin ("Message" gin_trgm_ops); +``` + +Left opt-in because a GIN index on a high-write table costs on every insert, and log +inserts are the highest-volume writes Milvaion makes. + ### Check for Missing Indexes ```sql diff --git a/docs/portaldocs/13-dashboard-screenshots.md b/docs/portaldocs/13-dashboard-screenshots.md index 69aebdb..2213bba 100644 --- a/docs/portaldocs/13-dashboard-screenshots.md +++ b/docs/portaldocs/13-dashboard-screenshots.md @@ -35,11 +35,28 @@ Browse all scheduled jobs with filtering and sorting capabilities. ![Jobs List](./images/jobs-list.png) **Features:** -- Filter by status (Active, Disabled, Deleted) -- Search by job name or type -- Sort by next run time, last execution, or creation date +- Search by job name or tag +- Card and table views - Quick actions (Enable/Disable, Trigger, Edit, Delete) +**Filters:** + +| Filter | Values | +|--------|--------| +| Status | Active / Inactive | +| Type | Any job type registered by a worker | +| Schedule | Recurring / One-time | +| Worker | Any registered worker | +| Source | Milvaion / External (Quartz, Hangfire) | + +Filters combine, and the active count appears next to a **Clear filters** button. The Type +and Worker lists are built from the worker registry rather than from the jobs currently on +screen, so an option does not disappear the moment you select it. + +A one-time job that has already run keeps its **Active** badge and gains a **Completed** +badge — it is not disabled, it simply has nothing left to do. See +**[Core Concepts](03-core-concepts.md)**. + ### Job Details View comprehensive information about a specific job. @@ -105,6 +122,47 @@ View execution logs for debugging and monitoring. --- +## Upcoming Executions + +Shows what runs next — scheduled jobs and workflows on one timeline, earliest first. + +Unlike the job list, which shows how jobs are *configured*, this screen reads the live +schedule the dispatcher polls. It is the place to look after a deploy, or when somebody +asks why a job did not run. + +**Columns:** +- When — a live countdown plus the absolute time +- Name and tags +- Type — Job or Workflow +- Schedule — the cron expression, or "one-time" +- Target — worker and job name +- Status + +**Filters:** time window (1 hour to 7 days), jobs or workflows, name search, and an +**Only problems** toggle. + +**Status values:** + +| Status | Meaning | +|--------|---------| +| Scheduled | The dispatcher holds this time. It will fire. | +| Projected | Computed from the cron expression — workflows only. | +| Not scheduled | Active and recurring, but nothing will run it. | +| Invalid cron | The expression could not be parsed. | + +Entries with no run time sort to the bottom and are marked down the left edge, so they +stay findable in a long list. A banner at the top reports how many there are, with a +shortcut to filter down to just those. + +Two banners signal trouble rather than data: one when the Redis scheduler is unreachable — +without it, an outage would look like an empty schedule — and one when there are more +recurring jobs than the health check inspects, meaning the count is a floor rather than a +total. + +See **[Monitoring](10-monitoring.md)** for the underlying endpoint. + +--- + ## Workers ### Worker List diff --git a/docs/portaldocs/19-alerting.md b/docs/portaldocs/19-alerting.md index 0658a59..074c21e 100644 --- a/docs/portaldocs/19-alerting.md +++ b/docs/portaldocs/19-alerting.md @@ -273,7 +273,7 @@ Configure alerting using environment variables in your `docker-compose.yml`: ```yaml services: milvaion-api: - image: milvasoft/milvaion:latest + image: milvasoft/milvaion-api:latest environment: # Base URL for action links - MilvaionConfig__Alerting__MilvaionAppUrl=https://milvaion.example.com @@ -380,7 +380,7 @@ spec: spec: containers: - name: milvaion-api - image: milvasoft/milvaion:latest + image: milvasoft/milvaion-api:latest envFrom: - configMapRef: name: milvaion-alerting-config diff --git a/docs/portaldocs/23-local-job-testing.md b/docs/portaldocs/23-local-job-testing.md new file mode 100644 index 0000000..6e9700d --- /dev/null +++ b/docs/portaldocs/23-local-job-testing.md @@ -0,0 +1,600 @@ +--- +id: local-job-testing +title: Local Job Testing +sidebar_position: 23 +description: Test your job implementations locally without RabbitMQ, Redis, or a running Milvaion environment. +--- + + +# Local Job Testing + +This guide explains how to unit-test your job implementations locally. The `Milvasoft.Milvaion.Sdk.Worker` package lets you execute any job and inspect its result without a running RabbitMQ, Redis, or database. + +## Why Test Locally? + +When you develop a new job in your worker, the typical deployment cycle looks like this: + +``` +Code → Build → Docker image → Deploy to Dev → Activate job → Trigger manually → Check logs +``` + +This cycle is slow and requires a live environment for every iteration. Local testing short-circuits this by running the exact same job execution path directly in a test process: + +| | Local Test | Live Environment | +|---|---|---| +| **RabbitMQ** | Not required | Required | +| **Redis** | Not required | Required | +| **Database** | Not required | Required | +| **Milvaion API** | Not required | Required | +| **Feedback speed** | Milliseconds | Minutes | +| **Debugger** | Full support | Not available | +| **Result inspection** | Programmatic | Dashboard only | + +> **Tip**: Local tests do not replace live testing in a Dev environment. Use them to iterate on business logic fast, then validate end-to-end in Dev before promoting. + +--- + +## Template — Pre-configured Test Project + +If you created your worker from the **Milvaion Worker Template**, a test project is already included. You don't need to create or configure anything. + +The template generates the following structure out of the box: + +``` +MyWorker/ +├── MyWorker.csproj ← your worker +├── MyJobs.cs +├── Program.cs +└── MyWorker.Tests/ + ├── MyWorker.Tests.csproj ← test project (pre-configured) + └── SampleJobTests.cs ← ready-to-run example tests +``` + +The test project already has: +- `xUnit`, `FluentAssertions`, and `Microsoft.NET.Test.Sdk` referenced +- A `ProjectReference` pointing to your worker +- Sample tests for every job type included in the template + +To verify everything works right after scaffolding, just run: + +```bash +dotnet test MyWorker.Tests +``` + +All sample tests should pass immediately with no additional setup. + +> If you used the template, skip to [JobTestRunner API](#jobtestrunner-api) and start writing tests for your own jobs. + +--- + +## Setup + +### 1. Create a Test Project + +Add an xUnit test project alongside your worker: + +```bash +dotnet new xunit -n MyWorker.Tests +cd MyWorker.Tests +``` + +### 2. Add References + +Add the testing SDK and a reference to your worker project: + +```bash +dotnet add reference ../MyWorker/MyWorker.csproj +dotnet add package Milvasoft.Milvaion.Sdk.Worker +dotnet add package FluentAssertions +``` + +Your `.csproj` should look like: + +```xml + + + + net10.0 + true + disable + + + + + + + + + + + + + + + +``` + +--- + +## JobTestRunner API + +`JobTestRunner` is a fluent builder that wraps `JobExecutor` — the same component that runs your jobs in production. + +### Builder Methods + +| Method | Description | Default | +|--------|-------------|---------| +| `For(IJobBase job)` | Sets the job instance to run | — | +| `WithJobData(T data)` | Serializes `data` as JSON and passes it as job data | `{}` | +| `WithJobData(string json)` | Sets raw JSON string as job data | `{}` | +| `WithTimeout(int seconds)` | Sets execution timeout; `0` disables timeout | `30` | +| `WithWorkerId(string id)` | Sets the worker ID in the execution context | `"local-test"` | +| `WithCancellationToken(CancellationToken)` | Passes a cancellation token | `None` | +| `WithLoggerFactory(ILoggerFactory)` | Captures logs through a custom logger | Console | + +All methods return the builder, so they can be chained freely. + +### Return Value: `JobExecutionResult` + +`RunAsync()` returns a `JobExecutionResult` record with the following fields: + +| Field | Type | Description | +|-------|------|-------------| +| `Status` | `JobOccurrenceStatus` | `Completed`, `Failed`, `Cancelled`, `TimedOut` | +| `DurationMs` | `long` | Wall-clock time of the job in milliseconds | +| `Exception` | `string` | Exception message and inner exception chain (if any) | +| `Result` | `string` | Serialized return value for `IJobWithResult` jobs | +| `Logs` | `List` | All log entries written via `context.Log*()` | +| `IsPermanentFailure` | `bool` | `true` if a `PermanentJobException` was thrown | +| `CorrelationId` | `Guid` | Correlation ID of this test run | + +--- + +## Basic Usage + +### Simplest Test + +```csharp +[Fact] +public async Task MyJob_ShouldComplete_WhenRunNormally() +{ + var result = await JobTestRunner + .For(new MyJob()) + .RunAsync(); + + result.Status.Should().Be(JobOccurrenceStatus.Completed); + result.Exception.Should().BeNull(); +} +``` + +### With Typed Job Data + +When your job reads typed data via `context.GetData()`, pass it with `WithJobData()`: + +```csharp +[Fact] +public async Task SendInvoiceJob_ShouldComplete_WhenDataIsValid() +{ + var jobData = new InvoiceJobData + { + CustomerId = 42, + InvoiceId = 1001, + Currency = "USD" + }; + + var result = await JobTestRunner + .For(new SendInvoiceJob()) + .WithJobData(jobData) + .WithTimeout(60) + .RunAsync(); + + result.Status.Should().Be(JobOccurrenceStatus.Completed); + result.Exception.Should().BeNull(); + result.DurationMs.Should().BeGreaterThan(0); +} +``` + +### With Raw JSON + +When you want full control over the raw payload: + +```csharp +[Fact] +public async Task MyJob_ShouldHandleMissingFields_Gracefully() +{ + var result = await JobTestRunner + .For(new MyJob()) + .WithJobData("""{"customerId": 99}""") + .RunAsync(); + + result.Status.Should().Be(JobOccurrenceStatus.Completed); +} +``` + +--- + +## Testing Failure Scenarios + +### Exception Handling + +```csharp +[Fact] +public async Task ProcessOrderJob_ShouldFail_WhenOrderNotFound() +{ + var result = await JobTestRunner + .For(new ProcessOrderJob()) + .WithJobData(new OrderJobData { OrderId = -1 }) + .RunAsync(); + + result.Status.Should().Be(JobOccurrenceStatus.Failed); + result.Exception.Should().Contain("Order not found"); + result.IsPermanentFailure.Should().BeFalse(); +} +``` + +### Permanent Failures + +If your job throws `PermanentJobException`, the result marks the failure as permanent (no retry): + +```csharp +// In your job +throw new PermanentJobException("Invalid invoice data — will not retry."); +``` + +```csharp +// In your test +[Fact] +public async Task ProcessOrderJob_ShouldFailPermanently_WhenDataIsInvalid() +{ + var result = await JobTestRunner + .For(new ProcessOrderJob()) + .WithJobData(new OrderJobData { OrderId = 0 }) + .RunAsync(); + + result.Status.Should().Be(JobOccurrenceStatus.Failed); + result.IsPermanentFailure.Should().BeTrue(); +} +``` + +--- + +## Testing Timeouts + +Use `WithTimeout()` to verify your job respects execution time limits: + +```csharp +[Fact] +public async Task LongRunningJob_ShouldTimeOut_WhenExceedsLimit() +{ + var result = await JobTestRunner + .For(new DataSyncJob()) + .WithTimeout(1) // 1 second — much shorter than the job's actual runtime + .RunAsync(); + + result.Status.Should().Be(JobOccurrenceStatus.TimedOut); + result.Exception.Should().Contain("timeout"); +} +``` + +> **Note**: The timeout in tests uses the same `CancellationTokenSource` mechanism as production. If your job passes `context.CancellationToken` to all async operations, it will be cancelled correctly. + +--- + +## Testing Cancellation + +Test that your job stops cleanly when cancelled: + +```csharp +[Fact] +public async Task MyJob_ShouldCancel_WhenTokenCancelled() +{ + using var cts = new CancellationTokenSource(); + await cts.CancelAsync(); + + var result = await JobTestRunner + .For(new MyJob()) + .WithCancellationToken(cts.Token) + .RunAsync(); + + result.Status.Should().Be(JobOccurrenceStatus.Cancelled); +} + +[Fact] +public async Task MyJob_ShouldCancelMidExecution_WhenTokenCancelledAfterDelay() +{ + using var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(500)); + + var result = await JobTestRunner + .For(new MyJob()) + .WithCancellationToken(cts.Token) + .WithTimeout(0) // Disable runner timeout; only use the external token + .RunAsync(); + + result.Status.Should().BeOneOf(JobOccurrenceStatus.Cancelled, JobOccurrenceStatus.Completed); +} +``` + +--- + +## Testing Jobs with Return Values + +For jobs implementing `IAsyncJobWithResult` or `IJobWithResult`, the serialized result is available in `result.Result`: + +```csharp +[Fact] +public async Task GenerateReportJob_ShouldReturnFilePath_WhenSucceeds() +{ + var result = await JobTestRunner + .For(new GenerateReportJob()) + .WithJobData(new ReportJobData { ReportType = "monthly", Month = 6 }) + .WithTimeout(120) + .RunAsync(); + + result.Status.Should().Be(JobOccurrenceStatus.Completed); + result.Result.Should().NotBeNullOrEmpty(); + result.Result.Should().Contain("monthly"); +} +``` + +--- + +## Testing Jobs with Dependency Injection + +When your job has constructor-injected services, build the job manually using a `ServiceCollection`: + +```csharp +[Fact] +public async Task SendInvoiceJob_ShouldSendEmail_WhenEmailServiceAvailable() +{ + // Arrange — wire up services + var services = new ServiceCollection(); + services.AddLogging(b => b.AddConsole()); + services.AddScoped(); // or a stub/mock + services.AddScoped(); + var sp = services.BuildServiceProvider(); + + // Resolve the job from DI (same as production) + var job = sp.GetRequiredService(); + + // Act + var result = await JobTestRunner + .For(job) + .WithJobData(new InvoiceJobData { InvoiceId = 1001 }) + .RunAsync(); + + // Assert + result.Status.Should().Be(JobOccurrenceStatus.Completed); +} +``` + +### Using Mocks + +For unit tests where you want to isolate the job from real services, use Moq: + +```csharp +[Fact] +public async Task SendInvoiceJob_ShouldCallEmailService_ExactlyOnce() +{ + // Arrange + var emailServiceMock = new Mock(); + emailServiceMock + .Setup(x => x.SendAsync(It.IsAny(), It.IsAny(), It.IsAny())) + .Returns(Task.CompletedTask); + + var job = new SendInvoiceJob(emailServiceMock.Object); + + // Act + var result = await JobTestRunner + .For(job) + .WithJobData(new InvoiceJobData { InvoiceId = 1001, RecipientEmail = "client@example.com" }) + .RunAsync(); + + // Assert + result.Status.Should().Be(JobOccurrenceStatus.Completed); + emailServiceMock.Verify( + x => x.SendAsync("client@example.com", It.IsAny(), It.IsAny()), + Times.Once); +} +``` + +--- + +## Capturing Logs in Tests + +By default `JobTestRunner` writes to the console. To capture log output in xUnit test output, supply a custom `ILoggerFactory`: + +```csharp +using Microsoft.Extensions.Logging; +using Xunit.Abstractions; + +public class MyJobTests(ITestOutputHelper output) +{ + private ILoggerFactory CreateLoggerFactory() => + LoggerFactory.Create(b => + { + b.SetMinimumLevel(LogLevel.Debug); + b.AddConsole(); + // Optional: pipe to xUnit output via a custom provider + }); + + [Fact] + public async Task MyJob_ShouldLogStartAndCompletion() + { + var result = await JobTestRunner + .For(new MyJob()) + .WithLoggerFactory(CreateLoggerFactory()) + .RunAsync(); + + // Logs written via context.Log*() are available in result.Logs + result.Logs.Should().Contain(l => l.Message.Contains("started")); + result.Logs.Should().Contain(l => l.Message.Contains("completed")); + } +} +``` + +> **Note**: `result.Logs` contains every entry written by `context.LogInformation()`, `context.LogWarning()`, and `context.LogError()` — the same entries that appear in the Milvaion dashboard occurrence detail view. + +--- + +## Recommended Test Structure + +Organize your test file to cover all meaningful scenarios for each job: + +``` +MyWorker.Tests/ +├── MyWorker.Tests.csproj +└── Jobs/ + ├── ProcessOrderJobTests.cs + ├── SendInvoiceJobTests.cs + └── DataSyncJobTests.cs +``` + +A typical test class per job: + +```csharp +public class ProcessOrderJobTests +{ + // ── Happy path ────────────────────────────────────────── + [Fact] + public async Task ShouldComplete_WhenOrderIsValid() { ... } + + // ── Job data edge cases ───────────────────────────────── + [Fact] + public async Task ShouldComplete_WhenJobDataIsNull() { ... } + + [Fact] + public async Task ShouldFail_WhenOrderIdIsNegative() { ... } + + // ── Resilience ────────────────────────────────────────── + [Fact] + public async Task ShouldFailPermanently_WhenOrderIsDuplicate() { ... } + + [Fact] + public async Task ShouldTimeOut_WhenExceedsTimeLimit() { ... } + + [Fact] + public async Task ShouldCancel_WhenTokenCancelled() { ... } +} +``` + +--- + +## Complete Example + +The `SampleWorker.Tests` project in the repository demonstrates the full pattern across multiple job types: + +```csharp +public class SampleJobTests +{ + private ILoggerFactory CreateLoggerFactory() => + LoggerFactory.Create(b => b.AddConsole().SetMinimumLevel(LogLevel.Debug)); + + [Fact] + public async Task TestJob_ShouldComplete_WhenRunNormally() + { + var result = await JobTestRunner + .For(new TestJob()) + .WithLoggerFactory(CreateLoggerFactory()) + .RunAsync(); + + result.Status.Should().Be(JobOccurrenceStatus.Completed); + result.Exception.Should().BeNull(); + result.DurationMs.Should().BeGreaterThan(0); + } + + [Fact] + public async Task SampleSendEmailJob_ShouldComplete_WhenValidDataProvided() + { + var jobData = new EmailJobData + { + To = "test@example.com", + Subject = "Local test email", + Body = "Hello from local test!" + }; + + var result = await JobTestRunner + .For(new SampleSendEmailJob()) + .WithJobData(jobData) + .WithTimeout(120) + .WithLoggerFactory(CreateLoggerFactory()) + .RunAsync(); + + result.Status.Should().Be(JobOccurrenceStatus.Completed); + } + + [Fact] + public async Task AlwaysFailingJob_ShouldFail_WithExceptionMessage() + { + var result = await JobTestRunner + .For(new AlwaysFailingJob()) + .WithLoggerFactory(CreateLoggerFactory()) + .RunAsync(); + + result.Status.Should().Be(JobOccurrenceStatus.Failed); + result.Exception.Should().Contain("always fails"); + } + + [Fact] + public async Task HaveResultJob_ShouldReturnSerializedResult() + { + var result = await JobTestRunner + .For(new HaveResultJob()) + .WithLoggerFactory(CreateLoggerFactory()) + .RunAsync(); + + result.Status.Should().Be(JobOccurrenceStatus.Completed); + result.Result.Should().Contain("Test Product"); + } + + [Fact] + public async Task TestJob_ShouldTimeOut_WhenTimeoutIsVeryShort() + { + var result = await JobTestRunner + .For(new TestJob()) + .WithTimeout(1) + .WithLoggerFactory(CreateLoggerFactory()) + .RunAsync(); + + result.Status.Should().Be(JobOccurrenceStatus.TimedOut); + } + + [Fact] + public async Task TestJob_ShouldBeCancelled_WhenTokenCancelled() + { + using var cts = new CancellationTokenSource(); + await cts.CancelAsync(); + + var result = await JobTestRunner + .For(new TestJob()) + .WithCancellationToken(cts.Token) + .WithLoggerFactory(CreateLoggerFactory()) + .RunAsync(); + + result.Status.Should().Be(JobOccurrenceStatus.Cancelled); + } +} +``` + +--- + +## What's Not Tested + +Local tests use `JobExecutor` directly and therefore do not cover: + +| Concern | Where to Test | +|---------|--------------| +| RabbitMQ message routing | Integration tests / Dev environment | +| Retry + DLQ behavior | Integration tests / Dev environment | +| Concurrent execution limits | Dev environment | +| Heartbeat / zombie detection | Dev environment | +| Dashboard visibility | Dev environment | +| Redis cancellation channel | Integration tests | + +--- + +## What's Next? + +- **[Implementing Jobs](05-implementing-jobs.md)** - Advanced job patterns: DI, error handling, long-running jobs +- **[Configuration](06-configuration.md)** - Timeout and retry configuration for production +- **[Reliability](08-reliability.md)** - Retry, DLQ, and permanent failure handling +- **[Monitoring](10-monitoring.md)** - Viewing job logs and occurrence details in the dashboard diff --git a/docs/portaldocs/24-api-keys.md b/docs/portaldocs/24-api-keys.md new file mode 100644 index 0000000..4a8f6cf --- /dev/null +++ b/docs/portaldocs/24-api-keys.md @@ -0,0 +1,120 @@ +--- +id: api-keys +title: "Api Keys" +sidebar_label: "Api Keys" +sidebar_position: 121 +description: "Create and manage api keys so CI pipelines, scripts and MCP clients can call the Milvaion API without an interactive login." +keywords: [milvaion api key, api authentication, service account, ci integration] +--- + +# Api Keys + +An api key is a non-interactive credential. It lets something that cannot log in through a browser — a CI pipeline, a deployment script, an MCP client — call the Milvaion API on its own. + +Keys carry the same permissions as users, drawn from the same catalog. A key is therefore never "admin by default": it can do exactly what you granted it and nothing else. + +## Creating a Key + +Go to **User Management → Api Keys → Create Api Key**. + +| Field | Notes | +|-------|-------| +| **Name** | How you will recognise it later. Name it after the thing that will hold it: `CI pipeline`, `Claude Code - Bugra`. | +| **Description** | Optional. Useful when somebody finds the key in six months and wonders what breaks if they revoke it. | +| **Expires** | 30 days, 90 days, 1 year, or never. Fixed at creation — see [Rotating a Key](#rotating-a-key). | +| **Permissions** | The same permission catalog used by roles. Grant the minimum. | + +:::danger The key is shown once + +Milvaion does not store the key, only a signed record of it. The create dialog is the only place it ever appears. If you lose it, revoke it and create another — there is no way to recover it. + +::: + +## Using a Key + +Send it in the `X-ApiKey` header: + +```bash +curl -X PATCH http://localhost:5000/api/v1/jobs \ + -H "Content-Type: application/json" \ + -H "X-ApiKey: $MILVAION_API_KEY" \ + -d '{"pageNumber": 1, "rowCount": 20}' +``` + +Every endpoint that accepts an interactive session also accepts an api key. The permission requirements are identical — a key without `ScheduledJobManagement.List` gets the same 403 a user without it would. + +The one exception is the api key endpoints themselves. Those require an interactive session, so a leaked key cannot mint replacements for itself and survive being revoked. + +## Choosing Permissions + +Grant the narrowest set that does the job. Two common shapes: + +**Read-only monitoring** — a dashboard, an alerting script, an [MCP client](./25-mcp-server.md) you want to be safe by construction: + +- `ScheduledJobManagement.List`, `ScheduledJobManagement.Detail` +- `FailedOccurrenceManagement.List` +- `WorkerManagement.List` +- `WorkflowManagement.List` + +**CI pipeline that triggers a job after deploy** — the above plus: + +- `ScheduledJobManagement.Trigger` + +Avoid granting `App.SuperAdmin` to a key. It bypasses every other check, which defeats the point of scoping the key at all. + +## Revoking and Deleting + +**Revoke** stops the key working immediately while keeping the record, so the audit trail still explains what the key was and who created it. This is what you want in almost every case, including a suspected leak. + +**Delete** removes the record entirely. Reserve it for housekeeping — keys created by mistake, or long-dead integrations. + +Revocation takes effect at once across every API replica. Authentication caches key lookups, but revoking invalidates that cache explicitly rather than waiting for it to expire. + +## Rotating a Key + +Expiry is fixed at creation, and the key material cannot be regenerated for an existing record. To rotate: + +1. Create a new key with the same permissions. +2. Deploy it wherever the old one is used. +3. Revoke the old key. + +The **Last Used** column tells you whether step 2 actually landed everywhere. If the old key is still being used, something you forgot is still holding it. + +## Rotating the Signing Secret + +All keys are signed with `MilvaionConfig.ApiKey.Secret`. Rotating that secret invalidates **every** key at once, so it is a deliberate, disruptive operation — do it if you believe the secret itself leaked. + +```json +{ + "MilvaionConfig": { + "ApiKey": { + "Secret": "your-new-secret", + "Version": 2 + } + } +} +``` + +Increment `Version` at the same time. Keys issued under an older version are then rejected with a clear message about a retired signing secret, instead of failing signature validation with no explanation. After rotating, re-create every key that is still in use. + +:::warning Keep the secret out of source control + +Anyone holding `ApiKey.Secret` can mint valid keys for any permission set. Supply it through an environment variable or a secret store in production: + +```bash +MilvaionConfig__ApiKey__Secret=... +``` + +::: + +## Auditing + +Key creation, update, revocation and deletion are recorded as user activities, visible under **User Management → Activity Logs**. + +Each key also tracks **Last Used**, written at most once every few minutes to avoid a database write per request. Treat it as "used recently", not as a precise access log. + +## Next Steps + +- **[MCP Server](./25-mcp-server.md)** — point an AI coding assistant at Milvaion using a read-only key +- **[Enterprise Features](./22-enterprise-features.md)** — roles, permissions and activity tracking +- **[Security](./12-security.md)** — hardening guidance for the whole deployment diff --git a/docs/portaldocs/25-mcp-server.md b/docs/portaldocs/25-mcp-server.md new file mode 100644 index 0000000..ab1a3eb --- /dev/null +++ b/docs/portaldocs/25-mcp-server.md @@ -0,0 +1,400 @@ +--- +id: mcp-server +title: "MCP Server" +sidebar_label: "MCP Server" +sidebar_position: 122 +description: "Connect Claude Code, Cursor or GitHub Copilot to Milvaion over the Model Context Protocol and ask about jobs, failures and workers in plain language." +keywords: [milvaion mcp, model context protocol, claude code, cursor, copilot, ai job scheduler] +--- + +# MCP Server + +Milvaion exposes a [Model Context Protocol](https://modelcontextprotocol.io/) server at `/mcp`. Point an AI coding assistant at it and you can ask things like *"which jobs failed last night and why?"* instead of clicking through the dashboard. + +## Which Way Round Does This Work? + +This is the part people usually get backwards, so it is worth being explicit. + +```text +Your machine Your Milvaion server +┌────────────────────────┐ ┌──────────────────┐ +│ Claude Code / Cursor / │ │ Milvaion API │ +│ Copilot │ ──MCP──▶ │ /mcp │ +│ │ │ │ +│ • the model runs here │ ◀──JSON── │ • jobs │ +│ • your subscription │ │ • logs │ +│ • your tokens │ │ • metrics │ +└────────────────────────┘ └──────────────────┘ +``` + +**Milvaion never calls a language model.** It does not store an OpenAI, Anthropic or Google key, and it does not consume tokens. It is the data source; the model lives entirely in your editor, on your subscription. + +The only credential involved is a **Milvaion api key**, which your editor uses to authenticate to Milvaion. + +## Setup + +### 1. Create a read-only api key + +Follow [Api Keys](./24-api-keys.md) and grant only: + +- `ScheduledJobManagement.List`, `ScheduledJobManagement.Detail` +- `FailedOccurrenceManagement.List` +- `WorkerManagement.List` +- `WorkflowManagement.List` + +With this set the assistant can investigate anything and change nothing. Add `ScheduledJobManagement.Trigger` or `WorkflowManagement.Trigger` later if you want it to be able to run jobs. + +### 2. Register the server in your client + +Milvaion is a **remote HTTP** MCP server, so any client that supports streamable HTTP transport can connect. What differs between clients is the config file location and, annoyingly, the key names. + +| Client | Config location | Top-level key | URL field | +|--------|-----------------|---------------|-----------| +| Claude Code | `.mcp.json` or `claude mcp add` | `mcpServers` | `url` | +| Cursor | `.cursor/mcp.json` | `mcpServers` | `url` | +| VS Code / Copilot | `.vscode/mcp.json` | **`servers`** | `url` | +| Windsurf | `~/.codeium/windsurf/mcp_config.json` | `mcpServers` | **`serverUrl`** | +| Gemini CLI | `~/.gemini/settings.json` | `mcpServers` | `httpUrl` | +| Claude Desktop | Connectors UI | — | — | +| ChatGPT | Connectors UI | — | — | + +Getting `servers` and `mcpServers` the wrong way round, or `url` instead of `serverUrl`, is the single most common reason a client silently lists no tools. + +--- + +#### Claude Code + +No file editing needed: + +```bash +claude mcp add --transport http milvaion \ + https://milvaion.yourcompany.com/mcp \ + --header "X-ApiKey: $MILVAION_API_KEY" +``` + +Add `--scope project` to write it to `.mcp.json` and share it with the team. Verify with `claude mcp list`, or type `/mcp` inside a session to see the tool list. + +#### Cursor + +`.cursor/mcp.json` in the project, or the global equivalent from **Settings → MCP**: + +```json +{ + "mcpServers": { + "milvaion": { + "type": "http", + "url": "https://milvaion.yourcompany.com/mcp", + "headers": { + "X-ApiKey": "your-api-key" + } + } + } +} +``` + +#### VS Code / GitHub Copilot + +`.vscode/mcp.json`. Note the top-level key is `servers`, and `type` is required: + +```json +{ + "inputs": [ + { + "id": "milvaion-key", + "type": "promptString", + "description": "Milvaion api key", + "password": true + } + ], + "servers": { + "milvaion": { + "type": "http", + "url": "https://milvaion.yourcompany.com/mcp", + "headers": { + "X-ApiKey": "${input:milvaion-key}" + } + } + } +} +``` + +The `inputs` block makes VS Code prompt for the key at runtime instead of storing it in the file. + +:::warning Agent mode is required + +MCP tools do not appear in Copilot's default Ask mode. Switch the chat to **Agent** mode or the server will look connected while nothing can call it. + +::: + +#### Windsurf + +`~/.codeium/windsurf/mcp_config.json`. Windsurf uses `serverUrl`, not `url`: + +```json +{ + "mcpServers": { + "milvaion": { + "serverUrl": "https://milvaion.yourcompany.com/mcp", + "headers": { + "X-ApiKey": "your-api-key" + } + } + } +} +``` + +#### Gemini CLI + +`~/.gemini/settings.json` for all projects, or `.gemini/settings.json` for one. Gemini CLI expands `${VAR}` from your environment, so the key never has to sit in the file: + +```json +{ + "mcpServers": { + "milvaion": { + "httpUrl": "https://milvaion.yourcompany.com/mcp", + "headers": { + "X-ApiKey": "${MILVAION_API_KEY}" + } + } + } +} +``` + +Restart the CLI after editing. `includeTools` and `excludeTools` let you narrow the surface further on the client side — though scoping the api key is the stronger control, since it is enforced server side. + +#### Claude Desktop + +Claude Desktop's `claude_desktop_config.json` validates **stdio servers only**. Putting a `url` in it does nothing — the entry is ignored, which is why the tools never show up. + +Two options: + +1. **Connectors UI** — Settings → Connectors → Add custom connector, and paste the `/mcp` URL. This path requires **HTTPS**, so a `localhost` instance will not work. +2. **`mcp-remote` bridge** — run a local stdio process that forwards to your HTTP endpoint: + +```json +{ + "mcpServers": { + "milvaion": { + "command": "npx", + "args": [ + "-y", "mcp-remote", + "http://localhost:5000/mcp", + "--header", "X-ApiKey:${MILVAION_API_KEY}" + ], + "env": { "MILVAION_API_KEY": "your-api-key" } + } + } +} +``` + +For local development, Claude Code is far less friction than either of these. + +#### ChatGPT + +Settings → Connectors → **Advanced → Developer mode**, then add a connector pointing at your `/mcp` URL. + +:::warning ChatGPT limitations worth knowing before you try + +- **HTTPS only, remote only.** No stdio, and no `localhost` — expose the endpoint through a tunnel or a real domain first. +- **Write tools need a Business, Enterprise or Edu workspace.** On individual Plus and Pro plans, custom connectors are restricted to read and fetch operations, so `trigger_job` and anything else that changes state will not be callable. +- **The connector must be enabled per conversation.** Turning it on once in settings is not enough; most "it stopped working" reports are this. + +::: + +For a read-only Milvaion key the Plus/Pro restriction does not matter much, since the fifteen reading tools are the ones you would grant anyway. + +#### Google AI Studio + +AI Studio does not currently expose custom MCP connectors in the same way. For a Google-based workflow, use **Gemini CLI** as documented above. + +--- + +:::tip Keep the key out of the repository + +Every client above supports either environment variable expansion or a runtime prompt. Use it. A config file with a live key in it will eventually be committed, and a key in git history means creating a new one and revoking the old. + +::: + +### 3. Ask something + +> Which jobs failed last night? + +> Job `daily-invoice-export` has been failing since Tuesday. Read the logs and tell me what changed. + +> Is there a worker alive that can run `SendReportJob`? + +## Available Tools + +Every tool is gated by the permission in the right hand column. A key that lacks a permission cannot call the tool, so the tool surface an assistant actually sees is decided entirely by how you scope the key. + +### Reading + +| Tool | What it does | Permission | +|------|--------------|------------| +| `get_overview` | Dashboard statistics: counts, throughput, worker health | `ScheduledJobManagement.List` | +| `list_jobs` | Scheduled jobs, with free text search | `ScheduledJobManagement.List` | +| `get_job` | One job in full: schedule, worker, job data, retry and timeout settings | `ScheduledJobManagement.Detail` | +| `list_tags` | Distinct tags in use across jobs | `ScheduledJobManagement.List` | +| `list_occurrences` | Executions with status, duration and result | `ScheduledJobManagement.List` | +| `get_occurrence` | One execution with its logs and exception detail | `ScheduledJobManagement.Detail` | +| `list_failures` | Jobs that exhausted their retries and reached the dead letter queue | `FailedOccurrenceManagement.List` | +| `get_failure` | One dead letter record with exception and resolution notes | `FailedOccurrenceManagement.Detail` | +| `list_workers` | Workers with heartbeat status and executable job types | `WorkerManagement.List` | +| `get_worker` | One worker in full | `WorkerManagement.Detail` | +| `list_workflows` | Workflows | `WorkflowManagement.List` | +| `get_workflow` | One workflow with its step graph | `WorkflowManagement.Detail` | +| `list_workflow_runs` | Workflow runs and their status | `WorkflowManagement.List` | +| `get_workflow_run` | One run, steps in order with dependencies by name | `WorkflowManagement.Detail` | +| `list_activity_logs` | Who changed what, and when | `ActivityLogManagement.List` | +| `summarize_logs` | Execution log counts by level, category, exception, job and message | `ScheduledJobManagement.Detail` | +| `search_logs` | Individual execution log lines | `ScheduledJobManagement.Detail` | +| `list_reports` | Metric report metadata and freshness, without payloads | `ScheduledJobManagement.List` | +| `get_latest_report` | Newest report of a metric type, with its data | `ScheduledJobManagement.Detail` | +| `get_report` | One report by id, for comparing against an older one | `ScheduledJobManagement.Detail` | +| `get_system_health` | Health of the scheduler, database, Redis and RabbitMQ | `SystemAdministration.List` | +| `get_queue_stats` | RabbitMQ queue depths and consumer counts | `SystemAdministration.List` | +| `get_queue_info` | Depth detail for one queue | `SystemAdministration.List` | +| `get_job_statistics` | Aggregate counters across jobs and executions | `SystemAdministration.List` | +| `get_database_statistics` | Size, index efficiency, cache hit ratio, bloat | `SystemAdministration.List` | +| `get_resource_usage` | CPU, memory and disk of the API process, optionally per background service | `SystemAdministration.List` | +| `get_configuration` | Effective scheduler configuration | `SystemAdministration.List` | +| `list_notifications` | Alerts Milvaion itself raised | `InternalNotificationManagement.List` | +| `list_permissions` | The permission catalog, for explaining what to grant | `PermissionManagement.List` | + +### Running and pausing + +| Tool | What it does | Permission | +|------|--------------|------------| +| `trigger_job` | Runs a job immediately, outside its schedule | `ScheduledJobManagement.Trigger` | +| `cancel_occurrence` | Cancels a running execution | `ScheduledJobManagement.Trigger` | +| `set_job_active` | Pauses or resumes a job without deleting it | `ScheduledJobManagement.Update` | +| `trigger_workflow` | Starts a workflow run immediately | `WorkflowManagement.Trigger` | +| `cancel_workflow_run` | Cancels a run in progress | `WorkflowManagement.Trigger` | + +### Editing + +| Tool | What it does | Permission | +|------|--------------|------------| +| `create_job` | Creates a scheduled job | `ScheduledJobManagement.Create` | +| `update_job` | Changes name, description, tags, cron, job data or timeout | `ScheduledJobManagement.Update` | +| `resolve_failures` | Marks dead letter records resolved with a note | `FailedOccurrenceManagement.Update` | + +### Deleting + +| Tool | What it does | Permission | +|------|--------------|------------| +| `delete_job` | Deletes a job and its history | `ScheduledJobManagement.Delete` | +| `delete_occurrences` | Deletes execution records | `ScheduledJobManagement.Delete` | +| `delete_failures` | Deletes dead letter records | `FailedOccurrenceManagement.Delete` | +| `delete_worker` | Removes a worker registration | `WorkerManagement.Delete` | +| `delete_workflow` | Deletes a workflow and its run history | `WorkflowManagement.Delete` | + +### Filtering + +The list tools filter rather than make the assistant page through everything: + +| Tool | Filters | +|------|---------| +| `list_jobs` | `tag`, `workerId`, `isActive`, free text | +| `list_occurrences` | `jobId`, `status`, `workerId`, `since`, `until`, free text | +| `list_failures` | `jobId`, `resolved`, `since`, `until`, free text | +| `list_activity_logs` | `since`, `until` | +| `summarize_logs` | `jobId`, `level`, `since`, `until`, free text | +| `search_logs` | `jobId`, `occurrenceId`, `level`, `category`, `exceptionType`, `since`, `until`, free text | +| `list_workflow_runs` | `workflowId` | + +All times are UTC. `since` on its own is the common case — "what failed since midnight" needs one bound, not two. + +`get_occurrence` returns the tail of the log rather than all of it, defaulting to the last 100 lines with a `logLines` parameter to raise it. A job that logs in a loop can otherwise fill an assistant's entire context in one call. + +### Prompts + +Three prompt templates ship with the server, selectable in clients that support them: + +| Prompt | What it does | +|--------|--------------| +| `diagnose_job` | Walks a failing job in order: failures, pattern, logs, worker health, then recent config changes | +| `overnight_review` | Reviews a recent window and groups failures by cause rather than listing them | +| `explain_workflow` | Describes a workflow's steps, branching and data flow in plain language | + +These matter more than they look. Left to itself a model tends to start with whichever tool it read first; the prompts put a working order in front of it. + +### Not exposed + +Users, roles, permissions, api keys, dispatcher control, configuration and content management have no tools and are not reachable over MCP, regardless of what the key is granted. Workflow creation and editing are also absent: authoring a directed graph with conditions and data mappings belongs in the visual builder, not in a tool call. + +:::tip Scope the key, not the tool list + +You do not choose which tools to expose — you choose what the key can do. Granting only `List` and `Detail` permissions leaves an assistant with the fifteen reading tools and nothing else, which is the right default for most people. + +::: + +## Security Model + +**Authentication happens at the endpoint.** `/mcp` requires a valid credential; an anonymous request never reaches a tool. + +**Authorization happens per tool.** Each tool checks its own permission before doing any work. A key without `ScheduledJobManagement.Trigger` calling `trigger_job` gets an error naming the missing permission — the assistant is told exactly what it lacks, so it stops retrying and can tell you what to grant. + +**Side effects are attributed.** Anything triggered, cancelled or resolved through MCP is recorded with the key's name, so the history distinguishes it from a human acting in the dashboard. + +**Concurrency policies are never bypassed.** `trigger_job` always dispatches with force disabled. Overriding a job's concurrency policy stays a deliberate human action in the dashboard. + +**Fields cannot be cleared by accident.** `update_job` only changes the arguments it is given; an omitted argument is left alone rather than blanked. The trade-off is that clearing a field on purpose has to be done in the dashboard, which is the safer way round. + +**Tools declare what they do.** Every tool carries the MCP annotations clients use to decide whether to prompt: reading tools are marked read-only, `delete_job` and the other four deletions are marked destructive, and the `set_*` and `update_*` tools are marked idempotent. A client that offers "always allow" for a tool can then treat `list_jobs` and `delete_job` differently, which without these it cannot. + +:::warning Treat write permissions as production access + +`Trigger` lets an assistant cause real work to run. `Update` lets it change when your jobs run. `Delete` is irreversible. The tool descriptions tell the model to confirm before doing any of this, and to prefer `set_job_active` over `delete_job` — but those are prompts, not guarantees. + +Grant `Create`, `Update` and `Delete` only where you would be comfortable with the same person having dashboard access. For everything else, a read-only key is the right answer. + +::: + +## Verifying It Works + +`/mcp` speaks JSON-RPC over HTTP, so you can exercise it with `curl` before involving a client. + +List the tools: + +```bash +curl -X POST http://localhost:5000/mcp \ + -H "Content-Type: application/json" \ + -H "Accept: application/json, text/event-stream" \ + -H "X-ApiKey: $MILVAION_API_KEY" \ + -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' +``` + +Call one: + +```bash +curl -X POST http://localhost:5000/mcp \ + -H "Content-Type: application/json" \ + -H "Accept: application/json, text/event-stream" \ + -H "X-ApiKey: $MILVAION_API_KEY" \ + -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"get_overview","arguments":{}}}' +``` + +Without the header the same request returns 401, which is the quickest confirmation that the endpoint is protected. + +## Troubleshooting + +| Symptom | Cause | +|---------|-------| +| **401 Unauthorized** | Missing or malformed `X-ApiKey` header, or the key was revoked or expired. | +| **Api key was issued with a retired signing secret** | `ApiKey.Version` was incremented after the key was created. Create a new key. | +| **"does not have the '...' permission"** | Working as intended. Grant that permission to the key, or use a different key. | +| **404 on /mcp** | The API predates MCP support, or a reverse proxy is not forwarding the path. | +| **Client connects but lists no tools** | Usually a proxy stripping the `Accept: text/event-stream` header. | + +## Deployment Notes + +The server runs in **stateless** mode, so any API replica can serve any request and no sticky sessions are needed behind a load balancer. Server-initiated features that depend on a persistent session — sampling, elicitation, roots — are therefore not available. + +If Milvaion is behind a reverse proxy, make sure `/mcp` is forwarded and that `Content-Type` and `Accept` headers survive the hop. + +## Next Steps + +- **[Api Keys](./24-api-keys.md)** — creating and scoping the credential this needs +- **[Monitoring](./10-monitoring.md)** — the metrics behind `get_overview` +- **[Workflows](./20-workflows.md)** — what `list_workflows` is showing you diff --git a/dotnet-tools.json b/dotnet-tools.json new file mode 100644 index 0000000..b0e38ab --- /dev/null +++ b/dotnet-tools.json @@ -0,0 +1,5 @@ +{ + "version": 1, + "isRoot": true, + "tools": {} +} \ No newline at end of file diff --git a/for multi-arch support b/for multi-arch support deleted file mode 100644 index 1250959..0000000 --- a/for multi-arch support +++ /dev/null @@ -1,314 +0,0 @@ - - SSUUMMMMAARRYY OOFF LLEESSSS CCOOMMMMAANNDDSS - - Commands marked with * may be preceded by a number, _N. - Notes in parentheses indicate the behavior if _N is given. - A key preceded by a caret indicates the Ctrl key; thus ^K is ctrl-K. - - h H Display this help. - q :q Q :Q ZZ Exit. - --------------------------------------------------------------------------- - - MMOOVVIINNGG - - e ^E j ^N CR * Forward one line (or _N lines). - y ^Y k ^K ^P * Backward one line (or _N lines). - f ^F ^V SPACE * Forward one window (or _N lines). - b ^B ESC-v * Backward one window (or _N lines). - z * Forward one window (and set window to _N). - w * Backward one window (and set window to _N). - ESC-SPACE * Forward one window, but don't stop at end-of-file. - d ^D * Forward one half-window (and set half-window to _N). - u ^U * Backward one half-window (and set half-window to _N). - ESC-) RightArrow * Right one half screen width (or _N positions). - ESC-( LeftArrow * Left one half screen width (or _N positions). - ESC-} ^RightArrow Right to last column displayed. - ESC-{ ^LeftArrow Left to first column. - F Forward forever; like "tail -f". - ESC-F Like F but stop when search pattern is found. - r ^R ^L Repaint screen. - R Repaint screen, discarding buffered input. - --------------------------------------------------- - Default "window" is the screen height. - Default "half-window" is half of the screen height. - --------------------------------------------------------------------------- - - SSEEAARRCCHHIINNGG - - /_p_a_t_t_e_r_n * Search forward for (_N-th) matching line. - ?_p_a_t_t_e_r_n * Search backward for (_N-th) matching line. - n * Repeat previous search (for _N-th occurrence). - N * Repeat previous search in reverse direction. - ESC-n * Repeat previous search, spanning files. - ESC-N * Repeat previous search, reverse dir. & spanning files. - ^O^N ^On * Search forward for (_N-th) OSC8 hyperlink. - ^O^P ^Op * Search backward for (_N-th) OSC8 hyperlink. - ^O^L ^Ol Jump to the currently selected OSC8 hyperlink. - ESC-u Undo (toggle) search highlighting. - ESC-U Clear search highlighting. - &_p_a_t_t_e_r_n * Display only matching lines. - --------------------------------------------------- - A search pattern may begin with one or more of: - ^N or ! Search for NON-matching lines. - ^E or * Search multiple files (pass thru END OF FILE). - ^F or @ Start search at FIRST file (for /) or last file (for ?). - ^K Highlight matches, but don't move (KEEP position). - ^R Don't use REGULAR EXPRESSIONS. - ^S _n Search for match in _n-th parenthesized subpattern. - ^W WRAP search if no match found. - ^L Enter next character literally into pattern. - --------------------------------------------------------------------------- - - JJUUMMPPIINNGG - - g < ESC-< * Go to first line in file (or line _N). - G > ESC-> * Go to last line in file (or line _N). - p % * Go to beginning of file (or _N percent into file). - t * Go to the (_N-th) next tag. - T * Go to the (_N-th) previous tag. - { ( [ * Find close bracket } ) ]. - } ) ] * Find open bracket { ( [. - ESC-^F _<_c_1_> _<_c_2_> * Find close bracket _<_c_2_>. - ESC-^B _<_c_1_> _<_c_2_> * Find open bracket _<_c_1_>. - --------------------------------------------------- - Each "find close bracket" command goes forward to the close bracket - matching the (_N-th) open bracket in the top line. - Each "find open bracket" command goes backward to the open bracket - matching the (_N-th) close bracket in the bottom line. - - m_<_l_e_t_t_e_r_> Mark the current top line with . - M_<_l_e_t_t_e_r_> Mark the current bottom line with . - '_<_l_e_t_t_e_r_> Go to a previously marked position. - '' Go to the previous position. - ^X^X Same as '. - ESC-m_<_l_e_t_t_e_r_> Clear a mark. - --------------------------------------------------- - A mark is any upper-case or lower-case letter. - Certain marks are predefined: - ^ means beginning of the file - $ means end of the file - --------------------------------------------------------------------------- - - CCHHAANNGGIINNGG FFIILLEESS - - :e [_f_i_l_e] Examine a new file. - ^X^V Same as :e. - :n * Examine the (_N-th) next file from the command line. - :p * Examine the (_N-th) previous file from the command line. - :x * Examine the first (or _N-th) file from the command line. - ^O^O Open the currently selected OSC8 hyperlink. - :d Delete the current file from the command line list. - = ^G :f Print current file name. - --------------------------------------------------------------------------- - - MMIISSCCEELLLLAANNEEOOUUSS CCOOMMMMAANNDDSS - - -_<_f_l_a_g_> Toggle a command line option [see OPTIONS below]. - --_<_n_a_m_e_> Toggle a command line option, by name. - __<_f_l_a_g_> Display the setting of a command line option. - ___<_n_a_m_e_> Display the setting of an option, by name. - +_c_m_d Execute the less cmd each time a new file is examined. - - !_c_o_m_m_a_n_d Execute the shell command with $SHELL. - #_c_o_m_m_a_n_d Execute the shell command, expanded like a prompt. - |XX_c_o_m_m_a_n_d Pipe file between current pos & mark XX to shell command. - s _f_i_l_e Save input to a file. - v Edit the current file with $VISUAL or $EDITOR. - V Print version number of "less". - --------------------------------------------------------------------------- - - OOPPTTIIOONNSS - - Most options may be changed either on the command line, - or from within less by using the - or -- command. - Options may be given in one of two forms: either a single - character preceded by a -, or a name preceded by --. - - -? ........ --help - Display help (from command line). - -a ........ --search-skip-screen - Search skips current screen. - -A ........ --SEARCH-SKIP-SCREEN - Search starts just after target line. - -b [_N] .... --buffers=[_N] - Number of buffers. - -B ........ --auto-buffers - Don't automatically allocate buffers for pipes. - -c ........ --clear-screen - Repaint by clearing rather than scrolling. - -d ........ --dumb - Dumb terminal. - -D xx_c_o_l_o_r . --color=xx_c_o_l_o_r - Set screen colors. - -e -E .... --quit-at-eof --QUIT-AT-EOF - Quit at end of file. - -f ........ --force - Force open non-regular files. - -F ........ --quit-if-one-screen - Quit if entire file fits on first screen. - -g ........ --hilite-search - Highlight only last match for searches. - -G ........ --HILITE-SEARCH - Don't highlight any matches for searches. - -h [_N] .... --max-back-scroll=[_N] - Backward scroll limit. - -i ........ --ignore-case - Ignore case in searches that do not contain uppercase. - -I ........ --IGNORE-CASE - Ignore case in all searches. - -j [_N] .... --jump-target=[_N] - Screen position of target lines. - -J ........ --status-column - Display a status column at left edge of screen. - -k _f_i_l_e ... --lesskey-file=_f_i_l_e - Use a compiled lesskey file. - -K ........ --quit-on-intr - Exit less in response to ctrl-C. - -L ........ --no-lessopen - Ignore the LESSOPEN environment variable. - -m -M .... --long-prompt --LONG-PROMPT - Set prompt style. - -n ......... --line-numbers - Suppress line numbers in prompts and messages. - -N ......... --LINE-NUMBERS - Display line number at start of each line. - -o [_f_i_l_e] .. --log-file=[_f_i_l_e] - Copy to log file (standard input only). - -O [_f_i_l_e] .. --LOG-FILE=[_f_i_l_e] - Copy to log file (unconditionally overwrite). - -p _p_a_t_t_e_r_n . --pattern=[_p_a_t_t_e_r_n] - Start at pattern (from command line). - -P [_p_r_o_m_p_t] --prompt=[_p_r_o_m_p_t] - Define new prompt. - -q -Q .... --quiet --QUIET --silent --SILENT - Quiet the terminal bell. - -r -R .... --raw-control-chars --RAW-CONTROL-CHARS - Output "raw" control characters. - -s ........ --squeeze-blank-lines - Squeeze multiple blank lines. - -S ........ --chop-long-lines - Chop (truncate) long lines rather than wrapping. - -t _t_a_g .... --tag=[_t_a_g] - Find a tag. - -T [_t_a_g_s_f_i_l_e] --tag-file=[_t_a_g_s_f_i_l_e] - Use an alternate tags file. - -u -U .... --underline-special --UNDERLINE-SPECIAL - Change handling of backspaces, tabs and carriage returns. - -V ........ --version - Display the version number of "less". - -w ........ --hilite-unread - Highlight first new line after forward-screen. - -W ........ --HILITE-UNREAD - Highlight first new line after any forward movement. - -x [_N[,...]] --tabs=[_N[,...]] - Set tab stops. - -X ........ --no-init - Don't use termcap init/deinit strings. - -y [_N] .... --max-forw-scroll=[_N] - Forward scroll limit. - -z [_N] .... --window=[_N] - Set size of window. - -" [_c[_c]] . --quotes=[_c[_c]] - Set shell quote characters. - -~ ........ --tilde - Don't display tildes after end of file. - -# [_N] .... --shift=[_N] - Set horizontal scroll amount (0 = one half screen width). - - --exit-follow-on-close - Exit F command on a pipe when writer closes pipe. - --file-size - Automatically determine the size of the input file. - --follow-name - The F command changes files if the input file is renamed. - --header=[_L[,_C[,_N]]] - Use _L lines (starting at line _N) and _C columns as headers. - --incsearch - Search file as each pattern character is typed in. - --intr=[_C] - Use _C instead of ^X to interrupt a read. - --lesskey-context=_t_e_x_t - Use lesskey source file contents. - --lesskey-src=_f_i_l_e - Use a lesskey source file. - --line-num-width=[_N] - Set the width of the -N line number field to _N characters. - --match-shift=[_N] - Show at least _N characters to the left of a search match. - --modelines=[_N] - Read _N lines from the input file and look for vim modelines. - --mouse - Enable mouse input. - --no-keypad - Don't send termcap keypad init/deinit strings. - --no-histdups - Remove duplicates from command history. - --no-number-headers - Don't give line numbers to header lines. - --no-search-header-lines - Searches do not include header lines. - --no-search-header-columns - Searches do not include header columns. - --no-search-headers - Searches do not include header lines or columns. - --no-vbell - Disable the terminal's visual bell. - --redraw-on-quit - Redraw final screen when quitting. - --rscroll=[_C] - Set the character used to mark truncated lines. - --save-marks - Retain marks across invocations of less. - --search-options=[EFKNRW-] - Set default options for every search. - --show-preproc-errors - Display a message if preprocessor exits with an error status. - --proc-backspace - Process backspaces for bold/underline. - --PROC-BACKSPACE - Treat backspaces as control characters. - --proc-return - Delete carriage returns before newline. - --PROC-RETURN - Treat carriage returns as control characters. - --proc-tab - Expand tabs to spaces. - --PROC-TAB - Treat tabs as control characters. - --status-col-width=[_N] - Set the width of the -J status column to _N characters. - --status-line - Highlight or color the entire line containing a mark. - --use-backslash - Subsequent options use backslash as escape char. - --use-color - Enables colored text. - --wheel-lines=[_N] - Each click of the mouse wheel moves _N lines. - --wordwrap - Wrap lines at spaces. - - - --------------------------------------------------------------------------- - - LLIINNEE EEDDIITTIINNGG - - These keys can be used to edit text being entered - on the "command line" at the bottom of the screen. - - RightArrow ..................... ESC-l ... Move cursor right one character. - LeftArrow ...................... ESC-h ... Move cursor left one character. - ctrl-RightArrow ESC-RightArrow ESC-w ... Move cursor right one word. - ctrl-LeftArrow ESC-LeftArrow ESC-b ... Move cursor left one word. - HOME ........................... ESC-0 ... Move cursor to start of line. - END ............................ ESC-$ ... Move cursor to end of line. - BACKSPACE ................................ Delete char to left of cursor. - DELETE ......................... ESC-x ... Delete char under cursor. - ctrl-BACKSPACE ESC-BACKSPACE ........... Delete word to left of cursor. - ctrl-DELETE .... ESC-DELETE .... ESC-X ... Delete word under cursor. - ctrl-U ......... ESC (MS-DOS only) ....... Delete entire line. - UpArrow ........................ ESC-k ... Retrieve previous command line. - DownArrow ...................... ESC-j ... Retrieve next command line. - TAB ...................................... Complete filename & cycle. - SHIFT-TAB ...................... ESC-TAB Complete filename & reverse cycle. - ctrl-L ................................... Complete filename, list all. diff --git a/src/Milvaion.Api/AppStartup/ApplicationBuilderExtensions.cs b/src/Milvaion.Api/AppStartup/ApplicationBuilderExtensions.cs index 666eb88..9d701de 100644 --- a/src/Milvaion.Api/AppStartup/ApplicationBuilderExtensions.cs +++ b/src/Milvaion.Api/AppStartup/ApplicationBuilderExtensions.cs @@ -1,9 +1,6 @@ using Microsoft.AspNetCore.Authentication.JwtBearer; using Microsoft.AspNetCore.Localization; -using Milvaion.Application.Utils.Constants; using Milvaion.Application.Utils.Extensions; -using Milvaion.Application.Utils.Models.Options; -using Milvaion.Application.Utils.PermissionManager; using Milvaion.Domain.Enums; using Milvasoft.Core.MultiLanguage.Manager; using Milvasoft.Identity.Concrete.Options; diff --git a/src/Milvaion.Api/AppStartup/McpExtensions.cs b/src/Milvaion.Api/AppStartup/McpExtensions.cs new file mode 100644 index 0000000..678c77f --- /dev/null +++ b/src/Milvaion.Api/AppStartup/McpExtensions.cs @@ -0,0 +1,80 @@ +using Milvaion.Api.Mcp; + +namespace Milvaion.Api.AppStartup; + +/// +/// Registration and mapping for the Milvaion MCP server. +/// +public static class McpExtensions +{ + /// + /// Route the MCP endpoint is served from. + /// + public const string McpRoute = "/mcp"; + + /// + /// Adds the MCP server and its tools. + /// + public static IServiceCollection AddMilvaionMcp(this IServiceCollection services) + { + services.AddHttpContextAccessor(); + services.AddScoped(); + + services.AddMcpServer(options => + { + options.ServerInfo = new() + { + Name = "milvaion", + Version = typeof(McpExtensions).Assembly.GetName().Version?.ToString() ?? "1.0.0" + }; + + options.ServerInstructions = + "Milvaion is a distributed job scheduler. Jobs are definitions with a schedule; occurrences are " + + "individual executions of those jobs. When investigating a problem, start with list_failures " + + "rather than list_occurrences - it is a much smaller set of jobs that exhausted their retries. " + + "Use get_occurrence to read the logs and exception for a specific execution. " + + "For questions about the system as a whole rather than one execution - what is going wrong, " + + "which job is noisiest, what errors are new - call summarize_logs first: it returns counts " + + "whose size does not grow with the log volume. Only then use search_logs, narrowed by what " + + "the summary pointed at. " + + "For anything about resource consumption - memory, CPU, disk - call get_resource_usage. It reads " + + "the API process directly, so there is no need for host access and no reason to say you have " + + "none; pass includeBackgroundServices to see which background service is growing. " + + "trigger_job and trigger_workflow cause real work to run in the user's environment; ask before " + + "calling them. " + + "Tools whose id parameter is a list - delete_failures, resolve_failures, delete_occurrences - are " + + "bulk operations: collect every id first and make one call, never a call per record."; + }) + .WithHttpTransport(options => + { + // Stateless keeps any API replica able to serve any request, which matters because Milvaion is + // routinely run behind a load balancer with several API instances. It also rules out + // server-to-client requests such as sampling, which none of these tools need. + options.Stateless = true; + }) + // The assembly must be passed explicitly. The parameterless overload scans Assembly.GetEntryAssembly(), + // which is this API when it runs normally but the test host when it runs under WebApplicationFactory - + // so tool discovery either finds nothing or fails outright, and the failure surfaces as the host never + // being built. + .WithToolsFromAssembly(typeof(McpExtensions).Assembly) + .WithPromptsFromAssembly(typeof(McpExtensions).Assembly); + + return services; + } + + /// + /// Maps the MCP endpoint. + /// + /// + /// Authorization is required at the endpoint, so an anonymous caller never reaches a tool. The default policy + /// accepts both the login token and an api key, which means a developer can point an MCP client at this + /// endpoint with nothing more than a key created in the dashboard. Per-permission checks happen inside the + /// tools themselves - see . + /// + public static WebApplication MapMilvaionMcp(this WebApplication app) + { + app.MapMcp(McpRoute).RequireAuthorization(); + + return app; + } +} diff --git a/src/Milvaion.Api/AppStartup/MissingResxKeyFinder.cs b/src/Milvaion.Api/AppStartup/MissingResxKeyFinder.cs index dda7c72..4f885a4 100644 --- a/src/Milvaion.Api/AppStartup/MissingResxKeyFinder.cs +++ b/src/Milvaion.Api/AppStartup/MissingResxKeyFinder.cs @@ -1,8 +1,5 @@ using Bogus.DataSets; using Milvaion.Application; -using Milvaion.Application.Utils.Constants; -using Milvaion.Application.Utils.PermissionManager; -using Milvaion.Domain; using Milvaion.Domain.Enums; using Milvasoft.Core.Helpers; using System.Reflection; @@ -19,18 +16,45 @@ public static partial class MissingResxKeyFinder /// /// Finds missing keys in .resx files. /// + /// + /// A development-time convenience, called before the host is built. It must never throw: anything escaping + /// here kills the entry point before an IHost exists, which surfaces as the unhelpful + /// "The entry point exited without ever building an IHost" and takes the whole application - or the whole + /// integration test suite - down with it. + /// + /// The paths are resolved from the current working directory, which is the API project only when it is run + /// directly. Under a test host it points somewhere else entirely, so the folders legitimately do not exist. + /// + /// public static void FindAndPrintToConsole() { #if !DEBUG return; #endif - string projectFolderPath = Directory.GetParent(Directory.GetCurrentDirectory()).FullName; + string projectFolderPath = Directory.GetParent(Directory.GetCurrentDirectory())?.FullName; string resxFolderProjectPath = Directory.GetCurrentDirectory(); string resxFolderPath = Path.Combine(resxFolderProjectPath, "LocalizationResources", "Resources"); - if (string.IsNullOrWhiteSpace(projectFolderPath) || string.IsNullOrWhiteSpace(resxFolderPath)) + // Existence, not just non-emptiness: the reader throws DirectoryNotFoundException on a missing folder. + if (string.IsNullOrWhiteSpace(projectFolderPath) || !Directory.Exists(projectFolderPath) || !Directory.Exists(resxFolderPath)) return; + try + { + RunCheck(projectFolderPath, resxFolderPath); + } + catch (Exception ex) + { + // Report and carry on. A broken resx check is never a reason to stop the application starting. + Console.ForegroundColor = ConsoleColor.Yellow; + Console.WriteLine($"Resx key check skipped: {ex.Message}"); + Console.ResetColor(); + } + } + + private static void RunCheck(string projectFolderPath, string resxFolderPath) + { + var nameofReferences = FindNameofReferencesInLocalizer(projectFolderPath); HashSet keysInNameOfReferences = []; diff --git a/src/Milvaion.Api/AppStartup/Program.cs b/src/Milvaion.Api/AppStartup/Program.cs index dcc09cb..03d86c8 100644 --- a/src/Milvaion.Api/AppStartup/Program.cs +++ b/src/Milvaion.Api/AppStartup/Program.cs @@ -1,13 +1,11 @@ -using Milvaion.Api; +using Microsoft.Extensions.Options; +using Milvaion.Api; using Milvaion.Api.AppStartup; using Milvaion.Api.Hubs; using Milvaion.Api.Middlewares; using Milvaion.Api.Migrations; using Milvaion.Application; -using Milvaion.Application.Utils.Constants; using Milvaion.Application.Utils.LinkedWithFormatters; -using Milvaion.Application.Utils.Models.Options; -using Milvaion.Domain; using Milvaion.Infrastructure; using Milvaion.Infrastructure.Services.RabbitMQ; using Milvasoft.Components.Rest; @@ -17,6 +15,8 @@ try { + MissingResxKeyFinder.FindAndPrintToConsole(); + var builder = WebApplication.CreateBuilder(new WebApplicationOptions { WebRootPath = GlobalConstant.WWWRoot @@ -31,9 +31,9 @@ // Add services to the container. var services = builder.Services; - var fineConfig = builder.Configuration.GetSection(nameof(MilvaionConfig)).Get(); + var config = builder.Configuration.GetSection(nameof(MilvaionConfig)).Get(); - services.AddSingleton(fineConfig); + services.AddSingleton(config); services.AddControllers().AddApplicationPart(PresentationAssembly.Assembly); @@ -45,6 +45,8 @@ services.AddAuthorization(builder.Configuration); + services.AddMilvaionMcp(); + services.AddHttpContextAccessor(); services.AddMultiLanguageSupport(builder.Configuration); @@ -82,6 +84,19 @@ // Configure the HTTP request pipeline. var app = builder.Build(); + // PathBase support: when BasePath is configured (e.g. "/milvaion"), all middleware and endpoints (controllers, hubs, static files, SPA fallback) are automatically scoped to it. + // UseRouting() MUST be called explicitly right after UsePathBase so that route matching happens on the already-stripped path, not the full original path. + if (!string.IsNullOrWhiteSpace(config?.BasePath)) + { + app.UsePathBase(config.BasePath); + app.UseRouting(); + } + + var effectiveBasePath = string.IsNullOrWhiteSpace(config?.BasePath) ? "/" : config.BasePath; + + if (app.Logger.IsEnabled(LogLevel.Information)) + app.Logger.LogInformation("Milvaion starting on base path: {BasePath}", effectiveBasePath); + app.UseCorsFromConfiguration(builder.Configuration); #region Configure @@ -92,7 +107,7 @@ // Prometheus metrics endpoint - /api/metrics app.UsePrometheusMetrics(builder.Configuration); - // Serve static files (wwwroot) - Use Microsoft.AspNetCore.Builder extension directly + // Serve static files (wwwroot) StaticFileExtensions.UseStaticFiles(app, new StaticFileOptions { OnPrepareResponse = ctx => @@ -118,20 +133,39 @@ // Map SignalR hub app.MapHub("/hubs/jobs"); + // Must come before MapFallbackToFile, otherwise the SPA fallback would swallow /mcp. + app.MapMilvaionMcp(); + app.UseScalarWithOpenApi(); // SPA Fallback - Serve React app for all non-API routes // Must be LAST (after MapControllers, MapHub, etc.) app.MapFallbackToFile("index.html"); + // Runtime config endpoint — serves base path and other boot-time settings to the SPA. + // Must be reachable BEFORE authentication so the browser can load it as a plain diff --git a/src/MilvaionUI/package.json b/src/MilvaionUI/package.json index d821392..1797a93 100644 --- a/src/MilvaionUI/package.json +++ b/src/MilvaionUI/package.json @@ -11,7 +11,6 @@ }, "dependencies": { "@microsoft/signalr": "^8.0.0", - "@xyflow/react": "^12.10.1", "axios": "^1.6.7", "cronstrue": "^2.49.0", "moment": "^2.30.1", diff --git a/src/MilvaionUI/src/App.jsx b/src/MilvaionUI/src/App.jsx index 83acdfa..8c3a31b 100644 --- a/src/MilvaionUI/src/App.jsx +++ b/src/MilvaionUI/src/App.jsx @@ -1,45 +1,78 @@ +import { lazy, Suspense } from 'react' +import { SkeletonCard } from './components/Skeleton' import { BrowserRouter as Router, Routes, Route, Navigate } from 'react-router-dom' import { ThemeProvider } from './contexts/ThemeContext' import Layout from './components/Layout' import ProtectedRoute from './components/ProtectedRoute' import Login from './pages/Login/Login' -import Dashboard from './pages/Dashboard' -import JobList from './pages/Jobs/JobList' -import JobDetail from './pages/Jobs/JobDetail' -import JobForm from './pages/Jobs/JobForm' -import OccurrenceDetail from './pages/Occurrences/OccurrenceDetail' -import WorkerList from './pages/Workers/WorkerList' -import ExecutionList from './pages/Executions/ExecutionList' -import Tags from './pages/Tags' -import AdminDashboard from './pages/Admin/AdminDashboard' -import Configuration from './pages/Configuration' -import FailedOccurrenceList from './pages/FailedOccurrences/FailedOccurrenceList' -import FailedOccurrenceDetail from './pages/FailedOccurrences/FailedOccurrenceDetail' -import UserList from './pages/UserManagement/UserList' -import RoleList from './pages/UserManagement/RoleList' -import ActivityLogList from './pages/UserManagement/ActivityLogList' -import Profile from './pages/Profile/Profile' -import WorkflowList from './pages/Workflows/WorkflowList' -import WorkflowDetail from './pages/Workflows/WorkflowDetail' -import WorkflowForm from './pages/Workflows/WorkflowForm' -import WorkflowRunDetail from './pages/Workflows/WorkflowRunDetail' -import WorkflowBuilder from './pages/Workflows/WorkflowBuilder/WorkflowBuilder' -import ReportDashboard from './pages/Reports/ReportDashboard' -import FailureRateTrendReport from './pages/Reports/FailureRateTrendReport' -import PercentileDurationsReport from './pages/Reports/PercentileDurationsReport' -import TopSlowJobsReport from './pages/Reports/TopSlowJobsReport' -import WorkerThroughputReport from './pages/Reports/WorkerThroughputReport' -import WorkerUtilizationTrendReport from './pages/Reports/WorkerUtilizationTrendReport' -import CronScheduleVsActualReport from './pages/Reports/CronScheduleVsActualReport' -import JobHealthScoreReport from './pages/Reports/JobHealthScoreReport' -import WorkflowSuccessRateReport from './pages/Reports/WorkflowSuccessRateReport' -import WorkflowStepBottleneckReport from './pages/Reports/WorkflowStepBottleneckReport' -import WorkflowDurationTrendReport from './pages/Reports/WorkflowDurationTrendReport' + +/* + * Pages are loaded on demand. Imported statically they all landed in one bundle, + * and the heavy dependencies came with them - opening the dashboard downloaded + * the workflow canvas (reactflow) and the report charts (recharts) too. + * + * Login and Layout stay eager: both are needed for the first paint, so deferring + * them would only lengthen the blank screen. + */ +const ActivityLogList = lazy(() => import('./pages/UserManagement/ActivityLogList')) +/* `/admin` now serves the redesigned Monitoring page. `AdminDashboard` is left in the + tree so switching back is this one line. */ +const AdminDashboard = lazy(() => import('./pages/Admin/Monitoring')) +const ApiKeyList = lazy(() => import('./pages/UserManagement/ApiKeyList')) +/* The redesigned page lives in its own folder alongside the spec that drives it. The old + `pages/Configuration.jsx` is untouched; switching back is this one line. */ +const Configuration = lazy(() => import('./pages/Configuration/Configuration')) +const CronScheduleVsActualReport = lazy(() => import('./pages/Reports/CronScheduleVsActualReport')) +/* Redesigned dashboard. The old `pages/Dashboard.jsx` is untouched; this is the switch. */ +const Dashboard = lazy(() => import('./pages/Dashboard/Dashboard')) +const ExecutionList = lazy(() => import('./pages/Executions/ExecutionList')) +const FailedOccurrenceDetail = lazy(() => import('./pages/FailedOccurrences/FailedOccurrenceDetail')) +const FailedOccurrenceList = lazy(() => import('./pages/FailedOccurrences/FailedOccurrenceList')) +const FailureRateTrendReport = lazy(() => import('./pages/Reports/FailureRateTrendReport')) +const JobDetail = lazy(() => import('./pages/Jobs/JobDetail')) +const JobForm = lazy(() => import('./pages/Jobs/JobForm')) +const JobHealthScoreReport = lazy(() => import('./pages/Reports/JobHealthScoreReport')) +const JobList = lazy(() => import('./pages/Jobs/JobList')) +const OccurrenceDetail = lazy(() => import('./pages/Occurrences/OccurrenceDetail')) +const PercentileDurationsReport = lazy(() => import('./pages/Reports/PercentileDurationsReport')) +const Profile = lazy(() => import('./pages/Profile/Profile')) +const ReportDashboard = lazy(() => import('./pages/Reports/ReportDashboard')) +const RoleList = lazy(() => import('./pages/UserManagement/RoleList')) +const Tags = lazy(() => import('./pages/Tags')) +const TopSlowJobsReport = lazy(() => import('./pages/Reports/TopSlowJobsReport')) +const UpcomingExecutions = lazy(() => import('./pages/UpcomingExecutions/UpcomingExecutions')) +const UserList = lazy(() => import('./pages/UserManagement/UserList')) +const WorkerList = lazy(() => import('./pages/Workers/WorkerList')) +const WorkerThroughputReport = lazy(() => import('./pages/Reports/WorkerThroughputReport')) +const WorkerUtilizationTrendReport = lazy(() => import('./pages/Reports/WorkerUtilizationTrendReport')) +const WorkflowBuilder = lazy(() => import('./pages/Workflows/WorkflowBuilder/WorkflowBuilder')) +const WorkflowDetail = lazy(() => import('./pages/Workflows/WorkflowDetail')) +const WorkflowDurationTrendReport = lazy(() => import('./pages/Reports/WorkflowDurationTrendReport')) +const WorkflowForm = lazy(() => import('./pages/Workflows/WorkflowForm')) +const WorkflowList = lazy(() => import('./pages/Workflows/WorkflowList')) +const WorkflowRunDetail = lazy(() => import('./pages/Workflows/WorkflowRunDetail')) +const WorkflowStepBottleneckReport = lazy(() => import('./pages/Reports/WorkflowStepBottleneckReport')) +const WorkflowSuccessRateReport = lazy(() => import('./pages/Reports/WorkflowSuccessRateReport')) + +/** + * Shown while a page chunk is downloading. + * + * Invisible on a local network; on a slow connection it is the difference between + * a blank screen and something that looks like it is working. + */ +function PageFallback() { + return ( +
+ +
+ ) +} function App() { + const basename = (window.__MILVAION_CONFIG__?.basePath ?? import.meta.env.VITE_BASE_PATH ?? '/') || '/' return ( - + } /> @@ -48,46 +81,50 @@ function App() { element={ - - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> + }> + + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> - } /> - + } /> + + } diff --git a/src/MilvaionUI/src/components/AuditInfoCard.css b/src/MilvaionUI/src/components/AuditInfoCard.css index 120c0a9..5dead49 100644 --- a/src/MilvaionUI/src/components/AuditInfoCard.css +++ b/src/MilvaionUI/src/components/AuditInfoCard.css @@ -23,7 +23,7 @@ } .audit-info-btn:hover { - color: var(--accent-color); + color: var(--accent-text); border-color: var(--accent-color); background-color: var(--bg-hover); } diff --git a/src/MilvaionUI/src/components/CollapsibleSection.css b/src/MilvaionUI/src/components/CollapsibleSection.css new file mode 100644 index 0000000..b05b67d --- /dev/null +++ b/src/MilvaionUI/src/components/CollapsibleSection.css @@ -0,0 +1,139 @@ +/* Başlığından katlanabilen bölüm. */ + +.collapsible-section { + /* --transition-base "180ms ease" - süre ve yumuşatma birlikte. Geçiş + gecikmesi olarak yalnızca süre gerekiyor ve oraya "180ms ease" + yazılamıyor, o yüzden süre ayrıca duruyor. İkisi birlikte değişmeli. */ + --collapsible-duration: 180ms; +} + +.collapsible-head { + display: flex; + align-items: center; + gap: var(--space-3); + cursor: pointer; + user-select: none; +} + +.collapsible-head h2 { + display: flex; + align-items: center; + gap: var(--space-2); + flex: 1; + min-width: 0; + margin: 0; +} + +.collapsible-head:hover h2 { + color: var(--accent-color); +} + +/* Eylem düğmeleri imleci geri normale çeviriyor: çubuğun geri kalanı katlama + yüzeyi ama bunlar kendi işlerini yapıyor. */ +.collapsible-actions { + display: flex; + align-items: center; + gap: var(--space-2); + cursor: default; +} + +.collapsible-chevron { + display: inline-flex; + align-items: center; + justify-content: center; + width: 30px; + height: 30px; + flex-shrink: 0; + border: none; + border-radius: var(--radius-sm); + background: transparent; + color: var(--text-secondary); + cursor: pointer; + transition: background var(--transition-fast), color var(--transition-fast); +} + +/* + * Chevron'un kendisi. Artık döndürülmüyor - açık ve kapalı için iki ayrı glif var, + * gerekçesi bileşenin içinde yazılı. + * + * Buradaki eski kural `.collapsible-chevron svg` idi ve hiç eşleşmiyordu, çünkü `Icon` + * bileşeni `` değil `` üretiyor. Yani dönme zaten hiçbir zaman çalışmamıştı; + * bölüm açılıp kapanırken ok sabit kalıyordu. Seçiciyi düzeltmek dönmeyi devreye soktu ve + * asıl sorunu görünür hale getirdi: kare olmayan bir kutuyu çevirmek glifi yatayda + * kaydırıyor. + */ +.collapsible-chevron .icon { + line-height: 1; +} + +.collapsible-head:hover .collapsible-chevron { + background: var(--bg-hover); + color: var(--accent-color); +} + +.collapsible-chevron:focus-visible { + outline: 2px solid var(--accent-color); + outline-offset: 2px; +} + +/* ── Açılma kapanma ────────────────────────────────────────────────────────── + İçerik yüksekliği önceden bilinmiyor - tablo kaç satır, DAG kaç düğüm belli + değil - o yüzden sabit bir max-height veremiyoruz; çok küçük seçilirse içerik + kesiliyor, çok büyük seçilirse animasyon kısa içerikte anlamsız hızlanıyor. + + grid-template-rows 0fr → 1fr geçişi bu sorunu çözüyor: tarayıcı gerçek + yüksekliği kendisi hesaplıyor ve arada animasyon yapıyor. */ + +.collapsible-outer { + display: grid; + grid-template-rows: 0fr; + + /* Kapalıyken sekme sırasından ve ekran okuyucudan çıkıyor; aria-hidden tek + başına klavye erişimini kapatmıyor. + + visibility ara değer almayan bir özellik, yani anında değişiyor. Geçiş + tanımlanmazsa kapanırken içerik kutu daralmaya başlamadan yok oluyor ve + geriye boş bir kutunun kapanışı kalıyor. Kapanışta gecikme süre kadar, + açılışta sıfır: içerik hemen görünüyor, kapanırken sona kadar duruyor. */ + visibility: hidden; + transition: grid-template-rows var(--transition-base), + visibility 0s linear var(--collapsible-duration); +} + +.collapsible-section.is-open .collapsible-outer { + grid-template-rows: 1fr; + visibility: visible; + transition: grid-template-rows var(--transition-base), + visibility 0s linear 0s; +} + +/* Grid alanı sıfıra inerken içeriğin taşmaması için. */ +.collapsible-inner { + overflow: hidden; + min-height: 0; +} + +.collapsible-body { + margin-top: var(--space-4); + /* İçerik de solup görünüyor; yalnızca yükseklik değişimi, kapanırken metnin + sıkışarak yok olması gibi görünüyordu. */ + opacity: 0; + transition: opacity var(--transition-fast); +} + +.collapsible-section.is-open .collapsible-body { + opacity: 1; + transition-delay: 60ms; +} + +@media (prefers-reduced-motion: reduce) { + /* Gecikme de sıfırlanıyor: animasyon yokken içeriğin kapandıktan sonra bir + süre daha görünür kalması için sebep kalmıyor. */ + /* Chevron artık geçişsiz - glif değişiyor, dönmüyor - o yüzden burada yalnızca + katlanan gövde kalıyor. */ + .collapsible-outer, + .collapsible-section.is-open .collapsible-outer, + .collapsible-body { + transition: none; + } +} diff --git a/src/MilvaionUI/src/components/CollapsibleSection.jsx b/src/MilvaionUI/src/components/CollapsibleSection.jsx new file mode 100644 index 0000000..ed3e71a --- /dev/null +++ b/src/MilvaionUI/src/components/CollapsibleSection.jsx @@ -0,0 +1,84 @@ +import Icon from './Icon' +import { useLocalStorageState } from '../hooks/useLocalStorageState' +import './CollapsibleSection.css' + +/* eslint-disable react/prop-types */ + +/** + * Başlığından katlanabilen bölüm. + * + * Açık/kapalı tercihi storageKey altında localStorage'da tutuluyor, yani her + * bölüm kendi durumunu ayrı hatırlıyor. Sayfayı her açtığında aynı bölümü kapatmak + * zorunda kalmak, katlamayı işe yaramaz hale getiriyor. + * + * @param {string} storageKey Tercihin saklanacağı anahtar. + * @param {React.ReactNode} title Başlık içeriği. + * @param {string} icon Başlığın solundaki ikon adı. + * @param {React.ReactNode} actions Başlık çubuğundaki düğmeler. Tıklamaları katlamayı tetiklemiyor. + * @param {boolean} defaultOpen Kayıtlı tercih yokken açık mı gelsin. + */ +function CollapsibleSection({ + storageKey, + title, + icon, + actions = null, + defaultOpen = true, + className = '', + children, +}) { + const [open, setOpen] = useLocalStorageState(storageKey, defaultOpen) + + const toggle = () => setOpen(o => !o) + + return ( +
+ {/* Çubuğun tamamı tıklanabilir. Klavye erişimi sağdaki düğmeden geliyor; + çubuğu da düğme yapmak, içindeki eylem düğmelerini geçersiz hale getirirdi. */} +
+

+ {icon && } + {title} +

+ + {actions && ( +
e.stopPropagation()}> + {actions} +
+ )} + + +
+ + {/* Kapalıyken de DOM'da duruyor: yüksekliği animasyonla değiştirmek için + tarayıcının içeriği ölçebilmesi gerekiyor. Görünürlük CSS'te kapatılıyor, + böylece kapalı bölüm sekmeyle gezilemiyor ve ekran okuyucuya okunmuyor. */} +
+
+
{children}
+
+
+
+ ) +} + +export default CollapsibleSection diff --git a/src/MilvaionUI/src/components/DatabaseStatistics/DatabaseStatistics.css b/src/MilvaionUI/src/components/DatabaseStatistics/DatabaseStatistics.css index 86f04ab..6004c5c 100644 --- a/src/MilvaionUI/src/components/DatabaseStatistics/DatabaseStatistics.css +++ b/src/MilvaionUI/src/components/DatabaseStatistics/DatabaseStatistics.css @@ -37,9 +37,9 @@ } .db-stats-card .refresh-btn-small:hover { - background: rgba(99, 102, 241, 0.1); - border-color: #6366f1; - color: #6366f1; + background: var(--accent-glow); + border-color: var(--accent-color); + color: var(--accent-text); } .db-stats-card .card-content { @@ -60,11 +60,10 @@ /* Summary Section */ .db-stat-summary { - margin-bottom: 2rem; padding: 1.5rem; - background: rgba(99, 102, 241, 0.1); + background: var(--accent-glow); border-radius: 8px; - border: 1px solid rgba(99, 102, 241, 0.2); + border: 1px solid rgba(var(--accent-color-rgb), 0.2); } .stat-item-large { @@ -74,7 +73,7 @@ } .stat-icon-large { - color: #6366f1; + color: var(--accent-text); } .stat-value-large { @@ -148,7 +147,7 @@ .progress-fill { height: 100%; - background: linear-gradient(90deg, #6366f1 0%, #747bff 100%); + background: var(--accent-color); border-radius: 3px; transition: width 0.3s ease; } @@ -387,8 +386,8 @@ align-items: flex-start; gap: 0.75rem; padding: 0.75rem; - background: rgba(99, 102, 241, 0.1); - border: 1px solid rgba(99, 102, 241, 0.2); + background: var(--accent-glow); + border: 1px solid rgba(var(--accent-color-rgb), 0.2); border-radius: 6px; font-size: 0.875rem; color: var(--text-secondary); diff --git a/src/MilvaionUI/src/components/DatabaseStatistics/DatabaseStatistics.jsx b/src/MilvaionUI/src/components/DatabaseStatistics/DatabaseStatistics.jsx index 221a0f0..d8ea4b8 100644 --- a/src/MilvaionUI/src/components/DatabaseStatistics/DatabaseStatistics.jsx +++ b/src/MilvaionUI/src/components/DatabaseStatistics/DatabaseStatistics.jsx @@ -9,14 +9,14 @@ const [indexEfficiency, setIndexEfficiency] = useState(null) const [cacheHitRatio, setCacheHitRatio] = useState(null) const [tableBloat, setTableBloat] = useState(null) const [totalDbSize, setTotalDbSize] = useState({ bytes: 0, formatted: 'N/A' }) - + const [loading, setLoading] = useState({ tables: true, indexes: true, cache: true, bloat: true }) - + const [errors, setErrors] = useState({ tables: null, indexes: null, @@ -42,7 +42,7 @@ const loadTableSizes = async () => { const response = await api.get('/admin/database-statistics/tables') const data = response?.data || response setTableSizes(data) - + // Calculate total size if (data && data.length > 0) { const totalBytes = data.reduce((sum, t) => sum + t.sizeBytes, 0) @@ -128,14 +128,8 @@ const loadTableBloat = async () => { if (isLoading) { return ( -
-
-

- - Database Statistics -

-
-
+
+
Loading...
@@ -143,16 +137,7 @@ const loadTableBloat = async () => { } return ( -
-
-

- - Database Statistics -

- -
+
{/* Total Database Size */}
diff --git a/src/MilvaionUI/src/components/JobSelect.css b/src/MilvaionUI/src/components/JobSelect.css new file mode 100644 index 0000000..2b52706 --- /dev/null +++ b/src/MilvaionUI/src/components/JobSelect.css @@ -0,0 +1,154 @@ +/* + * Server-backed job picker. + * + * Prefixed throughout: this renders inside the workflow form and the builder's config + * panel, two places with their own dropdown and option styling, and stylesheets here load + * globally. + */ + +.mv-jobselect { + position: relative; + width: 100%; +} + +.mv-jobselect-control { + display: flex; + align-items: center; + gap: 0.5rem; + width: 100%; + padding: 0.55rem 0.7rem; + background-color: var(--bg-card); + border: 1px solid var(--border-color); + border-radius: var(--radius-md); + color: var(--text-primary); + font-size: 0.925rem; + text-align: left; + cursor: pointer; + transition: border-color var(--transition-base); +} + +.mv-jobselect-control:hover:not(:disabled), +.mv-jobselect-control:focus-visible { + border-color: var(--accent-color); + outline: none; +} + +.mv-jobselect-control:disabled { + opacity: 0.5; + cursor: not-allowed; +} + +/* Nothing chosen yet: the text is a prompt, not a value. */ +.mv-jobselect-control.is-empty .mv-jobselect-label { + color: var(--text-muted); +} + +.mv-jobselect-label { + flex: 1; + min-width: 0; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.mv-jobselect-clear { + display: inline-flex; + align-items: center; + color: var(--text-muted); + border-radius: var(--radius-sm); +} + +.mv-jobselect-clear:hover { + color: var(--error-color); +} + +/* ── Menu ────────────────────────────────────────────────────────────────── */ + +.mv-jobselect-menu { + position: absolute; + top: calc(100% + 4px); + left: 0; + right: 0; + z-index: 50; + background-color: var(--bg-card); + border: 1px solid var(--border-color); + border-radius: var(--radius-md); + box-shadow: var(--shadow-md); + overflow: hidden; +} + +.mv-jobselect-search { + display: flex; + align-items: center; + gap: 0.5rem; + padding: 0.55rem 0.7rem; + border-bottom: 1px solid var(--border-color); +} + +.mv-jobselect-search > span:first-child { + color: var(--text-muted); + flex-shrink: 0; +} + +.mv-jobselect-search input { + flex: 1; + min-width: 0; + border: none; + outline: none; + background: transparent; + color: var(--text-primary); + font-size: 0.9rem; +} + +.mv-jobselect-results { + max-height: 280px; + overflow-y: auto; +} + +.mv-jobselect-option { + display: flex; + flex-direction: column; + gap: 2px; + width: 100%; + padding: 0.5rem 0.7rem; + background: transparent; + border: none; + border-bottom: 1px solid var(--border-color); + color: var(--text-primary); + text-align: left; + cursor: pointer; +} + +.mv-jobselect-option:last-child { + border-bottom: none; +} + +.mv-jobselect-option:hover { + background-color: var(--bg-hover); +} + +.mv-jobselect-option.is-selected { + background-color: var(--accent-glow); +} + +.mv-jobselect-option-name { + font-size: 0.9rem; + font-weight: 500; +} + +.mv-jobselect-option-meta { + font-size: 0.75rem; + color: var(--text-muted); +} + +.mv-jobselect-hint { + padding: 0.7rem; + font-size: 0.825rem; + color: var(--text-muted); + text-align: center; +} + +/* Görünmez tetikleyici: açılır listenin sonuna yaklaşınca sonraki sayfa isteniyor. */ +.mv-jobselect-sentinel { + height: 1px; +} diff --git a/src/MilvaionUI/src/components/JobSelect.jsx b/src/MilvaionUI/src/components/JobSelect.jsx new file mode 100644 index 0000000..1bbbefa --- /dev/null +++ b/src/MilvaionUI/src/components/JobSelect.jsx @@ -0,0 +1,246 @@ +import { useState, useEffect, useRef, useCallback } from 'react' +import Icon from './Icon' +import jobService from '../services/jobService' +import useInfiniteScroll from '../hooks/useInfiniteScroll' +import './JobSelect.css' + +/* eslint-disable react/prop-types */ + +/** How many jobs one page of the dropdown holds. */ +const _pageSize = 50 + +/** + * Picks one job, searching on the server. + * + * The workflow screens used to render a plain ` setSearch(e.target.value)} + placeholder="Search jobs..." + autoFocus + /> +
+ +
+ {loading &&
Searching…
} + + {!loading && results.length === 0 && ( +
+ {debouncedSearch ? 'No jobs match that search.' : 'No jobs found.'} +
+ )} + + {!loading && results.map(job => ( + + ))} + + {/* Sentinel plus status. Without the closing line someone scrolling a long + list has no way to tell "that is all of them" from "still loading". */} + {!loading && hasMore &&
} + + {loadingMore &&
Loading more…
} + + {!loading && !hasMore && results.length > 0 && ( +
End of list
+ )} +
+
+ )} +
+ ) +} + +export default JobSelect diff --git a/src/MilvaionUI/src/components/JsonEditor.css b/src/MilvaionUI/src/components/JsonEditor.css index 8d79578..9d6d778 100644 --- a/src/MilvaionUI/src/components/JsonEditor.css +++ b/src/MilvaionUI/src/components/JsonEditor.css @@ -1,165 +1,31 @@ -/* JSON Editor Component Styles */ -.json-editor { +/* + * The form wrapper around a JSON field. + * + * The control itself - toolbar, tree, textarea, validity - is `JsonView`, styled in + * `JsonView.css`. What is left here is the label and the hint, which belong to the form + * rather than to the editor. + */ + +.json-editor-field { display: flex; flex-direction: column; - gap: 0.5rem; + gap: 0.4rem; + text-align: left; } .json-editor-label { + font-size: 0.875rem; font-weight: 500; - color: var(--text-primary); - font-size: 0.9rem; -} - -.json-editor-label .required { - color: #ef4444; - margin-left: 0.25rem; -} - -.json-editor-container { - display: flex; - flex-direction: column; - border: 1px solid var(--border-color); - border-radius: 8px; - overflow: hidden; - transition: border-color 0.2s; -} - -.json-editor.has-error .json-editor-container { - border-color: #ef4444; -} - -.json-editor-container:focus-within { - border-color: #6366f1; - box-shadow: 0 0 0 3px rgba(99, 102, 241, 0.1); -} - -.json-editor.has-error .json-editor-container:focus-within { - border-color: #ef4444; - box-shadow: 0 0 0 3px rgba(239, 68, 68, 0.1); -} - -/* Toolbar */ -.json-editor-toolbar { - display: flex; - justify-content: space-between; - align-items: center; - padding: 0.5rem 0.75rem; - background: var(--bg-secondary); - border-bottom: 1px solid var(--border-color); - gap: 0.5rem; - flex-wrap: wrap; -} - -.toolbar-left, -.toolbar-right { - display: flex; - align-items: center; - gap: 0.5rem; -} - -.toolbar-btn { - display: inline-flex; - align-items: center; - gap: 0.375rem; - padding: 0.375rem 0.625rem; - background: var(--bg-tertiary); - border: 1px solid var(--border-color); - border-radius: 6px; color: var(--text-secondary); - font-size: 0.75rem; - font-weight: 500; - cursor: pointer; - transition: all 0.2s; } -.toolbar-btn:hover:not(:disabled) { - background: var(--bg-hover); - border-color: var(--accent-color); - color: var(--accent-color); -} - -.toolbar-btn:disabled { - opacity: 0.5; - cursor: not-allowed; -} - -.toolbar-btn span { - display: none; -} - -@media (min-width: 480px) { - .toolbar-btn span { - display: inline; - } -} - -/* Validation Badge */ -.validation-badge { - display: inline-flex; - align-items: center; - gap: 0.25rem; - padding: 0.25rem 0.5rem; - border-radius: 4px; - font-size: 0.7rem; - font-weight: 600; - text-transform: uppercase; - letter-spacing: 0.5px; -} - -.validation-badge.valid { - background: rgba(16, 185, 129, 0.12); - color: #10b981; -} - -.validation-badge.invalid { - background: rgba(244, 67, 54, 0.15); - color: #ef4444; -} - -/* Textarea */ -.json-editor-textarea { - width: 100%; - padding: 0.875rem; - font-family: 'Consolas', 'Monaco', 'Courier New', monospace; - font-size: 0.875rem; - line-height: 1.5; - background: var(--bg-primary); - border: none; - color: var(--text-primary); - resize: vertical; - min-height: 150px; - outline: none; -} - -.json-editor-textarea::placeholder { - color: var(--text-muted); -} - -.json-editor-textarea.invalid { - background: rgba(244, 67, 54, 0.03); -} - -/* Error Message */ -.json-editor-error { - display: flex; - align-items: flex-start; - gap: 0.375rem; - padding: 0.5rem 0.75rem; - background: rgba(239, 68, 68, 0.1); - border-top: 1px solid rgba(239, 68, 68, 0.2); - color: #ef4444; - font-size: 0.75rem; - font-family: 'Consolas', 'Monaco', 'Courier New', monospace; -} - -.json-editor-error span { - flex: 1; - word-break: break-word; +.json-editor-label .required { + margin-left: 2px; + color: var(--error-color); } -/* Hint */ .json-editor-hint { + font-size: 0.775rem; + line-height: 1.5; color: var(--text-muted); - font-size: 0.8rem; } diff --git a/src/MilvaionUI/src/components/JsonEditor.jsx b/src/MilvaionUI/src/components/JsonEditor.jsx index 4a384c5..f82e947 100644 --- a/src/MilvaionUI/src/components/JsonEditor.jsx +++ b/src/MilvaionUI/src/components/JsonEditor.jsx @@ -1,211 +1,51 @@ -import { useState, useEffect, useCallback } from 'react' -import Icon from './Icon' +import JsonView from './JsonView' import './JsonEditor.css' +/* eslint-disable react/prop-types */ + /** - * JSON Editor component with beautify and validation features. - * - * @param {Object} props - * @param {string} props.value - JSON string value - * @param {Function} props.onChange - Callback when value changes (receives event-like object or string) - * @param {string} props.name - Input name for form handling - * @param {string} props.placeholder - Placeholder text - * @param {number} props.rows - Number of textarea rows - * @param {boolean} props.required - Whether field is required - * @param {string} props.label - Optional label text - * @param {string} props.hint - Optional hint text below input + * A labelled JSON field. + * + * The editing behaviour now lives in `JsonView`, which is also what every read-only JSON + * surface in the application uses. This is the form wrapper around it: label, required + * marker, hint. + * + * The props are unchanged from the previous implementation - `value`, an `onChange` shaped + * like a DOM change event, `name`, `rows`, `placeholder`, `required`, `label`, `hint` - so + * the three screens that use it did not have to be touched. What they gain, without asking, + * is everything the read-only viewer had and the old editor did not: a tree view of what + * they are typing, search across it, and per-field copy. What the read-only screens gain in + * return is the validity reporting that used to exist only here. */ -function JsonEditor({ - value = '', - onChange, - name = 'json', +function JsonEditor({ + value = '', + onChange, + name = 'json', placeholder = '{"key": "value"}', rows = 8, required = false, label, - hint + hint, }) { - const [error, setError] = useState(null) - const [isValid, setIsValid] = useState(true) - - // Validate JSON on value change - const validateJson = useCallback((jsonString) => { - if (!jsonString || jsonString.trim() === '') { - setError(null) - setIsValid(true) - return true - } - - try { - JSON.parse(jsonString) - setError(null) - setIsValid(true) - return true - } catch (e) { - setError(e.message) - setIsValid(false) - return false - } - }, []) - - // Validate on mount and value change - useEffect(() => { - validateJson(value) - }, [value, validateJson]) - - // Handle input change - const handleChange = (e) => { - const newValue = e.target.value - validateJson(newValue) - - // Call onChange with event-like object for compatibility - if (onChange) { - onChange({ - target: { - name: name, - value: newValue - } - }) - } - } - - // Beautify JSON - const handleBeautify = () => { - if (!value || value.trim() === '') return - - try { - const parsed = JSON.parse(value) - const beautified = JSON.stringify(parsed, null, 2) - - if (onChange) { - onChange({ - target: { - name: name, - value: beautified - } - }) - } - setError(null) - setIsValid(true) - } catch (e) { - setError(e.message) - setIsValid(false) - } - } - - // Minify JSON - const handleMinify = () => { - if (!value || value.trim() === '') return - - try { - const parsed = JSON.parse(value) - const minified = JSON.stringify(parsed) - - if (onChange) { - onChange({ - target: { - name: name, - value: minified - } - }) - } - setError(null) - setIsValid(true) - } catch (e) { - setError(e.message) - setIsValid(false) - } - } - - // Copy to clipboard - const handleCopy = async () => { - if (!value) return - - try { - await navigator.clipboard.writeText(value) - } catch (e) { - console.error('Failed to copy:', e) - } - } - return ( -
+
{label && ( -