feature/auth#27
Merged
Merged
Conversation
- AssetCreate/AssetUpdate: enrich_meta adds created_via/updated_via from context['via']
- Validate via in {server,mcp,api}, default to 'api' when not provided
- Preserve @meta rejection behavior
- EvaluationCreate: add created_via with same validation/default
- Index templates: add @meta.created_via and @meta.updated_via (keyword) where @meta exists
- Judgements search: rated-ai/rated-human use @meta.updated_via (mcp vs not mcp) instead of updated_by
- Judgements set: script and upsert include created_via/updated_via
- Flask: pass via='server' to all create/update/set API calls
- FastMCP: pass via='mcp' to all create/update/set API calls
- Setup: pass via for ledger writes, add 1.2.0 migration for existing indices
- Agent: pass via='server' for conversation updates
- Unit tests: via validation/default, judgements filter, evaluation created_via
Made-with: Cursor
- src/server/tls.py: Central TLS config loader (TLS_ENABLED default true, require TLS_CERT_FILE + TLS_KEY_FILE when enabled, clear startup errors) - src/server/flask.py: Use TLS config, ssl_context tuple, run via python -m - src/server/fastmcp.py: Same TLS behavior for MCP HTTP transport - .env-reference: TLS_ENABLED/TLS_CERT_FILE/TLS_KEY_FILE docs - webpack.config.js: DEV_API_URL for optional dev TLS/proxy - Dockerfile-server, Dockerfile-server-mcp: Document cert volume mount - docker-compose.yml: Cert volume mounts, TLS_ENABLED=false for default - scripts/quickstart.sh: --generate-tls-cert for self-signed local dev - docs/docs/v1.1/reference/security.md: Native TLS docs, self-signed guide - tests/unit/test_tls.py: Unit tests for TLS config - tests/docker-compose.yml: TLS_ENABLED=false for integration tests Made-with: Cursor
- Add es_studio_endpoint() returning endpoint config without credentials
- Add es_from_credentials(api_key or username/password, optional cloud_id/url)
for constructing Elasticsearch client for studio endpoint
- Error when no credentials; prefer cloud_id when available else url
- Update API modules to accept optional es_client, fallback to es('studio') when None:
workspaces, displays, scenarios, strategies, judgements, benchmarks,
evaluations, conversations, setup
- content.py accepts optional es_client for consistency; uses content cluster
service-account by default
- Update utils.search_assets(), search_tags() to accept optional es_client
- Worker behavior unchanged (uses es('studio') by default)
- Add unit tests for es_client, es_from_credentials, es_studio_endpoint
- Add unit tests for API es_client fallback/override
- Add integration test for workspaces CRUD with explicit es_client
- Fix setup test monkeypatches for new es_client signatures
Made-with: Cursor
- Add es_studio_endpoint() returning endpoint config without credentials
- Add es_from_credentials(api_key or username/password, optional cloud_id/url)
for constructing Elasticsearch client for studio endpoint
- Error when no credentials; prefer cloud_id when available else url
- Update API modules to accept optional es_client, fallback to es('studio') when None:
workspaces, displays, scenarios, strategies, judgements, benchmarks,
evaluations, conversations, setup
- content.py accepts optional es_client for consistency; uses content cluster
service-account by default
- Update utils.search_assets(), search_tags() to accept optional es_client
- Worker behavior unchanged (uses es('studio') by default)
- Add unit tests for es_client, es_from_credentials, es_studio_endpoint
- Add unit tests for API es_client fallback/override
- Add integration test for workspaces CRUD with explicit es_client
- Fix setup test monkeypatches for new es_client signatures
Made-with: Cursor
- Add src/server/auth.py: - validate_credentials() for Basic/ApiKey against _security/_authenticate - create_session_api_key() calling POST /_security/api_key - encode_jwt() / decode_jwt() for session tokens - AUTH_ENABLED config (default true) - Update src/server/flask.py: - POST /api/auth/login, POST /api/auth/logout, GET /api/auth/session - before_request auth middleware for /api/* except login and /healthz - Decode cookie JWT, create per-request ES client via es_from_credentials - Attach g.user and g.es_client; AUTH_ENABLED=false uses service-account - Pass g.user and g.es_client to all API route handlers - Update .env-reference with AUTH_ENABLED, JWT_SECRET, SESSION_EXPIRY - Add PyJWT dependency - Add unit tests (auth module, flask middleware/routes) - Add integration tests for login/session/auth (AUTH_ENABLED=false) - Set AUTH_ENABLED=false in test docker-compose Made-with: Cursor
- Add src/server/auth.py: - validate_credentials() for Basic/ApiKey against _security/_authenticate - create_session_api_key() calling POST /_security/api_key - encode_jwt() / decode_jwt() for session tokens - AUTH_ENABLED config (default true) - Update src/server/flask.py: - POST /api/auth/login, POST /api/auth/logout, GET /api/auth/session - before_request auth middleware for /api/* except login and /healthz - Decode cookie JWT, create per-request ES client via es_from_credentials - Attach g.user and g.es_client; AUTH_ENABLED=false uses service-account - Pass g.user and g.es_client to all API route handlers - Update .env-reference with AUTH_ENABLED, JWT_SECRET, SESSION_EXPIRY - Add PyJWT dependency - Add unit tests (auth module, flask middleware/routes) - Add integration tests for login/session/auth (AUTH_ENABLED=false) - Set AUTH_ENABLED=false in test docker-compose Made-with: Cursor
- Add src/server/auth.py: - validate_credentials() for Basic/ApiKey against _security/_authenticate - create_session_api_key() calling POST /_security/api_key - encode_jwt() / decode_jwt() for session tokens - AUTH_ENABLED config (default true) - Update src/server/flask.py: - POST /api/auth/login, POST /api/auth/logout, GET /api/auth/session - before_request auth middleware for /api/* except login and /healthz - Decode cookie JWT, create per-request ES client via es_from_credentials - Attach g.user and g.es_client; AUTH_ENABLED=false uses service-account - Pass g.user and g.es_client to all API route handlers - Update .env-reference with AUTH_ENABLED, JWT_SECRET, SESSION_EXPIRY - Add PyJWT dependency - Add unit tests (auth module, flask middleware/routes) - Add integration tests for login/session/auth (AUTH_ENABLED=false) - Set AUTH_ENABLED=false in test docker-compose Made-with: Cursor
- Add es_studio_endpoint() returning endpoint config without credentials
- Add es_from_credentials(api_key or username/password, optional cloud_id/url)
for constructing Elasticsearch client for studio endpoint
- Error when no credentials; prefer cloud_id when available else url
- Update API modules to accept optional es_client, fallback to es('studio') when None:
workspaces, displays, scenarios, strategies, judgements, benchmarks,
evaluations, conversations, setup
- content.py accepts optional es_client for consistency; uses content cluster
service-account by default
- Update utils.search_assets(), search_tags() to accept optional es_client
- Worker behavior unchanged (uses es('studio') by default)
- Add unit tests for es_client, es_from_credentials, es_studio_endpoint
- Add unit tests for API es_client fallback/override
- Add integration test for workspaces CRUD with explicit es_client
- Fix setup test monkeypatches for new es_client signatures
Made-with: Cursor
- Add Login.js: Basic and API key modes, submit to /api/auth/login, success redirect, failure message - Add AuthContext.js: session check on load via /api/auth/session, user/isAuthenticated/login/logout, register 401 handler - Add auth401handler.js: registry for 401 handler (avoids circular imports) - Update client.js: Axios 401 interceptor clears auth and redirects to /login - Update Routes.js: AuthGuard, /login route, redirect when unauthenticated - Update App.js: wrap with AuthProvider - Update Page.js: show username and Sign out when authenticated - Update api.js: auth_login, auth_logout, auth_session - Incorporate Issue 4 client.py: es_from_credentials, es_studio_endpoint - When AUTH_ENABLED=false, session returns system user, no login required - Tests: test_login_accepts_api_key, test_auth_login_returns_system_when_disabled Made-with: Cursor
- Add mcp_auth module: parse Authorization (Basic, ApiKey, Bearer), validate via auth module, create per-request ES client via es_from_credentials - Add MCPAuthMiddleware: enforce auth per request/tool except /healthz and healthz_mcp; AUTH_ENABLED=false uses default user and singleton ES client - Wire all MCP tools with ctx, user, and es_client from context - Replace hardcoded user="ai" with authenticated username when auth enabled - Update fastmcp_proxy.py docs for credential forwarding - Update security and mcp-tools docs for MCP client credential config - Unit tests: parse header, enforcement, exemptions, disabled mode, context - Integration test: healthz exemption (skips if MCP server not running) Made-with: Cursor
- Add es_studio_endpoint() returning endpoint config without credentials
- Add es_from_credentials(api_key or username/password, optional cloud_id/url)
for constructing Elasticsearch client for studio endpoint
- Error when no credentials; prefer cloud_id when available else url
- Update API modules to accept optional es_client, fallback to es('studio') when None:
workspaces, displays, scenarios, strategies, judgements, benchmarks,
evaluations, conversations, setup
- content.py accepts optional es_client for consistency; uses content cluster
service-account by default
- Update utils.search_assets(), search_tags() to accept optional es_client
- Worker behavior unchanged (uses es('studio') by default)
- Add unit tests for es_client, es_from_credentials, es_studio_endpoint
- Add unit tests for API es_client fallback/override
- Add integration test for workspaces CRUD with explicit es_client
- Fix setup test monkeypatches for new es_client signatures
Made-with: Cursor
- AssetCreate/AssetUpdate: enrich_meta adds created_via/updated_via from context['via']
- Validate via in {server,mcp,api}, default to 'api' when not provided
- Preserve @meta rejection behavior
- EvaluationCreate: add created_via with same validation/default
- Index templates: add @meta.created_via and @meta.updated_via (keyword) where @meta exists
- Judgements search: rated-ai/rated-human use @meta.updated_via (mcp vs not mcp) instead of updated_by
- Judgements set: script and upsert include created_via/updated_via
- Flask: pass via='server' to all create/update/set API calls
- FastMCP: pass via='mcp' to all create/update/set API calls
- Setup: pass via for ledger writes, add 1.2.0 migration for existing indices
- Agent: pass via='server' for conversation updates
- Unit tests: via validation/default, judgements filter, evaluation created_via
Made-with: Cursor
- src/server/tls.py: Central TLS config loader (TLS_ENABLED default true, require TLS_CERT_FILE + TLS_KEY_FILE when enabled, clear startup errors) - src/server/flask.py: Use TLS config, ssl_context tuple, run via python -m - src/server/fastmcp.py: Same TLS behavior for MCP HTTP transport - .env-reference: TLS_ENABLED/TLS_CERT_FILE/TLS_KEY_FILE docs - webpack.config.js: DEV_API_URL for optional dev TLS/proxy - Dockerfile-server, Dockerfile-server-mcp: Document cert volume mount - docker-compose.yml: Cert volume mounts, TLS_ENABLED=false for default - scripts/quickstart.sh: --generate-tls-cert for self-signed local dev - docs/docs/v1.1/reference/security.md: Native TLS docs, self-signed guide - tests/unit/test_tls.py: Unit tests for TLS config - tests/docker-compose.yml: TLS_ENABLED=false for integration tests Made-with: Cursor
- Add src/server/auth.py: - validate_credentials() for Basic/ApiKey against _security/_authenticate - create_session_api_key() calling POST /_security/api_key - encode_jwt() / decode_jwt() for session tokens - AUTH_ENABLED config (default true) - Update src/server/flask.py: - POST /api/auth/login, POST /api/auth/logout, GET /api/auth/session - before_request auth middleware for /api/* except login and /healthz - Decode cookie JWT, create per-request ES client via es_from_credentials - Attach g.user and g.es_client; AUTH_ENABLED=false uses service-account - Pass g.user and g.es_client to all API route handlers - Update .env-reference with AUTH_ENABLED, JWT_SECRET, SESSION_EXPIRY - Add PyJWT dependency - Add unit tests (auth module, flask middleware/routes) - Add integration tests for login/session/auth (AUTH_ENABLED=false) - Set AUTH_ENABLED=false in test docker-compose Made-with: Cursor
- Add mcp_auth module: parse Authorization (Basic, ApiKey, Bearer), validate via auth module, create per-request ES client via es_from_credentials - Add MCPAuthMiddleware: enforce auth per request/tool except /healthz and healthz_mcp; AUTH_ENABLED=false uses default user and singleton ES client - Wire all MCP tools with ctx, user, and es_client from context - Replace hardcoded user="ai" with authenticated username when auth enabled - Update fastmcp_proxy.py docs for credential forwarding - Update security and mcp-tools docs for MCP client credential config - Unit tests: parse header, enforcement, exemptions, disabled mode, context - Integration test: healthz exemption (skips if MCP server not running) Made-with: Cursor
- Add Login.js: Basic and API key modes, submit to /api/auth/login, success redirect, failure message - Add AuthContext.js: session check on load via /api/auth/session, user/isAuthenticated/login/logout, register 401 handler - Add auth401handler.js: registry for 401 handler (avoids circular imports) - Update client.js: Axios 401 interceptor clears auth and redirects to /login - Update Routes.js: AuthGuard, /login route, redirect when unauthenticated - Update App.js: wrap with AuthProvider - Update Page.js: show username and Sign out when authenticated - Update api.js: auth_login, auth_logout, auth_session - Incorporate Issue 4 client.py: es_from_credentials, es_studio_endpoint - When AUTH_ENABLED=false, session returns system user, no login required - Tests: test_login_accepts_api_key, test_auth_login_returns_system_when_disabled Made-with: Cursor
- .env-reference: complete docs for AUTH_ENABLED, JWT_SECRET, SESSION_EXPIRY, TLS_ENABLED, TLS_CERT_FILE, TLS_KEY_FILE; clarified service-account roles for ELASTICSEARCH_* and CONTENT_* - docs/v1.1/reference/security.md: add auth config table, align with final auth+TLS model - Add docs: getting-started-auth, getting-started-tls, mcp-client-auth, auth-tls-migration - scripts/quickstart.sh: auth/tls setup prompts and cert generation alignment - Add unit test test_env_reference.py to verify env vars in .env-reference - Fix test_api_setup mock for _append_applied_version via param Made-with: Cursor
The setup API's _append_applied_version was extended with a via parameter for issue 2 (meta via). The unit test mock must accept this keyword arg. Made-with: Cursor
Docker Compose may return exit code 1 with 'No such container' due to a known race on macOS even when containers start successfully. Use check=False and rely on wait_for_es/wait_for_esrs to verify services. Made-with: Cursor
- Add _request_with_retry for setup/upgrade HTTP calls to handle transient connection resets (Connection reset by peer) after heavy ES operations - Add retry logic to delete_index_templates in conftest for ES connection issues - Use retries in clean_data fixture for POST /api/setup Made-with: Cursor
Add down -v --remove-orphans before up to avoid container name conflicts and 'No such container' errors. Use --force-recreate on up for reliable container creation. Add short sleep after down for Docker to settle. Made-with: Cursor
- Add es_client=None to setup.check(), run(), upgrade() to accept the parameter passed from Flask routes (fixes 500 on /api/setup) - Add pytest-asyncio>=0.24 to avoid Package object collection error - Add -p no:asyncio for integration tests in CI workflow Made-with: Cursor
- Add pytest-asyncio>=0.24 to requirements-dev.txt to fix 'Package' object has no attribute 'obj' collection error with pytest 8.x - Add _post_setup_with_retry in conftest for clean_data fixture to handle transient 'Remote end closed connection' errors when calling /api/setup Made-with: Cursor
- pytest.ini: add -p no:asyncio to fix pytest-asyncio collection error - tests: add esrs-mcp service to docker-compose for integration tests - tests/conftest: use absolute paths, cwd for docker compose; wait for MCP - tests/integration/test_mcp_auth: use services fixture, skip when MCP unavailable Made-with: Cursor
- Add pytest-asyncio>=0.24.0 for pytest 8 compatibility (fixes Package.obj) - Pre-up docker compose down with --remove-orphans and 5s sleep - Retry docker compose up up to 3 times on failure - Add requests_get_retry/requests_post_retry for API calls in setup/upgrade tests Made-with: Cursor
- test_flask_serves_https_with_valid_certs: HTTPS with self-signed certs, verify=False - test_flask_serves_http_when_tls_disabled: HTTP when TLS disabled - test_flask_fails_to_start_with_missing_cert: non-zero exit when cert missing - test_https_without_trust_fails_verification: SSL error with verify=True Uses openssl for cert generation, subprocess for Flask, requests for HTTP(S). Marked with integration_tls. Skip if openssl not available. Made-with: Cursor
…verify AUTH_ENABLED - Fix 1: _request_user() extracts username from g.user dict so API modules receive string for created_by (Elasticsearch keyword field) - Fix 2: Add test_decode_expired_token_raises, test_decode_tampered_token_raises, test_decode_wrong_secret_raises in test_auth.py - Fix 3: flask.py already uses auth.AUTH_ENABLED (verified) - Update evaluations_run to use user string directly (started_by) Made-with: Cursor
…verify AUTH_ENABLED - Fix 1: _request_user() extracts username from g.user dict so API modules receive string for created_by (Elasticsearch keyword field) - Fix 2: Add test_decode_expired_token_raises, test_decode_tampered_token_raises, test_decode_wrong_secret_raises in test_auth.py - Fix 3: flask.py already uses auth.AUTH_ENABLED (verified) - Update evaluations_run to use user string directly (started_by) Made-with: Cursor
…verify AUTH_ENABLED - Fix 1: _request_user() extracts username from g.user dict so API modules receive string for created_by (Elasticsearch keyword field) - Fix 2: Add test_decode_expired_token_raises, test_decode_tampered_token_raises, test_decode_wrong_secret_raises in test_auth.py - Fix 3: flask.py already uses auth.AUTH_ENABLED (verified) - Update evaluations_run to use user string directly (started_by) Made-with: Cursor
Import auth.AUTH_ENABLED in mcp_auth.py instead of parsing env independently. mcp_auth_middleware continues to use mcp_auth.AUTH_ENABLED (now delegated). Made-with: Cursor
…ests - Move /login route outside AuthGuard and ResourceProvider to prevent 401 redirect loops when ResourceProvider makes API calls on mount - Add LoginRoute wrapper to redirect authenticated users away from /login - Export AuthGuard for testing - Add Jest, React Testing Library, jest-environment-jsdom - Add Login.test.js: basic/apikey modes, toggle, disabled submit, error display - Add AuthContext.test.js: session check, isAuthenticated on 200/401 - Add AuthGuard.test.js: renders children when auth, redirects when not Made-with: Cursor
Made-with: Cursor
Made-with: Cursor
Made-with: Cursor
Made-with: Cursor
…Display username of humans who made judgements.
… for testing development installations
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v1.2.0
Features
AUTH_ENABLED(default:true). (Issue 4) (2a9b84a, 7866176)Authorizationheader (Basic, ApiKey, Bearer schemes). (Issue 5) (fb62d8e)TLS_ENABLED(default:true). (Issue 3) (4d6afd4)@meta.created_viachannel-tracking to asset documents ("server","mcp","api"), and@meta.updated_viawhere assets support update semantics. (Issue 2) (1e773f8)Improvements
rated-human/rated-aifilter now uses@meta.updated_via(channel), while retaining@meta.updated_bychecks for compatibility with existing data. (635198f)1.2.0to addcreated_viaandupdated_viafields to existing indices. (Issue 7) (ad3919e)Breaking changes
AUTH_ENABLEDnow exists and defaults totrue. When enabled, API routes require login andAUTH_JWT_SECRETmust be configured. (468957f)TLS_ENABLEDnow exists and defaults totrue. When enabled,TLS_CERT_FILEandTLS_KEY_FILEare required at startup. (468957f)Bug fixes