Merged
Conversation
Introduce multi-host Docker client and per-request Docker host selection, add NATS messaging initialization and WebSocket hub bridge, and update many API handlers and frontend components to use the new flow. Key changes: initialize docker.NewMultiHostClient and middleware.DockerHostSelector, use resolveDockerClient in handlers, expose X-Docker-Host via CORS, add initMessaging and bridgeNATSToHub stubs, defer logger.Sync, and keep a default dockerClient for compatibility. Also update go.mod (add nats.go, zap, uuid, deps), move .air.toml to configs and adjust docker-compose.dev, and add numerous new API handlers, middleware, messaging and frontend files to support multi-host and real-time features.
Introduce a config validation subsystem to detect and remediate config drift and to stream related events to the UI. Adds internal/configvalidator with Validator (snapshot, validate, restore, delete), starts a messaging hub for SSE/WebSocket, and publishes config_drift/config_missing/config_restored events. Persist snapshots with a new config_snapshots DB table and corresponding database methods and model types. Expose REST endpoints and handlers (/config/validation) and register SSE + WS event routes; update server startup to initialize the validator, snapshot on boot, and run validation. Add frontend support: API client for config validation, ConfigValidation page, SSE hook (useSSE), config event hook (useConfigEvents), and new log UI components (LogViewer, LogFilterPanel, TraefikAnalytics) with exports. Minor: add SSE handler in messaging hub, adjust NATS init signature and bridging, update imports/routes, and a tiny .gitignore whitespace fix.
Introduce backend features and refactors: add new API handlers for Hub (list/install/remove/upgrade), Simulation (status/toggle) and Alert inspection, plus a Gin rate-limiting middleware. Harden input handling (hub item name validation, numeric alert ID), tighten WebSocket origin checks, and make docker interactions safer by replacing shell echo/sed patterns with container file APIs (WriteFileToContainer, FileExists, AppendLineToFileInContainer) to prevent shell interpolation issues. Other changes: register hub and simulation routes in server startup and apply the rate limiter middleware; add GetSettings handler; improve resolveDockerClient to scope clients to request context; replace some raw http.Get calls with a shared ExternalHTTPClient.
Normalize CrowdSec console status semantics and add enrollment preferences, finalization and preview features. Backend: consolidate approval/phase fields in ConsoleStatus, keep legacy flags for compatibility, add enroll_disable_context to Settings and DB migrations, persist/read/update enrollment preferences, add enroll finalize endpoint (restarts container and polls status), add PreviewDiscordConfig and raw YAML support, improve whitelist parsing to handle IPv6 and add an endpoint to remove IPs from whitelists. Models updated to include new fields (approved, management_enabled, phase, raw_yaml) and routes wired for the new handlers. Frontend: introduce SearchContext provider, add ResultsSummary, ScenarioName and TimeDisplay components, enhance QueryError to show details, add breadcrumbs support in PageHeader and update EnrollDialog to surface/save the disable-context preference, adjust polling/approval logic and add finalize flow. Misc: minor UI icon/layout tweaks and small fixes across components.
Refactor and extend CrowdSec Hub handling: replace simple type/name endpoints with category-aware handlers (list, install, remove, manual YAML apply), add category metadata and ordering, sanitize filenames and validate modes. Persist hub preferences and operation history in the DB (schema additions + methods to upsert/list/get operations and preferences). Update routes to pass the database, add new API handlers for preferences and history, and introduce new models for Hub requests, preferences and operation records. Minor frontend updates and dependency bumps to integrate the new Hub UI page and API.
Remove the platform-specific @rollup/rollup-linux-x64-gnu entry from web/package.json. The package-lock.json was updated by npm to reflect the removal and related metadata changes.
Update bouncer health logic to require a valid key and consider pulls within 60 minutes as "connected", mark valid-but-old as "stale", otherwise "disconnected" (applied to bouncers handler and health diagnostics). Clean up frontend: remove search context, shortcut and search input from Header, memoize breadcrumbs, remove location-driven scope handling and footer from Layout, move footer/copyright into Sidebar. Replace hardcoded sidebar version with import.meta.env.VITE_APP_VERSION and add Vite config to inject package.json version. Misc: adjust sidebar theme button sizing/layout and minor logging whitespace fix in DeleteBouncer.
Add logic to preserve and match existing YAML list indentation when appending entries. Introduce appendToYAMLList helper in whitelist handler and use it to append CrowdSec whitelist items instead of hardcoding 4-space indentation. Change Traefik append calls to pass list items as "- <value>" and update Client.AppendLineToFileInContainer to insert after the last "- " item in a YAML block, detect existing indentation, and apply it to the inserted line. Also update the function comment to reflect the new behavior.
Move response-shaping logic out of queryFn and into react-query's select. queryFn now returns response.data.data directly, while select handles both array and object formats and returns a typed Bouncer[] (or an empty array) for consistent downstream usage.
Make response handling more defensive across two pages: Bouncers.tsx now reads response.data.data into a local `raw` and adapts to either an array or a wrapped object (e.g. { bouncers: [...] }). If the backend returns an object, it scans values for the first array and returns it; otherwise it falls back to an empty array. HubBrowser.tsx now returns `response.data.data ?? null` to explicitly yield null when data is absent. These changes make the UI resilient to varying backend response shapes.
Introduce a feature_configs table and DB helpers to store, retrieve, and mark feature configuration as applied. Add detection and apply workflows for captcha and Discord notifications (handlers: save/apply/detect) and refactor captcha setup to use helper functions for writing captcha.html and restarting containers. Add a compose scanner to read env vars from docker-compose files without modifying them. Expose new API routes and update the web client types and APIs. Also add UI components (FeatureWizard, StepProgress) and model types (FeatureConfig, FeatureDetectionResult) to support the new flows.
Stable-Version-Release-V1.0.0
Introduce a stepwise apply pipeline for captcha and Discord notification configuration with optional ?step=N re-run support. Backend: define named pipeline steps, validate step query param, run individual steps and return per-step results; use RestartContainerWithTimeout for container restarts and propagate errors; add errString helper to common package. Frontend: allow apply API to accept an optional step parameter, add retry UI and logic (StepProgress shows Retry button and loader), and implement single-step retry handling in Captcha and Notifications pages; update notification status UI to reflect individual step outcomes.
Introduce a simple in-memory TTL cache and integrate it across API handlers to reduce repeated cscli/docker calls and improve performance. Key changes: - Add internal/cache: a concurrency-safe TTLCache with Get/Set. - Instantiate cache in cmd/server and pass into RegisterServicesRoutes; handlers accept optional TTLCache. - Cache decisions (including a lightweight summary), metrics, and alerts analysis with short TTLs. - Add CLIFlag and appendCLIFlags helpers in handlers/common.go and refactor decision add/delete handlers to use them. - Replace file copy helper in ImportDecisions with writing file bytes into the container via WriteFileToContainer. - Add getExternalIP helper to try multiple external IP services; use it in GetPublicIP and WhitelistCurrentIP. - Add FollowContainerLogs to the Docker client and update StreamLogs to follow container logs natively (streaming ReadCloser), simplifying polling logic and reducing duplicate-send logic. - Web UI: add getDecisionsSummary API call and update Dashboard to use the summary endpoint and reduce auto-refresh frequency/labels. These changes aim to reduce load, improve responsiveness, simplify log streaming, and centralize common helpers.
Stop using yaml.FlowStyle and normalize sequence node styles when creating notifications nodes; ensure notifications nodes are explicit SequenceNodes and reset Style to zero where needed. Rely on yaml.Encoder to emit document separators instead of manually inserting "---" between encoded docs, and strip only the leading "---\n" to match CrowdSec's expected multi-document format. Also set encoder indentation to 1 and add comments explaining encoder behavior to prevent duplicate separators and maintain correct profiles.yaml formatting.
Add a reusable parseDecisionNode helper and fallback parsing for manual decisions that may appear on the top-level alert object (not only inside a decisions array), fixing missing entries in Decisions UI and Decisions Analysis. Import jsonparser and set AlertID/CreatedAt appropriately when falling back. Also remove the `?? null` coercion in HubBrowser query to avoid forcing valid responses to null and causing a blank page. Affected files: internal/api/handlers/common.go, internal/api/handlers/dashboard.go, internal/api/handlers/dashboard_analysis.go, web/src/pages/HubBrowser.tsx.
Add debug notes and fixes for multiple issues: strip preamble text before JSON parsing in hub handlers to avoid broken Hub Browser; add missing Hub categories (appsec-configs, appsec-rules, contexts) in frontend types/tabs/parse logic; implement fallback parsing for manually-added (flat) decisions so they appear in Decisions UI; correct notifications YAML formatting (use block sequences and 1-space indent). Also update README and docs to a simplified stable 1.0.0 onboarding flow (compose examples, networks, volumes, quick-start), add DEBUG_FINDINGS.md, and apply related frontend/backend file edits (hub handlers, hub API, HubBrowser/HubCategory, docs site files and minor package-lock cleanups). These changes address parsing bugs and improve documentation for the stable release.
Serve frontend assets from /assets with long-term caching while forcing no-cache for index.html to avoid stale hashed asset references. Update README with embedded images, docs link, license, support, and acknowledgments; bump docker image tag to hhftechnology/crowdsec-manager:v1.0.0. Remove now-unreferenced image files from images/. Normalize frontend API query handlers to return explicit nulls/defaults (using ?? null / ?? []), adjust related type annotations, and tweak a component prop to avoid runtime undefined values.
Handle noisy cscli hub output by extracting the first JSON payload instead of assuming clean stdout. backend: add firstJSONStartIndex and parseHubJSONOutput to strip preamble and decode either object or array, replace ad-hoc slicing in hub handlers, and add unit tests. frontend: add parseHubJSONString and extractHubArray to robustly parse mixed text/JSON responses (including iterative trimming) and accept alternate keys for appsec configs/rules. This makes hub listing resilient to informational logs surrounding the JSON payload.
parseCLIJSONToBytes / parseCLIJSONOutput helpers and rename firstJSONStartIndex to firstCLIJSONStartIndex; update multiple API handlers to normalize and parse cscli CLI JSON output before decoding. Replace direct json.Unmarshal([]byte(output)) use with normalized bytes and add improved error handling, logging, and appropriate HTTP responses where parsing/normalization fails. Updated related tests and minor cleanup across alerts_inspect, allowlists, bouncers, common, dashboard, dashboard_analysis, health_diagnostics, hub, hub_parse_test, ip, scenarios and simulation handlers.
Replace README image references with new screenshot filenames and add a comprehensive API endpoints section. Also add multiple new screenshot assets under images/ (various dashboard, analysis, management and hub views). The README now lists grouped API endpoints (Health, IP, Whitelist, Allowlist, Scenarios, Captcha, Logs, Backup, Update, Services, CrowdSec, Traefik, Config, Notifications, Cron, Profiles, Hosts, Terminal, Config Validation, Hub, Simulation, Events) to document the manager's HTTP API.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 737a733932
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // IsIPBlocked checks if an IP is blocked | ||
| func IsIPBlocked(dockerClient *docker.Client, cfg *config.Config) gin.HandlerFunc { | ||
| return func(c *gin.Context) { | ||
| dockerClient = resolveDockerClient(c, dockerClient) |
There was a problem hiding this comment.
This line reassigns the closure-captured dockerClient variable, which is shared by all concurrent requests for this handler instance. Under load, requests can race and overwrite each other’s request-scoped client/context, so one request may execute Docker commands with another request’s selected host or a canceled context. Use a per-request local variable instead of mutating the captured one.
Useful? React with 👍 / 👎.
Comment on lines
+174
to
+176
| cacheKey := "metrics" | ||
| if len(ttlCache) > 0 && ttlCache[0] != nil { | ||
| if cached, ok := ttlCache[0].Get(cacheKey); ok { |
There was a problem hiding this comment.
The cache key is global ("metrics") even though the Docker client is resolved per request from the selected host. In multi-host mode, a response cached from host A will be returned to host B until TTL expiry, so operators can see incorrect metrics for the currently selected host.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.