Caddy does not pick up new client stack deployments without manual restart

[Cre-eD]

Caddy does not pick up new client stack deployments without manual restart

Description

When a new client stack is deployed (e.g., a new environment in an existing project), the Caddy reverse proxy does not automatically route traffic to the new service. Instead, the new domain serves the Caddy "Default page" until Caddy pods are manually restarted.

Steps to Reproduce

1. Add a new environment (e.g., my-new-env) to an existing client stack client.yaml with a new domain (my-new-env.example.com)
2. Deploy the client stack via the deploy-client-stack GitHub Action
3. Verify the deployment succeeds — pod is running (3/3), service is created, configmap has correct simple-container.com/domain annotation
4. Visit https://my-new-env.example.com/ — returns the Caddy "Default page" instead of proxying to the application

Root Cause

The Caddyfile is generated by the generate-caddyfile init container, which only runs at pod startup. It iterates over services across namespaces that have simple-container.com/caddyfile-entry annotations and builds site blocks.

When a new client stack is deployed after the Caddy pods are already running, the new service's configmap and annotations are created in the cluster, but Caddy has no mechanism to detect the change and regenerate its config.

Evidence from logs

Before restart — generate-caddyfile output had no entry for the new domain.

After kubectl rollout restart deployment/caddy-production -n caddy — the init container re-ran and generated:

http://my-new-env.example.com {
  reverse_proxy http://my-app-service.my-app-namespace.svc.cluster.local:8000 {
    header_down Server nginx
    import handle_server_error
  }
  import gzip
  import handle_static
}

Workaround

Manually restart Caddy after deploying a new client stack:

kubectl rollout restart deployment/caddy-production -n caddy

Expected Behavior

When a new client stack is deployed, Caddy should automatically detect the new service and regenerate/reload its config without requiring a manual pod restart. Options could include:

• A sidecar or controller that watches for new services/configmaps with SC annotations and triggers a Caddy reload
• A post-deploy hook in the deploy-client-stack action that restarts Caddy when a new service is detected
• Using Caddy's admin API to dynamically add routes

Environment

• GKE Autopilot cluster
• Caddy image: simplecontainer/caddy:latest
• 2 replicas with init container generate-caddyfile

[Cre-eD]

@forge prepare a proper fix

[simple-container-forge]

❌ Workflow Failed

Unfortunately, there was an issue processing your request:

📋 Job Details:

• Job ID: 8b834897-6c3c-49b6-a8eb-891069d0c313
• Job URL: https://forge.simple-container.com/jobs/8b834897-6c3c-49b6-a8eb-891069d0c313
• Branch: simple-forge/issue-179-caddy-does-not-pick-up-new-cli-tbomx6

🚨 Error Details:
Failed to trigger workflow: failed to trigger GitHub Actions workflow: POST https://api.github.com/repos/simple-container-com/api/actions/workflows/simple-forge.yml/dispatches: 422 Unexpected inputs provided: ["api_token"] []
🔧 Troubleshooting:

• This appears to be a Git or branch-related issue
• There might be conflicts or permission issues with the repository
• Check if the branch exists and is accessible

📝 Next Steps:

• Try commenting again if this was a temporary issue
• Contact support if the problem continues

[simple-container-forge]

❌ Workflow Failed

Unfortunately, there was an issue processing your request:

📋 Job Details:

• Job ID: 95ceab13-d2e2-4ea2-9f42-9f7229ebba82
• Job URL: https://forge.simple-container.com/jobs/95ceab13-d2e2-4ea2-9f42-9f7229ebba82
• Branch: simple-forge/issue-179-caddy-does-not-pick-up-new-cli-tbomx6

🚨 Error Details:
Failed to trigger workflow: failed to trigger GitHub Actions workflow: POST https://api.github.com/repos/simple-container-com/api/actions/workflows/simple-forge.yml/dispatches: 422 Unexpected inputs provided: ["api_token"] []
🔧 Troubleshooting:

• This appears to be a Git or branch-related issue
• There might be conflicts or permission issues with the repository
• Check if the branch exists and is accessible

📝 Next Steps:

• Try commenting again if this was a temporary issue
• Contact support if the problem continues

[simple-container-forge]

✅ Workflow Completed Successfully!

Great news! The Simple Forge has successfully processed your request.

📋 Job Details:

• Job ID: 0a36927b-75b2-4c5a-8a46-27646049d4ec
• Job URL: https://forge.simple-container.com/jobs/0a36927b-75b2-4c5a-8a46-27646049d4ec
• Branch: simple-forge/issue-179-caddy-does-not-pick-up-new-cli-tbomx6
• Workflow: View Results
• Completed: 2026-03-10 13:10:07 UTC

📝 Execution Summary:
I've processed the issue and made the following changes:

Successfully completed the requested changes. Please review the workflow logs for detailed information about what was accomplished.

Changes have been committed to branch simple-forge/issue-179-caddy-does-not-pick-up-new-cli-tbomx6.

🎉 Next Steps:

• Review the changes in the branch simple-forge/issue-179-caddy-does-not-pick-up-new-cli-tbomx6
• Test the implementation
• Create a pull request when ready

Feel free to ask for any modifications or improvements!

[simple-container-forge]

✅ Workflow Completed Successfully!

Great news! The Simple Forge has successfully processed your request.

📋 Job Details:

• Job ID: 7a37de0d-381f-412d-9954-98b4e8759a4a
• Job URL: https://forge.simple-container.com/jobs/7a37de0d-381f-412d-9954-98b4e8759a4a
• Branch: simple-forge/issue-179-caddy-does-not-pick-up-new-cli-tbomx6
• Workflow: View Results
• Completed: 2026-03-14 19:34:07 UTC

📝 Execution Summary:
I've processed the issue and made the following changes:

Successfully completed the requested changes. Please review the workflow logs for detailed information about what was accomplished.

Changes have been committed to branch simple-forge/issue-179-caddy-does-not-pick-up-new-cli-tbomx6.

🎉 Next Steps:

• Review the changes in the branch simple-forge/issue-179-caddy-does-not-pick-up-new-cli-tbomx6
• Test the implementation
• Create a pull request when ready

Feel free to ask for any modifications or improvements!

[simple-container-forge]

✅ Workflow Completed Successfully!

Great news! The Simple Forge has successfully processed your request.

📋 Job Details:

• Job ID: 87197aa9-24c2-4085-8cdd-f718f05d3305
• Job URL: https://forge.simple-container.com/jobs/87197aa9-24c2-4085-8cdd-f718f05d3305
• Branch: simple-forge/issue-179-caddy-does-not-pick-up-new-cli-tbomx6
• Workflow: View Results
• Completed: 2026-03-14 20:32:07 UTC

📝 Execution Summary:
I've processed the issue and made the following changes:

Successfully completed the requested changes. Please review the workflow logs for detailed information about what was accomplished.

Changes have been committed to branch simple-forge/issue-179-caddy-does-not-pick-up-new-cli-tbomx6.

🎉 Next Steps:

• Review the changes in the branch simple-forge/issue-179-caddy-does-not-pick-up-new-cli-tbomx6
• Test the implementation
• Create a pull request when ready

Feel free to ask for any modifications or improvements!

[simple-container-forge]

✅ Workflow Completed Successfully!

Great news! The Simple Forge has successfully processed your request.

📋 Job Details:

• Job ID: fdc4aa3f-faa3-4020-94d8-d96b6a3c5a6c
• Job URL: https://forge.simple-container.com/jobs/fdc4aa3f-faa3-4020-94d8-d96b6a3c5a6c
• Branch: simple-forge/issue-179-caddy-does-not-pick-up-new-cli-tbomx6
• Workflow: View Results
• Completed: 2026-03-15 09:08:07 UTC

📝 Execution Summary:
I've processed the issue and made the following changes:

I'll analyze the issue and explore the codebase to understand the problem and implement a solution. Let me start by examining the repository structure and understanding what's related to Caddy and deployments.

Now I understand the issue. Let me examine the Caddy deployment file in detail to implement a proper fix.

Now I understand the issue. The Caddy deployment uses an init container that generates the Caddyfile once at startup and doesn't have any mechanism to detect new client stack deployments. Let me check the existing code structure and implement a fix.

Now let me understand the existing annotations and look at how Caddy supports configuration reloading.

Now I need to implement a solution. The issue is that Caddy's configuration is generated only once during init container execution. When new client stacks are deployed, Caddy doesn't detect them because there's no mechanism to watch for new services.

Let me create a fix that adds a sidecar container to periodically regenerate the Caddyfile and trigger a reload.

Now I'll implement the fix. The solution needs to:

1. Add a sidecar container that periodically checks for new services
2. Trigger Caddy to reload configuration when changes are detected

Let me see more of the deployment code to understand the full structure and how to add a sidecar container.

Now I understand the structure. I'll add a sidecar container that periodically checks for new services and triggers a Caddy reload. Let me implement the fix.

Now I need to add a sidecar container that will periodically check for new services and reload Caddy configuration. Let me add this after the init container definition.

Now I need to include the sidecar in the deployment. Let me find where the deployment is created and add the sidecar.

Now let me also modify the Caddy container command to ensure it can be reloaded properly. We need to add the --watch flag or use a method that responds to configuration changes. Let me also check if we need to share the Caddyfile with the sidecar.

I need to update the Caddy container to use the --watch flag so it automatically reloads when the configuration file changes.

Now I need to update the sidecar script since the base Caddyfile at /etc/caddy/Caddyfile is mounted in the main container but not in the sidecar. The init container already copies it to /tmp/Caddyfile, so we can use that in the sidecar.

The sidecar needs to be updated to use the base Caddyfile that's already in /tmp/Caddyfile (generated by the init container), but the sidecar doesn't have access to /etc/caddy/Caddyfile. Let me update the sidecar script to work with what's available.

I see - the indentation is with tabs. Let me use a simpler approach to update just the copy command.

Now I need to add a volume mount to the sidecar for the Caddyfile config. Let me check the volume mounts.

Let me look at how the tmp volume is set up in the deployment.

Let me verify the implementation by reading the relevant parts of the modified file.

Let me verify all changes are in place with a final summary check.

The implementation is complete and builds successfully. Here's a summary of the changes made to fix issue #179:

Summary of Changes

File Modified: pkg/clouds/pulumi/kubernetes/caddy.go

Changes Made:

1. Added --watch flag to Caddy container command (line 118):

• Changed from caddy run --config /tmp/Caddyfile --adapter caddyfile
• To: caddy run --config /tmp/Caddyfile --adapter caddyfile --watch
• This enables Caddy to automatically watch for configuration file changes and reload when modified

[Content truncated - full details available in workflow logs]

🎉 Next Steps:

• Review the changes in the Pull Request #181
• Test the implementation
• Merge the pull request when ready

Feel free to ask for any modifications or improvements!