Skip to content

DS-4699 4.3.0 Swirl support and general improvements - #9

Merged
peterswirl merged 3 commits into
mainfrom
DS-4699
Aug 28, 2025
Merged

DS-4699 4.3.0 Swirl support and general improvements#9
peterswirl merged 3 commits into
mainfrom
DS-4699

Conversation

@peterswirl

@peterswirl peterswirl commented Aug 28, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Goal

Support 4.3.0 swirl and replace existing preview for prospects

Improvements

General

  • Documentation covers MacOS
  • Reload AI prompts at initial and subsequent system starts
  • Commenting for OIDC providers in example
  • System supports MacOS and Linux
  • System handles installation to any path

4.3.0 Support

  • Health check moves on to image
  • Branding refactor
  • Removal of defunct env variables

Testing

Linux Ubuntu 24.04

  1. Setup System via Standard image_test
  2. Clear image's files and environment
  3. Rsync this branch
  4. Reset docker and start system

Between each run we use the std clear process
systemctl stop swirl && ./DevUtils/reset_compose_environment.sh && rm -f /app/.swirl*

  • installation sets up systemd service
  • ACME certbot run works with MS Login and Search
  • Bring Your Own Cert (BYOC) works with self-signed cert

MacOS

  • initial setup
  • run without tls as service

General

  • diff between this branch and docker-compose-internal main reports no changes for the common fileset
diff --color -r docker-compose-internal/README.md docker-compose/README.md
61d60
<
diff --color -r docker-compose-internal/env.example docker-compose/env.example
5,6c5,6
< POSTGRES_VERSION=16
< REDIS_VERSION=8
---
> POSTGRES_VERSION=
> REDIS_VERSION=

Peter Kahn
- Documentation covers MacOS
a62b220 
- Reload AI prompts at initial and subsequent system starts
- Health check moves on to image (4.3.0 and above)
- Branding refactor (4.3.0 and above)
- Commenting for OIDC providers in example
- Removal of defunct env variables
- System supports MacOS and Linux
- System handles installation to any path
@peterswirl peterswirl self-assigned this Aug 28, 2025
@github-actions

github-actions Bot commented Aug 28, 2025

Copy link
Copy Markdown

URL Check Summary

URL Check Summary

Passed: 23
Excluded: 4
Failed: 1
Unhandled Lines: 0

Failed URLs:

@peterswirl
peterswirl marked this pull request as ready for review August 28, 2025 15:33
@peterswirl
peterswirl merged commit 4999235 into main Aug 28, 2025
2 checks passed
@peterswirl
peterswirl deleted the DS-4699 branch August 28, 2025 15:46
dnicodemus added a commit that referenced this pull request Dec 23, 2025
@joaoferreiraswirl @joaoferreirackl
@dnicodemus @peterswirl
[DS-4100/DS-4061] Multiple OIDC & BYOC (#11)
0a145f4 
* Initialize develop

* feat(infra): include base files

* chore(infra): update shell and compose files to include phase 1 features

* feat(infra): include github workflow for integration tests

* chore(infra): update nginx docker entrypoint with fswatch

* chore(infra): refactor healthy check step for integration tests

* chore(infra): update health check timeout to 600

* chore(infra): use variables for setting all image tags

* chore(infra): include health checks for tika, ttm and certbot services

* chore(infra): include scripts for health check and image pull

* chore(infra): include image storage as artifacts

* fix(infra): add missing prefix for docker variables

* chore(fix): include path prefix for running shell scripts

* fix(infra): add missing checkout step

* chore(infra): include free space commands before saving images

* chore(infra): save docker images only as tar and remove additional compression

* chore(infra): convert generated images to .tar.gz using maximum compression

* chore(infra): test compression using zstd instead of gzip

* chore(infra): update runner to ubuntu-latest-m

* chore(infra): change compose-test to use ubuntu-latest-m runner

* chore(infra): test removing pull-images step

* chore(infra): include fail-fast condition to false

* fix(infra): include tag variables for swirl release components

* chore(infra): update volume mappings for nginx service

* chore(infra): update scripts to include file creations and posix compliance

* chore(infra): update matrix values

* chore(infra): update pr template to remove typos

* chore(infra): include act pre-push githook

* chore(infra): update branch conditions for integration test workflow

* chore(infra): generate new routes for nginx health checks

* chore(infra): include certbot liveness methods and health checks

* chore(infra): update health check params for certbot and nginx

* feat(infra): include base files

* chore(infra): update shell and compose files to include phase 1 features

* feat(infra): include github workflow for integration tests

* chore(infra): update nginx docker entrypoint with fswatch

* chore(infra): refactor healthy check step for integration tests

* chore(infra): update health check timeout to 600

* chore(infra): use variables for setting all image tags

* chore(infra): include health checks for tika, ttm and certbot services

* chore(infra): include scripts for health check and image pull

* chore(infra): include image storage as artifacts

* fix(infra): add missing prefix for docker variables

* chore(fix): include path prefix for running shell scripts

* fix(infra): add missing checkout step

* chore(infra): include free space commands before saving images

* chore(infra): save docker images only as tar and remove additional compression

* chore(infra): convert generated images to .tar.gz using maximum compression

* chore(infra): test compression using zstd instead of gzip

* chore(infra): update runner to ubuntu-latest-m

* chore(infra): change compose-test to use ubuntu-latest-m runner

* chore(infra): test removing pull-images step

* chore(infra): include fail-fast condition to false

* fix(infra): include tag variables for swirl release components

* chore(infra): update volume mappings for nginx service

* chore(infra): update scripts to include file creations and posix compliance

* chore(infra): update matrix values

* chore(infra): update pr template to remove typos

* chore(infra): include act pre-push githook

* chore(infra): update branch conditions for integration test workflow

* chore(infra): generate new routes for nginx health checks

* chore(infra): include certbot liveness methods and health checks

* chore(infra): update health check params for certbot and nginx

* [DS-3888] Update repository README (#4)

* chore(infra): update repository README file

* chore(infra): remove .github readme file

* WIP Changes

* add chat to AI tablee

* Some changes

* remove detailed doc and leave just the links to additional doc.

* Added link to support ticket

* chore(infra): update swirl service health checks

---------

Co-authored-by: Joao Ferreira <joao.ferreira@ckl.io>
Co-authored-by: dnicodemus-la <dave.nicodemus@gmail.com>
Co-authored-by: dnicodemus <94639381+dnicodemus@users.noreply.github.com>

* [DS-3888] Update health checks (#6)

* chore(infra): update healthchecks params to matching values

* chore(infra): chmod swirl health check script

---------

Co-authored-by: Joao Ferreira <joao.ferreira@ckl.io>

* fix: include missing ( in README.md file (#7)

Co-authored-by: Joao Ferreira <joao.ferreira@ckl.io>

* DS-4074 add mcp (#8)

* Add mcp
* Update env example with correct values and user configuration sections

* [DS-4100/DS-4061] Multiple OIDC & BYOC (#9)

* chore(infra): update example env file to use alphabetical variables

* chore(infra): update swirl-load.sh to support multiple oidc script

* chore(infra): include additional variables for multiple oidc support

* chore(infra): remove unused update_json function

* chore(infra): include missing vars in example file

* chore(infra): move entrypoint out of root directory

* chore(infra): update reference variable for fqdn

* chore(infra): include multiple oidc support and byoc solution

---------

Co-authored-by: Joao Ferreira <joao.ferreira@ckl.io>

* Revert "DS-4074 add mcp (#8)"

This reverts commit 09c45b46e8de80e25972a117a4c0b287742171ed.

---------

Co-authored-by: Joao Ferreira <joao.ferreira@ckl.io>
Co-authored-by: dnicodemus-la <dave.nicodemus@gmail.com>
Co-authored-by: dnicodemus <94639381+dnicodemus@users.noreply.github.com>
Co-authored-by: Peter Kahn <peter@swirl.today>
dnicodemus added a commit that referenced this pull request Dec 23, 2025
@joaoferreiraswirl @joaoferreirackl
@dnicodemus @peterswirl Peter Kahn
[DS-4061/DS-4074/DS-4100/DS-4247/DS-4315] Merge develop changes int…
f858161 
…o `main` (#19)

* Initialize develop

* feat(infra): include base files

* chore(infra): update shell and compose files to include phase 1 features

* feat(infra): include github workflow for integration tests

* chore(infra): update nginx docker entrypoint with fswatch

* chore(infra): refactor healthy check step for integration tests

* chore(infra): update health check timeout to 600

* chore(infra): use variables for setting all image tags

* chore(infra): include health checks for tika, ttm and certbot services

* chore(infra): include scripts for health check and image pull

* chore(infra): include image storage as artifacts

* fix(infra): add missing prefix for docker variables

* chore(fix): include path prefix for running shell scripts

* fix(infra): add missing checkout step

* chore(infra): include free space commands before saving images

* chore(infra): save docker images only as tar and remove additional compression

* chore(infra): convert generated images to .tar.gz using maximum compression

* chore(infra): test compression using zstd instead of gzip

* chore(infra): update runner to ubuntu-latest-m

* chore(infra): change compose-test to use ubuntu-latest-m runner

* chore(infra): test removing pull-images step

* chore(infra): include fail-fast condition to false

* fix(infra): include tag variables for swirl release components

* chore(infra): update volume mappings for nginx service

* chore(infra): update scripts to include file creations and posix compliance

* chore(infra): update matrix values

* chore(infra): update pr template to remove typos

* chore(infra): include act pre-push githook

* chore(infra): update branch conditions for integration test workflow

* chore(infra): generate new routes for nginx health checks

* chore(infra): include certbot liveness methods and health checks

* chore(infra): update health check params for certbot and nginx

* feat(infra): include base files

* chore(infra): update shell and compose files to include phase 1 features

* feat(infra): include github workflow for integration tests

* chore(infra): update nginx docker entrypoint with fswatch

* chore(infra): refactor healthy check step for integration tests

* chore(infra): update health check timeout to 600

* chore(infra): use variables for setting all image tags

* chore(infra): include health checks for tika, ttm and certbot services

* chore(infra): include scripts for health check and image pull

* chore(infra): include image storage as artifacts

* fix(infra): add missing prefix for docker variables

* chore(fix): include path prefix for running shell scripts

* fix(infra): add missing checkout step

* chore(infra): include free space commands before saving images

* chore(infra): save docker images only as tar and remove additional compression

* chore(infra): convert generated images to .tar.gz using maximum compression

* chore(infra): test compression using zstd instead of gzip

* chore(infra): update runner to ubuntu-latest-m

* chore(infra): change compose-test to use ubuntu-latest-m runner

* chore(infra): test removing pull-images step

* chore(infra): include fail-fast condition to false

* fix(infra): include tag variables for swirl release components

* chore(infra): update volume mappings for nginx service

* chore(infra): update scripts to include file creations and posix compliance

* chore(infra): update matrix values

* chore(infra): update pr template to remove typos

* chore(infra): include act pre-push githook

* chore(infra): update branch conditions for integration test workflow

* chore(infra): generate new routes for nginx health checks

* chore(infra): include certbot liveness methods and health checks

* chore(infra): update health check params for certbot and nginx

* [DS-3888] Update repository README (#4)

* chore(infra): update repository README file

* chore(infra): remove .github readme file

* WIP Changes

* add chat to AI tablee

* Some changes

* remove detailed doc and leave just the links to additional doc.

* Added link to support ticket

* chore(infra): update swirl service health checks

---------

Co-authored-by: Joao Ferreira <joao.ferreira@ckl.io>
Co-authored-by: dnicodemus-la <dave.nicodemus@gmail.com>
Co-authored-by: dnicodemus <94639381+dnicodemus@users.noreply.github.com>

* [DS-3888] Update health checks (#6)

* chore(infra): update healthchecks params to matching values

* chore(infra): chmod swirl health check script

---------

Co-authored-by: Joao Ferreira <joao.ferreira@ckl.io>

* fix: include missing ( in README.md file (#7)

Co-authored-by: Joao Ferreira <joao.ferreira@ckl.io>

* DS-4074 add mcp (#8)

* Add mcp
* Update env example with correct values and user configuration sections

* [DS-4100/DS-4061] Multiple OIDC & BYOC (#9)

* chore(infra): update example env file to use alphabetical variables

* chore(infra): update swirl-load.sh to support multiple oidc script

* chore(infra): include additional variables for multiple oidc support

* chore(infra): remove unused update_json function

* chore(infra): include missing vars in example file

* chore(infra): move entrypoint out of root directory

* chore(infra): update reference variable for fqdn

* chore(infra): include multiple oidc support and byoc solution

---------

Co-authored-by: Joao Ferreira <joao.ferreira@ckl.io>

* DS-4315 docker login (#14)

* DS-4315 docker login script for conversion from trial to actual

* set name

---------

Co-authored-by: Peter Kahn <peter@swirl.com>

* Revert "DS-4074 add mcp (#8)" (#16)

This reverts commit 09c45b46e8de80e25972a117a4c0b287742171ed.

Co-authored-by: Joao Ferreira <joao.ferreira@ckl.io>

* DS-4315 backup restore (#15)

* DS-4315 backup and restore scripts

* fix spacing

* refactore restore

* - breakup into functions
- error handling

* fix unpack

---------

Co-authored-by: Peter Kahn <peter@swirl.com>

* [DS-4247] Remove use of inotify for docker compose system (#13)

* Revert "DS-4074 add mcp (#8)"

This reverts commit 09c45b46e8de80e25972a117a4c0b287742171ed.

* fix(infra): include absolute path for scripts and environment files

* fix(infra): remove missing prefix for flag file

* workaround for Certificate not yet due for renewal problem (#12)

* workaround for Certificate not yet due for renewal problem

* drop the daemon setting, we do not want the action on line  43 to run in the background.  We need it to fully complete prior to starting the web server

Also, `set -e` means we don't need to do cmd && cmd as 1st fail on any line terminates the script

---------

Co-authored-by: Peter Kahn <peter@swirl.com>

* chore(infra): remove curl installation in swirl-load script

* chore(infra): implement native polling for nginx configuration

* chore(infra): removes unused mount template for nginx

* chore(infra): update paths for services in docker compose

* chore(infra): remove certbot and nginx logic from swirl-service script

* chore(infra): include cert_conf variable in tls template

* chore(infra): include nginx and certbot configs inside respective entrypoints

* chore(infra): include volume for certbot data

* chore(infra): include sleep on nginx entrypoint to wait for dns propagation

* chore(infra): update directory for letsencrypt config on certbot entrypoint

* chore(infra): include logging info for nginx entrypoint

* chore(infra): remove macos fallback in nginx entrypoint

* - combine swirl.py && daphne
- drop the 300s wait

* chore(infra): include restart policy for certbot service

* chore(infra): include nginx.template for proper debugging

* chore(infra): apply changes from 0.148.1 vm offer

* chore(infra): update folder structure to match vm offer

* fix(infra): update scripts to match new folder structure

* chore(infra): update paths for certbot conf files

* chore(infra): include snippet for restarting swirl on both linux and macos

* chore(infra): revert snippet for macos and linux compatibility

* chore(infra): implements dynamic script directory for referencing paths inside swirl-service

* chore(infra): update references for certificate files

* fix(infra): using updated path for swirl service script

* chore(infra): implement logic for supporting macos execution on swirl service script

* chore(infra): include healthchecks for all services

* chore(infra): remove health checks for tika, ttm and certbot

* fix(infra): using correct extension for certificate files

* chore(infra): include healthcheck for certbot

* chore(infra): upgrade postgres to 16

* chore(infra): update tab for tls templates

* fix(infra): use correct key and certificate paths for certbot

* chore(infra): replace getent with nslookup for macos compatibility

* fix(infra): using correct syntax for nslookup command

* chore(infra): update readme file to include certbot conf instructions

* Ds 4247 updates (#18)

* change ES7 to version from swsi repo

* fix nslookup issue - getent uses hosts, nslookup does not

---------

Co-authored-by: Peter Kahn <peter@swirl.com>
Co-authored-by: João Ferreira <joao.ferreira@swirlaiconnect.com>

* chore(infra): replace echo with log for tls configs

* chore(infra): specify read only permissions for volumes in nginx service

* chore(infra): using app prefix for certbot conf volume

* chore(infra): update absolute paths for certbot files and symlinks

* chore(infra): bump default swirl images to 4.2.1.0

* chore(infra): remove unused exit trap call

* chore(infra): remove debug flag from swirl.py start

---------

Co-authored-by: Joao Ferreira <joao.ferreira@ckl.io>
Co-authored-by: Peter Kahn <peter@swirl.today>
Co-authored-by: Peter Kahn <peter@swirl.com>

---------

Co-authored-by: Joao Ferreira <joao.ferreira@ckl.io>
Co-authored-by: dnicodemus-la <dave.nicodemus@gmail.com>
Co-authored-by: dnicodemus <94639381+dnicodemus@users.noreply.github.com>
Co-authored-by: Peter Kahn <peter@swirl.today>
Co-authored-by: Peter Kahn <peter@swirl.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@joaoferreiraswirl joaoferreiraswirl Awaiting requested review from joaoferreiraswirl
1 more reviewer
@dnicodemus dnicodemus dnicodemus approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@peterswirl peterswirl

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@peterswirl @dnicodemus