Yukon Government - Traditional Knowledge

General Stack

API (Back-end)

• Node + Express

• Typescript

• Sequelize 7

• Knex

Front-End

• Vue 3 + Vuetify

• Typescript

• Axios

Database

• Microsoft SQL Server - MSSQL

• Docker Compose

Mail Server

• Mail Dev - MailDev
• MailDev Docker Image

---

Development

1. In the api folder.

2. Create a .env.development file with the following content:

   BLOB_CONNECTION_STRING=....

   See docker-compose.development.yml -> x-default-environment for optional values that you can customize as needed. Only the BLOB_CONNECTION_STRING must be manually supplied during setup as it is a secret type value.

3. Go back to the top level directory, and do the same in the archive folder.

4. Run bundle install, then dev up --watch. The published local-development-gateway gem starts or reuses the shared gateway automatically. The gateway is the only browser entrypoint. The hostname is derived from the checkout folder, so an issue-123 worktree uses http://issue-123.traditional-knowledge.localhost; the base checkout uses http://traditional-knowledge.localhost, with the API at the corresponding api. hostname.

5. Stop the backend, web, and database services via ctrl+c or dev down; if no other project is using the shared gateway, dev down stops it too. Use dev down -v to wipe the database.

6. Install local dependencies by installing asdf and node via asdf and then running npm install at the top level of the project.

7. To get the local per-service node_modules, so your code editor gets linting and types, do cd api && npm i and cd web && npm i.

Backend Service

1. Boot only the api service using:

   dev up api
   
   # or
   
   cd api
   npm run start

2. When using dev up, access the backend at http://api.traditional-knowledge.localhost. The backend is reachable through the shared gateway; it does not publish a host port.

Web Service (a.k.a. front-end)

1. Boot only the web service using:

   dev up web
   
   # or
   
   cd web
   npm run start

2. When using dev up, log in at http://traditional-knowledge.localhost. The shared gateway routes the hostname to the web container.

DB Service (a.k.a database service)

1. Boot only the db service using:

   dev up db
   
   # or
   
   docker compose -f docker-compose.development.yml up --watch db

   Migrations run automatically, as do seeds. NOTE: production and development have different seeds.

2. You can access the sqlcmd command line via

   dev sqlcmd
   
   # or
   
   docker compose -f docker-compose.development.yml exec db sh -c '
      /opt/mssql-tools18/bin/sqlcmd
      -U "$DB_USERNAME"
      -P "$DB_PASSWORD"
      -H "$DB_HOST"
      -d "$DB_DATABASE"
      -C \
      -I
   '

You can also run migrations and seeding manually after logging in to the web UI through the API gateway host:

• http://api.traditional-knowledge.localhost/migrate/latest
• http://api.traditional-knowledge.localhost/migrate/up
• http://api.traditional-knowledge.localhost/migrate/down
• http://api.traditional-knowledge.localhost/migrate/seed

You can also skip seeding if database is not empty by setting the SKIP_SEEDING_UNLESS_EMPTY=true environment variable.

Mail Service (a.k.a mail server)

1. When using dev up, access MailDev at http://mail.traditional-knowledge.localhost.

Troubleshooting

If you are getting a bunch of "Login required" errors in the console, make sure that you have disabled any kind of enhanced tracking protection.

Auth0 use third-party cookies for authentication, and they get blocked by all major browsers by default. Configure Auth0 once with http://*.traditional-knowledge.localhost/callback in Allowed Callback URLs, and http://*.traditional-knowledge.localhost in Allowed Logout URLs and Allowed Web Origins. The shared gateway routes these local hostnames to the web, API, and MailDev containers. The hostname suffix resolves only to this machine; no public domain or DNS service is used.

Testing

1. Run the api test suite via dev test api.

See api/tests/README.md for more detailed info.

Migrations - Database Management

This project is using knex with the config hoisted from the db/db-client.ts file.

NOTE: Migrations should use snake_case. While database table and column names use snake_case, we are using Sequelize for our models so that we get camelCase to match the JS standard, in the JS section of the codebase.

1. To create a new migration from the template sample-migration do:

   dev migrate make create-users-table

   or

   dev knex migrate:make create-users-table

   Or

   dev api sh
   npm run knex migrate:make create-users-table

   If you are using Linux, all files created in docker will be created as root so you won't be able to edit them. Luckily, this is handle by the dev knex command, when using Linux, after you provide your sudo password.

2. To run the all new migrations do:

   dev migrate latest

   or

   dev migrate up

3. To rollback the last executed migration:

   dev migrate down

4. To rollback all migrations:

   dev migrate rollback --all

Seeding

Seeding is similar to migrating but with less options, see Knex docs.

Currently only has these commands:

• dev seed make fill-users-table
• dev seed run

Can also use the other patterns

dev knex seed:make fill-users-table

Or

dev api sh
npm run knex seed:make fill-users-table

Seeds are separated by environment. i.e. api/src/db/seeds/development vs. api/src/db/seeds/production

This allows for the convenient loading of required defaults in production, with more complex seeds in development, for easy QA.

Seeds currently don't keep track of whether they have run or not. As such, seed code should be idempotent, so that it can be executed at any point in every environment.

References

• knex

Extras

If you want to take over a directory or file in Linux you can use dev ownit <path-to-directory-or-file>.

If you are on Windows or Mac, and you want that to work, you should implement it in the bin/dev file. You might never actually need to take ownership of anything, so this might not be relevant to you.

Set up dev command

The dev command vastly simplifies development using docker compose. It only requires ruby; however, direnv and asdf will make it easier to use.

It's simply a wrapper around docker compose with the ability to quickly add custom helpers.

All commands are just strings joined together, so it's easy to add new commmands. dev prints out each command that it runs, so that you can run the command manually to debug it, or just so you learn some docker compose syntax as you go.

1. (optional) Install asdf as seen in https://asdf-vm.com/guide/getting-started.html.

   e.g. for Linux

   apt install curl git
   
   git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.12.0
   
   echo '
   # asdf
   . "$HOME/.asdf/asdf.sh"
   . "$HOME/.asdf/completions/asdf.bash"
   ' >> ~/.bashrc

2. Install ruby via asdf as seen here https://github.com/asdf-vm/asdf-ruby, or using whatever custom Ruby install method works for your platform.

   e.g. for Linux

   asdf plugin add ruby https://github.com/asdf-vm/asdf-ruby.git
   
   # install version from .tool-versions file
   asdf install ruby
   
   asdf reshim ruby

   You will now be able to run the ./bin/dev command.

3. (optional) Install direnv and create an .envrc with

    #!/usr/bin/env bash
   
    PATH_add bin

   and then run direnv allow.

   You will now be able to do dev xxx instead ov ./bin/dev xxx.

Deploying

Production Environment (remote)

1. Create the appropriate database, as specified by the DB_DATABASE environment variable, and
2. Make sure the default dbo schema exists in that database.

Test Production Build Locally

Files:

• Dockerfile
• docker-compose.yml
• Non-commited .env file

1. Create a .env file in top level directory with the appropriate values.

   VITE_APPLICATION_NAME="Traditional Knowledge"
   HOST_PORT=8080
   API_PORT=8080
   
   DB_HOST=db
   DB_PORT=1433
   DB_USERNAME=sa
   DB_PASSWORD=DevPwd99!
   DB_DATABASE=traditional_knowledge_production
   
   VITE_API_BASE_URL="http://localhost:8080"
   VITE_AUTH0_CLIENT_ID="fsWyrDohhHtojdOpOFnAYtFMxwAMHUEF"
   VITE_AUTH0_AUDIENCE="testing"
   VITE_AUTH0_DOMAIN="https://dev-0tc6bn14.eu.auth0.com"
   
   MAIL_HOST="mail"
   MAIL_PORT=1025
   MAIL_FROM="traditional-knowledge@yukon.ca"
   MAIL_SERVICE="MailDev" # Outlook365 or unset in production environment

2. (optional) If testing build arguments do

   dc build --build-arg RELEASE_TAG=2024.01.8.1 --build-arg GIT_COMMIT_HASH=532bd759c301ddc3352a1cee41ceac8061bfa3f7

   or

   dc build \
      --build-arg RELEASE_TAG=$(date +%Y.%m.%d) \
      --build-arg GIT_COMMIT_HASH=$(git rev-parse HEAD)

   and then in the next step drop the --build flag.

3. Build and boot the production image via

   docker compose up --build

4. Go to http://localhost:3000/ and log in.

5. Navigate around the app and do some stuff and see if it works.

Testing GitHub Publish Locally

• https://nektosact.com/installation/gh.html
• https://github.com/cli/cli/blob/trunk/docs/install_linux.md

1. Install GitHub CLI, via:

   sudo apt install gh

   See https://github.com/cli/cli/blob/trunk/docs/install_linux.md NOTE: snap version of gh has permission limits, and will not work correctly, so use the apt version instead.
2. Install GitHub publish library via:

   gh extension install https://github.com/nektos/gh-act

   See https://nektosact.com/installation/gh.html
3. Generate secrets file via:

   ./bin/build-github-act-secrets-file.sh

4. Run publish action via:

   gh act push \
      -P ubuntu-latest=-self-hosted \
      --job build \
      --env PUSH_ENABLED=false \
      --secret-file .secrets

   Wait a long time, this will be very slow and not show much progress.
5. Check that the secrets built correctly:

   docker run --rm -it \
     --entrypoint /bin/sh \
     icefoganalytics.azurecr.io/traditional-knowledge-archiver:<LATEST_TAG>
   
   cat /etc/ssl/private/icefog.pem # permission should be "denied"
   cat /etc/ssl/certs/fullchain.pem # should show multi-line output