From 2be7a2f994644276206bee623fe453e4897f14bc Mon Sep 17 00:00:00 2001 From: jnasbyupgrade Date: Tue, 7 Oct 2025 15:27:17 -0500 Subject: [PATCH 01/11] Add Claude Code support files - Add .claude/settings.json with references to test repos - Add CLAUDE.md documenting the meta-framework architecture - Include git commit guidelines Co-Authored-By: Claude --- .claude/settings.json | 8 ++ CLAUDE.md | 168 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 176 insertions(+) create mode 100644 .claude/settings.json create mode 100644 CLAUDE.md diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..8ab782c --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,8 @@ +{ + "permissions": { + "additionalDirectories": [ + "../pgxntool-test/", + "../pgxntool-test-template/" + ] + } +} diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..225aec5 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,168 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Git Commit Guidelines + +**IMPORTANT**: When creating commit messages, do not attribute commits to yourself (Claude). Commit messages should reflect the work being done without AI attribution in the message body. The standard Co-Authored-By trailer is acceptable. + +## Critical: What This Repo Actually Is + +**pgxntool is NOT a standalone project.** It is a meta-framework that exists ONLY to be embedded into PostgreSQL extension projects via `git subtree`. This repo cannot be built, tested, or run directly. + +**Think of it like this**: pgxntool is to PostgreSQL extensions what a Makefile template library is to C projects - it's infrastructure code that gets copied into other projects, not a project itself. + +## Three-Repository Development Pattern + +This codebase uses an unusual three-repository testing pattern: + +1. **pgxntool/** (this repo) - The framework code that gets embedded into extension projects +2. **../pgxntool-test-template/** - A minimal "dummy" PostgreSQL extension that has pgxntool embedded (serves as an example consumer) +3. **../pgxntool-test/** - The test harness that clones the template, exercises pgxntool's functionality, and validates outputs + +**To test changes to pgxntool**, you must: +1. Ensure changes are visible to pgxntool-test (via git commit or using the rsync mechanism in the test harness) +2. Run tests from `../pgxntool-test/` +3. The test harness will clone `../pgxntool-test-template/`, sync in pgxntool changes, and validate behavior + +## How Extension Developers Use pgxntool + +Extension projects include pgxntool via git subtree: + +```bash +git subtree add -P pgxntool --squash git@github.com:decibel/pgxntool.git release +pgxntool/setup.sh +``` + +After setup, their Makefile typically contains just: +```makefile +include pgxntool/base.mk +``` + +## Architecture: Two-Phase Build System + +### Phase 1: Meta Generation (`build_meta.sh`) +- Processes `META.in.json` (template with placeholders/empty values) +- Strips out X_comment fields and empty values +- Produces clean `META.json` + +### Phase 2: Variable Extraction (`meta.mk.sh`) +- Parses `META.json` using `JSON.sh` (a bash-based JSON parser) +- Generates `meta.mk` with Make variables: + - `PGXN` - distribution name + - `PGXNVERSION` - version number + - `EXTENSIONS` - list of extensions provided + - `EXTENSION_*_VERSION` - per-extension versions + - `EXTENSION_VERSION_FILES` - auto-generated versioned SQL files +- `base.mk` includes `meta.mk` via `-include` + +### The Magic of base.mk + +`base.mk` provides a complete PGXS-based build system: +- Auto-detects extension SQL files in `sql/` +- Auto-detects C modules in `src/*.c` +- Auto-detects tests in `test/sql/*.sql` +- Auto-generates versioned extension files (`extension--version.sql`) +- Handles Asciidoc → HTML conversion +- Integrates with PGXN distribution format +- Manages git tagging and release packaging + +## File Structure for Consumer Projects + +Projects using pgxntool follow this layout: +``` +project/ +├── Makefile # include pgxntool/base.mk +├── META.in.json # Template metadata (customize for your extension) +├── META.json # Auto-generated from META.in.json +├── extension.control # Standard PostgreSQL control file +├── pgxntool/ # This repo, embedded via git subtree +├── sql/ +│ └── extension.sql # Base extension SQL +├── src/ # Optional C code (*.c files) +├── test/ +│ ├── deps.sql # Load extension and test dependencies +│ ├── sql/*.sql # Test SQL files +│ └── expected/*.out # Expected test outputs +└── doc/ # Optional docs (*.adoc, *.asciidoc) +``` + +## Commands for Extension Developers (End Users) + +These are the commands extension developers use (documented for context): + +```bash +make # Build extension (generates versioned SQL, docs) +make test # Full test: testdeps → install → installcheck → show diffs +make results # Run tests and update expected output files +make html # Generate HTML from Asciidoc sources +make tag # Create git branch for current META.json version +make dist # Create PGXN .zip (auto-tags, places in ../) +make pgxntool-sync # Update to latest pgxntool via git subtree pull +``` + +## Development Workflow (for pgxntool Contributors) + +When modifying pgxntool: + +1. **Make changes** in this repo (pgxntool/) +2. **Test changes** by running `make test` in `../pgxntool-test/` + - The test harness clones `../pgxntool-test-template/` + - If pgxntool is dirty, it rsyncs your uncommitted changes + - Runs setup, builds, tests, and validates outputs +3. **Examine results** in `../pgxntool-test/results/` and `../pgxntool-test/diffs/` +4. **Update expected outputs** if needed: `make sync-expected` in `../pgxntool-test/` +5. **Commit changes** once tests pass + +## Key Implementation Details + +### PostgreSQL Version Handling +- `MAJORVER` = version × 10 (e.g., 9.6 → 96, 13 → 130) +- Tests use `--load-language=plpgsql` for versions < 13 +- Version detection via `pg_config --version` + +### Test System (pg_regress based) +- Tests in `test/sql/*.sql`, outputs compared to `test/expected/*.out` +- Setup via `test/pgxntool/setup.sql` (loads pgTap and deps.sql) +- `.IGNORE: installcheck` prevents build failures on test errors +- `make results` updates expected outputs after test runs + +### Document Generation +- Auto-detects `asciidoctor` or `asciidoc` +- Generates HTML from `*.adoc` and `*.asciidoc` in `$(DOC_DIRS)` +- HTML required for `make dist`, optional for `make install` +- Template-based rules via `ASCIIDOC_template` + +### Distribution Packaging +- `make dist` creates `../PGXN-VERSION.zip` +- Always creates git branch tag matching version +- Uses `git archive` to package +- Validates repo is clean before tagging + +### Subtree Sync Support +- `make pgxntool-sync` pulls latest release +- Multiple sync targets: release, stable, local variants +- Uses `git subtree pull --squash` +- Requires clean repo (no uncommitted changes) + +## Critical Gotchas + +1. **Empty Variables**: If `DOCS` or `MODULES` is empty, base.mk sets to empty to prevent PGXS errors +2. **testdeps Pattern**: Never add recipes to `testdeps` - create separate target and make it a prerequisite +3. **META.json is Generated**: Always edit `META.in.json`, never `META.json` directly +4. **Control File Versions**: No automatic validation that `.control` matches `META.json` version +5. **PGXNTOOL_NO_PGXS_INCLUDE**: Setting this skips PGXS inclusion (for special scenarios) +6. **Distribution Placement**: `.zip` files go in parent directory (`../`) to avoid repo clutter + +## Scripts + +- **setup.sh** - Initializes pgxntool in a new extension project (copies templates, creates directories) +- **build_meta.sh** - Strips empty fields from META.in.json to create META.json +- **meta.mk.sh** - Parses META.json via JSON.sh and generates meta.mk with Make variables +- **JSON.sh** - Third-party bash JSON parser (MIT licensed) +- **safesed** - Utility for safe sed operations + +## Related Repositories + +- **../pgxntool-test/** - Test harness for validating pgxntool functionality +- **../pgxntool-test-template/** - Minimal extension project used as test subject From 0e83d4316d0f4ab18cb9c6f95821cd59a2268a64 Mon Sep 17 00:00:00 2001 From: jnasbyupgrade Date: Tue, 7 Oct 2025 16:28:56 -0500 Subject: [PATCH 02/11] Add Claude local settings to _.gitignore This ensures all projects using pgxntool will ignore Claude Code local configuration files Co-Authored-By: Claude --- _.gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/_.gitignore b/_.gitignore index 3eb345a..82933e8 100644 --- a/_.gitignore +++ b/_.gitignore @@ -1,6 +1,9 @@ # Editor files .*.swp +# Claude Code local settings +.claude/*.local.json + # Explicitly exclude META.json! !/META.json From b096d7958e534b24009ff9ce495ef832224df5e0 Mon Sep 17 00:00:00 2001 From: jnasbyupgrade Date: Tue, 7 Oct 2025 16:32:56 -0500 Subject: [PATCH 03/11] Exclude *.md files from git archive Add *.md to export-ignore to prevent markdown files (including CLAUDE.md) from being included in extension distributions Co-Authored-By: Claude --- .gitattributes | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitattributes b/.gitattributes index c602ea0..ed57585 100644 --- a/.gitattributes +++ b/.gitattributes @@ -2,3 +2,4 @@ *.asc export-ignore *.adoc export-ignore *.html export-ignore +*.md export-ignore From 313e5f254f8282731c4fedf124d27eb8ca87ced6 Mon Sep 17 00:00:00 2001 From: jnasbyupgrade Date: Tue, 11 Nov 2025 13:39:21 -0600 Subject: [PATCH 04/11] Add shared commit.md and document META.json generation Add `.claude/commands/commit.md` with comprehensive commit workflow that will be shared with pgxntool-test via symlink. This ensures consistent commit standards across both repos. Document META.json generation process in `build_meta.sh` to explain why we generate from template (PGXN.org doesn't like empty optional fields) and future possibilities (could generate control files from template). Co-Authored-By: Claude --- .claude/commands/commit.md | 78 ++++++++++++++++++++++++++++++++++++++ build_meta.sh | 19 ++++++++++ 2 files changed, 97 insertions(+) create mode 100644 .claude/commands/commit.md diff --git a/.claude/commands/commit.md b/.claude/commands/commit.md new file mode 100644 index 0000000..5ddbd74 --- /dev/null +++ b/.claude/commands/commit.md @@ -0,0 +1,78 @@ +--- +description: Create a git commit following project standards and safety protocols +allowed-tools: Bash(git status:*), Bash(git log:*), Bash(git add:*), Bash(git diff:*), Bash(git commit:*), Bash(make test:*) +--- + +# commit + +Create a git commit following all project standards and safety protocols for pgxntool-test. + +**CRITICAL REQUIREMENTS:** + +1. **Git Safety**: Never update `git config`, never force push to `main`/`master`, never skip hooks unless explicitly requested + +2. **Commit Attribution**: Do NOT add "Generated with Claude Code" to commit message body. The standard Co-Authored-By trailer is acceptable per project CLAUDE.md. + +3. **Testing**: ALL tests must pass before committing: + - Run `make test` + - Check the output carefully for any "not ok" lines + - Count passing vs total tests + - **If ANY tests fail: STOP. Do NOT commit. Ask the user what to do.** + - There is NO such thing as an "acceptable" failing test + - Do NOT rationalize failures as "pre-existing" or "unrelated" + +**WORKFLOW:** + +1. Run in parallel: `git status`, `git diff --stat`, `git log -10 --oneline` + +2. Check test status - THIS IS MANDATORY: + - Run `make test 2>&1 | tee /tmp/test-output.txt` + - Check for failing tests: `grep "^not ok" /tmp/test-output.txt` + - If ANY tests fail: STOP immediately and inform the user + - Only proceed if ALL tests pass + +3. Analyze changes and draft concise commit message following this repo's style: + - Look at `git log -10 --oneline` to match existing style + - Be factual and direct (e.g., "Fix BATS dist test to create its own distribution") + - Focus on "why" when it adds value, otherwise just describe "what" + - List items in roughly decreasing order of impact + - Keep related items grouped together + - **In commit messages**: Wrap all code references in backticks - filenames, paths, commands, function names, variables, make targets, etc. + - Examples: `helpers.bash`, `make test-recursion`, `setup_sequential_test()`, `TEST_REPO`, `.envs/`, `01-meta.bats` + - Prevents markdown parsing issues and improves clarity + +4. **PRESENT the proposed commit message to the user and WAIT for approval before proceeding** + +5. After receiving approval, stage changes appropriately using `git add` + +6. **VERIFY staged files with `git status`**: + - If user did NOT specify a subset: Confirm ALL modified/untracked files are staged + - If user specified only certain files: Confirm ONLY those files are staged + - STOP and ask user if staging doesn't match intent + +7. After verification, commit using `HEREDOC` format: +```bash +git commit -m "$(cat <<'EOF' +Subject line (imperative mood, < 72 chars) + +Additional context if needed, wrapped at 72 characters. + +Co-Authored-By: Claude +EOF +)" +``` + +8. Run `git status` after commit to verify success + +9. If pre-commit hook modifies files: Check authorship (`git log -1 --format='%an %ae'`) and branch status, then amend if safe or create new commit + +**REPOSITORY CONTEXT:** + +This is pgxntool-test, a test harness for the pgxntool framework. Key facts: +- Tests live in `tests/` directory +- `.envs/` contains test environments (gitignored) + +**RESTRICTIONS:** +- DO NOT push unless explicitly asked +- DO NOT commit files with actual secrets (`.env`, `credentials.json`, etc.) +- Never use `-i` flags (`git commit -i`, `git rebase -i`, etc.) diff --git a/build_meta.sh b/build_meta.sh index 70d2273..1576f68 100755 --- a/build_meta.sh +++ b/build_meta.sh @@ -1,5 +1,24 @@ #!/bin/bash +# Build META.json from META.in.json template +# +# WHY META.in.json EXISTS: +# META.in.json serves as a template that: +# 1. Shows all possible PGXN metadata fields (both required and optional) with comments +# 2. Can have empty placeholder fields like "key": "" or "key": [ "", "" ] +# 3. Users edit this to fill in their extension's metadata +# +# WHY WE GENERATE META.json: +# The reason we generate META.json from a template is to eliminate empty fields that +# are optional; PGXN.org gets upset about them. In the future it's possible we'll do +# more here (for example, if we added more info to the template we could use it to +# generate control files). +# +# WHY WE COMMIT META.json: +# PGXN.org requires META.json to be present in submitted distributions. We choose +# to commit it to git instead of manually adding it to distributions for simplicity +# (and since it generally only changes once for each new version). + set -e error () { From f58dd7da03c1d18993ea72ab23c102d496dd006e Mon Sep 17 00:00:00 2001 From: jnasbyupgrade Date: Tue, 11 Nov 2025 16:32:11 -0600 Subject: [PATCH 05/11] Add misc Claude stuff --- .claude/settings.json | 12 ++++++++++++ .gitignore | 1 + CLAUDE.md | 1 + 3 files changed, 14 insertions(+) diff --git a/.claude/settings.json b/.claude/settings.json index 8ab782c..e7d75ad 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -1,5 +1,17 @@ { "permissions": { + "allow": [ + "Bash(cat:*)", + "Bash(make test:*)", + "Bash(tee:*)", + "Bash(echo:*)", + "Bash(git show:*)", + "Bash(git log:*)", + "Bash(ls:*)", + "Bash(find:*)", + "Bash(git checkout:*)", + "Bash(head:*)" + ], "additionalDirectories": [ "../pgxntool-test/", "../pgxntool-test-template/" diff --git a/.gitignore b/.gitignore index a01ee28..5ffb236 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ .*.swp +.claude/*.local.json diff --git a/CLAUDE.md b/CLAUDE.md index 225aec5..72fdb6a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -166,3 +166,4 @@ When modifying pgxntool: - **../pgxntool-test/** - Test harness for validating pgxntool functionality - **../pgxntool-test-template/** - Minimal extension project used as test subject +- Never produce any kind of metrics or estimates unless you have data to back them up. If you do have data you MUST reference it. \ No newline at end of file From 067b7d4bd3ba8eb2a960e29eb9b7467c19ac597e Mon Sep 17 00:00:00 2001 From: jnasbyupgrade Date: Thu, 13 Nov 2025 16:51:19 -0600 Subject: [PATCH 06/11] Document testing guidelines and critical warnings Add testing section to `CLAUDE.md` with critical rules: never use `make installcheck` directly, never run `make results` without verification, database connection requirements. Enhance `README.asc` to recommend `make test`, document `make results` verification workflow, and emphasize pgTap benefits. Co-Authored-By: Claude --- CLAUDE.md | 27 +++++++++++++++++++++++++++ README.asc | 12 +++++++++++- 2 files changed, 38 insertions(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 72fdb6a..036df2e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -101,6 +101,33 @@ make dist # Create PGXN .zip (auto-tags, places in ../) make pgxntool-sync # Update to latest pgxntool via git subtree pull ``` +## Testing with pgxntool + +### Critical Testing Rules + +**NEVER use `make installcheck` directly**. Always use `make test` instead. The `make test` target ensures: +- Clean builds before testing +- Proper test isolation +- Correct test dependency installation +- Proper cleanup and result comparison + +**Database Connection Requirement**: PostgreSQL must be running before executing `make test`. If you get connection errors (e.g., "could not connect to server"), stop and ask the user to start PostgreSQL. + +**Claude Code MUST NEVER run `make results`**. This target updates test expected output files and requires manual human verification of test changes before execution. The workflow is: +1. Human runs `make test` and examines diffs +2. Human manually verifies changes are correct +3. Human manually runs `make results` to update expected files + +### Test Output Mechanics + +pgxntool uses PostgreSQL's pg_regress test framework: +- **Actual test output**: Written to `test/results/` directory +- **Expected output**: Stored in `test/expected/` directory +- **Test comparison**: `make test` compares actual vs expected and shows diffs +- **Updating expectations**: `make results` copies `test/results/` → `test/expected/` + +When tests fail, examine the diff output carefully. The actual test output in `test/results/` shows what your code produced, while `test/expected/` shows what was expected. + ## Development Workflow (for pgxntool Contributors) When modifying pgxntool: diff --git a/README.asc b/README.asc index c2c6683..50797e5 100644 --- a/README.asc +++ b/README.asc @@ -41,6 +41,8 @@ This will build any .html files that can be created. See <<_Document_Handling>>. === test Runs unit tests via the PGXS `installcheck` target. Unlike a simple `make installcheck` though, the `test` rule has the following prerequisites: clean testdeps install installcheck. All of those are PGXS rules, except for `testdeps`. +NOTE: While you can still run `make installcheck` or any other valid PGXS make target directly, it's recommended to use `make test` when using pgxntool. The `test` target ensures clean builds, proper test isolation, and correct dependency installation. + === testdeps This rule allows you to ensure certain actions have taken place before running tests. By default it has a single prerequisite, `pgtap`, which will attempt to install http://pgtap.org[pgtap] from PGXN. This depneds on having the pgxn client installed. @@ -60,10 +62,18 @@ If you want to over-ride the default dependency on `pgtap` you should be able to WARNING: It will probably cause problems if you try to create a `testdeps` rule that has a recipe. Instead of doing that, put the recipe in a separate rule and make that rule a prerequisite of `testdeps` as show in the example. === results -Because `make test` ultimately runs `installcheck`, it's using the Postgres test suite. Unfortunately, that suite is based on running `diff` between a raw output file and expected results. I *STRONGLY* recommend you use http://pgtap.org[pgTap] instead! The extra effort of learning pgTap will quickly pay for itself. https://github.com/decibel/trunklet-format/blob/master/test/sql/base.sql[This example] might help get you started. +Because `make test` ultimately runs `installcheck`, it's using the Postgres test suite. Unfortunately, that suite is based on running `diff` between a raw output file and expected results. I *STRONGLY* recommend you use http://pgtap.org[pgTap] instead! With pgTap, it's MUCH easier to determine whether a test is passing or not - tests explicitly pass or fail rather than requiring you to examine diff output. The extra effort of learning pgTap will quickly pay for itself. https://github.com/decibel/trunklet-format/blob/master/test/sql/base.sql[This example] might help get you started. No matter what method you use, once you know that all your tests are passing correctly, you need to create or update the test output expected files. `make results` does that for you. +IMPORTANT: *`make results` requires manual verification first*. The correct workflow is: + +1. Run `make test` and examine the diff output +2. Manually verify that the differences are correct and expected +3. Only then run `make results` to update the expected output files in `test/expected/` + +Never run `make results` without first verifying the test changes are correct. The `results` target copies files from `test/results/` to `test/expected/`, so running it blindly will make incorrect output become the new expected behavior. + === tag `make tag` will create a git branch for the current version of your extension, as determined by the META.json file. The reason to do this is so you can always refer to the exact code that went into a released version. From 484523de63b08bb7547a6abb3c68e0fe0607fa91 Mon Sep 17 00:00:00 2001 From: jnasbyupgrade Date: Wed, 10 Dec 2025 18:07:30 -0600 Subject: [PATCH 07/11] Exclude .claude/ from distributions --- .gitattributes | 1 + README.html | 802 ------------------------------------------------- 2 files changed, 1 insertion(+), 802 deletions(-) delete mode 100644 README.html diff --git a/.gitattributes b/.gitattributes index ed57585..5b9a578 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,4 +1,5 @@ .gitattributes export-ignore +.claude/ export-ignore *.asc export-ignore *.adoc export-ignore *.html export-ignore diff --git a/README.html b/README.html deleted file mode 100644 index ae4a597..0000000 --- a/README.html +++ /dev/null @@ -1,802 +0,0 @@ - - - - - - - - -PGXNtool - - - - - -
-
-
-
-

PGXNtool is meant to make developing new Postgres extensions for PGXN easier.

-
-
-

Currently, it consists a base Makefile that you can include instead of writing your own, a template META.json, and some test framework. More features will be added over time.

-
-
-

If you find any bugs or have ideas for improvements, please open an issue.

-
-
-
-
-

1. Install

-
-
-

This assumes that you’ve already initialized your extension in git.

-
-
- - - - - -
-
Note
-		 -The --squash is important! Otherwise you’ll clutter your repo with a bunch of commits you probably don’t want. -
-
-
-
-
git subtree add -P pgxntool --squash git@github.com:decibel/pgxntool.git release
-pgxntool/setup.sh
-
-
-
-

TODO: Create a nice script that will init a new project for you.

-
-
-
-
-

2. Usage

-
-
-

Typically, you can just create a simple Makefile that does nothing but include base.mk:

-
-
-
-
include pgxntool/base.mk
-
-
-
-
-
-

3. make targets

-
-
-

These are the make targets that are provided by base.mk

-
-
- - - - - -
-
Note
-		 -all the targets normally provided by Postgres PGXS still work. -
-
-
-

3.1. html

-
-

This will build any .html files that can be created. See [_Document_Handling].

-
-
-
-

3.2. test

-
-

Runs unit tests via the PGXS installcheck target. Unlike a simple make installcheck though, the test rule has the following prerequisites: clean testdeps install installcheck. All of those are PGXS rules, except for testdeps.

-
-
-
-

3.3. testdeps

-
-

This rule allows you to ensure certain actions have taken place before running tests. By default it has a single prerequisite, pgtap, which will attempt to install pgtap from PGXN. This depneds on having the pgxn client installed.

-
-
-

You can add any other dependencies you want by simply adding another testdeps rule. For example:

-
-
-

testdeps example from test_factory

-
-
-
-
testdeps: check_control
-
-.PHONY: check_control
-check_control:
-	grep -q "requires = 'pgtap, test_factory'" test_factory_pgtap.control
-
-
-
-

If you want to over-ride the default dependency on pgtap you should be able to do that with a makefile override. If you need help with that, please open an issue.

-
-
- - - - - -
-
Warning
-		 -It will probably cause problems if you try to create a testdeps rule that has a recipe. Instead of doing that, put the recipe in a separate rule and make that rule a prerequisite of testdeps as show in the example. -
-
-
-
-

3.4. results

-
-

Because make test ultimately runs installcheck, it’s using the Postgres test suite. Unfortunately, that suite is based on running diff between a raw output file and expected results. I STRONGLY recommend you use pgTap instead! The extra effort of learning pgTap will quickly pay for itself. This example might help get you started.

-
-
-

No matter what method you use, once you know that all your tests are passing correctly, you need to create or update the test output expected files. make results does that for you.

-
-
-
-

3.5. tag

-
-

make tag will create a git branch for the current version of your extension, as determined by the META.json file. The reason to do this is so you can always refer to the exact code that went into a released version.

-
-
-

If there’s already a tag for the current version that probably means you forgot to update META.json, so you’ll get an error. If you’re certain you want to over-write the tag, you can do make forcetag, which removes the existing tag (via make rmtag) and creates a new one.

-
-
- - - - - -
-
Warning
-		 -You will be very unhappy if you forget to update the .control file for your extension! There is an open issue to improve this. -
-
-
-
-

3.6. dist

-
-

make dist will create a .zip file for your current version that you can upload to PGXN. The file is named after the PGXN name and version (the top-level "name" and "version" attributes in META.json). The .zip file is placed in the parent directory so as not to clutter up your git repo.

-
-
- - - - - -
-
Note
-		 -Part of the clean recipe is cleaning up these .zip files. If you accidentally clean before uploading, just run make dist-only. -
-
-
-
-

3.7. pgxntool-sync

-
-

This rule will pull down the latest released version of PGXNtool via git subtree pull.

-
-
- - - - - -
-
Note
-		 -Your repository must be clean (no modified files) in order to run this. Running this command will produce a git commit of the merge. -
-
-
- - - - - -
-
Tip
-		 -There is also a pgxntool-sync-% rule if you need to do more advanced things. -
-
-
-
-
-
-

4. Document Handling

-
-
-

PGXNtool supports generation and installation of document files. There are several variables and rules that control this behavior.

-
-
-

It is recommended that you commit any generated documentation files (such as HTML generated from Asciidoc) into git. -That way users will have these files installed when they install your extension. -If any generated files are missing (or out-of-date) during installation, PGXNtool will build them if Asciidoc is present on the system.

-
-
-

4.1. Document Variables

-
-
-
DOC_DIRS
-
-

Directories to look for documents in. -Defined as += doc.

-
-
DOCS
-
-

PGXS variable. -See The DOCS variable below.

-
-
DOCS_HTML
-
-

Document HTML files. -PGXNtool appends `$(ASCIIDOC_HTML) to this variable.

-
-
ASCIIDOC
-
-

Location of asciidoc or equivalent executable. -If not set PGXNtool will search for first asciidoctor, then asciidoc.

-
-
ASCIIDOC_EXTS
-
-

File extensions to consider as Asciidoc. -Defined as += adoc asciidoc.

-
-
ASCIIDOC_FILES
-
-

Asciidoc input files. -PGXNtool searches each $(DOC_DIRS) directory, looking for files with any $(ASCIIDOC_EXTS) extension. -Any files found are added to ASCIIDOC_FILES using +=.

-
-
ASCIIDOC_FLAGS
-
-

Additional flags to pass to Asciidoc.

-
-
ASCIIDOC_HTML
-
-

PGXNtool replaces each $(ASCIIDOC_EXTS) in $(ASCIIDOC_FILES) with html. -The result is appended to ASCIIDOC_HTML using +=.

-
-
-
-
-
-

4.2. Document Rules

-
-

If Asciidoc is found (or $(ASCIIDOC) is set), the html rule will be added as a prerequisite to the install and installchec rules. -That will ensure that docs are generated for install and test, but only if Asciidoc is available. -The dist rule will always depend on html though, to ensure html files are up-to-date before creating a distribution.

-
-
-

The html rule simply depends on `$(ASCIIDOC_HTML). -This rule is always present.

-
-
-

For each Asciidoc extension in $(ASCIIDOC_EXTS) a rule is generated to build a .html file from that extension using $(ASCIIDOC). -These rules are generated from ASCIIDOC_template:

-
-
-
ASCIIDOC_template
-
-
define ASCIIDOC_template
-%.html: %.$(1) (1)
-ifndef ASCIIDOC
-	$$(warning Could not find "asciidoc" or "asciidoctor". Add one of them to your PATH,)
-	$$(warning or set ASCIIDOC to the correct location.)
-	$$(error Could not build %$$@)
-endif # ifndef ASCIIDOC
-	$$(ASCIIDOC) $$(ASCIIDOC_FLAGS) $$<
-endef # define ASCIIDOC_template
-
-
-
-
    -
  1. -

    $(1) is replaced by the extension.

    -
    2. -
-
-
-

These rules will always exist, even if $(ASCIIDOC) isn’t set (ie: if Asciidoc wasn’t found on the system). -These rules will throw an error if they are run if $(ASCIIDOC) isn’t defined. -On a normal user system that should never happen, because the html rule won’t be included in install or installcheck.

-
-
-
-

4.3. The DOCS variable

-
-

This variable has special meaning to PGXS. -See the Postgres documentation for full details.

-
-
-

If DOCS is defined when PGXS is included then rules will be added to install everything defined by $(DOCS) in PREFIX/share/doc/extension.

-
-
- - - - - -
-
Note
-		 -If DOCS is defined but empty some of the PGXS targets will error out. -Because of this, base.mk will forcibly define it to be NULL if it’s empty. -
-
-
-

PGXNtool appends all files found in all $(DOC_DIRS) to DOCS.

-
-
-
-
-
-

5. Copyright

-
-
-

Copyright (c) 2015 Jim Nasby <Jim.Nasby@BlueTreble.com>

-
-
-

PGXNtool is released under a BSD license. Note that it includes JSON.sh, which is released under a MIT license.

-
-
-
-
- - - \ No newline at end of file From 411033ade2f5ed4aecf96ce80eff88822aa33d6c Mon Sep 17 00:00:00 2001 From: jnasbyupgrade Date: Fri, 12 Dec 2025 15:18:22 -0600 Subject: [PATCH 08/11] Improve make results handling of .source files and add .gitattributes validation - Add `make_results.sh` script to handle copying results while respecting `output/*.source` files as source of truth - Update `base.mk` to properly handle ephemeral files from `.source` files: - Track `TEST_OUT_SOURCE_FILES` and `TEST_EXPECTED_FROM_SOURCE` - Add ephemeral files to `EXTRA_CLEAN` so `make clean` removes them - Create `test/results/` directory automatically - Remove rule that created `test/output/` directory (it's an optional input) - Add validation in `dist-only` target to ensure `.gitattributes` is committed before creating distribution (git archive only respects export-ignore for committed files) - Update `README.asc` and `CLAUDE.md` to document that development should be done from pgxntool-test repository Co-Authored-By: Claude --- CLAUDE.md | 14 + README.asc | 4 + README.html | 868 ++++++++++++++++++++++++++++++++++++++++++++++++ base.mk | 44 ++- make_results.sh | 28 ++ 5 files changed, 948 insertions(+), 10 deletions(-) create mode 100644 README.html create mode 100755 make_results.sh diff --git a/CLAUDE.md b/CLAUDE.md index 036df2e..57812d2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,6 +12,20 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co **Think of it like this**: pgxntool is to PostgreSQL extensions what a Makefile template library is to C projects - it's infrastructure code that gets copied into other projects, not a project itself. +## Development Workflow: Work from pgxntool-test + +**CRITICAL**: All development work on pgxntool should be done from the `../pgxntool-test/` repository, NOT from this repository. + +**Why?** Because: +- This repository contains ONLY the framework files that get embedded into extension projects +- All development tools, test infrastructure, and convenience scripts live in `../pgxntool-test/` +- Adding development tools here would pollute every extension project that uses pgxntool + +**What this means**: +- Clone and work from `../pgxntool-test/` when developing pgxntool +- Edit files in `../pgxntool/` but run all commands from `../pgxntool-test/` +- All testing, documentation of testing, and development utilities belong in pgxntool-test + ## Three-Repository Development Pattern This codebase uses an unusual three-repository testing pattern: diff --git a/README.asc b/README.asc index 50797e5..c077fe4 100644 --- a/README.asc +++ b/README.asc @@ -23,6 +23,10 @@ pgxntool/setup.sh TODO: Create a nice script that will init a new project for you. +== Development + +If you want to contribute to pgxntool development, work from the https://github.com/decibel/pgxntool-test[pgxntool-test] repository, not from this repository. That repository contains the test infrastructure and development tools needed to validate changes to pgxntool. This repository contains only the framework files that get embedded into extension projects via `git subtree`. + == Usage Typically, you can just create a simple Makefile that does nothing but include base.mk: diff --git a/README.html b/README.html new file mode 100644 index 0000000..975d3f9 --- /dev/null +++ b/README.html @@ -0,0 +1,868 @@ + + + + + + + + +PGXNtool + + + + + +
+
+
+
+

PGXNtool is meant to make developing new Postgres extensions for PGXN easier.

+
+
+

Currently, it consists a base Makefile that you can include instead of writing your own, a template META.json, and some test framework. More features will be added over time.

+
+
+

If you find any bugs or have ideas for improvements, please open an issue.

+
+
+
+
+

1. Install

+
+
+

This assumes that you’ve already initialized your extension in git.

+
+
+ + + + + +
+
Note
+		 +The --squash is important! Otherwise you’ll clutter your repo with a bunch of commits you probably don’t want. +
+
+
+
+
git subtree add -P pgxntool --squash git@github.com:decibel/pgxntool.git release
+pgxntool/setup.sh
+
+
+
+

TODO: Create a nice script that will init a new project for you.

+
+
+
+
+

2. Development

+
+
+

If you want to contribute to pgxntool development, work from the pgxntool-test repository, not from this repository. That repository contains the test infrastructure and development tools needed to validate changes to pgxntool. This repository contains only the framework files that get embedded into extension projects via git subtree.

+
+
+
+
+

3. Usage

+
+
+

Typically, you can just create a simple Makefile that does nothing but include base.mk:

+
+
+
+
include pgxntool/base.mk
+
+
+
+
+
+

4. make targets

+
+
+

These are the make targets that are provided by base.mk

+
+
+ + + + + +
+
Note
+		 +all the targets normally provided by Postgres PGXS still work. +
+
+
+

4.1. html

+
+

This will build any .html files that can be created. See [_Document_Handling].

+
+
+
+

4.2. test

+
+

Runs unit tests via the PGXS installcheck target. Unlike a simple make installcheck though, the test rule has the following prerequisites: clean testdeps install installcheck. All of those are PGXS rules, except for testdeps.

+
+
+ + + + + +
+
Note
+		 +While you can still run make installcheck or any other valid PGXS make target directly, it’s recommended to use make test when using pgxntool. The test target ensures clean builds, proper test isolation, and correct dependency installation. +
+
+
+
+

4.3. testdeps

+
+

This rule allows you to ensure certain actions have taken place before running tests. By default it has a single prerequisite, pgtap, which will attempt to install pgtap from PGXN. This depneds on having the pgxn client installed.

+
+
+

You can add any other dependencies you want by simply adding another testdeps rule. For example:

+
+
+

testdeps example from test_factory

+
+
+
+
testdeps: check_control
+
+.PHONY: check_control
+check_control:
+	grep -q "requires = 'pgtap, test_factory'" test_factory_pgtap.control
+
+
+
+

If you want to over-ride the default dependency on pgtap you should be able to do that with a makefile override. If you need help with that, please open an issue.

+
+
+ + + + + +
+
Warning
+		 +It will probably cause problems if you try to create a testdeps rule that has a recipe. Instead of doing that, put the recipe in a separate rule and make that rule a prerequisite of testdeps as show in the example. +
+
+
+
+

4.4. results

+
+

Because make test ultimately runs installcheck, it’s using the Postgres test suite. Unfortunately, that suite is based on running diff between a raw output file and expected results. I STRONGLY recommend you use pgTap instead! With pgTap, it’s MUCH easier to determine whether a test is passing or not - tests explicitly pass or fail rather than requiring you to examine diff output. The extra effort of learning pgTap will quickly pay for itself. This example might help get you started.

+
+
+

No matter what method you use, once you know that all your tests are passing correctly, you need to create or update the test output expected files. make results does that for you.

+
+
+ + + + + +
+
Important
+		 +make results requires manual verification first. The correct workflow is: +
+
+
+
    +
  1. +

    Run make test and examine the diff output

    +
    2. +
  2. +

    Manually verify that the differences are correct and expected

    +
    3. +
  3. +

    Only then run make results to update the expected output files in test/expected/

    +
    4. +
+
+
+

Never run make results without first verifying the test changes are correct. The results target copies files from test/results/ to test/expected/, so running it blindly will make incorrect output become the new expected behavior.

+
+
+
+

4.5. tag

+
+

make tag will create a git branch for the current version of your extension, as determined by the META.json file. The reason to do this is so you can always refer to the exact code that went into a released version.

+
+
+

If there’s already a tag for the current version that probably means you forgot to update META.json, so you’ll get an error. If you’re certain you want to over-write the tag, you can do make forcetag, which removes the existing tag (via make rmtag) and creates a new one.

+
+
+ + + + + +
+
Warning
+		 +You will be very unhappy if you forget to update the .control file for your extension! There is an open issue to improve this. +
+
+
+
+

4.6. dist

+
+

make dist will create a .zip file for your current version that you can upload to PGXN. The file is named after the PGXN name and version (the top-level "name" and "version" attributes in META.json). The .zip file is placed in the parent directory so as not to clutter up your git repo.

+
+
+ + + + + +
+
Note
+		 +Part of the clean recipe is cleaning up these .zip files. If you accidentally clean before uploading, just run make dist-only. +
+
+
+
+

4.7. pgxntool-sync

+
+

This rule will pull down the latest released version of PGXNtool via git subtree pull.

+
+
+ + + + + +
+
Note
+		 +Your repository must be clean (no modified files) in order to run this. Running this command will produce a git commit of the merge. +
+
+
+ + + + + +
+
Tip
+		 +There is also a pgxntool-sync-% rule if you need to do more advanced things. +
+
+
+
+
+
+

5. Document Handling

+
+
+

PGXNtool supports generation and installation of document files. There are several variables and rules that control this behavior.

+
+
+

It is recommended that you commit any generated documentation files (such as HTML generated from Asciidoc) into git. +That way users will have these files installed when they install your extension. +If any generated files are missing (or out-of-date) during installation, PGXNtool will build them if Asciidoc is present on the system.

+
+
+

5.1. Document Variables

+
+
+
DOC_DIRS
+
+

Directories to look for documents in. +Defined as += doc.

+
+
DOCS
+
+

PGXS variable. +See The DOCS variable below.

+
+
DOCS_HTML
+
+

Document HTML files. +PGXNtool appends `$(ASCIIDOC_HTML) to this variable.

+
+
ASCIIDOC
+
+

Location of asciidoc or equivalent executable. +If not set PGXNtool will search for first asciidoctor, then asciidoc.

+
+
ASCIIDOC_EXTS
+
+

File extensions to consider as Asciidoc. +Defined as += adoc asciidoc.

+
+
ASCIIDOC_FILES
+
+

Asciidoc input files. +PGXNtool searches each $(DOC_DIRS) directory, looking for files with any $(ASCIIDOC_EXTS) extension. +Any files found are added to ASCIIDOC_FILES using +=.

+
+
ASCIIDOC_FLAGS
+
+

Additional flags to pass to Asciidoc.

+
+
ASCIIDOC_HTML
+
+

PGXNtool replaces each $(ASCIIDOC_EXTS) in $(ASCIIDOC_FILES) with html. +The result is appended to ASCIIDOC_HTML using +=.

+
+
+
+
+
+

5.2. Document Rules

+
+

If Asciidoc is found (or $(ASCIIDOC) is set), the html rule will be added as a prerequisite to the install and installchec rules. +That will ensure that docs are generated for install and test, but only if Asciidoc is available. +The dist rule will always depend on html though, to ensure html files are up-to-date before creating a distribution.

+
+
+

The html rule simply depends on `$(ASCIIDOC_HTML). +This rule is always present.

+
+
+

For each Asciidoc extension in $(ASCIIDOC_EXTS) a rule is generated to build a .html file from that extension using $(ASCIIDOC). +These rules are generated from ASCIIDOC_template:

+
+
+
ASCIIDOC_template
+
+
define ASCIIDOC_template
+%.html: %.$(1) # (1)
+ifndef ASCIIDOC
+	$$(warning Could not find "asciidoc" or "asciidoctor". Add one of them to your PATH,)
+	$$(warning or set ASCIIDOC to the correct location.)
+	$$(error Could not build %$$@)
+endif # ifndef ASCIIDOC
+	$$(ASCIIDOC) $$(ASCIIDOC_FLAGS) $$<
+endef # define ASCIIDOC_template
+
+
+
+
    +
  1. +

    $(1) is replaced by the extension.

    +
    2. +
+
+
+

These rules will always exist, even if $(ASCIIDOC) isn’t set (ie: if Asciidoc wasn’t found on the system). +These rules will throw an error if they are run if $(ASCIIDOC) isn’t defined. +On a normal user system that should never happen, because the html rule won’t be included in install or installcheck.

+
+
+
+

5.3. The DOCS variable

+
+

This variable has special meaning to PGXS. +See the Postgres documentation for full details.

+
+
+

If DOCS is defined when PGXS is included then rules will be added to install everything defined by $(DOCS) in PREFIX/share/doc/extension.

+
+
+ + + + + +
+
Note
+		 +If DOCS is defined but empty some of the PGXS targets will error out. +Because of this, base.mk will forcibly define it to be NULL if it’s empty. +
+
+
+

PGXNtool appends all files found in all $(DOC_DIRS) to DOCS.

+
+
+
+
+
+

6. Copyright

+
+
+

Copyright (c) 2015 Jim Nasby <Jim.Nasby@BlueTreble.com>

+
+
+

PGXNtool is released under a BSD license. Note that it includes JSON.sh, which is released under a MIT license.

+
+
+
+
+ + + \ No newline at end of file diff --git a/base.mk b/base.mk index a976ebb..ac661c5 100644 --- a/base.mk +++ b/base.mk @@ -31,10 +31,16 @@ PG_CONFIG ?= pg_config TESTDIR ?= test TESTOUT ?= $(TESTDIR) TEST_SOURCE_FILES += $(wildcard $(TESTDIR)/input/*.source) +TEST_OUT_SOURCE_FILES += $(wildcard $(TESTDIR)/output/*.source) TEST_OUT_FILES = $(subst input,output,$(TEST_SOURCE_FILES)) TEST_SQL_FILES += $(wildcard $(TESTDIR)/sql/*.sql) TEST_RESULT_FILES = $(patsubst $(TESTDIR)/sql/%.sql,$(TESTDIR)/expected/%.out,$(TEST_SQL_FILES)) TEST_FILES = $(TEST_SOURCE_FILES) $(TEST_SQL_FILES) +# Ephemeral files generated from source files (should be cleaned) +# input/*.source → sql/*.sql (converted by pg_regress) +TEST_SQL_FROM_SOURCE = $(patsubst $(TESTDIR)/input/%.source,$(TESTDIR)/sql/%.sql,$(TEST_SOURCE_FILES)) +# output/*.source → expected/*.out (converted by pg_regress) +TEST_EXPECTED_FROM_SOURCE = $(patsubst $(TESTDIR)/output/%.source,$(TESTDIR)/expected/%.out,$(TEST_OUT_SOURCE_FILES)) REGRESS = $(sort $(notdir $(subst .source,,$(TEST_FILES:.sql=)))) # Sort is to get unique list REGRESS_OPTS = --inputdir=$(TESTDIR) --outputdir=$(TESTOUT) # See additional setup below MODULES = $(patsubst %.c,%,$(wildcard src/*.c)) @@ -42,7 +48,7 @@ ifeq ($(strip $(MODULES)),) MODULES =# Set to NUL so PGXS doesn't puke endif -EXTRA_CLEAN = $(wildcard ../$(PGXN)-*.zip) $(EXTENSION_VERSION_FILES) +EXTRA_CLEAN = $(wildcard ../$(PGXN)-*.zip) $(EXTENSION_VERSION_FILES) $(TEST_SQL_FROM_SOURCE) $(TEST_EXPECTED_FROM_SOURCE) # Get Postgres version, as well as major (9.4, etc) version. # NOTE! In at least some versions, PGXS defines VERSION, so we intentionally don't use that variable @@ -70,7 +76,7 @@ DATA += $(wildcard *.control) # Don't have installcheck bomb on error .IGNORE: installcheck -installcheck: $(TEST_RESULT_FILES) $(TEST_OUT_FILES) $(TEST_SQL_FILES) $(TEST_SOURCE_FILES) +installcheck: $(TEST_RESULT_FILES) $(TEST_SQL_FILES) $(TEST_SOURCE_FILES) | $(TESTDIR)/sql/ $(TESTDIR)/expected/ $(TESTOUT)/results/ # # TEST SUPPORT @@ -89,25 +95,36 @@ test: testdeps install installcheck # make results: runs `make test` and copy all result files to expected # DO NOT RUN THIS UNLESS YOU'RE CERTAIN ALL YOUR TESTS ARE PASSING! +# +# pg_regress workflow: +# 1. Converts input/*.source → sql/*.sql (with token substitution) +# 2. Converts output/*.source → expected/*.out (with token substitution) +# 3. Runs tests, saving actual output in results/ +# 4. Compares results/ with expected/ +# +# NOTE: Both input/*.source and output/*.source are COMPLETELY OPTIONAL and are +# very rarely needed. pg_regress does NOT create the input/ or output/ directories +# - these are optional INPUT directories that users create if they need them. +# Most extensions will never need these directories. +# +# CRITICAL: Do NOT copy files that have corresponding output/*.source files, because +# those are the source of truth and will be regenerated by pg_regress from the .source files. +# Only copy files from results/ that don't have output/*.source counterparts. .PHONY: results results: test - rsync -rlpgovP $(TESTOUT)/results/ $(TESTDIR)/expected + @# Copy .out files from results/ to expected/, excluding those with output/*.source counterparts + @# .out files with output/*.source counterparts are generated from .source files and should NOT be overwritten + @$(PGXNTOOL_DIR)/make_results.sh $(TESTDIR) $(TESTOUT) # testdeps is a generic dependency target that you can add targets to .PHONY: testdeps testdeps: pgtap # These targets ensure all the relevant directories exist -$(TESTDIR)/sql: - @mkdir -p $@ -$(TESTDIR)/expected/: +$(TESTDIR)/sql $(TESTDIR)/expected/ $(TESTOUT)/results/: @mkdir -p $@ $(TEST_RESULT_FILES): | $(TESTDIR)/expected/ @touch $@ -$(TESTDIR)/output/: - @mkdir -p $@ -$(TEST_OUT_FILES): | $(TESTDIR)/output/ $(TESTDIR)/expected/ $(TESTDIR)/sql/ - @touch $@ # @@ -171,6 +188,13 @@ forcetag: rmtag tag dist: tag dist-only dist-only: + @# Check if .gitattributes exists but isn't committed + @if [ -f .gitattributes ] && ! git ls-files --error-unmatch .gitattributes >/dev/null 2>&1; then \ + echo "ERROR: .gitattributes exists but is not committed to git." >&2; \ + echo " git archive only respects export-ignore for committed files." >&2; \ + echo " Please commit .gitattributes for export-ignore to take effect." >&2; \ + exit 1; \ + fi git archive --prefix=$(PGXN)-$(PGXNVERSION)/ -o ../$(PGXN)-$(PGXNVERSION).zip $(PGXNVERSION) .PHONY: forcedist diff --git a/make_results.sh b/make_results.sh new file mode 100755 index 0000000..066e372 --- /dev/null +++ b/make_results.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Helper script for make results target +# Copies .out files from results/ to expected/, excluding those with output/*.source counterparts + +set -e + +TESTDIR="${1:-test}" +TESTOUT="${2:-${TESTDIR}}" + +mkdir -p "${TESTDIR}/expected" + +# Use nullglob so globs that don't match return nothing instead of the literal pattern +shopt -s nullglob + +for result_file in "${TESTOUT}/results"/*.out; do + test_name=$(basename "$result_file" .out) + + # Check if this file has a corresponding output/*.source file + # Only consider non-empty source files (empty files are likely leftovers from pg_regress) + if [ -f "${TESTDIR}/output/${test_name}.source" ] && [ -s "${TESTDIR}/output/${test_name}.source" ]; then + echo "WARNING: ${TESTOUT}/results/${test_name}.out exists but will NOT be copied" >&2 + echo " (excluded because ${TESTDIR}/output/${test_name}.source exists)" >&2 + else + # Copy the file - it doesn't have an output/*.source counterpart + cp "$result_file" "${TESTDIR}/expected/${test_name}.out" + fi +done + From a69fc107902f940c89c968237f753ae291f1675e Mon Sep 17 00:00:00 2001 From: jnasbyupgrade Date: Wed, 31 Dec 2025 13:33:56 -0600 Subject: [PATCH 09/11] Add pg_tle support and improve commit command workflow - Add `pgtle` make target to generate pg_tle registration SQL for extensions - Support pg_tle version ranges (1.0.0-1.5.0 and 1.5.0+) with appropriate API usage - Add `pgtle.sh` script to generate pg_tle SQL from extension control files and versioned SQL - Update `base.mk` to include pg_tle generation with proper dependencies on SQL files - Add pg_tle/ directory to EXTRA_CLEAN for `make clean` - Enhance `.claude/commands/commit.md` to require checking all 3 repos before committing - Add explicit guidance to commit all repos with changes together (no empty commits) - Document multi-repo commit workflow in steps - Update `CLAUDE.md` with pg_tle development context and documentation - Update `README.asc` and `README.html` with pg_tle usage documentation Co-Authored-By: Claude --- .claude/commands/commit.md | 69 ++- CLAUDE.md | 53 +++ README.asc | 214 +++++++++- README.html | 424 ++++++++++++++++++- _.gitignore | 6 +- base.mk | 103 ++++- meta.mk.sh | 3 +- pgtle.sh | 843 +++++++++++++++++++++++++++++++++++++ 8 files changed, 1694 insertions(+), 21 deletions(-) create mode 100755 pgtle.sh diff --git a/.claude/commands/commit.md b/.claude/commands/commit.md index 5ddbd74..8d6871c 100644 --- a/.claude/commands/commit.md +++ b/.claude/commands/commit.md @@ -7,6 +7,32 @@ allowed-tools: Bash(git status:*), Bash(git log:*), Bash(git add:*), Bash(git di Create a git commit following all project standards and safety protocols for pgxntool-test. +**FIRST: Check ALL three repositories for changes** + +**CRITICAL**: Before doing ANYTHING else, you MUST check git status in all 3 repositories to understand the full scope of changes: + +```bash +# Check pgxntool (main framework) +echo "=== pgxntool status ===" +cd ../pgxntool && git status + +# Check pgxntool-test (test harness) +echo "=== pgxntool-test status ===" +cd ../pgxntool-test && git status + +# Check pgxntool-test-template (test template) +echo "=== pgxntool-test-template status ===" +cd ../pgxntool-test-template && git status +``` + +**Why this matters**: Work on pgxntool frequently involves changes across all 3 repositories. You need to understand the complete picture before committing anywhere. + +**IMPORTANT**: If ANY of the 3 repositories have changes, you should commit ALL of them that have changes (unless the user explicitly says otherwise). This ensures related changes stay synchronized across the repos. + +**DO NOT create empty commits** - Only commit repos that actually have changes (modified/untracked files). If a repo has no changes, skip it. + +--- + **CRITICAL REQUIREMENTS:** 1. **Git Safety**: Never update `git config`, never force push to `main`/`master`, never skip hooks unless explicitly requested @@ -45,9 +71,17 @@ Create a git commit following all project standards and safety protocols for pgx 5. After receiving approval, stage changes appropriately using `git add` + **CRITICAL: Include ALL new files** + - Check `git status` for untracked files + - **ALL untracked files that are part of the feature/change MUST be staged** + - New scripts, new documentation, new helper files, etc. should all be included + - Do NOT leave new files uncommitted unless explicitly told to exclude them + - When in doubt, include the file - it's better to commit too much than to leave important files uncommitted + 6. **VERIFY staged files with `git status`**: - - If user did NOT specify a subset: Confirm ALL modified/untracked files are staged + - If user did NOT specify a subset: Confirm ALL modified AND untracked files are staged - If user specified only certain files: Confirm ONLY those files are staged + - **Check for any untracked files that should be part of this commit** - STOP and ask user if staging doesn't match intent 7. After verification, commit using `HEREDOC` format: @@ -66,11 +100,38 @@ EOF 9. If pre-commit hook modifies files: Check authorship (`git log -1 --format='%an %ae'`) and branch status, then amend if safe or create new commit +10. **Repeat steps 1-9 for each repository that has changes**: + - After committing one repo, move to the next repo with changes (usually: pgxntool → pgxntool-test → pgxntool-test-template) + - Draft appropriate commit messages that reference the primary changes + - **ONLY commit repos with actual changes** - skip any repo that has no modified/untracked files + - Ensure ALL repos with changes get committed before finishing + +**MULTI-REPO COMMIT CONTEXT:** + +**CRITICAL**: Work on pgxntool frequently involves changes across all 3 repositories simultaneously: +- **pgxntool** (this repo) - The main framework +- **pgxntool-test** (at `../pgxntool-test/`) - Test harness +- **pgxntool-test-template** (at `../pgxntool-test-template/`) - Test template + +**This is why you MUST check all 3 repositories at the start** (see FIRST step above). + +**DEFAULT BEHAVIOR: Commit ALL repos with changes together** - If any of the 3 repos have changes when you check them, you should plan to commit ALL repos that have changes (unless user explicitly specifies otherwise). This keeps related changes synchronized. **Do NOT create empty commits** - only commit repos with actual modified/untracked files. + +When committing changes that span repositories: +1. **Commit messages in pgxntool-test and pgxntool-test-template should reference the main changes in pgxntool** + - Example: "Add tests for pg_tle support (see pgxntool commit for implementation)" + - Example: "Update template for pg_tle feature (see pgxntool commit for details)" + +2. **When working across repos, commit in logical order:** + - Usually: pgxntool → pgxntool-test → pgxntool-test-template + - But adapt based on dependencies + **REPOSITORY CONTEXT:** -This is pgxntool-test, a test harness for the pgxntool framework. Key facts: -- Tests live in `tests/` directory -- `.envs/` contains test environments (gitignored) +This is pgxntool, a PostgreSQL extension build framework. Key facts: +- Main Makefile is `base.mk` +- Scripts live in root directory +- Documentation is in `README.asc` (generates `README.html`) **RESTRICTIONS:** - DO NOT push unless explicitly asked diff --git a/CLAUDE.md b/CLAUDE.md index 57812d2..f4f149d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -112,6 +112,9 @@ make results # Run tests and update expected output files make html # Generate HTML from Asciidoc sources make tag # Create git branch for current META.json version make dist # Create PGXN .zip (auto-tags, places in ../) +make pgtle # Generate pg_tle registration SQL (see pg_tle Support below) +make check-pgtle # Check pg_tle installation and report version +make install-pgtle # Install pg_tle registration SQL files into database make pgxntool-sync # Update to latest pgxntool via git subtree pull ``` @@ -186,6 +189,56 @@ When modifying pgxntool: - Uses `git subtree pull --squash` - Requires clean repo (no uncommitted changes) +### pg_tle Support + +pgxntool can generate pg_tle (Trusted Language Extensions) registration SQL for deploying extensions in AWS RDS/Aurora without filesystem access. + +**Usage:** `make pgtle` or `make pgtle PGTLE_VERSION=1.5.0+` + +**Output:** `pg_tle/{version_range}/{extension}.sql` + +**Version ranges:** +- `1.0.0-1.5.0` - pg_tle 1.0.0 through 1.4.x (no schema parameter) +- `1.5.0+` - pg_tle 1.5.0 and later (schema parameter support) + +**Installation targets:** + +- `make check-pgtle` - Checks if pg_tle is installed and reports the version. Reports version from `pg_extension` if extension has been created, or newest available version from `pg_available_extension_versions` if available but not created. Errors if pg_tle not available in cluster. Assumes `PG*` environment variables are configured. + +- `make install-pgtle` - Auto-detects pg_tle version and installs appropriate registration SQL files. Updates or creates pg_tle extension as needed. Determines which version range files to install based on detected version. Runs all generated SQL files via `psql` to register extensions with pg_tle. Assumes `PG*` environment variables are configured. + +**Version notation:** +- `X.Y.Z+` means >= X.Y.Z +- `X.Y.Z-A.B.C` means >= X.Y.Z and < A.B.C (note boundary) + +**Key implementation details:** +- Script: `pgxntool/pgtle-wrap.sh` (bash) +- Parses `.control` files for metadata (NOT META.json) +- Fixed delimiter: `$_pgtle_wrap_delimiter_$` (validated not in source) +- Each output file contains ALL versions and ALL upgrade paths +- Multi-extension support (multiple .control files) +- Output directory `pg_tle/` excluded from git +- Depends on `make all` to ensure versioned SQL files exist first +- Only processes versioned files (`sql/{ext}--{version}.sql`), not base files + +**SQL file handling:** +- **Version files** (`sql/{ext}--{version}.sql`): Generated automatically by `make all` from base `sql/{ext}.sql` file +- **Upgrade scripts** (`sql/{ext}--{v1}--{v2}.sql`): Created manually by users when adding new extension versions +- The script ensures the default_version file exists if the base file exists (creates it from base file if missing) +- All version files and upgrade scripts are discovered and included in the generated pg_tle registration SQL + +**Dependencies:** +Generated files depend on: +- Control file (metadata source) +- All SQL files (sql/{ext}--*.sql) - must run `make all` first +- Generator script itself + +**Limitations:** +- No C code support (pg_tle requires trusted languages only) +- PostgreSQL 14.5+ required (pg_tle not available on earlier versions) + +See `README-pgtle.md` for complete user documentation. + ## Critical Gotchas 1. **Empty Variables**: If `DOCS` or `MODULES` is empty, base.mk sets to empty to prevent PGXS errors diff --git a/README.asc b/README.asc index c077fe4..d1a2004 100644 --- a/README.asc +++ b/README.asc @@ -97,6 +97,102 @@ NOTE: Your repository must be clean (no modified files) in order to run this. Ru TIP: There is also a `pgxntool-sync-%` rule if you need to do more advanced things. +=== pgtle +Generates pg_tle (Trusted Language Extensions) registration SQL files for deploying extensions in managed environments like AWS RDS/Aurora. See <<_pg_tle_Support>> for complete documentation. + +`make pgtle` generates SQL files in `pg_tle/` subdirectories organized by pg_tle version ranges: +- `pg_tle/1.0.0-1.5.0/` - For pg_tle versions 1.0.0 through 1.4.x +- `pg_tle/1.5.0+/` - For pg_tle versions 1.5.0 and later + +You can limit generation to a specific version range using `PGTLE_VERSION`: +---- +make pgtle PGTLE_VERSION=1.5.0+ +---- + +=== check-pgtle +Checks if pg_tle is installed and reports the version. This target: +- Reports the version from `pg_extension` if `CREATE EXTENSION pg_tle` has been run in the database +- Errors if pg_tle is not available in the cluster + +This target assumes `PG*` environment variables are configured for `psql` connectivity. + +---- +make check-pgtle +---- + +=== run-pgtle +Registers all extensions with pg_tle by executing the generated pg_tle registration SQL files in a PostgreSQL database. This target: +- Requires pg_tle extension to be installed (checked via `check-pgtle`) +- Uses `pgtle.sh` to determine which version range directory to use based on the installed pg_tle version +- Runs all generated SQL files via `psql` to register your extensions with pg_tle + +This target assumes that running `psql` without any arguments will connect to the desired database. You can control this by setting the various PG* environment variables (and possibly using the `.pgpassword` file). See the PostgreSQL documentation for more details. + +NOTE: The `pgtle` target is a dependency, so `make run-pgtle` will automatically generate the SQL files if needed. + +---- +make run-pgtle +---- + +After running `make run-pgtle`, you can create your extension in the database: +---- +CREATE EXTENSION "your-extension-name"; +---- + +== Version-Specific SQL Files + +PGXNtool automatically generates version-specific SQL files from your base SQL file. These files follow the pattern `sql/{extension}--{version}.sql` and are used by PostgreSQL's extension system to install specific versions of your extension. + +=== How Version Files Are Generated + +When you run `make` (or `make all`), PGXNtool: + +1. Reads your `META.json` file to determine the extension version from `provides.{extension}.version` +2. Generates a Makefile rule that copies your base SQL file (`sql/{extension}.sql`) to the version-specific file (`sql/{extension}--{version}.sql`) +3. Executes this rule, creating the version-specific file with a header comment indicating it's auto-generated + +For example, if your `META.json` contains: +---- +"provides": { + "myext": { + "version": "1.2.3", + ... + } +} +---- + +Running `make` will create `sql/myext--1.2.3.sql` by copying `sql/myext.sql`. + +=== What Controls the Version Number + +The version number comes from `META.json` → `provides.{extension}.version`, *not* from your `.control` file's `default_version` field. The `.control` file's `default_version` is used by PostgreSQL to determine which version to install by default, but the actual version-specific file that gets generated is determined by what's in `META.json`. + +To change the version of your extension: +1. Update `provides.{extension}.version` in `META.json` +2. Run `make` to regenerate the version-specific file +3. Update `default_version` in your `.control` file to match (if needed) + +=== Committing Version Files + +Version-specific SQL files are now treated as permanent files that should be committed to your repository. This makes it much easier to test updates to extensions, as you can see exactly what SQL was included in each version. + +IMPORTANT: These files are auto-generated and include a header comment warning not to edit them. Any manual changes will be overwritten the next time you run `make`. To modify the extension, edit the base SQL file (`sql/{extension}.sql`) instead. + +=== Distribution Inclusion + +Version-specific files are automatically included in distributions created by `make dist`. They are part of the `DATA` variable that PGXS uses to determine which files to install. + +=== Multiple Versions + +If you need to support multiple versions of your extension: + +1. Create additional version-specific files manually (e.g., `sql/myext--1.0.0.sql`, `sql/myext--1.1.0.sql`) +2. Create upgrade scripts for version transitions (e.g., `sql/myext--1.0.0--1.1.0.sql`) +3. Update `META.json` to reflect the current version you're working on +4. Commit all version files and upgrade scripts to your repository + +The version file for the current version (specified in `META.json`) will be automatically regenerated when you run `make`, but other version files you create manually will be preserved. + == Document Handling PGXNtool supports generation and installation of document files. There are several variables and rules that control this behavior. @@ -172,7 +268,123 @@ Because of this, `base.mk` will forcibly define it to be NULL if it's empty. PGXNtool appends *all* files found in all `$(DOC_DIRS)` to `DOCS`. +== pg_tle Support +[[_pg_tle_Support]] +pgxntool can generate link:https://github.com/aws/pg_tle[pg_tle (Trusted Language Extensions)] registration SQL for deploying PostgreSQL extensions in managed environments like AWS RDS and Aurora where filesystem access is not available. + +For make targets, see: <<_pgtle>>, <<_check_pgtle>>, <<_run_pgtle>>. + +=== What is pg_tle? + +pg_tle is an AWS open-source framework that enables developers to create and deploy PostgreSQL extensions without filesystem access. Traditional PostgreSQL extensions require `.control` and `.sql` files on the filesystem, which isn't possible in managed services like RDS and Aurora. + +pg_tle solves this by: +- Storing extension metadata and SQL in database tables +- Using the `pgtle_admin` role for administrative operations +- Enabling `CREATE EXTENSION` to work in managed environments + +=== Quick Start + +Generate pg_tle registration SQL for your extension: + +---- +make pgtle +---- + +This creates files in `pg_tle/` subdirectories organized by pg_tle version ranges: +- `pg_tle/1.0.0-1.5.0/{extension}.sql` - For pg_tle versions 1.0.0 through 1.4.x +- `pg_tle/1.5.0+/{extension}.sql` - For pg_tle versions 1.5.0 and later + +=== Version Groupings + +pgxntool creates different sets of files for different pg_tle versions to allow using newer features (like schema specification) without breaking backwards compatibility. You should always try to use the newest version of pg_tle when possible. + +The version groupings that pgxntool recognizes: + +* *`1.0.0-1.5.0`*: For pg_tle versions 1.0.0 through 1.4.x. + +* *`1.5.0+`*: For pg_tle versions 1.5.0 and later. This group was created because pg_tle 1.5.0 added support for extensions to specify a schema via the `schema` parameter in `pgtle.install_extension()`. If your control file specifies `schema = 'public'` (or another schema), you must use the `1.5.0+` file. Otherwise, either file will work, but `1.0.0-1.5.0` is more compatible with older pg_tle versions. + +If additional version groups are needed in the future, we will attempt to maintain existing "version+ groups" (such as `1.5.0+`) for backwards compatibility. However, older "version+ groups" should be considered as *DEPRECATED*. Note that we may not be able to maintain full compatibility, given that we don't have any control over what the pg_tle project does. + +Note that pg_tle is currently backward-compatible, so you it is possible to run `1.0.0-1.5.0` files against newer versions of pg_tle, but there's really no reason to do so. + +=== Installation Example + +IMPORTANT: This is only a basic example. Always refer to the link:https://github.com/aws/pg_tle[main pg_tle documentation] for complete installation instructions and best practices. + +Basic installation steps: + +. Ensure pg_tle is installed and grant the `pgtle_admin` role to your user +. Run the generated SQL file: `psql -f pg_tle/1.5.0+/myextension.sql` +. Create your extension: `CREATE EXTENSION myextension;` + +=== Advanced Usage + +==== Generate Only Specific Version + +Generate only one version range: + +---- +make pgtle PGTLE_VERSION=1.5.0+ +---- + +==== Multi-Extension Projects + +If your project has multiple extensions (multiple `.control` files), `make pgtle` generates files for all of them: + +---- +myproject/ +├── ext1.control +├── ext2.control +└── pg_tle/ + ├── 1.0.0-1.5.0/ + │ ├── ext1.sql + │ └── ext2.sql + └── 1.5.0+/ + ├── ext1.sql + └── ext2.sql +---- + +=== How It Works +`make pgtle` does the following: + +. Parses control file(s): Extracts `comment`, `default_version`, `requires`, and `schema` fields +. Discovers SQL files: Finds all versioned files (`sql/{ext}--{version}.sql`) and upgrade scripts (`sql/{ext}--{ver1}--{ver2}.sql`) +. Wraps SQL content: Uses a fixed dollar-quote delimiter (`$_pgtle_wrap_delimiter_$`) to wrap SQL for pg_tle functions +. Generates registration SQL: Creates `pgtle.install_extension()` calls for each version, `pgtle.install_update_path()` for upgrades, and `pgtle.set_default_version()` for the default +. Version-specific output: Generates separate files for different pg_tle capability levels + +Each generated SQL file is wrapped in a transaction (`BEGIN;` ... `COMMIT;`) to ensure atomic installation. + +=== Troubleshooting + +==== "No versioned SQL files found" + +*Problem*: The script can't find `sql/{ext}--{version}.sql` files. + +*Solution*: Run `make` first to generate versioned files from your base `sql/{ext}.sql` file. + +==== "Control file not found" + +*Problem*: The script can't find `{ext}.control` in the current directory. + +*Solution*: Run `make pgtle` from your extension's root directory (where the `.control` file is). + +==== "SQL file contains reserved pg_tle delimiter" + +*Problem*: Your SQL files contain the string `$_pgtle_wrap_delimiter_$` (extremely unlikely). + +*Solution*: Don't use that dollar-quote delimiter in your code. + +==== Extension uses C code + +*Problem*: Your control file has `module_pathname`, indicating C code. + +*Solution*: pg_tle only supports trusted languages. You cannot use C extensions with pg_tle. The script will warn you but still generate files (which won't work). + +NOTE: there are several untrusted languages (such as plpython), and the only tests for C. == Copyright -Copyright (c) 2015 Jim Nasby +Copyright (c) 2025 Jim Nasby PGXNtool is released under a https://github.com/decibel/pgxntool/blob/master/LICENCE[BSD license]. Note that it includes https://github.com/dominictarr/JSON.sh[JSON.sh], which is released under a https://github.com/decibel/pgxntool/blob/master/JSON.sh.LICENCE[MIT license]. diff --git a/README.html b/README.html index 975d3f9..1b5cbc0 100644 --- a/README.html +++ b/README.html @@ -456,16 +456,38 @@

PGXNtool

  • 4.5. tag
  • 4.6. dist
  • 4.7. pgxntool-sync
    • +
  • 4.8. pgtle
    • +
  • 4.9. check-pgtle
    • +
  • 4.10. run-pgtle
    • -
  • 5. Document Handling +
  • 5. Version-Specific SQL Files +
    • +
  • 6. Document Handling + +
    • +
  • 7. pg_tle Support +
    • -
  • 6. Copyright
    • @@ -717,10 +739,186 @@

    +

    4.8. pgtle

    +
    +

    Generates pg_tle (Trusted Language Extensions) registration SQL files for deploying extensions in managed environments like AWS RDS/Aurora. See [_pg_tle_Support] for complete documentation.

    +
    +
    +

    make pgtle generates SQL files in pg_tle/ subdirectories organized by pg_tle version ranges: +- pg_tle/1.0.0-1.5.0/ - For pg_tle versions 1.0.0 through 1.4.x +- pg_tle/1.5.0+/ - For pg_tle versions 1.5.0 and later

    +
    +
    +

    You can limit generation to a specific version range using PGTLE_VERSION:

    +
    +
    +
    +
    make pgtle PGTLE_VERSION=1.5.0+
    +
    +
    + +
    +

    4.9. check-pgtle

    +
    +

    Checks if pg_tle is installed and reports the version. This target: +- Reports the version from pg_extension if CREATE EXTENSION pg_tle has been run in the database +- Errors if pg_tle is not available in the cluster

    +
    +
    +

    This target assumes PG* environment variables are configured for psql connectivity.

    +
    +
    +
    +
    make check-pgtle
    +
    +
    +
    +
    +

    4.10. run-pgtle

    +
    +

    Registers all extensions with pg_tle by executing the generated pg_tle registration SQL files in a PostgreSQL database. This target: +- Requires pg_tle extension to be installed (checked via check-pgtle) +- Uses pgtle.sh to determine which version range directory to use based on the installed pg_tle version +- Runs all generated SQL files via psql to register your extensions with pg_tle

    +
    +
    +

    This target assumes that running psql without any arguments will connect to the desired database. You can control this by setting the various PG* environment variables (and possibly using the .pgpassword file). See the PostgreSQL documentation for more details.

    +
    +
    + + + + + +
    +
    Note
    +    		 +The pgtle target is a dependency, so make run-pgtle will automatically generate the SQL files if needed. +
    +
    +
    +
    +
    make run-pgtle
    +
    +
    +
    +

    After running make run-pgtle, you can create your extension in the database:

    +
    +
    +
    +
    CREATE EXTENSION "your-extension-name";
    +
    +
    +
    -

    5. Document Handling

    +

    5. Version-Specific SQL Files

    +
    +
    +

    PGXNtool automatically generates version-specific SQL files from your base SQL file. These files follow the pattern sql/{extension}--{version}.sql and are used by PostgreSQL’s extension system to install specific versions of your extension.

    +
    +
    +

    5.1. How Version Files Are Generated

    +
    +

    When you run make (or make all), PGXNtool:

    +
    +
    +
      +
    1. +

      Reads your META.json file to determine the extension version from provides.{extension}.version

      +
      2. +
    2. +

      Generates a Makefile rule that copies your base SQL file (sql/{extension}.sql) to the version-specific file (sql/{extension}--{version}.sql)

      +
      3. +
    3. +

      Executes this rule, creating the version-specific file with a header comment indicating it’s auto-generated

      +
      4. +
    +
    +
    +

    For example, if your META.json contains:

    +
    +
    +
    +
    "provides": {
+  "myext": {
+    "version": "1.2.3",
+    ...
+  }
+}
    +
    +
    +
    +

    Running make will create sql/myext—​1.2.3.sql by copying sql/myext.sql.

    +
    +
    +
    +

    5.2. What Controls the Version Number

    +
    +

    The version number comes from META.json → provides.{extension}.version, not from your .control file’s default_version field. The .control file’s default_version is used by PostgreSQL to determine which version to install by default, but the actual version-specific file that gets generated is determined by what’s in META.json.

    +
    +
    +

    To change the version of your extension: +1. Update provides.{extension}.version in META.json +2. Run make to regenerate the version-specific file +3. Update default_version in your .control file to match (if needed)

    +
    +
    +
    +

    5.3. Committing Version Files

    +
    +

    Version-specific SQL files are now treated as permanent files that should be committed to your repository. This makes it much easier to test updates to extensions, as you can see exactly what SQL was included in each version.

    +
    +
    + + + + + +
    +
    Important
    +    		 +These files are auto-generated and include a header comment warning not to edit them. Any manual changes will be overwritten the next time you run make. To modify the extension, edit the base SQL file (sql/{extension}.sql) instead. +
    +
    +
    +
    +

    5.4. Distribution Inclusion

    +
    +

    Version-specific files are automatically included in distributions created by make dist. They are part of the DATA variable that PGXS uses to determine which files to install.

    +
    +
    +
    +

    5.5. Multiple Versions

    +
    +

    If you need to support multiple versions of your extension:

    +
    +
    +
      +
    1. +

      Create additional version-specific files manually (e.g., sql/myext—​1.0.0.sql, sql/myext—​1.1.0.sql)

      +
      2. +
    2. +

      Create upgrade scripts for version transitions (e.g., sql/myext—​1.0.0—​1.1.0.sql)

      +
      3. +
    3. +

      Update META.json to reflect the current version you’re working on

      +
      4. +
    4. +

      Commit all version files and upgrade scripts to your repository

      +
      5. +
    +
    +
    +

    The version file for the current version (specified in META.json) will be automatically regenerated when you run make, but other version files you create manually will be preserved.

    +
    +
    +
    +
    +
    +

    6. Document Handling

    PGXNtool supports generation and installation of document files. There are several variables and rules that control this behavior.

    @@ -731,7 +929,7 @@

    -

    5.1. Document Variables

    +

    6.1. Document Variables

    DOC_DIRS
    @@ -778,7 +976,7 @@

    <

    -

    5.2. Document Rules

    +

    6.2. Document Rules

    If Asciidoc is found (or $(ASCIIDOC) is set), the html rule will be added as a prerequisite to the install and installchec rules. That will ensure that docs are generated for install and test, but only if Asciidoc is available. @@ -820,7 +1018,7 @@

    -

    5.3. The DOCS variable

    +

    6.3. The DOCS variable

    This variable has special meaning to PGXS. See the Postgres documentation for full details.

    @@ -848,10 +1046,210 @@

    -

    6. Copyright

    +

    7. pg_tle Support

    +
    +

    pgxntool can generate pg_tle (Trusted Language Extensions) registration SQL for deploying PostgreSQL extensions in managed environments like AWS RDS and Aurora where filesystem access is not available.

    +
    -

    Copyright (c) 2015 Jim Nasby <Jim.Nasby@BlueTreble.com>

    +

    For make targets, see: pgtle, check-pgtle, run-pgtle.

    +
    +
    +

    7.1. What is pg_tle?

    +
    +

    pg_tle is an AWS open-source framework that enables developers to create and deploy PostgreSQL extensions without filesystem access. Traditional PostgreSQL extensions require .control and .sql files on the filesystem, which isn’t possible in managed services like RDS and Aurora.

    +
    +
    +

    pg_tle solves this by: +- Storing extension metadata and SQL in database tables +- Using the pgtle_admin role for administrative operations +- Enabling CREATE EXTENSION to work in managed environments

    +
    +
    +
    +

    7.2. Quick Start

    +
    +

    Generate pg_tle registration SQL for your extension:

    +
    +
    +
    +
    make pgtle
    +
    +
    +
    +

    This creates files in pg_tle/ subdirectories organized by pg_tle version ranges: +- pg_tle/1.0.0-1.5.0/{extension}.sql - For pg_tle versions 1.0.0 through 1.4.x +- pg_tle/1.5.0+/{extension}.sql - For pg_tle versions 1.5.0 and later

    +
    +
    +
    +

    7.3. Version Groupings

    +
    +

    pgxntool creates different sets of files for different pg_tle versions to allow using newer features (like schema specification) without breaking backwards compatibility. You should always try to use the newest version of pg_tle when possible.

    +
    +
    +

    The version groupings that pgxntool recognizes:

    +
    +
    +
      +
    • +

      1.0.0-1.5.0: For pg_tle versions 1.0.0 through 1.4.x.

      +
      • +
    • +

      1.5.0+: For pg_tle versions 1.5.0 and later. This group was created because pg_tle 1.5.0 added support for extensions to specify a schema via the schema parameter in pgtle.install_extension(). If your control file specifies schema = 'public' (or another schema), you must use the 1.5.0+ file. Otherwise, either file will work, but 1.0.0-1.5.0 is more compatible with older pg_tle versions.

      +
      • +
    +
    +
    +

    If additional version groups are needed in the future, we will attempt to maintain existing "version+ groups" (such as 1.5.0+) for backwards compatibility. However, older "version+ groups" should be considered as DEPRECATED. Note that we may not be able to maintain full compatibility, given that we don’t have any control over what the pg_tle project does.

    +
    +
    +

    Note that pg_tle is currently backward-compatible, so you it is possible to run 1.0.0-1.5.0 files against newer versions of pg_tle, but there’s really no reason to do so.

    +
    +
    +
    +

    7.4. Installation Example

    +
    + + + + + +
    +
    Important
    +    		 +This is only a basic example. Always refer to the main pg_tle documentation for complete installation instructions and best practices. +
    +
    +
    +

    Basic installation steps:

    +
    +
    +
      +
    1. +

      Ensure pg_tle is installed and grant the pgtle_admin role to your user

      +
      2. +
    2. +

      Run the generated SQL file: psql -f pg_tle/1.5.0+/myextension.sql

      +
      3. +
    3. +

      Create your extension: CREATE EXTENSION myextension;

      +
      4. +
    +
    +
    +
    +

    7.5. Advanced Usage

    +
    +

    7.5.1. Generate Only Specific Version

    +
    +

    Generate only one version range:

    +
    +
    +
    +
    make pgtle PGTLE_VERSION=1.5.0+
    +
    +
    +
    +
    +

    7.5.2. Multi-Extension Projects

    +
    +

    If your project has multiple extensions (multiple .control files), make pgtle generates files for all of them:

    +
    +
    +
    +
    myproject/
+├── ext1.control
+├── ext2.control
+└── pg_tle/
+    ├── 1.0.0-1.5.0/
+    │   ├── ext1.sql
+    │   └── ext2.sql
+    └── 1.5.0+/
+        ├── ext1.sql
+        └── ext2.sql
    +
    +
    +
    +
    +
    +

    7.6. How It Works

    +
    +

    make pgtle does the following:

    +
    +
    +
      +
    1. +

      Parses control file(s): Extracts comment, default_version, requires, and schema fields

      +
      2. +
    2. +

      Discovers SQL files: Finds all versioned files (sql/{ext}--{version}.sql) and upgrade scripts (sql/{ext}--{ver1}--{ver2}.sql)

      +
      3. +
    3. +

      Wraps SQL content: Uses a fixed dollar-quote delimiter ($pgtle_wrap_delimiter$) to wrap SQL for pg_tle functions

      +
      4. +
    4. +

      Generates registration SQL: Creates pgtle.install_extension() calls for each version, pgtle.install_update_path() for upgrades, and pgtle.set_default_version() for the default

      +
      5. +
    5. +

      Version-specific output: Generates separate files for different pg_tle capability levels

      +
      6. +
    +
    +
    +

    Each generated SQL file is wrapped in a transaction (BEGIN; …​ COMMIT;) to ensure atomic installation.

    +
    +
    +
    +

    7.7. Troubleshooting

    +
    +

    7.7.1. "No versioned SQL files found"

    +
    +

    Problem: The script can’t find sql/{ext}--{version}.sql files.

    +
    +
    +

    Solution: Run make first to generate versioned files from your base sql/{ext}.sql file.

    +
    +
    +
    +

    7.7.2. "Control file not found"

    +
    +

    Problem: The script can’t find {ext}.control in the current directory.

    +
    +
    +

    Solution: Run make pgtle from your extension’s root directory (where the .control file is).

    +
    +
    +
    +

    7.7.3. "SQL file contains reserved pg_tle delimiter"

    +
    +

    Problem: Your SQL files contain the string $pgtle_wrap_delimiter$ (extremely unlikely).

    +
    +
    +

    Solution: Don’t use that dollar-quote delimiter in your code.

    +
    +
    +
    +

    7.7.4. Extension uses C code

    +
    +

    Problem: Your control file has module_pathname, indicating C code.

    +
    +
    +

    Solution: pg_tle only supports trusted languages. You cannot use C extensions with pg_tle. The script will warn you but still generate files (which won’t work).

    +
    +
    + + + + + +
    +
    Note
    +    		 +there are several untrusted languages (such as plpython), and the only tests for C. +== Copyright +Copyright (c) 2025 Jim Nasby <Jim.Nasby@gmail.com> +

    PGXNtool is released under a BSD license. Note that it includes JSON.sh, which is released under a MIT license.

    @@ -859,9 +1257,11 @@

    +
    +
    diff --git a/_.gitignore b/_.gitignore index 82933e8..c79e4ff 100644 --- a/_.gitignore +++ b/_.gitignore @@ -16,8 +16,7 @@ meta.mk .deps/ # built targets -/sql/*--* -!/sql/*--*--*.sql +# Note: Version-specific files (sql/*--*.sql) are now tracked in git and should be committed # Test artifacts results/ @@ -27,3 +26,6 @@ regression.out # Misc tmp/ .DS_Store + +# pg_tle generated files +/pg_tle/ diff --git a/base.mk b/base.mk index ac661c5..affe732 100644 --- a/base.mk +++ b/base.mk @@ -48,7 +48,7 @@ ifeq ($(strip $(MODULES)),) MODULES =# Set to NUL so PGXS doesn't puke endif -EXTRA_CLEAN = $(wildcard ../$(PGXN)-*.zip) $(EXTENSION_VERSION_FILES) $(TEST_SQL_FROM_SOURCE) $(TEST_EXPECTED_FROM_SOURCE) +EXTRA_CLEAN = $(wildcard ../$(PGXN)-*.zip) $(TEST_SQL_FROM_SOURCE) $(TEST_EXPECTED_FROM_SOURCE) pg_tle/ # Get Postgres version, as well as major (9.4, etc) version. # NOTE! In at least some versions, PGXS defines VERSION, so we intentionally don't use that variable @@ -120,6 +120,107 @@ results: test .PHONY: testdeps testdeps: pgtap +# +# pg_tle support - Generate pg_tle registration SQL +# + +# User-configurable: specific pg_tle version to generate +# Leave empty to generate all versions (default) +# Example: make pgtle PGTLE_VERSION=1.5.0+ +PGTLE_VERSION ?= + +# pg_tle version ranges we support +# These correspond to different capability levels +PGTLE_VERSION_RANGES = 1.0.0-1.5.0 1.5.0+ + +# pg_tle version subdirectories +PGTLE_1_0_TO_1_5_DIR = pg_tle/1.0.0-1.5.0 +PGTLE_1_5_PLUS_DIR = pg_tle/1.5.0+ + +# Discover all extensions from control files in current directory +PGXNTOOL_CONTROL_FILES = $(wildcard *.control) +PGXNTOOL_EXTENSIONS = $(basename $(PGXNTOOL_CONTROL_FILES)) + +# Generate list of pg_tle output files +# If PGTLE_VERSION is set, generate only that version +# Otherwise, generate all version ranges +ifeq ($(PGTLE_VERSION),) + # Generate all versions (default) + PGTLE_FILES = $(foreach ext,$(PGXNTOOL_EXTENSIONS),\ + $(PGTLE_1_0_TO_1_5_DIR)/$(ext).sql \ + $(PGTLE_1_5_PLUS_DIR)/$(ext).sql) +else + # Generate only specified version + ifeq ($(PGTLE_VERSION),1.0.0-1.5.0) + PGTLE_FILES = $(foreach ext,$(PGXNTOOL_EXTENSIONS),\ + $(PGTLE_1_0_TO_1_5_DIR)/$(ext).sql) + else ifeq ($(PGTLE_VERSION),1.5.0+) + PGTLE_FILES = $(foreach ext,$(PGXNTOOL_EXTENSIONS),\ + $(PGTLE_1_5_PLUS_DIR)/$(ext).sql) + else + $(error Invalid PGTLE_VERSION: $(PGTLE_VERSION). Use 1.0.0-1.5.0 or 1.5.0+) + endif +endif + +# Main target +# Depend on 'all' to ensure versioned SQL files are generated first +# Depend on meta.mk (which defines EXTENSION_VERSION_FILES) and versioned SQL files +# to ensure they're generated first +# Depend on control files explicitly so changes trigger rebuilds +.PHONY: pgtle +pgtle: all meta.mk $(PGXNTOOL_CONTROL_FILES) $(PGTLE_FILES) + +# Enable secondary expansion for dynamic dependencies +.SECONDEXPANSION: + +# Pattern rule for generating pg_tle 1.0.0-1.5.0 files +# Dependencies: +# - Control file (metadata source) +# - Generator script (tool itself) +# - All SQL files for this extension (using secondary expansion) +# Note: We depend on $(EXTENSION_VERSION_FILES) at the pgtle target level +# to ensure all versioned files exist before pattern rules run +$(PGTLE_1_0_TO_1_5_DIR)/%.sql: %.control $(PGXNTOOL_DIR)/pgtle.sh $$(wildcard sql/$$*--*.sql) $$(wildcard sql/$$*.sql) + @mkdir -p $(PGTLE_1_0_TO_1_5_DIR) + @$(PGXNTOOL_DIR)/pgtle.sh \ + --extension $(basename $<) \ + --pgtle-version 1.0.0-1.5.0 + +# Pattern rule for generating pg_tle 1.5.0+ files +$(PGTLE_1_5_PLUS_DIR)/%.sql: %.control $(PGXNTOOL_DIR)/pgtle.sh $$(wildcard sql/$$*--*.sql) $$(wildcard sql/$$*.sql) + @mkdir -p $(PGTLE_1_5_PLUS_DIR) + @$(PGXNTOOL_DIR)/pgtle.sh \ + --extension $(basename $<) \ + --pgtle-version 1.5.0+ + +# +# pg_tle installation support +# + +# Check if pg_tle is installed and report version +# Only reports version if CREATE EXTENSION pg_tle has been run +# Errors if pg_tle extension is not installed +# Uses pgtle.sh to get version (avoids code duplication) +.PHONY: check-pgtle +check-pgtle: + @echo "Checking pg_tle installation..." + @PGTLE_VERSION=$$($(PGXNTOOL_DIR)/pgtle.sh --get-version 2>/dev/null); \ + if [ -n "$$PGTLE_VERSION" ]; then \ + echo "pg_tle extension version: $$PGTLE_VERSION"; \ + exit 0; \ + fi; \ + echo "ERROR: pg_tle extension is not installed" >&2; \ + echo " Run 'CREATE EXTENSION pg_tle;' first" >&2; \ + exit 1 + +# Run pg_tle registration SQL files +# Requires pg_tle extension to be installed (checked via check-pgtle) +# Uses pgtle.sh to determine which version range directory to use +# Assumes PG* environment variables are configured +.PHONY: run-pgtle +run-pgtle: pgtle + @$(PGXNTOOL_DIR)/pgtle.sh --run + # These targets ensure all the relevant directories exist $(TESTDIR)/sql $(TESTDIR)/expected/ $(TESTOUT)/results/: @mkdir -p $@ diff --git a/meta.mk.sh b/meta.mk.sh index a5da2ec..2851ae5 100755 --- a/meta.mk.sh +++ b/meta.mk.sh @@ -90,7 +90,8 @@ for ext in $provides; do echo "EXTENSION_${ext}_VERSION_FILE = sql/${ext}--\$(EXTENSION_${ext}_VERSION).sql" echo "EXTENSION_VERSION_FILES += \$(EXTENSION_${ext}_VERSION_FILE)" echo "\$(EXTENSION_${ext}_VERSION_FILE): sql/${ext}.sql META.json meta.mk" - echo ' cp $< $@' + echo " @echo '/* DO NOT EDIT - AUTO-GENERATED FILE */' > \$(EXTENSION_${ext}_VERSION_FILE)" + echo " @cat sql/${ext}.sql >> \$(EXTENSION_${ext}_VERSION_FILE)" done # vi: expandtab ts=2 sw=2 diff --git a/pgtle.sh b/pgtle.sh new file mode 100755 index 0000000..92103da --- /dev/null +++ b/pgtle.sh @@ -0,0 +1,843 @@ +#!/bin/bash +# +# pgtle.sh - Generate pg_tle registration SQL for PostgreSQL extensions +# +# Part of pgxntool: https://github.com/decibel/pgxntool +# +# SYNOPSIS +# pgtle.sh --extension EXTNAME [--pgtle-version VERSION] +# pgtle.sh --get-dir VERSION +# pgtle.sh --get-version +# pgtle.sh --run +# +# DESCRIPTION +# Generates pg_tle (Trusted Language Extensions) registration SQL from +# a pgxntool-based PostgreSQL extension. Reads the extension's .control +# file and SQL files, wrapping them for pg_tle deployment in managed +# environments like AWS RDS and Aurora. +# +# pg_tle enables extension installation without filesystem access by +# storing extension code in database tables. This script converts +# traditional PostgreSQL extensions into pg_tle-compatible SQL. +# +# OPTIONS +# --extension NAME +# Extension name (required). Must match a .control file basename +# in the current directory. +# +# --pgtle-version VERSION +# Generate for specific pg_tle version only (optional). +# Format: 1.0.0-1.5.0 or 1.5.0+ +# Default: Generate all supported versions +# +# --get-dir VERSION +# Returns the directory path for the given pg_tle version. +# Format: VERSION is a version string like "1.5.2" +# Output: Directory path like "pg_tle/1.5.0+" or "pg_tle/1.0.0-1.5.0" +# This option is used by make to determine which directory to use +# +# --get-version +# Returns the installed pg_tle version from the database. +# Output: Version string like "1.5.2" or empty if not installed +# Exit status: 0 if pg_tle is installed, 1 if not installed +# +# --run +# Runs the generated pg_tle registration SQL files. This option: +# - Detects the installed pg_tle version from the database +# - Determines the appropriate directory using --get-dir logic +# - Executes all SQL files in that directory via psql +# - Assumes PG* environment variables are configured for psql +# +# VERSION NOTATION +# X.Y.Z+ Works on pg_tle >= X.Y.Z +# X.Y.Z-A.B.C Works on pg_tle >= X.Y.Z and < A.B.C +# +# Note the boundary conditions: +# 1.5.0+ means >= 1.5.0 (includes 1.5.0) +# 1.0.0-1.5.0 means >= 1.0.0 and < 1.5.0 (excludes 1.5.0) +# +# SUPPORTED VERSIONS +# 1.0.0-1.5.0 pg_tle 1.0.0 through 1.4.x (no schema parameter) +# 1.5.0+ pg_tle 1.5.0 and later (schema parameter support) +# +# EXAMPLES +# # Generate all versions (default) +# pgtle.sh --extension myext +# +# # Generate only for pg_tle 1.5+ +# pgtle.sh --extension myext --pgtle-version 1.5.0+ +# +# # Get directory for a specific pg_tle version +# pgtle.sh --get-dir 1.5.2 +# # Output: pg_tle/1.5.0+ +# +# # Get installed pg_tle version from database +# pgtle.sh --get-version +# # Output: 1.5.2 (or empty if not installed) +# +# # Run generated pg_tle registration SQL files +# pgtle.sh --run +# +# OUTPUT +# Creates files in version-specific subdirectories: +# pg_tle/1.0.0-1.5.0/{extension}.sql +# pg_tle/1.5.0+/{extension}.sql +# +# Each file contains: +# - All versions of the extension +# - All upgrade paths between versions +# - Default version configuration +# - Complete installation instructions +# +# For --get-dir: Outputs the directory path to stdout. +# +# For --get-version: Outputs the installed pg_tle version to stdout, or empty if not installed. +# +# For --run: Executes SQL files and outputs progress messages to stderr. +# +# REQUIREMENTS +# - Must run from extension directory (where .control files are) +# - Extension must use only trusted languages (PL/pgSQL, SQL, PL/Perl, etc.) +# - No C code (module_pathname not supported by pg_tle) +# - Versioned SQL files must exist: sql/{ext}--{version}.sql +# +# EXIT STATUS +# 0 Success +# 1 Error (missing files, validation failure, C code detected, etc.) +# +# SEE ALSO +# pgxntool/README-pgtle.md - Complete user guide +# https://github.com/aws/pg_tle - pg_tle documentation +# + +set -eo pipefail + +# Error function - outputs to stderr but doesn't exit +# Usage: error "message" +error() { + echo "ERROR: $*" >&2 +} + +# Die function - outputs error message and exits with specified code +# Usage: die EXIT_CODE "message" +die() { + local exit_code=$1 + shift + error "$@" + exit "$exit_code" +} + +# Debug function +# Usage: debug LEVEL "message" +# Outputs message to stderr if DEBUG >= LEVEL +# Debug levels use multiples of 10 (10, 20, 30, 40, etc.) to allow for easy expansion +# - 10: Critical errors, important warnings +# - 20: Warnings, significant state changes +# - 30: General debugging, function entry/exit, array operations +# - 40: Verbose details, loop iterations +# - 50+: Maximum verbosity +# Enable with: DEBUG=30 pgtle.sh --extension myext +debug() { + local level=$1 + shift + local message="$*" + + if [ "${DEBUG:-0}" -ge "$level" ]; then + echo "DEBUG[$level]: $message" >&2 + fi +} + +# Constants +PGTLE_DELIMITER='$_pgtle_wrap_delimiter_$' +PGTLE_VERSIONS=("1.0.0-1.5.0" "1.5.0+") + +# Supported pg_tle version ranges and their capabilities +# Use a function instead of associative array for compatibility with bash < 4.0 +get_pgtle_capability() { + local version="$1" + case "$version" in + "1.0.0-1.5.0") + echo "no_schema_param" + ;; + "1.5.0+") + echo "schema_param" + ;; + *) + echo "unknown" + ;; + esac +} + +# Global variables (populated from control file) +EXTENSION="" +DEFAULT_VERSION="" +COMMENT="" +REQUIRES="" +SCHEMA="" +MODULE_PATHNAME="" +VERSION_FILES=() +UPGRADE_FILES=() + +debug 30 "Global arrays initialized: VERSION_FILES=${#VERSION_FILES[@]}, UPGRADE_FILES=${#UPGRADE_FILES[@]}" +PGTLE_VERSION="" # Empty = generate all +GET_DIR_VERSION="" # For --get-dir option + +# Arrays (populated from SQL discovery) +VERSION_FILES=() +UPGRADE_FILES=() + +# Parse and validate a version string +# Extracts numeric version (major.minor.patch) from version strings +# Handles versions with suffixes like "1.5.0alpha1", "2.0beta", "1.2.3dev" +# Returns: numeric version string (e.g., "1.5.0") or exits with error +parse_version() { + local version="$1" + + if [ -z "$version" ]; then + die 1 "Version string is empty" + fi + + # Extract numeric version part (major.minor.patch) + # Matches: 1.5.0, 1.5, 10.2.1alpha, 2.0beta1, etc. + # Pattern: start of string, then digits, dot, digits, optionally (dot digits), then anything + local numeric_version + if [[ "$version" =~ ^([0-9]+\.[0-9]+(\.[0-9]+)?) ]]; then + numeric_version="${BASH_REMATCH[1]}" + else + die 1 "Cannot parse version string: '$version' + Expected format: major.minor[.patch][suffix] + Examples: 1.5.0, 1.5, 2.0alpha1, 10.2.3dev" + fi + + # Ensure we have at least major.minor (add .0 if needed) + if [[ ! "$numeric_version" =~ \. ]]; then + die 1 "Invalid version format: '$version' (need at least major.minor)" + fi + + # If we only have major.minor, add .0 for patch + if [[ ! "$numeric_version" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + numeric_version="${numeric_version}.0" + fi + + echo "$numeric_version" +} + +# Convert version string to comparable integer +# Takes a numeric version string (major.minor.patch) and converts to integer +# Example: "1.5.0" -> 1005000 +# Encoding scheme: major * 1000000 + minor * 1000 + patch +# This limits each component to 0-999 to prevent overflow +version_to_number() { + local version="$1" + + # Parse major.minor.patch + local major minor patch + if [[ "$version" =~ ^([0-9]+)\.([0-9]+)\.([0-9]+)$ ]]; then + major="${BASH_REMATCH[1]}" + minor="${BASH_REMATCH[2]}" + patch="${BASH_REMATCH[3]}" + else + die 1 "version_to_number: Invalid numeric version format: '$version'" + fi + + # Check for overflow in encoding scheme + # Each component must be < 1000 to fit in the allocated space + if [ "$major" -ge 1000 ]; then + die 1 "version_to_number: Major version too large: $major (max 999) + Version: $version" + fi + if [ "$minor" -ge 1000 ]; then + die 1 "version_to_number: Minor version too large: $minor (max 999) + Version: $version" + fi + if [ "$patch" -ge 1000 ]; then + die 1 "version_to_number: Patch version too large: $patch (max 999) + Version: $version" + fi + + # Convert to comparable number: major * 1000000 + minor * 1000 + patch + echo $(( major * 1000000 + minor * 1000 + patch )) +} + +# Get directory for a given pg_tle version +# Takes a version string like "1.5.2" and returns the directory path +# Handles versions with suffixes (e.g., "1.5.0alpha1") +# Returns: "pg_tle/1.0.0-1.5.0" or "pg_tle/1.5.0+" +get_version_dir() { + local version="$1" + + if [ -z "$version" ]; then + die 1 "Version required for --get-dir (got empty string)" + fi + + # Parse and validate version + local numeric_version + numeric_version=$(parse_version "$version") + + # Convert versions to comparable numbers + local version_num + local threshold_num + version_num=$(version_to_number "$numeric_version") + threshold_num=$(version_to_number "1.5.0") + + # Compare: if version < 1.5.0, use 1.0.0-1.5.0 directory; otherwise use 1.5.0+ directory + if [ "$version_num" -lt "$threshold_num" ]; then + echo "pg_tle/1.0.0-1.5.0" + else + echo "pg_tle/1.5.0+" + fi +} + +# Get pg_tle version from installed extension +# Returns version string or empty if not installed +get_pgtle_version() { + psql --no-psqlrc --tuples-only --no-align --command "SELECT extversion FROM pg_extension WHERE extname = 'pg_tle';" 2>/dev/null | tr -d '[:space:]' || echo "" +} + +# Run pg_tle registration SQL files +# Detects installed pg_tle version and runs appropriate SQL files +run_pgtle_sql() { + echo "Running pg_tle registration SQL files..." >&2 + + # Get version from installed extension + local pgtle_version=$(get_pgtle_version) + if [ -z "$pgtle_version" ]; then + die 1 "pg_tle extension is not installed + Run 'CREATE EXTENSION pg_tle;' first, or use 'make check-pgtle' to verify" + fi + + # Get directory for this version + local pgtle_dir=$(get_version_dir "$pgtle_version") + if [ -z "$pgtle_dir" ]; then + die 1 "Failed to determine pg_tle directory for version $pgtle_version" + fi + + echo "Using pg_tle files for version $pgtle_version (directory: $pgtle_dir)" >&2 + + # Check if directory exists + if [ ! -d "$pgtle_dir" ]; then + die 1 "pg_tle directory $pgtle_dir does not exist + Run 'make pgtle' first to generate files" + fi + + # Run all SQL files in the directory + local sql_file + local found=0 + for sql_file in "$pgtle_dir"/*.sql; do + if [ -f "$sql_file" ]; then + found=1 + echo "Running $sql_file..." >&2 + psql --no-psqlrc --file="$sql_file" || exit 1 + fi + done + + if [ "$found" -eq 0 ]; then + die 1 "No SQL files found in $pgtle_dir + Run 'make pgtle' first to generate files" + fi + + echo "pg_tle registration complete" >&2 +} + +# Main logic +main() { + # Handle --get-dir, --get-version, --test-function, and --run options first (early exit, before other validation) + local args=("$@") + local i=0 + while [ $i -lt ${#args[@]} ]; do + if [ "${args[$i]}" = "--get-dir" ] && [ $((i+1)) -lt ${#args[@]} ]; then + get_version_dir "${args[$((i+1))]}" + exit 0 + elif [ "${args[$i]}" = "--get-version" ]; then + local version=$(get_pgtle_version) + if [ -n "$version" ]; then + echo "$version" + exit 0 + else + exit 1 + fi + elif [ "${args[$i]}" = "--test-function" ] && [ $((i+1)) -lt ${#args[@]} ]; then + # Hidden option for testing internal functions + # NOT a supported public interface - used only by the test suite + # Usage: pgtle.sh --test-function FUNC_NAME [ARGS...] + local func_name="${args[$((i+1))]}" + shift $((i+2)) # Remove script name and --test-function and func_name + + # Check if function exists + if ! declare -f "$func_name" >/dev/null 2>&1; then + die 1 "Function '$func_name' does not exist" + fi + + # Call the function with remaining arguments + "$func_name" "${args[@]:$((i+2))}" + exit $? + elif [ "${args[$i]}" = "--run" ]; then + run_pgtle_sql + exit 0 + fi + i=$((i+1)) + done + + # Parse other arguments + parse_args "$@" + + validate_environment + parse_control_file + discover_sql_files + + if [ -z "$PGTLE_VERSION" ]; then + # Generate all versions + for version in "${PGTLE_VERSIONS[@]}"; do + generate_pgtle_sql "$version" + done + else + # Generate specific version + generate_pgtle_sql "$PGTLE_VERSION" + fi +} + +parse_args() { + while [[ $# -gt 0 ]]; do + case $1 in + --extension) + EXTENSION="$2" + shift 2 + ;; + --pgtle-version) + PGTLE_VERSION="$2" + shift 2 + ;; + --get-dir) # This case should ideally not be hit due to early exit + GET_DIR_VERSION="$2" + shift 2 + ;; + --get-version) # This case should ideally not be hit due to early exit + shift + ;; + --test-function) # Hidden option for testing - not documented, not supported + shift 2 # Skip function name and --test-function + ;; + --run) # This case should ideally not be hit due to early exit + shift + ;; + *) + echo "Unknown option: $1" >&2 + exit 1 + ;; + esac + done + + if [ -z "$EXTENSION" ] && [ -z "$GET_DIR_VERSION" ]; then + die 1 "--extension is required (unless using --get-dir, --get-version, --test-function, or --run)" + fi +} + +validate_environment() { + # Check if control file exists + if [ ! -f "${EXTENSION}.control" ]; then + die 1 "Control file not found: ${EXTENSION}.control + Must run from extension directory" + fi +} + +parse_control_file() { + local control_file="${EXTENSION}.control" + + echo "Parsing control file: $control_file" >&2 + + # Parse key = value or key = 'value' format + while IFS= read -r line; do + # Skip comments and empty lines + [[ "$line" =~ ^[[:space:]]*# ]] && continue + [[ "$line" =~ ^[[:space:]]*$ ]] && continue + + # Extract key = value + if [[ "$line" =~ ^[[:space:]]*([a-z_]+)[[:space:]]*=[[:space:]]*(.*)[[:space:]]*$ ]]; then + local key="${BASH_REMATCH[1]}" + local value="${BASH_REMATCH[2]}" + + # Strip quotes if present (both single and double) + value="${value#\'}" + value="${value%\'}" + value="${value#\"}" + value="${value%\"}" + + # Trim trailing whitespace/comments + value="${value%%#*}" # Remove trailing comments + value="${value%% }" # Trim trailing spaces + + # Store in global variables + case "$key" in + default_version) DEFAULT_VERSION="$value" ;; + comment) COMMENT="$value" ;; + requires) REQUIRES="$value" ;; + schema) SCHEMA="$value" ;; + module_pathname) MODULE_PATHNAME="$value" ;; + esac + fi + done < "$control_file" + + # Validate required fields + if [ -z "$DEFAULT_VERSION" ]; then + die 1 "Control file missing default_version" + fi + + if [ -z "$COMMENT" ]; then + echo "WARNING: Control file missing comment, using extension name" >&2 + COMMENT="$EXTENSION extension" + fi + + # Warn about C code + if [ -n "$MODULE_PATHNAME" ]; then + cat >&2 <<-EOF + WARNING: Extension uses module_pathname (C code) + pg_tle only supports trusted languages (PL/pgSQL, SQL, etc.) + Generated SQL will likely not work + EOF + fi + + echo " default_version: $DEFAULT_VERSION" >&2 + echo " comment: $COMMENT" >&2 + if [ -n "$REQUIRES" ]; then + echo " requires: $REQUIRES" >&2 + fi + if [ -n "$SCHEMA" ]; then + echo " schema: $SCHEMA" >&2 + fi +} + +discover_sql_files() { + echo "Discovering SQL files for extension: $EXTENSION" >&2 + debug 30 "discover_sql_files: Starting discovery for extension: $EXTENSION" + + # Ensure default_version file exists and has content if base file exists + # This handles the case where make all hasn't generated it yet, or it exists but is empty + local default_version_file="sql/${EXTENSION}--${DEFAULT_VERSION}.sql" + local base_file="sql/${EXTENSION}.sql" + if [ -f "$base_file" ] && ([ ! -f "$default_version_file" ] || [ ! -s "$default_version_file" ]); then + debug 30 "discover_sql_files: Creating default_version file from base file" + cp "$base_file" "$default_version_file" + fi + + # Find versioned files: sql/{ext}--{version}.sql + # Use find to get proper null-delimited output, then filter out upgrade scripts + VERSION_FILES=() # Reset array + debug 30 "discover_sql_files: Reset VERSION_FILES array" + while IFS= read -r -d '' file; do + local basename=$(basename "$file" .sql) + local dash_count=$(echo "$basename" | grep -o -- "--" | wc -l | tr -d '[:space:]') + # Skip upgrade scripts (they have 2 dashes) + if [ "$dash_count" -ne 1 ]; then + continue + fi + # Error on empty version files + if [ ! -s "$file" ]; then + die 1 "Empty version file found: $file" + fi + VERSION_FILES+=("$file") + done < <(find sql/ -maxdepth 1 -name "${EXTENSION}--*.sql" -print0 2>/dev/null | sort -zV) + + # Find upgrade scripts: sql/{ext}--{ver1}--{ver2}.sql + # These have TWO occurrences of "--" in the filename + UPGRADE_FILES=() # Reset array + debug 30 "discover_sql_files: Reset UPGRADE_FILES array" + while IFS= read -r -d '' file; do + # Error on empty upgrade files + if [ ! -s "$file" ]; then + die 1 "Empty upgrade file found: $file" + fi + local basename=$(basename "$file" .sql) + local dash_count=$(echo "$basename" | grep -o -- "--" | wc -l | tr -d '[:space:]') + if [ "$dash_count" -eq 2 ]; then + UPGRADE_FILES+=("$file") + fi + done < <(find sql/ -maxdepth 1 -name "${EXTENSION}--*--*.sql" -print0 2>/dev/null | sort -zV) + + if [ ${#VERSION_FILES[@]} -eq 0 ]; then + die 1 "No versioned SQL files found for $EXTENSION + Expected pattern: sql/${EXTENSION}--{version}.sql + Run 'make' first to generate versioned files from sql/${EXTENSION}.sql" + fi + + echo " Found ${#VERSION_FILES[@]} version file(s):" >&2 + for f in "${VERSION_FILES[@]}"; do + echo " - $f" >&2 + done + + debug 30 "discover_sql_files: Checking UPGRADE_FILES array, count=${#UPGRADE_FILES[@]:-0}" + if [ ${#UPGRADE_FILES[@]:-0} -gt 0 ]; then + echo " Found ${#UPGRADE_FILES[@]} upgrade script(s):" >&2 + debug 30 "discover_sql_files: Iterating over ${#UPGRADE_FILES[@]} upgrade files" + for f in "${UPGRADE_FILES[@]}"; do + echo " - $f" >&2 + done + else + debug 30 "discover_sql_files: No upgrade files found" + fi +} + +extract_version_from_filename() { + local filename="$1" + local basename=$(basename "$filename" .sql) + + # Match patterns: + # - ext--1.0.0 → FROM_VERSION=1.0.0, TO_VERSION="" + # - ext--1.0.0--2.0.0 → FROM_VERSION=1.0.0, TO_VERSION=2.0.0 + + if [[ "$basename" =~ ^${EXTENSION}--([0-9][0-9.]*)(--([0-9][0-9.]*))?$ ]]; then + FROM_VERSION="${BASH_REMATCH[1]}" + TO_VERSION="${BASH_REMATCH[3]}" # Empty for non-upgrade files + return 0 + else + die 1 "Cannot parse version from filename: $filename + Expected format: ${EXTENSION}--{version}.sql or ${EXTENSION}--{ver1}--{ver2}.sql" + fi +} + +validate_delimiter() { + local sql_file="$1" + + if grep -qF "$PGTLE_DELIMITER" "$sql_file"; then + die 1 "SQL file contains reserved pg_tle delimiter: $sql_file + Found: $PGTLE_DELIMITER + This delimiter is used internally by pgtle.sh to wrap SQL content. + You must modify your SQL to not contain this string. If this poses a + serious problem, please open an issue at https://github.com/decibel/pgxntool/issues" + fi +} + +wrap_sql_content() { + local sql_file="$1" + + validate_delimiter "$sql_file" + + # Output wrapped SQL with proper indentation + echo " ${PGTLE_DELIMITER}" + cat "$sql_file" + echo " ${PGTLE_DELIMITER}" +} + +build_requires_array() { + # Input: "plpgsql, other_ext, another" + # Output: 'plpgsql', 'other_ext', 'another' + + # Split on comma, trim whitespace, quote each element + REQUIRES_ARRAY=$(echo "$REQUIRES" | \ + sed 's/[[:space:]]*,[[:space:]]*/\n/g' | \ + sed "s/^[[:space:]]*//;s/[[:space:]]*$//" | \ + sed "s/^/'/;s/$/'/" | \ + paste -sd, -) +} + +generate_header() { + local pgtle_version="$1" + local output_file="$2" + local version_count=${#VERSION_FILES[@]:-0} + local upgrade_count=${#UPGRADE_FILES[@]:-0} + + # Determine version compatibility message + local compat_msg + if [[ "$pgtle_version" == *"+"* ]]; then + local base_version="${pgtle_version%+}" + compat_msg="-- Works on pg_tle >= ${base_version}" + else + local min_version="${pgtle_version%-*}" + local max_version="${pgtle_version#*-}" + compat_msg="-- Works on pg_tle >= ${min_version} and < ${max_version}" + fi + + cat < $to_ver" + echo "SELECT pgtle.install_update_path(" + echo " '${EXTENSION}'," + echo " '${from_ver}'," + echo " '${to_ver}'," + wrap_sql_content "$upgrade_file" + echo ");" + echo +} + +generate_pgtle_sql() { + local pgtle_version="$1" + debug 30 "generate_pgtle_sql: Starting for version $pgtle_version, extension $EXTENSION" + + # Get capability using function (compatible with bash < 4.0) + local capability=$(get_pgtle_capability "$pgtle_version") + local version_dir="pg_tle/${pgtle_version}" + local output_file="${version_dir}/${EXTENSION}.sql" + + # Ensure arrays are initialized (defensive programming) + # Arrays should already be initialized at top level, but ensure they exist + debug 30 "generate_pgtle_sql: Checking array initialization" + debug 30 "generate_pgtle_sql: VERSION_FILES is ${VERSION_FILES+set}, count=${#VERSION_FILES[@]:-0}" + debug 30 "generate_pgtle_sql: UPGRADE_FILES is ${UPGRADE_FILES+set}, count=${#UPGRADE_FILES[@]:-0}" + + if [ -z "${VERSION_FILES+set}" ]; then + echo "WARNING: VERSION_FILES not set, initializing" >&2 + VERSION_FILES=() + fi + if [ -z "${UPGRADE_FILES+set}" ]; then + echo "WARNING: UPGRADE_FILES not set, initializing" >&2 + UPGRADE_FILES=() + fi + + # Create version-specific output directory if needed + mkdir -p "$version_dir" + + echo "Generating: $output_file (pg_tle $pgtle_version)" >&2 + + # Generate SQL to file + { + generate_header "$pgtle_version" "$output_file" + + cat < "$output_file" + + echo " ✓ Generated: $output_file" >&2 +} + +main "$@" + From c7b57055f8a522918a053247c557592a7f46ddc2 Mon Sep 17 00:00:00 2001 From: jnasbyupgrade Date: Thu, 8 Jan 2026 14:51:34 -0600 Subject: [PATCH 10/11] Add pg_tle 1.4.0-1.5.0 version range and refactor shared utilities Add new pg_tle version range to handle API changes in 1.4.0: - Split 1.0.0-1.5.0 range into 1.0.0-1.4.0 and 1.4.0-1.5.0 - Version 1.4.0 added uninstall function (backward-incompatible) - Version 1.5.0 added schema parameter (another boundary) - Update `base.mk` pattern rules for three version ranges - Update `pgtle.sh` documentation and logic for new range Extract shared utility functions into `lib.sh`: - Move `error()`, `die()`, and `debug()` functions from `pgtle.sh` - Update `pgtle.sh` and `setup.sh` to source `lib.sh` - Reduces code duplication and improves maintainability Refine `.gitattributes` export rules: - Add specific excludes: `CLAUDE.md`, `PLAN-*.md`, `.DS_Store` - Remove blanket `*.md` exclude (too broad) - Allows README.md and other docs to be included in distributions Add documentation about directory purity in `CLAUDE.md`: - Emphasize that pgxntool directory contains ONLY embedded files - Warn against adding temporary files or planning documents - Clarify that such files belong in pgxntool-test instead Co-Authored-By: Claude --- .gitattributes | 4 +- CLAUDE.md | 16 +++++++ base.mk | 32 +++++++++----- lib.sh | 40 +++++++++++++++++ pgtle.sh | 118 ++++++++++++++++++++++++++----------------------- setup.sh | 4 ++ 6 files changed, 147 insertions(+), 67 deletions(-) create mode 100644 lib.sh diff --git a/.gitattributes b/.gitattributes index 5b9a578..ef3e715 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,6 +1,8 @@ .gitattributes export-ignore .claude/ export-ignore +CLAUDE.md export-ignore +PLAN-*.md export-ignore +.DS_Store export-ignore *.asc export-ignore *.adoc export-ignore *.html export-ignore -*.md export-ignore diff --git a/CLAUDE.md b/CLAUDE.md index f4f149d..1b77984 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,6 +12,22 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co **Think of it like this**: pgxntool is to PostgreSQL extensions what a Makefile template library is to C projects - it's infrastructure code that gets copied into other projects, not a project itself. +## Critical: Directory Purity - NO Temporary Files + +**This directory contains ONLY files that get embedded into extension projects.** When extension developers run `git subtree add`, they pull the entire pgxntool directory into their project. + +**ABSOLUTE RULE**: NO temporary files, scratch work, or development tools may be added to this directory. + +**Examples of what NEVER belongs here:** +- Temporary files (scratch notes, test output, debugging artifacts) +- Development scripts or tools (these go in pgxntool-test/) +- Planning documents (PLAN-*.md files go in pgxntool-test/) +- Any file you wouldn't want in every extension project that uses pgxntool + +**CLAUDE.md exception**: CLAUDE.md exists here for AI assistant guidance, but is excluded from distributions via `.gitattributes export-ignore`. Same with `.claude/` directory. + +**Why this matters**: Any file you add here will be pulled into hundreds of extension projects via git subtree. Keep this directory lean and clean. + ## Development Workflow: Work from pgxntool-test **CRITICAL**: All development work on pgxntool should be done from the `../pgxntool-test/` repository, NOT from this repository. diff --git a/base.mk b/base.mk index affe732..39291cb 100644 --- a/base.mk +++ b/base.mk @@ -131,10 +131,11 @@ PGTLE_VERSION ?= # pg_tle version ranges we support # These correspond to different capability levels -PGTLE_VERSION_RANGES = 1.0.0-1.5.0 1.5.0+ +PGTLE_VERSION_RANGES = 1.0.0-1.4.0 1.4.0-1.5.0 1.5.0+ # pg_tle version subdirectories -PGTLE_1_0_TO_1_5_DIR = pg_tle/1.0.0-1.5.0 +PGTLE_1_0_TO_1_4_DIR = pg_tle/1.0.0-1.4.0 +PGTLE_1_4_TO_1_5_DIR = pg_tle/1.4.0-1.5.0 PGTLE_1_5_PLUS_DIR = pg_tle/1.5.0+ # Discover all extensions from control files in current directory @@ -147,18 +148,22 @@ PGXNTOOL_EXTENSIONS = $(basename $(PGXNTOOL_CONTROL_FILES)) ifeq ($(PGTLE_VERSION),) # Generate all versions (default) PGTLE_FILES = $(foreach ext,$(PGXNTOOL_EXTENSIONS),\ - $(PGTLE_1_0_TO_1_5_DIR)/$(ext).sql \ + $(PGTLE_1_0_TO_1_4_DIR)/$(ext).sql \ + $(PGTLE_1_4_TO_1_5_DIR)/$(ext).sql \ $(PGTLE_1_5_PLUS_DIR)/$(ext).sql) else # Generate only specified version - ifeq ($(PGTLE_VERSION),1.0.0-1.5.0) + ifeq ($(PGTLE_VERSION),1.0.0-1.4.0) PGTLE_FILES = $(foreach ext,$(PGXNTOOL_EXTENSIONS),\ - $(PGTLE_1_0_TO_1_5_DIR)/$(ext).sql) + $(PGTLE_1_0_TO_1_4_DIR)/$(ext).sql) + else ifeq ($(PGTLE_VERSION),1.4.0-1.5.0) + PGTLE_FILES = $(foreach ext,$(PGXNTOOL_EXTENSIONS),\ + $(PGTLE_1_4_TO_1_5_DIR)/$(ext).sql) else ifeq ($(PGTLE_VERSION),1.5.0+) PGTLE_FILES = $(foreach ext,$(PGXNTOOL_EXTENSIONS),\ $(PGTLE_1_5_PLUS_DIR)/$(ext).sql) else - $(error Invalid PGTLE_VERSION: $(PGTLE_VERSION). Use 1.0.0-1.5.0 or 1.5.0+) + $(error Invalid PGTLE_VERSION: $(PGTLE_VERSION). Use 1.0.0-1.4.0, 1.4.0-1.5.0, or 1.5.0+) endif endif @@ -173,18 +178,25 @@ pgtle: all meta.mk $(PGXNTOOL_CONTROL_FILES) $(PGTLE_FILES) # Enable secondary expansion for dynamic dependencies .SECONDEXPANSION: -# Pattern rule for generating pg_tle 1.0.0-1.5.0 files +# Pattern rule for generating pg_tle 1.0.0-1.4.0 files # Dependencies: # - Control file (metadata source) # - Generator script (tool itself) # - All SQL files for this extension (using secondary expansion) # Note: We depend on $(EXTENSION_VERSION_FILES) at the pgtle target level # to ensure all versioned files exist before pattern rules run -$(PGTLE_1_0_TO_1_5_DIR)/%.sql: %.control $(PGXNTOOL_DIR)/pgtle.sh $$(wildcard sql/$$*--*.sql) $$(wildcard sql/$$*.sql) - @mkdir -p $(PGTLE_1_0_TO_1_5_DIR) +$(PGTLE_1_0_TO_1_4_DIR)/%.sql: %.control $(PGXNTOOL_DIR)/pgtle.sh $$(wildcard sql/$$*--*.sql) $$(wildcard sql/$$*.sql) + @mkdir -p $(PGTLE_1_0_TO_1_4_DIR) + @$(PGXNTOOL_DIR)/pgtle.sh \ + --extension $(basename $<) \ + --pgtle-version 1.0.0-1.4.0 + +# Pattern rule for generating pg_tle 1.4.0-1.5.0 files +$(PGTLE_1_4_TO_1_5_DIR)/%.sql: %.control $(PGXNTOOL_DIR)/pgtle.sh $$(wildcard sql/$$*--*.sql) $$(wildcard sql/$$*.sql) + @mkdir -p $(PGTLE_1_4_TO_1_5_DIR) @$(PGXNTOOL_DIR)/pgtle.sh \ --extension $(basename $<) \ - --pgtle-version 1.0.0-1.5.0 + --pgtle-version 1.4.0-1.5.0 # Pattern rule for generating pg_tle 1.5.0+ files $(PGTLE_1_5_PLUS_DIR)/%.sql: %.control $(PGXNTOOL_DIR)/pgtle.sh $$(wildcard sql/$$*--*.sql) $$(wildcard sql/$$*.sql) diff --git a/lib.sh b/lib.sh new file mode 100644 index 0000000..379b6dc --- /dev/null +++ b/lib.sh @@ -0,0 +1,40 @@ +#!/bin/bash +# lib.sh - Common utility functions for pgxntool scripts +# +# This file is meant to be sourced by other scripts, not executed directly. +# Usage: source "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +# Error function - outputs to stderr but doesn't exit +# Usage: error "message" +error() { + echo "ERROR: $*" >&2 +} + +# Die function - outputs error message and exits with specified code +# Usage: die EXIT_CODE "message" +die() { + local exit_code=$1 + shift + error "$@" + exit "$exit_code" +} + +# Debug function +# Usage: debug LEVEL "message" +# Outputs message to stderr if DEBUG >= LEVEL +# Debug levels use multiples of 10 (10, 20, 30, 40, etc.) to allow for easy expansion +# - 10: Critical errors, important warnings +# - 20: Warnings, significant state changes +# - 30: General debugging, function entry/exit, array operations +# - 40: Verbose details, loop iterations +# - 50+: Maximum verbosity +# Enable with: DEBUG=30 scriptname.sh +debug() { + local level=$1 + shift + local message="$*" + + if [ "${DEBUG:-0}" -ge "$level" ]; then + echo "DEBUG[$level]: $message" >&2 + fi +} diff --git a/pgtle.sh b/pgtle.sh index 92103da..8fd2d17 100755 --- a/pgtle.sh +++ b/pgtle.sh @@ -27,13 +27,13 @@ # # --pgtle-version VERSION # Generate for specific pg_tle version only (optional). -# Format: 1.0.0-1.5.0 or 1.5.0+ +# Format: 1.0.0-1.4.0, 1.4.0-1.5.0, or 1.5.0+ # Default: Generate all supported versions # # --get-dir VERSION # Returns the directory path for the given pg_tle version. # Format: VERSION is a version string like "1.5.2" -# Output: Directory path like "pg_tle/1.5.0+" or "pg_tle/1.0.0-1.5.0" +# Output: Directory path like "pg_tle/1.5.0+", "pg_tle/1.4.0-1.5.0", or "pg_tle/1.0.0-1.4.0" # This option is used by make to determine which directory to use # # --get-version @@ -54,11 +54,13 @@ # # Note the boundary conditions: # 1.5.0+ means >= 1.5.0 (includes 1.5.0) -# 1.0.0-1.5.0 means >= 1.0.0 and < 1.5.0 (excludes 1.5.0) +# 1.4.0-1.5.0 means >= 1.4.0 and < 1.5.0 (excludes 1.5.0) +# 1.0.0-1.4.0 means >= 1.0.0 and < 1.4.0 (excludes 1.4.0) # # SUPPORTED VERSIONS -# 1.0.0-1.5.0 pg_tle 1.0.0 through 1.4.x (no schema parameter) -# 1.5.0+ pg_tle 1.5.0 and later (schema parameter support) +# 1.0.0-1.4.0 pg_tle 1.0.0 through 1.3.x (no uninstall function, no schema parameter) +# 1.4.0-1.5.0 pg_tle 1.4.0 through 1.4.x (has uninstall function, no schema parameter) +# 1.5.0+ pg_tle 1.5.0 and later (has uninstall function, schema parameter support) # # EXAMPLES # # Generate all versions (default) @@ -71,6 +73,9 @@ # pgtle.sh --get-dir 1.5.2 # # Output: pg_tle/1.5.0+ # +# pgtle.sh --get-dir 1.4.2 +# # Output: pg_tle/1.4.0-1.5.0 +# # # Get installed pg_tle version from database # pgtle.sh --get-version # # Output: 1.5.2 (or empty if not installed) @@ -80,7 +85,8 @@ # # OUTPUT # Creates files in version-specific subdirectories: -# pg_tle/1.0.0-1.5.0/{extension}.sql +# pg_tle/1.0.0-1.4.0/{extension}.sql +# pg_tle/1.4.0-1.5.0/{extension}.sql # pg_tle/1.5.0+/{extension}.sql # # Each file contains: @@ -112,55 +118,27 @@ set -eo pipefail -# Error function - outputs to stderr but doesn't exit -# Usage: error "message" -error() { - echo "ERROR: $*" >&2 -} - -# Die function - outputs error message and exits with specified code -# Usage: die EXIT_CODE "message" -die() { - local exit_code=$1 - shift - error "$@" - exit "$exit_code" -} - -# Debug function -# Usage: debug LEVEL "message" -# Outputs message to stderr if DEBUG >= LEVEL -# Debug levels use multiples of 10 (10, 20, 30, 40, etc.) to allow for easy expansion -# - 10: Critical errors, important warnings -# - 20: Warnings, significant state changes -# - 30: General debugging, function entry/exit, array operations -# - 40: Verbose details, loop iterations -# - 50+: Maximum verbosity -# Enable with: DEBUG=30 pgtle.sh --extension myext -debug() { - local level=$1 - shift - local message="$*" - - if [ "${DEBUG:-0}" -ge "$level" ]; then - echo "DEBUG[$level]: $message" >&2 - fi -} +# Source common library functions (error, die, debug) +PGXNTOOL_DIR="$(dirname "${BASH_SOURCE[0]}")" +source "$PGXNTOOL_DIR/lib.sh" # Constants PGTLE_DELIMITER='$_pgtle_wrap_delimiter_$' -PGTLE_VERSIONS=("1.0.0-1.5.0" "1.5.0+") +PGTLE_VERSIONS=("1.0.0-1.4.0" "1.4.0-1.5.0" "1.5.0+") # Supported pg_tle version ranges and their capabilities # Use a function instead of associative array for compatibility with bash < 4.0 get_pgtle_capability() { local version="$1" case "$version" in - "1.0.0-1.5.0") - echo "no_schema_param" + "1.0.0-1.4.0") + echo "no_uninstall_no_schema" + ;; + "1.4.0-1.5.0") + echo "has_uninstall_no_schema" ;; "1.5.0+") - echo "schema_param" + echo "has_uninstall_has_schema" ;; *) echo "unknown" @@ -262,7 +240,7 @@ version_to_number() { # Get directory for a given pg_tle version # Takes a version string like "1.5.2" and returns the directory path # Handles versions with suffixes (e.g., "1.5.0alpha1") -# Returns: "pg_tle/1.0.0-1.5.0" or "pg_tle/1.5.0+" +# Returns: "pg_tle/1.0.0-1.4.0", "pg_tle/1.4.0-1.5.0", or "pg_tle/1.5.0+" get_version_dir() { local version="$1" @@ -274,15 +252,40 @@ get_version_dir() { local numeric_version numeric_version=$(parse_version "$version") + # Check if the original version has a pre-release suffix + # Pre-release versions (alpha, beta, rc, dev) are considered BEFORE the release + # Example: 1.4.0alpha1 comes BEFORE 1.4.0, so it should use the 1.0.0-1.4.0 range + local has_prerelease=0 + if [[ "$version" =~ (alpha|beta|rc|dev) ]]; then + has_prerelease=1 + fi + # Convert versions to comparable numbers local version_num - local threshold_num + local threshold_1_4_num + local threshold_1_5_num version_num=$(version_to_number "$numeric_version") - threshold_num=$(version_to_number "1.5.0") - - # Compare: if version < 1.5.0, use 1.0.0-1.5.0 directory; otherwise use 1.5.0+ directory - if [ "$version_num" -lt "$threshold_num" ]; then - echo "pg_tle/1.0.0-1.5.0" + threshold_1_4_num=$(version_to_number "1.4.0") + threshold_1_5_num=$(version_to_number "1.5.0") + + # Compare and return appropriate directory: + # < 1.4.0 -> 1.0.0-1.4.0 + # >= 1.4.0 and < 1.5.0 -> 1.4.0-1.5.0 + # >= 1.5.0 -> 1.5.0+ + # + # Special handling for pre-release versions: + # If version equals a threshold but has a pre-release suffix, treat it as less than that threshold + # Example: 1.4.0alpha1 is treated as < 1.4.0, so it uses 1.0.0-1.4.0 + if [ "$version_num" -lt "$threshold_1_4_num" ]; then + echo "pg_tle/1.0.0-1.4.0" + elif [ "$version_num" -eq "$threshold_1_4_num" ] && [ "$has_prerelease" -eq 1 ]; then + # Pre-release of 1.4.0 is considered < 1.4.0 + echo "pg_tle/1.0.0-1.4.0" + elif [ "$version_num" -lt "$threshold_1_5_num" ]; then + echo "pg_tle/1.4.0-1.5.0" + elif [ "$version_num" -eq "$threshold_1_5_num" ] && [ "$has_prerelease" -eq 1 ]; then + # Pre-release of 1.5.0 is considered < 1.5.0 + echo "pg_tle/1.4.0-1.5.0" else echo "pg_tle/1.5.0+" fi @@ -703,7 +706,7 @@ generate_install_extension() { fi # Add schema parameter only for capability version 1.5.0+ - if [ "$capability" = "schema_param" ]; then + if [ "$capability" = "has_uninstall_has_schema" ]; then if [ -n "$SCHEMA" ]; then echo " , '${SCHEMA}' -- schema parameter (pg_tle 1.5.0+)" else @@ -783,17 +786,19 @@ generate_pgtle_sql() { cat < Date: Fri, 9 Jan 2026 12:00:22 -0600 Subject: [PATCH 11/11] Remove pgxntool-test-template and convert to two-repo pattern Remove references to pgxntool-test-template (now consolidated): - Template files moved to pgxntool-test/template/ - Remove template directory from `.claude/settings.json` - Update `CLAUDE.md` for two-repository pattern Convert `.claude/commands/commit.md` to symlink: - Development happens in pgxntool-test, avoid duplication - Symlink: `commit.md -> ../../../pgxntool-test/.claude/commands/commit.md` Related changes in pgxntool-test: - Consolidate template files into pgxntool-test/template/ - Simplify commit workflow to two-phase (remove amend step) Co-Authored-By: Claude --- .claude/commands/commit.md | 140 +------------------------------------ .claude/settings.json | 3 +- CLAUDE.md | 16 ++--- 3 files changed, 10 insertions(+), 149 deletions(-) mode change 100644 => 120000 .claude/commands/commit.md diff --git a/.claude/commands/commit.md b/.claude/commands/commit.md deleted file mode 100644 index 8d6871c..0000000 --- a/.claude/commands/commit.md +++ /dev/null @@ -1,139 +0,0 @@ ---- -description: Create a git commit following project standards and safety protocols -allowed-tools: Bash(git status:*), Bash(git log:*), Bash(git add:*), Bash(git diff:*), Bash(git commit:*), Bash(make test:*) ---- - -# commit - -Create a git commit following all project standards and safety protocols for pgxntool-test. - -**FIRST: Check ALL three repositories for changes** - -**CRITICAL**: Before doing ANYTHING else, you MUST check git status in all 3 repositories to understand the full scope of changes: - -```bash -# Check pgxntool (main framework) -echo "=== pgxntool status ===" -cd ../pgxntool && git status - -# Check pgxntool-test (test harness) -echo "=== pgxntool-test status ===" -cd ../pgxntool-test && git status - -# Check pgxntool-test-template (test template) -echo "=== pgxntool-test-template status ===" -cd ../pgxntool-test-template && git status -``` - -**Why this matters**: Work on pgxntool frequently involves changes across all 3 repositories. You need to understand the complete picture before committing anywhere. - -**IMPORTANT**: If ANY of the 3 repositories have changes, you should commit ALL of them that have changes (unless the user explicitly says otherwise). This ensures related changes stay synchronized across the repos. - -**DO NOT create empty commits** - Only commit repos that actually have changes (modified/untracked files). If a repo has no changes, skip it. - ---- - -**CRITICAL REQUIREMENTS:** - -1. **Git Safety**: Never update `git config`, never force push to `main`/`master`, never skip hooks unless explicitly requested - -2. **Commit Attribution**: Do NOT add "Generated with Claude Code" to commit message body. The standard Co-Authored-By trailer is acceptable per project CLAUDE.md. - -3. **Testing**: ALL tests must pass before committing: - - Run `make test` - - Check the output carefully for any "not ok" lines - - Count passing vs total tests - - **If ANY tests fail: STOP. Do NOT commit. Ask the user what to do.** - - There is NO such thing as an "acceptable" failing test - - Do NOT rationalize failures as "pre-existing" or "unrelated" - -**WORKFLOW:** - -1. Run in parallel: `git status`, `git diff --stat`, `git log -10 --oneline` - -2. Check test status - THIS IS MANDATORY: - - Run `make test 2>&1 | tee /tmp/test-output.txt` - - Check for failing tests: `grep "^not ok" /tmp/test-output.txt` - - If ANY tests fail: STOP immediately and inform the user - - Only proceed if ALL tests pass - -3. Analyze changes and draft concise commit message following this repo's style: - - Look at `git log -10 --oneline` to match existing style - - Be factual and direct (e.g., "Fix BATS dist test to create its own distribution") - - Focus on "why" when it adds value, otherwise just describe "what" - - List items in roughly decreasing order of impact - - Keep related items grouped together - - **In commit messages**: Wrap all code references in backticks - filenames, paths, commands, function names, variables, make targets, etc. - - Examples: `helpers.bash`, `make test-recursion`, `setup_sequential_test()`, `TEST_REPO`, `.envs/`, `01-meta.bats` - - Prevents markdown parsing issues and improves clarity - -4. **PRESENT the proposed commit message to the user and WAIT for approval before proceeding** - -5. After receiving approval, stage changes appropriately using `git add` - - **CRITICAL: Include ALL new files** - - Check `git status` for untracked files - - **ALL untracked files that are part of the feature/change MUST be staged** - - New scripts, new documentation, new helper files, etc. should all be included - - Do NOT leave new files uncommitted unless explicitly told to exclude them - - When in doubt, include the file - it's better to commit too much than to leave important files uncommitted - -6. **VERIFY staged files with `git status`**: - - If user did NOT specify a subset: Confirm ALL modified AND untracked files are staged - - If user specified only certain files: Confirm ONLY those files are staged - - **Check for any untracked files that should be part of this commit** - - STOP and ask user if staging doesn't match intent - -7. After verification, commit using `HEREDOC` format: -```bash -git commit -m "$(cat <<'EOF' -Subject line (imperative mood, < 72 chars) - -Additional context if needed, wrapped at 72 characters. - -Co-Authored-By: Claude -EOF -)" -``` - -8. Run `git status` after commit to verify success - -9. If pre-commit hook modifies files: Check authorship (`git log -1 --format='%an %ae'`) and branch status, then amend if safe or create new commit - -10. **Repeat steps 1-9 for each repository that has changes**: - - After committing one repo, move to the next repo with changes (usually: pgxntool → pgxntool-test → pgxntool-test-template) - - Draft appropriate commit messages that reference the primary changes - - **ONLY commit repos with actual changes** - skip any repo that has no modified/untracked files - - Ensure ALL repos with changes get committed before finishing - -**MULTI-REPO COMMIT CONTEXT:** - -**CRITICAL**: Work on pgxntool frequently involves changes across all 3 repositories simultaneously: -- **pgxntool** (this repo) - The main framework -- **pgxntool-test** (at `../pgxntool-test/`) - Test harness -- **pgxntool-test-template** (at `../pgxntool-test-template/`) - Test template - -**This is why you MUST check all 3 repositories at the start** (see FIRST step above). - -**DEFAULT BEHAVIOR: Commit ALL repos with changes together** - If any of the 3 repos have changes when you check them, you should plan to commit ALL repos that have changes (unless user explicitly specifies otherwise). This keeps related changes synchronized. **Do NOT create empty commits** - only commit repos with actual modified/untracked files. - -When committing changes that span repositories: -1. **Commit messages in pgxntool-test and pgxntool-test-template should reference the main changes in pgxntool** - - Example: "Add tests for pg_tle support (see pgxntool commit for implementation)" - - Example: "Update template for pg_tle feature (see pgxntool commit for details)" - -2. **When working across repos, commit in logical order:** - - Usually: pgxntool → pgxntool-test → pgxntool-test-template - - But adapt based on dependencies - -**REPOSITORY CONTEXT:** - -This is pgxntool, a PostgreSQL extension build framework. Key facts: -- Main Makefile is `base.mk` -- Scripts live in root directory -- Documentation is in `README.asc` (generates `README.html`) - -**RESTRICTIONS:** -- DO NOT push unless explicitly asked -- DO NOT commit files with actual secrets (`.env`, `credentials.json`, etc.) -- Never use `-i` flags (`git commit -i`, `git rebase -i`, etc.) diff --git a/.claude/commands/commit.md b/.claude/commands/commit.md new file mode 120000 index 0000000..e59a331 --- /dev/null +++ b/.claude/commands/commit.md @@ -0,0 +1 @@ +../../../pgxntool-test/.claude/commands/commit.md \ No newline at end of file diff --git a/.claude/settings.json b/.claude/settings.json index e7d75ad..e7bf5a9 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -13,8 +13,7 @@ "Bash(head:*)" ], "additionalDirectories": [ - "../pgxntool-test/", - "../pgxntool-test-template/" + "../pgxntool-test/" ] } } diff --git a/CLAUDE.md b/CLAUDE.md index 1b77984..cf29e89 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -42,18 +42,19 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co - Edit files in `../pgxntool/` but run all commands from `../pgxntool-test/` - All testing, documentation of testing, and development utilities belong in pgxntool-test -## Three-Repository Development Pattern +## Two-Repository Development Pattern -This codebase uses an unusual three-repository testing pattern: +This codebase uses a two-repository testing pattern: 1. **pgxntool/** (this repo) - The framework code that gets embedded into extension projects -2. **../pgxntool-test-template/** - A minimal "dummy" PostgreSQL extension that has pgxntool embedded (serves as an example consumer) -3. **../pgxntool-test/** - The test harness that clones the template, exercises pgxntool's functionality, and validates outputs +2. **../pgxntool-test/** - The test harness that creates test repositories, exercises pgxntool's functionality, and validates outputs + +The test harness contains template extension files in `../pgxntool-test/template/` which are used to create fresh test repositories. **To test changes to pgxntool**, you must: 1. Ensure changes are visible to pgxntool-test (via git commit or using the rsync mechanism in the test harness) 2. Run tests from `../pgxntool-test/` -3. The test harness will clone `../pgxntool-test-template/`, sync in pgxntool changes, and validate behavior +3. The test harness will create a fresh test repository from the template, sync in pgxntool changes, and validate behavior ## How Extension Developers Use pgxntool @@ -167,7 +168,7 @@ When modifying pgxntool: 1. **Make changes** in this repo (pgxntool/) 2. **Test changes** by running `make test` in `../pgxntool-test/` - - The test harness clones `../pgxntool-test-template/` + - The test harness creates a fresh test repository from `../pgxntool-test/template/` - If pgxntool is dirty, it rsyncs your uncommitted changes - Runs setup, builds, tests, and validates outputs 3. **Examine results** in `../pgxntool-test/results/` and `../pgxntool-test/diffs/` @@ -274,6 +275,5 @@ See `README-pgtle.md` for complete user documentation. ## Related Repositories -- **../pgxntool-test/** - Test harness for validating pgxntool functionality -- **../pgxntool-test-template/** - Minimal extension project used as test subject +- **../pgxntool-test/** - Test harness for validating pgxntool functionality (includes template extension files in `template/` directory) - Never produce any kind of metrics or estimates unless you have data to back them up. If you do have data you MUST reference it. \ No newline at end of file