Repository Index Generation

Overview

The generate-repository-index.sh script creates and maintains OSGi R5 repository metadata for all JAR bundles in the GOSS-Repository. The script is optimized to only process new or modified JAR files, making it efficient for incremental updates.

Usage

Basic Usage

cd /home/debian/repos/GridAPPSD/GOSS-Repository
./generate-repository-index.sh

This will:

Detect new/modified JAR files (via git or modification time)
Extract OSGi metadata from only those JARs
Preserve existing entries in index.xml
Generate updated index.xml and index.xml.gz

Force Full Regeneration

To regenerate metadata for all JARs (useful after corruption or major changes):

./generate-repository-index.sh --force

Custom Repository Path

By default, the script operates on the dependencies/ directory. To specify a different path:

./generate-repository-index.sh /path/to/repository

Or with force:

./generate-repository-index.sh --force /path/to/repository

How It Works

Change Detection

The script uses two methods to detect changes:

Git Repository (preferred):
- Uses git status --porcelain to find new, modified, or added JAR files
- Only processes JARs with status: ?? (untracked), M (modified), or A (added)
Modification Time (fallback):
- If not in a git repository, compares JAR modification times with index.xml
- Processes JARs newer than the existing index

Incremental Updates

When updating the index:

Parse Existing Index: Extracts all <resource> blocks from current index.xml
Identify Changed JARs: Creates a map of JAR paths that have changed
Preserve Unchanged: Copies resource blocks for unchanged JARs from old index
Process Changed: Extracts metadata and generates new resource blocks for changed JARs
Merge: Combines preserved and new entries into updated index.xml

Maven Metadata

The script also generates maven-metadata.xml files for each artifact directory, but only for directories containing changed JARs:

<metadata>
  <groupId>org.apache.jena.osgi</groupId>
  <artifactId>osgi</artifactId>
  <versioning>
    <latest>5.6.0</latest>
    <release>5.6.0</release>
    <versions>
      <version>4.1.0</version>
      <version>5.6.0</version>
    </versions>
    <lastUpdated>20251121151700</lastUpdated>
  </versioning>
</metadata>

Output Files

index.xml

OSGi R5 repository index containing metadata for all bundles:

<?xml version="1.0" encoding="UTF-8"?>
<repository increment="1763766000000" name="GOSS Archive Repository" xmlns="http://www.osgi.org/xmlns/repository/v1.0.0">
  <resource>
    <capability namespace="osgi.identity">
      <attribute name="osgi.identity" value="org.apache.jena.osgi"/>
      <attribute name="type" value="osgi.bundle"/>
      <attribute name="version" type="Version" value="5.6.0"/>
    </capability>
    <capability namespace="osgi.content">
      <attribute name="osgi.content" value="sha256-hash"/>
      <attribute name="url" value="dependencies/org.apache.jena.osgi/org.apache.jena.osgi-5.6.0.jar"/>
      <attribute name="size" type="Long" value="27262976"/>
      <attribute name="mime" value="application/vnd.osgi.bundle"/>
    </capability>
  </resource>
  <!-- More resources... -->
</repository>

index.xml.gz

Compressed version of index.xml for faster downloads.

maven-metadata.xml

Maven metadata for each artifact directory (e.g., dependencies/org.apache.jena.osgi/maven-metadata.xml).

Workflow Examples

Adding a New Bundle

# 1. Copy new bundle to repository
cp /path/to/mybundle-1.0.0.jar dependencies/com.example.mybundle/

# 2. Regenerate index (processes only the new JAR)
cd /home/debian/repos/GridAPPSD/GOSS-Repository
./generate-repository-index.sh

# 3. Commit changes
git add dependencies/com.example.mybundle/mybundle-1.0.0.jar
git add dependencies/com.example.mybundle/maven-metadata.xml
git add index.xml index.xml.gz
git commit -m "Add mybundle 1.0.0"

Updating an Existing Bundle

# 1. Replace or add new version
cp /path/to/mybundle-2.0.0.jar dependencies/com.example.mybundle/

# 2. Regenerate index
./generate-repository-index.sh

# Script will:
# - Remove old entry for mybundle-1.0.0 (if replaced)
# - Add new entry for mybundle-2.0.0
# - Update maven-metadata.xml with both versions

# 3. Commit
git add dependencies/com.example.mybundle/
git add index.xml index.xml.gz
git commit -m "Update mybundle to 2.0.0"

No Changes Scenario

If you run the script and no JARs have changed:

$ ./generate-repository-index.sh

[INFO] Generating repository index for: /home/debian/repos/GridAPPSD/GOSS-Repository/dependencies
[INFO] Git repository detected - will use git to detect changes
[INFO] Found 0 new/modified JAR(s) via git
[INFO] Updating Maven metadata...
[INFO] No Maven metadata updates needed
[INFO] Generating OSGi repository index...
[INFO] No changes detected - index is up to date

========================================
Repository Index Update Complete!
  Repository:       /home/debian/repos/GridAPPSD/GOSS-Repository/dependencies
  OSGi index:       index.xml
  Compressed index: index.xml.gz
========================================

[INFO] ✓ Done!

Performance

Before (Old Script)

Processed all JAR files every time (~500+ JARs in GOSS-Repository)
Regeneration time: ~2-5 minutes
Generated completely new index each time

After (Updated Script)

Processes only new/modified JAR files
Typical regeneration time: ~5-10 seconds for 1-2 new JARs
Preserves existing metadata, only updates what changed
--force option available for full regeneration when needed

Troubleshooting

"No existing index.xml found"

If this is the first run, the script will process all JARs. This is normal and expected.

Git Not Detecting Changes

Make sure JAR files are not in .gitignore. Check with:

git check-ignore dependencies/path/to/bundle.jar

Corrupted Index

If the index becomes corrupted, regenerate completely:

./generate-repository-index.sh --force

Missing unzip or jar Command

The script tries unzip first, then falls back to jar command. If both are missing:

# Install unzip (Debian/Ubuntu)
sudo apt-get install unzip

# Or ensure Java is installed (provides jar command)
java -version

Debug Mode

To see detailed debug output:

DEBUG=true ./generate-repository-index.sh

This will show:

Number of existing bundle entries parsed
Each JAR being processed
Maven metadata generation for each directory

GitHub Actions Automation

The repository includes a GitHub Actions workflow that automatically regenerates indexes when JAR files are pushed to the repository.

Workflow File

.github/workflows/regenerate-indexes.yml

When It Triggers

The workflow runs automatically when:

Any .jar file is pushed to the master branch
The workflow is manually triggered via GitHub UI (workflow_dispatch)

What It Does

Checkout Repository: Fetches the complete repository with full history
Set Up Environment: Installs required tools (Java 11, unzip, gzip)
Regenerate cnf/releaserepo indexes: Runs ./generate-repository-index.sh cnf/releaserepo
Regenerate dependencies indexes: Runs ./generate-repository-index.sh dependencies
Check for Changes: Detects if any index files were modified
Commit and Push: If changes detected, commits and pushes:
- cnf/releaserepo/index.xml
- cnf/releaserepo/index.xml.gz
- cnf/releaserepo/*/maven-metadata.xml
- dependencies/index.xml
- dependencies/index.xml.gz
- dependencies/*/maven-metadata.xml

Commit Message Format

When the workflow commits changes:

Auto-regenerate repository indexes

Generated by GitHub Actions after JAR file update

[skip ci]

The [skip ci] tag prevents the workflow from triggering itself recursively.

Manual Triggering

To manually trigger the workflow:

Go to the repository on GitHub
Click Actions tab
Select Regenerate Repository Indexes workflow
Click Run workflow button
Select the master branch
Click Run workflow

This is useful when:

You want to force regeneration without pushing new JARs
The workflow failed previously and you want to retry
You've made changes to the script and want to regenerate all indexes

Viewing Workflow Results

To see workflow execution:

Go to Actions tab on GitHub
Click on the workflow run to see detailed logs
Expand each step to see command output
Check for any errors or warnings

Workflow Permissions

The workflow uses ${{ secrets.GITHUB_TOKEN }} which has permissions to:

Read repository contents
Push commits to the repository
This token is automatically provided by GitHub Actions

Troubleshooting Workflow Issues

Workflow doesn't trigger:

Verify you pushed to the master branch (not main or another branch)
Verify you pushed at least one .jar file
Check workflow file syntax in .github/workflows/regenerate-indexes.yml

Workflow fails during index generation:

Check the workflow logs in the Actions tab
Look for script errors in the "Regenerate indexes" steps
Verify all JARs have valid OSGi metadata
Check for disk space or permission issues

Workflow fails to push:

Verify the repository allows GitHub Actions to push
Check that branch protection rules don't block GitHub Actions
Ensure the workflow has write permissions

Disabling the Workflow

To temporarily disable automatic index regeneration:

Rename or delete .github/workflows/regenerate-indexes.yml
Commit and push the change

To re-enable, restore the file.

Local vs Automated Workflow

When to run locally:

During development and testing
When you want immediate feedback
When working with --force flag for full regeneration

When to rely on the workflow:

For standard JAR additions in production
To ensure consistent index generation
To maintain audit trail of index changes

Both approaches are valid and produce the same results.

FilesExpand file tree

INDEXING.md

Latest commit

History