feat: add GitLab metadata processor#152
Merged
Merged
Conversation
- Refactor github_metadata.py to software_metadata.py with GitHub and GitLab support
- Breaking: gh_metadata_only_missing module option is renamed to metadata_only_missing
- GitLab does not support commit history (see nodiscc#48)
- processor can now be relatively easily extended to support other GrahQL API based services
- Updated documentations where applicable
- Updated relevant tests
Fixes nodiscc#48
This comment was marked as outdated.
This comment was marked as outdated.
- Fixes logic for missing repositories either not being found in search results or not being found in the API - Fixes issue that repositories, which are forks, where not being processed by the API
- Clean up code in `software_metadata.py` to make it more readable and maintainable - Updates default values for `batch_size_github` to `25` and `sleep_time` to `5`
nodiscc
previously requested changes
Jan 24, 2026
- Removes `execpt Exception` for more specific exceptions - Logs exception details for write software entry errors - Replaces Datetime Arithmetic Operations with `calendar.monthrange`
nodiscc
added a commit
to awesome-selfhosted/awesome-selfhosted-data
that referenced
this pull request
Mar 11, 2026
Point to the backend repository URL instead of the gitlab project subgroup - ref. nodiscc/hecat#152
Owner
|
I updated the Nirvati source code URL awesome-selfhosted/awesome-selfhosted-data#2138 I still need to review the main change in this PR, will do. |
nodiscc
requested changes
Mar 21, 2026
Co-authored-by: nodiscc <nodiscc@gmail.com>
nodiscc
approved these changes
Mar 22, 2026
nodiscc
left a comment
nodiscc
left a comment
Owner
There was a problem hiding this comment.
This looks good.
One thing to consider, who will provide the GITLAB_TOKEN for CI jobs that run on https://github.com/awesome-selfhosted/awesome-selfhosted-data?
Currently this is not a problem for the Github token, since Github actions issues an automatic token to all CI jobs, with minimal permissions and not tied to any user account. However for Gitlab I think one of us needs to provide an access token.
There is an old https://gitlab.com/awesome-selfhosted group (Github organization equivalent on Gitlab) which I had created a while ago. I'll check if it is possible to issue an access token on behalf of the group (with minimal access in case it is accidentally leaked by github actions)
Contributor
Author
This would be the best, otherwise I would say a dummy account would probably make the most sense. |
Owner
|
Using a service account + access token in the gitlab group appears to work perfectly
export GITHUB_TOKEN=****
export GITLAB_TOKEN=****
$ make test_update_software_metadata
...
...
INFO:software_metadata.py: processing GitHub batch 6/6 (size: 10)
DEBUG:software_metadata.py: batch contents: ['wordpress/wordpress', 'josephernest/writing', 'yacy/yacy_search_server', 'totonyus/ydl_api_ng', 'nbr23/youtube-dl-server', 'zammad/zammad', 'zerok-rts/zero-k', 'zim-desktop-wiki/zim-desktop-wiki', 'znc/znc', 'pikvm/ustreamer']
DEBUG:connectionpool.py: Starting new HTTPS connection (1): api.github.com:443
DEBUG:connectionpool.py: https://api.github.com:443 "POST /graphql HTTP/1.1" 200 None
DEBUG:software_metadata.py: Rate limit (Limit/Remain/Used/Reset): 5000/4994/6/1774208075
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/wordpress.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/zammad.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/yacy.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/zim.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/znc.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/µstreamer.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/writing.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/zero-k.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/youtubedl-server.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/ydl_api_ng.yml
INFO:software_metadata.py: processing 27 GitLab projects
INFO:software_metadata.py: processing GitLab batch 1/3 (size: 10)
DEBUG:software_metadata.py: batch contents: ['baserow/baserow', 'famedly/conduit', 'crafty-controller/crafty-4', 'davical-project/davical', 'elixire/elixire', 'dacid99/eonvelope', 'fdroid/fdroidserver', 'fmd-foss/fmd-server', 'gitlab-org/gitlab-foss', 'glitchtip/glitchtip']
DEBUG:connectionpool.py: Starting new HTTPS connection (1): gitlab.com:443
DEBUG:connectionpool.py: https://gitlab.com:443 "POST /api/graphql HTTP/1.1" 200 None
DEBUG:software_metadata.py: Rate limit (Limit/Remain/Used/Reset): 2000/1999/1/1774204560
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/baserow.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/conduit.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/crafty-controller.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/davical.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/elixire.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/eonvelope.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/f-droid.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/fmd-server.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/gitlab.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/glitchtip.yml
DEBUG:software_metadata.py: sleeping 5s between batches
INFO:software_metadata.py: processing GitLab batch 2/3 (size: 10)
DEBUG:software_metadata.py: batch contents: ['nickbusey/homelabos', 'hook0/hook0', 'mailman/hyperkitty', 'jirafeau/jirafeau', 'lazylibrarian/lazylibrarian', 'ogarcia/lesma', 'mayan-edms/mayan-edms', 'cyber5k/mistborn', 'noosfero/noosfero', 'mailman/postorius']
DEBUG:connectionpool.py: Starting new HTTPS connection (1): gitlab.com:443
DEBUG:connectionpool.py: https://gitlab.com:443 "POST /api/graphql HTTP/1.1" 200 None
DEBUG:software_metadata.py: Rate limit (Limit/Remain/Used/Reset): 2000/1998/2/1774204560
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/homelabos.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/hook0.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/hyperkitty.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/jirafeau.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/lazylibrarian.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/lesma.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/mayan-edms.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/mistborn.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/noosfero.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/postorius.yml
DEBUG:software_metadata.py: sleeping 5s between batches
INFO:software_metadata.py: processing GitLab batch 3/3 (size: 7)
DEBUG:software_metadata.py: batch contents: ['francoisjacquet/rosariosis', 'timvisee/send', 'simple-nixos-mailserver/nixos-mailserver', 'biolds1/sosse', 'tikiwiki/tiki', 'veloren/veloren', 'web-db/app']
DEBUG:connectionpool.py: Starting new HTTPS connection (1): gitlab.com:443
DEBUG:connectionpool.py: https://gitlab.com:443 "POST /api/graphql HTTP/1.1" 200 None
DEBUG:software_metadata.py: Rate limit (Limit/Remain/Used/Reset): 2000/1997/3/1774204560
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/rosariosis.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/send.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/simple-nixos-mailserver.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/sosse.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/tiki.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/veloren.yml
DEBUG:software_metadata.py: Writing file tests/awesome-selfhosted-data/software/webdb.yml
INFO:main.py: all steps completed# $ cat tests/awesome-selfhosted-data/software/jirafeau.yml
name: Jirafeau
website_url: https://gitlab.com/jirafeau/Jirafeau
source_code_url: https://gitlab.com/jirafeau/Jirafeau
description: One-click-fileshare project. Select your file, upload, and share a link. That's it.
licenses:
- AGPL-3.0
platforms:
- PHP
- Docker
tags:
- File Transfer - Single-click & Drag-n-drop Upload
stargazers_count: 29
updated_at: '2026-01-25'
archived: falseAdded to awesome-selfhosted-data environment secrets:
I will now merge this and create a new release + bump the hecat version in awesome-selfhosted-data Thanks again |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github_metadata.pyintosoftware_metadata.pywith support for GitHub and GitLab metadata providersgh_metadata_only_missingmodule option renamed tometadata_only_missingFixes #48
Fixes #148
Finally finished the second iteration after feedback from a friend that the first version was spaghetti code, which hopefully is now at least a bit better (also the reason why it took so long to fix this one issue, but at least I found another bug)...
The new invalid URL detection causes issues for GitLab projects using sub-groups.
GitHub URLs follow a strict
org/repoformat, so filtering out non-repository URLs is straightforward. GitLab supports sub-groups (org/subgroup/repo), making validation ambiguous.I currently see two options:
I see only the first option as viable. This means the currently only invalid GitLab project
nirvati-ug/nirvatineeds to be updated to either a real repo or just point to the org before using this new version.Output with Error catching (30 batch)
Output (25 batch)
While running several tests, I noticed that reducing the batch size by just 5 repos produced much more stable output. This change also allowed me to decrease the wait time significantly; down to 5 seconds on my system. It would be interesting to see how this performs on other machines, as there appears to be a network-related factor at play.