feat: listmonk sync by SuperVK · Pull Request #519 · GEWIS/gewisdb

SuperVK · 2026-03-28T17:29:29Z

Description

Draft PR for listmonk synchronization. Figuring things out as I go, but suggestions are welcome!

Things:

Listmonk does not allow automatic creation of API users. So that needs to happen manually. (how does mailman do this exactly?)

Related issues/external references

Fixes GH-NNN.

Types of changes

New feature (non-breaking change which adds functionality)

…eletion

rinkp · 2026-05-02T11:57:55Z

Edit: issue in GH-520 and fixed in GH-521

I get this error, but I suspect it is not caused by the PR

Last fetch of available listmonk lists was never. Possibly, the mailing list server is down or fetch mechanism is malfunctioning.

user@host:~/GitHub/gewisdb$ git log --max-count=1 
commit ef45965274be40ecdc09fa3b41454e20c1eab567 (HEAD, victor/feature/listmonk-sync)

user@host:~/GitHub/gewisdb$ docker compose logs --tail=15 -t web
web-1  | 2026-05-02T11:03:49.784353666Z 
web-1  | 2026-05-02T11:03:49.784356493Z orm:generate-proxies [--em EM] [--filter FILTER] [--object-manager [OBJECT-MANAGER]] [--] [<dest-path>]
web-1  | 2026-05-02T11:03:49.784359774Z 
web-1  | 2026-05-02T11:03:49.837303931Z [02-May-2026 13:03:49] NOTICE: fpm is running, pid 18
web-1  | 2026-05-02T11:03:49.837515675Z [02-May-2026 13:03:49] NOTICE: ready to handle connections
web-1  | 2026-05-02T11:05:00.068720934Z crond: can't change directory to '/home/www-data': No such file or directory
web-1  | 2026-05-02T11:05:00.068768042Z crond: USER www-data pid  77 cmd { . /code/config/bash.env && /usr/bin/php /code/web database:mailman:syncmembership -f -vv; } > /data/logs/cron-mailman-syncmembership.log 2>&1
web-1  | 2026-05-02T11:10:00.069978483Z crond: can't change directory to '/home/www-data': No such file or directory
web-1  | 2026-05-02T11:10:00.070008837Z crond: USER www-data pid  82 cmd { . /code/config/bash.env && /usr/bin/php /code/web database:listmonk:syncmembership -f -vv; } > /data/logs/cron-listmonk-syncmembership.log 2>&1
web-1  | 2026-05-02T11:30:00.072926087Z crond: can't change directory to '/home/www-data': No such file or directory
web-1  | 2026-05-02T11:30:00.072950848Z crond: USER www-data pid 224 cmd { . /code/config/bash.env && /usr/bin/php /code/web check:membership:renewal:graduate; } > /data/logs/cron-check-renewal.log 2>&1
web-1  | 2026-05-02T11:50:00.076091054Z crond: can't change directory to '/home/www-data': No such file or directory
web-1  | 2026-05-02T11:50:00.076126173Z crond: USER www-data pid 229 cmd { . /code/config/bash.env && /usr/bin/php /code/web database:mailman:fetch; } > /data/logs/cron-mailman-fetch.log 2>&1
web-1  | 2026-05-02T11:55:00.077334456Z crond: can't change directory to '/home/www-data': No such file or directory
web-1  | 2026-05-02T11:55:00.077362961Z crond: USER www-data pid 242 cmd { . /code/config/bash.env && /usr/bin/php /code/web database:listmonk:fetch; } > /data/logs/cron-listmonk-fetch.log 2>&1

rinkp

Amazing, thank you! Some remarks below

I realised we never wrote down some of the discussed constraints, so I have created GH-525.

There are a few bugs, pointed out in individual comments. Major remarks (repeated) below.

Structure is okay, now that we will get several syncing methods, I do propose to change the commands here to avoid duplicate code/overhead a bit:
https://github.com/GEWIS/gewisdb/pull/519/changes#r3176665997

One of our goals is a quick development environment for whomever wants to work on GEWISDB. As such, the automatic setup of the API user and initial sync with DB is necessary. (Tested) proposal given here.

Test overview

I checked the following scenario's (after fixing the bugs pointed out above):

Clean scenario, basic case
- docker volume prune -af, make rundev, make migrate, make seed (with the proposed changes)
- Created mailing list through UI
- Normal subscription
  - Subscribe member in GEWISDB
  - docker compose exec web ./web database:listmonk:syncmembership -f -vv
  - Verify member exists on listmonk side
  - Check that synced flag is set
- Regular unsubscription
  - Unsubscribe member in GEWISDB
  - docker compose exec web ./web database:listmonk:syncmembership -f -vv
  - Verify member does not exist on Listmonk side
  - Verify that unsubscription is processed on GEWISDB side
- External unsubscription
  - Subscribe member in GEWISDB
  - docker compose exec web ./web database:listmonk:syncmembership -f -vv
  - Verify member exists on listmonk side
  - Remove member on listmonk side
  - Setting MailingListMember lastSyncOn to <1 day ago
  - docker compose exec web ./web database:listmonk:syncmembership -f -vv
  - Verify now also unsubscribed in GEWISDB
- External subscription (rare, but supported)
  - Sign up and confirm member in Listmonk
  - Change ListmonkMailingList lastCheck to <1 day ago
  - docker compose exec web ./web database:listmonk:syncmembership -f -vv
  - Verify now also subscribed in GEWISDB, marked as synced
- External subscription of unknown email
  - Sign up and confirm non-GEWISDB-member in Listmonk
  - Change ListmonkMailingList lastCheck to <1 day ago
  - docker compose exec web ./web database:listmonk:syncmembership -f -vv
  - Verify that the address was removed in listmonk
Clean scenario, manually setting a mailman list
- Check if listmonk list cannot be set at the same time, fail. Logic in Form does not work, and frontend does not hide the option to set a listmonk list if the mailman list is set. After saving, both are present in the database. If you ensure the form does not allow saving the list like that, I think it is fine. I thought of implementing a preupdate and prepersist function in the model, but that seems slightly overkill.
- Switching from mailman list to listmonk list
  - Create list in GEWISDB
  - Link list to Mailman
  - Subscribe member in GEWISDB
  - docker compose exec web ./web database:mailman:syncmembership -vvv -f
  - Verify member status is synced
  - Unlink mailman list
  - Link to listmonk (through DB) because of the impossibility of saving a list
  - Next sync will cause unsubscriptions in GEWISDB. Instead, expected subscriptions in listmonk. In contrast to what I said before, linking a listmonk list (or mailman list for that matter), should set "toBeCreated = true" on all rows on the GEWISDB side. That will subscribe everyone who is not subscribed in the backend yet.
    For the initial mailman links this was not done (because the mailman backend was considered the source of truth), but for generic operations this seems to be the desired behaviour.

rinkp · 2026-05-02T13:13:46Z

    'laminas-cli' => [
        'commands' => [
            'database:mailinglist:maintenance' => MailingListMaintenanceCommand::class,
+            'database:local:syncmembership' => MailingListSyncLocalMembershipCommand::class,


Because we now will get 3 different sync methods, proposal to reduce the number of commands and instead implement an argument to distinguish them

Suggested change

'database:local:syncmembership' => MailingListSyncLocalMembershipCommand::class,

'database:mailinglist:sync-membership' => MailingListSyncMembershipCommand::class,

'database:mailinglist:fetch-lists' => MailingListFetchListsCommand::class,

rinkp · 2026-05-02T13:17:35Z

+
+class MailingListSyncLocalMembershipCommandFactory implements FactoryInterface
+{
+    public function __invoke(


No longer relevant after https://github.com/GEWIS/gewisdb/pull/519/changes#r3176665997, but this would have been the fix for the psalm error. You ignore MissingNativeTypeHint in phpcs.xml.dist, but not the MissingAnyTypeHint

Suggested change

public function __invoke(

/**

* @param string $requestedName

*/

public function __invoke(

rinkp · 2026-05-02T13:20:15Z

 */30    *   * * * { . /code/config/bash.env && /usr/bin/php /code/web check:membership:renewal:graduate; } > /data/logs/cron-check-renewal.log 2>&1
 0       2   * * * { . /code/config/bash.env && /usr/bin/php /code/web database:prospective-members:delete-expired; } > /data/logs/cron-delete-prospective.log 2>&1
-55      *   * * * { . /code/config/bash.env && /usr/bin/php /code/web database:mailman:fetch; } > /data/logs/cron-mailman-fetch.log 2>&1
+50      *   * * * { . /code/config/bash.env && /usr/bin/php /code/web database:mailman:fetch; } > /data/logs/cron-mailman-fetch.log 2>&1


After https://github.com/GEWIS/gewisdb/pull/519/changes#r3176665997, suggestion:

Suggested change

50 * * * * { . /code/config/bash.env && /usr/bin/php /code/web database:mailman:fetch; } > /data/logs/cron-mailman-fetch.log 2>&1

50 * * * * { . /code/config/bash.env && /usr/bin/php /code/web database:mailinglist:fetch-lists -b "mailman"; } > /data/logs/cron-mailinglist-fetch-lists-mailman.log 2>&1

rinkp · 2026-05-02T13:23:06Z

            '%s: %s',
-            $this->translate('Last fetch of mailman lists'),
-            $mailmanLastFetch?->format('Y-m-d H:i:s') ?: $this->translate('never')
+            $this->translate('Last fetch of listmonk lists'),


It seems you capitalise "listmonk" in all other end-user-visible places

Suggested change

$this->translate('Last fetch of listmonk lists'),

$this->translate('Last fetch of Listmonk lists'),

rinkp · 2026-05-02T13:25:02Z

    <!-- Legacy code that cannot be easily changed, so ignore for now -->
    <rule ref="Squiz.NamingConventions.ValidVariableName.MemberNotCamelCaps">
        <exclude-pattern>module/Database/src/Model/Decision.php</exclude-pattern>
+        <exclude-pattern>module/Database/src/Model/ListmonkMailingList.php</exclude-pattern>


Suggested change

<exclude-pattern>module/Database/src/Model/ListmonkMailingList.php</exclude-pattern>

rinkp · 2026-05-03T12:14:41Z

    ): ?MemberModel {
        // Check if we are performing a sync or not.
-        if ($this->mailmanService->isSyncLocked()) {
+        if ($this->mailmanService->isSyncLocked() || $this->listmonkService->isSyncLocked()) {


It feels wrong for the mailman sync to depend on the listmonk sync and vice versa.
I just realise that it may be possible to perform both syncs at the same time. Neither class should touch a list which the other is also processing, so I think it suffices here to simply only check if mailman sync is running.

rinkp · 2026-05-03T12:16:14Z

                <?= $this->formElementErrors($element) ?>
            </div>
        </div>
+        <?php endif; ?>


Are there cases where you want to show both fields? Or do you want to enforce unlinking the mailman list first, then saving and then setting the listmonk list.

Suggested change

<?php endif; ?>

<?php else: ?>

rinkp · 2026-05-03T12:31:31Z

@@ -66,6 +68,7 @@ public function editList(
        $list->setOnForm(boolval($data['onForm']));
        $list->setDefaultSub(boolval($data['defaultSub']));
        $list->setMailmanList($this->getMailmanService()->getMailingList($data['mailmanList']));


If the mailman field is not present during an update, this call now fails ensuring that a mailing list cannot be updated unless it has a mailman list set.

[2026-05-03 14:28:53] gewisdb.ERROR: TypeError: Database\Service\Mailman::getMailingList(): Argument #1 ($mailmanId) must be of type string, null given, called in /code/module/Database/src/Service/MailingList.php on line 70 and defined in /code/module/Database/src/Service/Mailman.php:359 Stack trace: #0 /code/module/Database/src/Service/MailingList.php(70): Database\Service\Mailman->getMailingList(NULL) #1 /code/module/Database/src/Controller/SettingsController.php(126): Database\Service\MailingList->editList(Object(Database\Model\MailingList), Array) #2 /code/vendor/laminas/laminas-mvc/src/Controller/AbstractActionController.php(72): Database\Controller\SettingsController->editListAction() #3 /code/vendor/laminas/laminas-eventmanager/src/EventManager.php(320): Laminas\Mvc\Controller\AbstractActionController->onDispatch(Object(Laminas\Mvc\MvcEvent)) #4 /code/vendor/laminas/laminas-eventmanager/src/EventManager.php(178): Laminas\EventManager\EventManager->triggerListeners(Object(Laminas\Mvc\MvcEvent), Object(Closure)) #5 /code/vendor/laminas/laminas-mvc/src/Controller/AbstractController.php(105): Laminas\EventManager\EventManager->triggerEventUntil(Object(Closure), Object(Laminas\Mvc\MvcEvent)) #6 /code/vendor/laminas/laminas-mvc/src/DispatchListener.php(117): Laminas\Mvc\Controller\AbstractController->dispatch(Object(Laminas\Http\PhpEnvironment\Request), Object(Laminas\Http\PhpEnvironment\Response)) #7 /code/vendor/laminas/laminas-eventmanager/src/EventManager.php(320): Laminas\Mvc\DispatchListener->onDispatch(Object(Laminas\Mvc\MvcEvent)) #8 /code/vendor/laminas/laminas-eventmanager/src/EventManager.php(178): Laminas\EventManager\EventManager->triggerListeners(Object(Laminas\Mvc\MvcEvent), Object(Closure)) #9 /code/vendor/laminas/laminas-mvc/src/Application.php(319): Laminas\EventManager\EventManager->triggerEventUntil(Object(Closure), Object(Laminas\Mvc\MvcEvent)) #10 /code/public/index.php(20): Laminas\Mvc\Application->run() #11 {main} [] []

rinkp · 2026-05-03T12:34:08Z

        $mailingListMemberMapper = $container->get(MailingListMemberMapper::class);
        /** @var MailmanService $mailmanService */
        $mailmanService = $container->get(MailmanService::class);
+        /** @var ListmonkService $listmonkService */


rinkp · 2026-05-03T12:34:37Z

    ): FrontPageService {
        $apiService = $container->get(ApiService::class);
        $mailmanService = $container->get(MailmanService::class);
+        $listmonkService = $container->get(ListmonkService::class);


Order (this one is very interesting)

feat: listmonk sync base

f2257de

SuperVK force-pushed the feature/listmonk-sync branch from e1c3e4a to f2257de Compare March 31, 2026 16:50

SuperVK added 15 commits March 31, 2026 19:47

chore: set up listmonk dev setup

da856b5

fix: add migration

e99cb41

chore: fix typo from LLM

98651b3

feat: add ui for editing/adding list

b33f178

fix: remove listmonk api errors

fa49d1d

feat: add listmonk sync to crontab

4e827b6

feat: add local mailinglist sync

009b94b

feat: update sync locking checks

41a1e88

fix: listmonk/mailman only sync their own lists

d248a28

chore: rename sync lock page

7dff9a2

feat: display which external is enabled on which list

dd272b1

feat: implement mailman as legacy option

241a992

feat: implement listmonk sync notifications

7895acb

feat: merge listmonk and mailman notifications for pending creation/d…

b5c8186

…eletion

fix: various small clean ups

7e6c528

SuperVK force-pushed the feature/listmonk-sync branch from 6d2f854 to 7e6c528 Compare April 29, 2026 18:07

SuperVK added 4 commits April 30, 2026 15:50

chore: clean up translations

cc3caf6

chore: make phpcs happy

e8a7ce6

fix: remove migration boilerplate

37e4750

chore: revert editing MemberFixture.php

ef45965

SuperVK marked this pull request as ready for review April 30, 2026 14:28

rinkp self-requested a review May 2, 2026 10:40

rinkp mentioned this pull request May 2, 2026

Warning cron daemon about non-existent home of www-data user #520

Closed

rinkp requested changes May 3, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: listmonk sync#519

feat: listmonk sync#519
SuperVK wants to merge 20 commits intoGEWIS:mainfrom
SuperVK:feature/listmonk-sync

SuperVK commented Mar 28, 2026 •

edited

Loading

Uh oh!

rinkp commented May 2, 2026 •

edited

Loading

Uh oh!

rinkp left a comment

Uh oh!

rinkp May 2, 2026

Uh oh!

rinkp May 2, 2026

Uh oh!

rinkp May 2, 2026

Uh oh!

rinkp May 2, 2026

Uh oh!

rinkp May 2, 2026

Uh oh!

rinkp May 3, 2026

Uh oh!

rinkp May 3, 2026

Uh oh!

rinkp May 3, 2026

Uh oh!

rinkp May 3, 2026

Uh oh!

rinkp May 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	'database:local:syncmembership' => MailingListSyncLocalMembershipCommand::class,
	'database:mailinglist:sync-membership' => MailingListSyncMembershipCommand::class,
	'database:mailinglist:fetch-lists' => MailingListFetchListsCommand::class,

	50 * * * * { . /code/config/bash.env && /usr/bin/php /code/web database:mailman:fetch; } > /data/logs/cron-mailman-fetch.log 2>&1
	50 * * * * { . /code/config/bash.env && /usr/bin/php /code/web database:mailinglist:fetch-lists -b "mailman"; } > /data/logs/cron-mailinglist-fetch-lists-mailman.log 2>&1

	$this->translate('Last fetch of listmonk lists'),
	$this->translate('Last fetch of Listmonk lists'),

Conversation

SuperVK commented Mar 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues/external references

Types of changes

Uh oh!

rinkp commented May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rinkp left a comment

Choose a reason for hiding this comment

Test overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SuperVK commented Mar 28, 2026 •

edited

Loading

rinkp commented May 2, 2026 •

edited

Loading