Missing error queue to notification api

[JanBrek]

Product versie / Product version

3.5.0

Omschrijf het probleem / Describe the bug

Yesterday a production issue due to notification api being offline (Due to another bug with opennotificatie redis).
This logically causes notifications not being sent to the notification api.
The problem was that there were error logs, but they just mentioned a failed attempt, but nothing on the actual data that failed to send. Hence we could not find out which notifications were missed. Luckily with the help of Sentry we were able to figure this out and recover.

TODO:

• implement storing of outgoing notifications and delivery attempts/response(s) in db in notifications-api-common (similar to Open Notificaties)
  • if a notification succeeds, remove the persisted notification (this way only failed notification are persisted)
• implement admin interface to see these notifications and do retries (for single notifications or in bulk, similar to Open Notificaties
• add a task/command to clean up old notifications in the database, with a configurable amount of retention days (default 60)

Upgrade notifications-api-common in these components:

• Open Zaak
• Open Object
• Open Klant
• Open Product
• Open VTB

Stappen om te reproduceren / Steps to reproduce

1. Bring ON down
2. Add a new object to the api
3. Wait until failure

Verwacht gedrag / Expected behavior

We need at least more/better information in the logs, but I prefer to have an error queue which I can view in the admin UI of objecten and start a retry for.

[stevenbal]

I agree, I think it would be good if Objects API (and our other applications that send notifications to Open Notificaties) also keep track of notifications they (try to) send, similar to how Open Notificaties itself does this. That way we can also make it possible to do retries manually via the admin UI. Since the logic to send notifications is already implemented in a library, I think we can implement this functionality there as well, and add this to all components.

I'd estimate 4-5 days to:

• implement storing of outgoing notifications and delivery attempts/response(s) in db
• implement admin interface to see these notifications and do retries (for single notifications or in bulk, similar to Open Notificaties
• add a task/command to clean up old notifications in the database

[alextreme]

@janmasGDH

[JanBrek]

With the additon: 'Add a task/command to clean up old notifications in the database' -> Succeeded will not be saved / deleted immediately after success. Failed will be kept for a configurable time with default 60 days
Scoped to all Maykin apps which send notifications

Comment Steven: Estimate 6 days total for all Maykin apps

Tech check OK for 6 days
@janmasGDH

[janmasGDH]

Approved