Epg feature, compatibility improvements and bugfixes

.github/workflows/ci.yml

@@ -25,6 +25,21 @@ jobs:
       - name: Config and protocol tests
         run: python3 tests/test_config.py
 
+      - name: Authentication state machine test
+        run: python3 tests/test_auth.py
+
+      - name: Request transport and retry test
+        run: python3 tests/test_transport.py
+
+      - name: Channel listing and paging test
+        run: python3 tests/test_listing.py
+
+      - name: Programme guide test
+        run: python3 tests/test_epg.py
+
+      - name: Scheduled refresh test
+        run: python3 tests/test_schedule.py
+
       - name: Mock portal integration test
         run: python3 tests/test_mock_portal.py
 

.github/workflows/release.yml

@@ -40,6 +40,13 @@ jobs:
       # entry does not happen. A changelog nobody is obliged to write is a
       # changelog that stops being written, and the reader of a release page
       # deserves better than a list of commit subjects.
+      #
+      # Only the part of the section above the "<!-- details -->" marker is
+      # published, though: a release page is read by someone deciding whether to
+      # upgrade, who wants a line per subject, while the prose below the marker
+      # is the reasoning behind each one and belongs to whoever goes looking for
+      # it in the file. A section without the marker is published whole, so an
+      # entry written before this split still releases.
       - name: Take the release notes from CHANGELOG.md
         run: |
           python3 - "${GITHUB_REF_NAME#v}" <<'PY' > release-notes.md
@@ -53,7 +60,10 @@ jobs:
           )
           if section is None or not section.group(1).strip():
               sys.exit(f"CHANGELOG.md has no entry for {version}")
-          print(section.group(1).strip())
+          summary = section.group(1).split("<!-- details -->")[0].strip()
+          if not summary:
+              sys.exit(f"the {version} entry is all detail and no summary")
+          print(summary)
           PY
 
       - name: Install test dependencies

CHANGELOG.md

@@ -1,5 +1,194 @@
 # Changelog
 
+## 0.9.3
+
+**After upgrading, restart Dispatcharr, then press Test portals and Re-fetch
+all.** The restart is what puts every worker on the new code, since plugins are
+loaded once per process. Sync on its own would then report every portal as
+unchanged and fetch nothing — it compares your settings against what was last
+published, and none of them changed, only the code did. Test writes nothing and
+is where a portal that now wants credentials will say so.
+
+- **Portals can now serve their programme guide.** `epg=1` on a portal line
+  writes an XMLTV file and registers it under **EPGs**; off unless asked for,
+  because a guide is by far the largest thing a sync downloads.
+- **Distalker can be put on a schedule.** *Refresh every (hours)* re-fetches
+  each portal on that interval — channels, and the guide where `epg=1` is set —
+  without anyone pressing Sync. Left at `0`, everything stays manual.
+- **Channels on portals that answer with a resolved link now play**, at sync
+  time and at tune time both, and stop duplicating themselves on every sync.
+- **The portal now decides which authentication it gets**, so an expired or
+  blocked account says so in the provider's own words instead of arriving as an
+  empty channel list.
+- **Portals that will not list their channels in one request now sync**, a page
+  at a time, and a sync survives a portal having a bad minute.
+- **The channel is named in the request as well as in the command**, for the
+  providers that could not find it in the command alone. Reported and first
+  fixed by [@shayward](https://github.com/shayward).
+- Channel logos and archive flags are read as the portal sends them.
+
+<!-- details -->
+
+**Refreshing by itself**
+
+- **Distalker can now be put on a schedule.** Set *Refresh every (hours)* and
+  each portal is re-fetched on that interval — channels, and the guide where
+  `epg=1` is set — without anyone pressing Sync. Left at `0`, nothing changes:
+  everything stays manual, as before.
+- The clock is Dispatcharr's, not ours. A plugin cannot register a Celery task
+  a stock install can consume, so the setting drives the refresh interval of
+  the M3U accounts this plugin owns, and the event Dispatcharr emits at the end
+  of each refresh is what calls back to re-fetch the portal.
+- Refreshing an M3U account that is not one of ours does nothing, and a
+  scheduled run cannot follow another within thirty minutes — the second is the
+  echo of the first, and answering it would loop. In practice it does more than
+  that: twelve accounts sharing one interval all fire at the same second, and
+  eleven of those are turned away.
+- A sync started by hand no longer comes back as a scheduled one. Every synced
+  portal asks Dispatcharr to re-read its playlist, and that re-read emits the
+  event the schedule listens for — so adding a single portal could set off a
+  re-fetch of every other one.
+- A scheduled sync runs inside the task that woke it rather than in a thread of
+  its own. The thread was right for the button, which answers a browser and
+  cannot block; it was wrong here, where the worker hosting it is reaped when
+  the pool scales down — taking the sync with it, silently.
+- **A scheduled sync now appears in the log.** It always ran; it just wrote
+  everything it did to a logger that prints from the process serving the
+  panel and not from the one running the schedule. Channel counts, guide
+  sizes and failures alike went nowhere, which made a refresh that worked
+  indistinguishable from one that never fired.
+- The portal whose own refresh woke the sync is asked a second time, a minute
+  later, to re-read its playlist. Its first request is refused: the sync runs
+  inside that account's refresh task, which holds the account's lock until we
+  are done. Left as it was, that one portal would be downloaded on every cycle
+  and read on the next — a day late on a daily schedule, and a different
+  portal each time, since they race to be the one that wakes us.
+
+**Guide**
+
+- **Distalker can now fetch a portal's programme guide.** Add `epg=1` to a
+  portal line and the next sync writes an XMLTV file and registers it under
+  **EPGs** as `Distalker: <portal>` — the same arrangement as the M3U account
+  it already creates for the channels. Matching is automatic: the `tvg-id` the
+  playlist has always carried is what Dispatcharr joins on.
+- Off unless asked for, because it is by a wide margin the largest thing a sync
+  downloads — a portal answers for its whole line-up at once, so a
+  13,000-channel provider means roughly 100 MB for a single day. `epg_hours=48`
+  raises the default 24; a guide past 200 MB is abandoned with a message saying
+  to lower it.
+- A guide that fails never fails the sync around it. You keep the channel list.
+- Channels the portal has no programmes for are left out rather than written as
+  empty entries, and removing `epg=1` deactivates the EPG source instead of
+  deleting it.
+- **Only whole-grid guides are read.** A portal that serves programmes one
+  channel at a time is reported as having no guide — which, measured across
+  twelve portals, is close to the truth: that route covered none of the
+  channels anyone had configured, at the price of one request per channel.
+- A portal with no guide says so plainly and suggests removing `epg=1`, rather
+  than reporting the same message as a portal that answered with something
+  unrecognisable — which is now reported as exactly that, and asks to be.
+
+**Playing**
+
+- **Channels a portal answers about with a resolved link now play.** Portals
+  are meant to name a channel with a marker and turn that into a link when
+  asked; some answer with the link itself, and then cannot read it back — one
+  spliced part of the URL into its own stream parameter, another returned that
+  parameter empty. Either way the channel was unplayable. Such a command is now
+  rebuilt into the marker the portal expects.
+- It also stops those channels duplicating. The links carry a token that
+  changes on every request, and Dispatcharr identifies a stream partly by its
+  URL — so every sync invented a new stream and stranded the previous one. One
+  portal was producing 647 duplicates an hour.
+- A command that already looks like a marker is left untouched, which is every
+  channel on every other portal tested — their identities do not move, and no
+  duplicate is created by the fix itself. The sync log says how many were
+  rewritten.
+- **The same portals are also caught at tune time.** The rewrite above needs
+  the portal to number its channels; a listing that does not still hands back a
+  resolved link, as does every portal synced before this release. One family of
+  them answers by gluing its own base in front of whatever it was given, so a
+  command that was already a URL comes back with `/user/pass/` twice and gets
+  401 — while the command itself plays. That reply is now recognised by its
+  shape and the command played instead, with a line on the channel's log
+  saying so. Nothing is asked of the provider to decide it: a check request
+  would take a connection slot from the tune that is about to need it.
+- **The channel is now named in the request as well as in the command.** Some
+  providers cannot find it in the command they are handed back and answer about
+  no channel at all — the link arrives with its stream number left empty, and
+  plays nothing. Both shapes a command takes here carry that number, and it is
+  sent alongside, so those portals resolve the channel that was asked for.
+  Reported and first fixed by [@shayward](https://github.com/shayward).
+- Sent only when there is one to send, so a portal that never needed it sees
+  the request it has always seen, and the number can only ever be one the
+  portal itself wrote.
+
+**Connecting**
+
+- **The portal now decides which authentication it gets.** Distalker reads the
+  `status` its profile request comes back with and does what it asks: nothing
+  further when the session is already good, or `do_auth` followed by a second
+  profile call when the portal says it wants credentials. It used to pick the
+  flow itself from whether a username and password happened to be configured,
+  which was wrong in both directions — it ran a device-ID step at portals that
+  wanted a password, and had no way to tell a refused account from an empty
+  channel list.
+- A portal that refuses the account now says so in the provider's own words —
+  "subscription expired", "blocked" — instead of failing later as a channel
+  list that came back empty and a suggestion to check the MAC address.
+- A portal that wants credentials and has none on its line now fails the sync
+  with that as the message, rather than appearing to work.
+- Expired sessions are recognised from the plain-text `Authorization failed.`
+  some portals answer with, and from HTTP 401/403, instead of being reported as
+  a portal talking nonsense.
+- The box's profile now carries the full identity every other Stalker client
+  sends, `signature` included — which had been a documented setting that no
+  request ever contained, so setting it configured nothing.
+- Portals with no profile endpoint at all keep working on the MAC alone, with a
+  warning. That tolerance is deliberate: most portals this plugin meets are not
+  Ministra and answer with less than it would.
+- The `device_id_auth` line key is gone. Nothing needs to be changed: it was
+  never something to write on a portal line, only a value the plugin derived
+  for itself, and lines are re-read on every sync.
+- The MAC and the session token now travel in the query string as well as in
+  the cookie and the `Authorization` header. Portals read one form or the
+  other, and sending both costs nothing.
+- **A portal reached at the wrong path now finds itself.** Ministra answers on
+  both `…/c/portal.php` and `…/server/load.php`, and installs differ in which
+  they expose; being handed the one your provider does not serve used to mean a
+  404 and no suggestion. The other path is now tried once, and the log says
+  which one worked so you can put it on the portal line and stop paying for the
+  failed request. Only a 404 or a reply that is not JSON earns the second
+  attempt — a portal that is merely down answers the same way on both.
+
+**Syncing**
+
+- **Portals that will not list their channels in one request now sync.** Some
+  cap `get_all_channels`, some never implemented it; either way the portal was
+  unusable, since the empty answer was reported as a probable wrong MAC. The
+  line-up is now collected a page at a time instead when that happens — slower
+  by a long way on a big bouquet, and the only way those portals work at all.
+  The sync log says when it has fallen back and how far along it is.
+- A portal that refuses the session is not paged as a second attempt, and an
+  empty listing from both routes still reports the original "check the MAC
+  address" message, which remains the likelier explanation.
+- Channel logos survive two shapes that used to come out broken: a logo served
+  from another scheme than `http(s)` was treated as a filename and glued behind
+  the portal's logo path, and an inline `data:` image got the same treatment.
+  The first is now left alone, the second dropped — Dispatcharr keeps this in a
+  URL field, where a base64 payload does not belong.
+- **A sync survives a portal having a bad minute.** Requests made while
+  syncing are attempted up to three times, one then two then four seconds
+  apart, where a single dropped connection or gateway error used to cost the
+  whole line-up until the next scheduled run. Only failures that another
+  attempt could fix are repeated: a refused login, a blocked account or a
+  missing endpoint still fails immediately.
+- Nothing is retried at tune time, deliberately. A source that is not answering
+  has to fail fast enough for Dispatcharr to move to the next one, which is the
+  same reason `-reconnect` is not in the default ffmpeg arguments. "Test
+  portals" does not retry either — it answers a click, and three attempts at
+  the portal timeout outlast the browser waiting for it.
+
 ## 0.9.2
 
 **Playing**

CONTRIBUTING.md

@@ -33,6 +33,11 @@ script with its own `__main__` block:
 
 ```bash
 python3 tests/test_config.py       # portal-line parsing, STB defaults, pseudo-URLs
+python3 tests/test_auth.py         # which authentication a portal asks for
+python3 tests/test_transport.py    # what a request carries, and when it is retried
+python3 tests/test_listing.py      # reading a portal's channel list
+python3 tests/test_epg.py          # the guide, and the tvg-id both halves share
+python3 tests/test_schedule.py     # being woken by Dispatcharr's own clock
 python3 tests/test_registry.py     # surviving the settings panel
 python3 tests/test_manifest.py     # plugin.json vs plugin.py, and run()'s plumbing
 python3 tests/test_fallback.py     # non-portal sources on a Distalker channel
@@ -91,6 +96,17 @@ which the workflow gates on two things — the tag must match `plugin.json`, and
 `CHANGELOG.md` must have a section for that version, which becomes the release
 body. A changelog nobody is obliged to write is one that stops being written.
 
+That section is written in two parts, separated by a `<!-- details -->` line:
+
+- Above it, what upgrading gets you — a line per subject, plus anything that has
+  to be done by hand after installing. This is what the release page shows, and
+  it is read by someone deciding whether to upgrade at all.
+- Below it, why each of those exists: the portal behaviour that forced it, what
+  was tried first, what is deliberately not done. Nobody reads that on a release
+  page, and everybody wants it six months later when the same thing breaks.
+
+A section with no marker is published whole, so nothing has to be back-filled.
+
 Keep changes to `.github/workflows/` in their own commit: pushing one needs a
 token carrying the `workflow` scope, which not every contributor's will have,
 and splitting it is what makes the rest shippable when it does not.