Conversation
The flow was chosen here, from whether a username and password happened to be configured: credentials meant do_auth, their absence meant a device-ID step whose failure was swallowed as a warning. That guess was wrong in both directions. Portals that want a password were handed a device-ID step and then went on to serve an empty channel list, which the plugin reported as a probable wrong MAC -- and an account the provider had blocked looked exactly the same, because nothing ever read the answer that said so. The portal already states which one it wants. get_profile comes back with a status: 0 is a session that is good, 2 is a request for credentials, and anything else is a refusal carrying the provider's own block_msg or msg. Follow it instead, and the refusal reaches the user in the words their reseller wrote. Two tolerances are deliberate, because most portals this meets are not Ministra and answer with less than it would. A profile with no status at all counts as good -- pvr.stalker calls that a failure, which would break every clone that worked yesterday. A portal that does not answer get_profile at all stays a warning and proceeds on the MAC alone; only an explicit refusal is fatal, which is what PortalAuthError now marks. That type also fixes a hole at tune time: an expired session is answered with plain text, an HTTP 403, or a hollow create_link success, and only the first of those was recognised. The resolver re-authenticates on the class rather than on any failure, so a portal that is merely unreachable no longer costs a pointless second round-trip before Dispatcharr fails over. The profile finally carries the whole STB identity, signature included -- documented as a setting since 0.3.0 and present in no request until now. device_id_auth goes with the guess it existed for; it was derived, never written, so nothing needs migrating.
Every other test file is a step of its own, so a failure names itself in the run summary rather than hiding inside a neighbour.
Two changes with one cause: portals that are not Ministra. The MAC goes in a cookie and the token in an Authorization header because that is what a MAG box sends and what Ministra reads. It is not what every portal reads. open-tv authenticates against real portals using only the query parameters -- no cookie, no header -- so a portal that ignores ours would be unusable here while working there. Sending both forms costs a handful of characters per request, and nothing has ever been seen to object to the one it does not want. The retry is the same admission applied to time rather than to shape. A dropped connection or a gateway error during a sync used to cost the user their whole line-up until the next scheduled run, for a portal that was merely busy. Three attempts, one then two then four seconds apart. What it must not do is the part worth reviewing. Retries default to off and the resolver leaves them there: a source that is not answering has to fail now, or Dispatcharr never fails over to the one that would have worked -- the reasoning already spelled out in the ffmpeg arguments. "Test portals" also leaves them off, because it runs on the request thread and three attempts at a 60s timeout outlast the proxy in front of it. And only failures another attempt could fix are repeated: a refusal is left alone, since retrying a rejected login is how a MAC gets itself banned. do_auth moves onto the same request path, which gains it the retry and types a rejected password as PortalAuthError -- it was the one credential failure still arriving as a bare PortalError.
Same reason as the authentication step: a failure that names itself.
Two logo shapes came out broken. A logo carrying any scheme other than http(s) fell through the "must be a filename" branch and was glued behind .../misc/logos/320/, producing a URL that pointed nowhere; and an inline data: image got the same treatment, when the right answer is to drop it -- it is a valid logo, and a base64 payload has no business in the URL field Dispatcharr stores this in. Both follow pvr.stalker's DetermineLogoURI. Channels also carry enable_tv_archive and tv_archive_duration, which are now read into ChannelEntry and deliberately not published. Dispatcharr does pick those attributes up from an M3U and turns them into is_catchup/catchup_days, but playing catch-up back is Xtream-only -- built from a server URL and credentials a portal source has none of. Advertising it would put a catch-up badge on channels whose catch-up cannot play. Captured now so that the day that path stops being Xtream-shaped, the data is already arriving; the reasoning is on ChannelEntry, where the next person will look. Row parsing moves out of get_all_channels into _channel_from_row, which is what the paginated listing still to come will need to share.
Some portals cap get_all_channels, some never implemented it. Either way they were unusable here: the empty answer came back as "check the MAC address", which was the wrong advice and the end of it. They can still be browsed a page at a time, which is what the MAG interface does anyway, so that is now the fallback -- hundreds of requests and several minutes on a large bouquet, and worth it against a portal that otherwise cannot be synced at all. Three things end the walk, and all three are needed because each covers a portal the others miss. The page count the portal implies from total_items and max_page_items stops at exactly the right place, and is the bound pvr.stalker uses. An empty page covers portals that run out politely, which is open-tv's only guard. A page whose rows have all been read already covers portals that clamp p to their last page and answer forever -- the case that makes open-tv's loop infinite, since it declares both counts in its DTO and then never reads them. A cap far above any real line-up catches the rest. Order of preference is unchanged where it works: one request first, always, because it is what nearly every portal supports and is enormously cheaper. Paging is not tried against a session the portal has refused, since it would be refused identically and a second rejected login is how a MAC gets noticed; and when neither route finds anything, the original message survives rather than being replaced by something vaguer -- an empty listing really is a wrong MAC far more often than it is a portal needing pages. The sync passes a progress callback so four silent minutes can account for themselves. A callback and not a logger: this module stays Django-free.
Two leftovers from reading pvr.stalker. Portal.watchdog() had no caller and never could have one. Keeping a Stalker session warm means calling get_events every 'timeslot' seconds, which needs something alive between requests; the sync is a task that ends and the resolver becomes ffmpeg. A ping method that exists and is never called reads as a feature, so it is gone and a comment says why, with a test to stop it coming back. Ministra answers on both <base>/c/portal.php and <base>/server/load.php, and installs differ in which they expose. Being handed the one a provider does not serve meant a 404 and no suggestion. The handshake -- every session's first request, so the only place a wrong path shows up -- now tries the other one, and says which worked so the user can put it on the portal line. Only PortalEndpointError earns that second attempt: a 404, or a reply that is not JSON at all. A portal that is down, unwell, or refusing the MAC answers identically on both paths, and at tune time a wasted round-trip is time Dispatcharr is not spending on the next source. self.url is kept apart from cfg.url because logos resolve against the configured URL: swapping the API path must not move them.
Portals answer get_epg_info with their whole line-up's guide, keyed by the
same channel id the playlist already carries in tvg-id -- put there in
anticipation of exactly this, so nothing had to be rewritten to make the two
meet. It becomes an XMLTV file and an EPGSource pointing at it: the mirror
image of the M3U account each portal already gets, and for the same reason.
Dispatcharr treats a source with a file_path and no url as a first-class
case, so the guide needs no HTTP endpoint either.
Off unless a line says epg=1, because it is by a wide margin the largest
thing a sync fetches. A portal with 13,000 channels answers a single day
with something around 100 MB, and the decoded form costs several times more
again -- so the reply is streamed to a scratch file and its size checked
before anything is decoded, and the document is generated by emptying that
structure as it writes rather than building a second copy of it.
None of which can fail the sync around it. The guide is an extra; a run that
ends with a working line-up and no guide is a good outcome, and one that
loses the line-up over a guide is not.
Three things learned from real portals rather than from reading:
Channels with no programmes are left out. A <channel> with nothing under it
still becomes a row in the EPG picker, and on the portal measured only 704
of 4,635 channels had a guide -- the rest would have been 3,900 permanent
empty promises.
Programmes overlap. The same show arrives twice with two start times and one
end, which is what a guide corrected in place looks like from outside, and
two candidates spanning one minute is an arbitrary answer to "what is on
now". Earliest start wins; touching exactly is not overlapping and is kept.
An empty guide is not an error but an unrecognised one is. Eight portals out
of twelve answer {"js": {"data": []}} -- thousands of channels and no
programmes for any of them, which is a property of the provider. Treating a
shape nobody has met as the same thing is how it would stay unmet, so that
one is reported and asks to be.
epg and epg_hours join the keys a change to which forces a fetch. Without
that the plan calls every portal unchanged, fetches nothing, and the setting
appears to do nothing at all -- and format_portal_line, which keeps only
what it is handed, would have deleted them from the line it rewrites.
Everything this plugin does has always waited for someone to press Sync. Not by choice: a plugin's @shared_task cannot be consumed on a stock install -- the consumer that resolves a task name is the prefork parent, which never imports plugins -- so the periodic task an earlier version registered was published into the void twice a day and was removed in 0.9.4. That made the guide close to useless, since a programme guide nobody refreshes is a programme guide about yesterday. There is a clock already running, though. Dispatcharr schedules a refresh per M3U account and ends each one by logging m3u_refresh, and a plugin action listing that event in its manifest is dispatched when it fires. So 'Refresh every (hours)' is written to the refresh interval of the accounts this plugin owns, and the event that comes back is what re-fetches the portal behind it. Nothing new is scheduled; an existing timer is answered. Three guards decide whether an event may act, and two of them protect somebody else rather than this feature. Zero hours means the user never asked, and an upgrade that started contacting portals by itself would be a poor surprise. The account name must be one of ours, because m3u_refresh fires for every M3U on the install -- without that, refreshing an unrelated playlist would set off a round of portal logins. The third is load-bearing rather than defensive. A scheduled sync ends by asking Dispatcharr to re-read the playlist it has just written -- otherwise the channel list sits one cycle behind for ever -- and that re-read emits the event that started it. The cooldown is what stops the pair going round, which is also why the interval cannot usefully be finer than an hour, and why it refuses when Redis cannot answer: the sync lock guards a button and should not fail on a cache outage, this guards a loop nobody asked for. Exactly one re-read task is dispatched, never both. refresh_single_m3u_account refreshes the groups itself, and asking for refresh_m3u_groups alongside it put them in a race for the same per-account lock -- the loser reported "Failed to refresh M3U groups" at the user and left the account in Pending Setup, which looked like a portal fault and was not. The interval is applied on every sync rather than only when portals are fetched. It is a global setting, so no portal ever looks changed because of it: the plan finds nothing to fetch and returns early, and a setting applied only on the fetching path would appear to do nothing whatsoever.
Same shape as the other steps: one per file, so a failure names itself.
The schedule announced itself and then produced nothing. The event arrives inside a Celery task, and the thread we handed the work to is a real daemon thread in that worker process -- which the pool reaps when it scales down. The log shows the sync starting and, forty seconds later, five processes going away with it. The thread is right for the button, which answers a browser and cannot block; here the Celery task is already the place long work belongs, and it allows an hour before its soft limit. Running it in the caller also means the panel reports what happened rather than that something started, and lets the one silent outcome be said out loud: a scheduled run refused by the lock used to look exactly like one that never fired. A sync started by hand claimed no cooldown of its own, so its own echo came back as a scheduled run -- every synced portal asks Dispatcharr to re-read its playlist, and that re-read emits the event we listen for. Adding one portal could set off a re-fetch of all twelve. It now pushes the window forward before dispatching, which delays the next scheduled run by half an hour: the right trade, since everything was just fetched.
A portal serving its guide one channel at a time is reported as having none, and that is worth stating rather than leaving to be discovered. Measured across twelve portals, the per-channel route covered none of the channels anyone had actually configured, at the price of one request per channel -- so reporting those portals as having no guide is very nearly true, but only someone who read this would know why.
A portal is meant to name a channel with a marker and turn that marker into a playable link when asked. Some answer the listing with the link itself, and then cannot read it back: handed one, a provider looked for the channel id inside it, failed, and spliced a piece of the URL into the stream parameter. Another returned the same parameter empty. Either way the channel did not play. It broke the channel's identity too. The stream hash is partly the URL, ours encodes this command, and these links carry a token that changes on every request -- so each sync invented a new stream and stranded the last one. One portal was producing 647 duplicates an hour. Narrow on purpose: a command that already looks like a marker is returned untouched, which is every channel on every other portal tested, so no identity moves and the fix creates no duplicates of its own. A command with no URL at all is left alone as well -- VOD commands look like that, and guessing at them would be worse than leaving it to the portal. Confirmed on the provider concerned: the rewritten markers resolve to links with the right stream id, and the channels play.
The portal used across five test files now carries the same host and MAC the README and the manifest placeholder use. Fixtures are read as examples of what to write on a portal line, so they should look like the examples.
Two things learned from a scheduled cycle that looked like it had not run. It had run, and correctly -- six portals fetched, their playlists and a guide written, all confirmed from the files' timestamps. What was missing was any trace of it. `logging.getLogger(__name__)` inside the plugin reaches the container log from a uWSGI worker and not from a Celery one, so every 'synced' line ever recorded came from the button, and the scheduled path wrote its counts, its sizes and its failures to a logger with nowhere to put them. Twice that absence was read as a schedule that never fired. Dispatcharr hands every action a logger that prints from both; the sync is given it now rather than inventing its own. The other is real rather than cosmetic. The sync runs inside Dispatcharr's refresh task for the account that woke it, and that task holds the account's lock for as long as we do -- so the reparse requested at the end of that one portal is refused, with a warning from Dispatcharr as its only trace. Left alone, that portal is downloaded on every cycle and read on the following one, for ever: an hour behind on an hourly schedule, a day behind on a daily one. It is asked again once the sync has returned and the lock has gone. A delay rather than any coordination, because being refused a second time leaves things exactly where not asking would have.
The same illness the marker rewrite cures, met at the other end. A portal whose listing answers with resolved links is handed one back at create_link, and one family of them builds the reply by gluing its own base in front of whatever it was given: a command that was already a complete URL comes back carrying /user/pass/ twice. That path answers 401 on every tune. The command itself answers 302 and plays. The rewrite at sync time is the better cure and it stays the first one, but it only fires on a row carrying an id -- a listing without one still stores the resolved link, as does every portal synced before it existed. So the reply is now recognised by its shape and the command played instead, with the token dropped: that token was minted for a path that does not exist. Structural, and settled without asking the provider anything. The obvious alternative is a probe request before the tune, and it costs a connection slot at the exact moment the tune needs it -- providers count those, and a subscription with one line would spend it on the check and fail the playback it was checking for. Narrow, because a false positive throws away a good token: same scheme and host, and the reply's path has to be the command's path exactly, behind a prefix that is itself a directory of it. A portal answering the same path with a token added -- what a working one does -- is untouched. The resolver prints the portal's warnings on both paths now. They were printed after login() and never on the cached-token path, which is the path almost every tune takes, so anything create_link had to say went nowhere. The token is still cached before the link is asked for: the handshake worked either way, and a channel the portal refuses must not send every later tune through a fresh one to learn the same thing. Reported by a user on 0.9.2 whose portal serves Stalker over an Xtream backend, with the 401 and the working raw command both verified by hand. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A portal is meant to find the channel in the command it is handed back. Some cannot, and rather than say so they answer about no channel at all: the link arrives with its stream number left empty, and plays nothing. Naming the channel a second time, in a stream parameter beside the command, is what those providers need on every request. Reported by @shayward, who fixed it first and opened the pull request this takes from. Two things had to change for it to work here. His reads the number out of the command's query string, and canonical_cmd has since rebuilt those commands into markers -- so by tune time there is no query left to read, and it would return nothing on exactly the installs that need it. It is read from the marker's path as well now, which is the only shape left after a sync. And it goes through quote() like the command beside it: a command whose URL is quoted -- the shape extract_link exists for -- otherwise carries its closing quote into the query, and a value with an ampersand in it would arrive as a parameter of its own. Sent only when there is one to send. An empty parameter is a question no portal asked to be asked, and this request is made against every provider, not the two that reported a problem. Whatever is sent came from the portal itself -- its own stream number, or the id it gave the channel in its listing -- so this cannot name a channel the portal did not name first. Confirmed neutral on a portal that never needed it: the channel resolves to the same link, with the same stream number filled in as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The changelog has been written for whoever comes back to it later: a subject, then the portal behaviour that forced each decision and what was tried first. That is the right thing to keep, and the wrong thing to publish -- a release page is read once, by someone deciding whether to upgrade, and 140 lines of reasoning is not what answers that question. So each version section now opens with a line per subject and a "<!-- details -->" marker, below which the existing prose stays untouched. Only what is above it becomes the release body; the file still reads top to bottom, summary then detail, and there is no second file to keep in step with this one. The upgrade note gains the restart it was missing. Test and Re-fetch were named because a sync compares settings that did not change, but neither button helps while half the workers are still running the old code -- plugins are loaded once per process, so the restart has to come first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The release body is now the part of the section above "<!-- details -->", so the reasoning kept in CHANGELOG.md for later stops being pushed at a reader who only wants to know whether to upgrade. GitHub's generated commit list still follows it, for anyone who wants to go further from the release page itself. A section with no marker is published whole, so 0.9.1 and 0.9.2 do not have to be back-filled to keep releasing; an entry that is all detail and no summary fails the build, on the same grounds as one with no entry at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.