Media: add method to discover whether playback is allowed before calling `play()`

[mounirlamouri]

/CC @cpearce @jernoble

With autoplay policies, a media element may not be allowed to play depending on various rules: browser setting, user behaviour on the page, etc. It was suggested to expose this as a permission but because the behaviour is often dependant on past user behaviour on the page [1], it may makes sense to expose the information at a media element level. @cpearce suggested canAutoplay(). The version that returns a promise would work for me.

We could also look into making it more generic such as canPlay() or willPlay() which could be used for other things than autoplay and mostly behave like a dry run and would return the same thing as play() except that it will not start the playback. It may be a bit more painful to handle but may solve more use cases. It would also help resolve situations where autoplay is allowed but playback wouldn't work anyways. Note that one issue with reflecting autoplay for a media element is that it may depend on the availability of the metadata (available tracks for example). We would need to specify what's the expected behaviour with regards to this: should we load the metadata in order to get the information or not? I would rather load all the needed information and call this willPlay() as side effects from calling a method like this may be more expected.

[1]: Chrome allows playback after one user activation on the page; Safari allows playback after one successful playback on the page, etc.

[domenic]

Can you explain why a promise-based version is preferable to a boolean version? How would this require time to determine?

I guess maybe the loading-metadata concern from your second paragraph is part of it. Is there anything else worth noting about that part of the design?

[guest271314]

Off-topic

@domenic Does this functionality not already exist at Chromium and Firefox by using flags and preferences?

Chromium/Chrome

--no-user-gesture-required Autoplay policy that does not require any user gesture.
--user-gesture-required Autoplay policy to require a user gesture in order to play.
--user-gesture-required-for-cross-origin Autoplay policy to require a user gesture in ordor to play for cross origin iframes.
--document-user-activation-required Autoplay policy that requires a document user activation
--ignore-autoplay-restrictions Ignores all autoplay restrictions. It will ignore the current autoplay policy and all restrictions such as playback in a background tab. It should only be enabled for testing.

Firefox/Nightly

media.autoplay.enabled
media.autoplay.enabled.user-gestures-needed
media.block-autoplay-until-in-foreground

(Following your policy statement for WHATWG making it clear that non-maintainers should not ask any questions in WHATWG threads to the OP regarding their issue, asked you, an WHATWG "Owner" the question. If your intent was to actually state that you do not want this user to ask any questions at your board whatsoever, then you should state that, else may be browsing these boards at any given time, and have interest in a topic or subject matter, prompting yet further questions at your board, without any regard for whether the question is "appreciated" or not).

[domenic]

Off-topic

@guest271314 please stop hijacking other peoples' threads with your questions. These are not "boards"; this is an issue tracker where we are trying to get work done.

[guest271314]

Off-topic

@domenic That still does not answer the questions. These are boards, no higher and mightier than any other board.

Your use of the term "hijacking" is ridiculous. You would know the difference if your board was "hijacked".

Am interested in HTML. If you do not want input or question outside of what you conceive then ban this user. Else, have questions, and shall ask them.

[domenic]

Off-topic

@guest271314 As you persist in such hijacking and state your further intentions to continue to do so unless banned, you are now banned. I'm sorry it's come to this. Perhaps in the future you can come to understand the difference between issue trackers for getting work done, and message boards for idle discussion. Questions and input are welcome, _but not in the middle of other peoples' issue threads_.

Apologies to those in this thread trying to solve a technical problem for the above off-topic discursions; dealing with this user's pattern of behavior has been an ongoing issue for some time and it's unfortunate that it spilled over into your thread. If anyone wants to discuss those matters further whatwg/meta is the appropriate venue. Otherwise, let's get back to canAutoplay/canPlay/willPlay.

[cpearce]

@domenic

Can you explain why a promise-based version is preferable to a boolean version? How would this require time to determine?

For Firefox, we may need to call out from our content process to our parent process in order to access a whitelist, and that should be asynchronous. We've found synchronous IPC between processes to be a significant source of UI jank, so we don't want to introduce more.

Also, if we end up having to spin up any kind of demuxing/decoding/parsing infrastructure as @mounirlamouri is suggesting above, that definitely needs to be async, as there's latency involved and our infrastructure there is async anyway.

[cpearce]

We could also look into making it more generic such as canPlay() or willPlay() which could be used for other things than autoplay and mostly behave like a dry run and would return the same thing as play() except that it will not start the playback. It may be a bit more painful to handle but may solve more use cases.

I'm wary of feature creeping here. We have one concrete use case here, what others do we have?

It would also help resolve situations where autoplay is allowed but playback wouldn't work anyways.

It has always irked me that HTMLMediaElement.canPlayType() was not async, as we may need to do disk I/O to load the system's decoders' shared libraries/DLLs from disk and test if they work, and it's pretty horrible to block JS on disk I/O.

And it has irked me that HTMLMediaElement.canPlayType() returns "maybe", but to really be sure we can play some content types on some platforms we need to actually run the decoder and produce decoded samples. We can't just parse the metadata, we need at least enough data for the first frame. So I don't think it's worthwhile going down the path of writing an API that accepts init segments and produces a result.

Also, it would be good to avoid having to download an init segment for a resource in order to tell whether the resource contains an audio track. The motivation for requesting this API was to avoid downloading media data that won't be played.

Note that one issue with reflecting autoplay for a media element is that it may depend on the availability of the metadata (available tracks for example). We would need to specify what's the expected behaviour with regards to this: should we load the metadata in order to get the information or not? I would rather load all the needed information and call this willPlay() as side effects from calling a method like this may be more expected.

Firefox's current draft block-autoplay design takes into account whether the user has interacted with the document, whether the media element is muted or volume != 0, whether the media resource has audio tracks, and whether the media element's document is already whitelisted.

So we may allow autoplay based on properties of the document hosting the media element, or properties of the media resource in question, or properties of the media element hosting the resource.

It's also worth pointing out that AFAIK all browsers that are blocking autoplay are only blocking autoplay of audible media elements. What's interesting to web authors is whether they can play audible media, as it's reasonable, at least in today's environment, to assume that they can inaudibly play media. So maybe the API could just assume it's being asked whether it can play audibly?

So assuming that, in the Firefox and Chrome implementation, the API would basically serve as reporting whether the document had been activated by a user gesture or was whitelisted. I'm not sure how well that maps to Safari's model of blocking autoplay.

[mounirlamouri]

Regarding the audible aspect, I think we shouldn't assume it can always autoplay. Chrome Android currently has an option to always block autoplay even for muted content. We don't intend to provide this on desktop at the moment but it's something many users are asking for and I wouldn't bet that this might not happen one day in a form or another in some browser. We may want to not paint ourselves in a corner. (As a side note, Chrome currently doesn't autoplay media with no audio track, we require muted to be use but that's something we intend to fix.)

Thinking more about restricting to autoplay or not, I would rather have something generic because it will answer more questions (eg. allowed to autoplay but can't play). This said, thinking more about it, I realise that we may hit an issue because of Safari iOS' implementation: last I checked, it relied on a user gesture on the stack in order to allow playback which means that checking asynchronously if playback is allowed wouldn't work very well. We would either need something synchronous or something asynchronous that would be specific to autoplay. Alternatively, we could have something like willPlay({gesture: true}); but I might be going down the rabbit hole :)

I would also prefer to have an async method here as we have been suffering for autoplay logic being mostly sync in Blink. I'm afraid some websites may be relaying on the following behaviour:

media.play();
if (media.paused) // autoplay blocked

To stay safe, we have been sending all the information we have down to the renderer process aggressively. It will be easier to make all this async when the platform will have a better way to deal with autoplay detection. (Arguably media.play().catch() may be it but still feels like a hack.)

[jernoble]

@mounirlamouri

This said, thinking more about it, I realise that we may hit an issue because of Safari iOS' implementation: last I checked, it relied on a user gesture on the stack in order to allow playback which means that checking asynchronously if playback is allowed wouldn't work very well.

I had the same thought, and was in the middle of composing a reply to that effect, when I realized WebKit could just adapt our policy to allow playback from the resolve handler of a canAutoplay() promise. While I think a synchronous API would be fine (I don't think we'd need to do anything requiring asynchronous calls to implement canAutoplay()), we could certainly implement a promised-based API as well.

[jernoble]

@mounirlamouri

We could also look into making it more generic such as canPlay() or willPlay() which could be used for other things than autoplay and mostly behave like a dry run and would return the same thing as play() except that it will not start the playback.

That sounds a lot like load(). Why not just modify load() to return a promise?

[domenic]

That sounds a lot like load(). Why not just modify load() to return a promise?

Proposed by @foolip a couple years ago: #554

[jernoble]

What would canAutoplay() return for an un-muted video element whose readyState was HAVE_NOTHING? (Assuming the UA blocked autoplaying elements with audio tracks.)

1. Return false? Returning true would definitely be incorrect, so if canAutoplay() returns a bool, then false is the only choice here.
2. Return 'maybe'? Borrowing from canPlayType() just perpetuates the hilarity of that API, but it's arguably the correct answer.
3. Return a Promise which resolves when the readyState reaches HAVE_METADATA? An unresolved promise as an answer has a relatively unambiguous meaning of "I can't tell you now".

I'm starting to come around to 3, but returning a Promise raises further questions: does canAutoplay() initiate the media element load algorithm? If not, does it reject with INVALID_STATE error, or does it stay unresolved until something else causes a load() to occur?