Remove exclude list and use XML layout flag "cacheable" instead?

[jissereitsma]

The current implementation of this module comes up with an exclude list. The README says about this option that it "(...) automatically excludes non-cacheable URLs, so this is only needed for custom cached URLs that load private data via JavaScript".

Indeed, this does not apply to non-cacheable URLs. A cacheable URL would be cacheable in a Full Page Cache storage like Varnish. And JavaScript could be used to retrieve private data via AJAX. But what is the scenario where a cached page would have private data in JavaScript that would need to be excluded from bfache?

To my opinion, we are talking about a scenario that never happens. If this never happens, this code just sits there for now reason, making it more difficult to maintain and configure.

So I hope that somebody is able to shed some light on this.

[jissereitsma]

This is being discussed here as well: #2 (comment)

[rhoerr]

@JaJuMa Thoughts?

[jissereitsma]

(I know this thread is a bit duplicate. But I wanted to discuss this separately from issue #2 which also deals with the PR and VCL matters.)

[JaJuMa]

@jissereitsma A valid question, thanks for raising it 🙏

But if I may, the phrase “a scenario that never happens” made me shiver a bit 😉
That’s a very bold statement - in Magento-land I’ve seen too many edge cases over the years to rule anything out entirely.

So, what kind of scenarios are we talking about?
In general, this feature is meant for publicly cacheable pages that still display highly time-sensitive or context-sensitive content via JavaScript.

The probably most common and realistic case is a change of login state while the user navigates cached pages.
This is where many bfcache solutions fall short, while our current implementation already handles it automatically with JavaScript safeguards that refresh the page on login state change to prevent inconsistencies.

Other cases where an exclusion rule could be useful include:

• Flash sales or countdown timers where stale data would be very visible.
• Personalization or A/B testing content injected into otherwise cacheable pages.
• Custom extensions that surface user-specific data via AJAX on cacheable pages.

You’re right that these aren’t things you’d typically find on a vanilla Magento store. They are edge cases.
And in many situations, a better approach is indeed to refresh only the affected section on bfcache restore rather than excluding the whole page.

So the real question is less “do these cases exist?” and more “do we provide a simple, low-effort fallback for them, or do we require every store to implement custom bfcache restore logic themselves?”

As for the concern that this code “just sits there and makes things harder to maintain”:
The exclusion logic is pretty lightweight - it doesn’t add a lot complexity to the core module.
It’s also clearly labeled in the configuration as “optional for most sites”, so there’s no risk of merchants being confused into thinking it’s required.
For the majority of projects it will stay unused, but when an edge case does appear, having it built-in saves developers from needing to implement custom bfcache restore logic from scratch.

Our reasoning for including it in our module comes down to two points:

1. Flexibility & reduced complexity for merchants - a safety valve when stores do encounter one of these edge cases.
2. Awareness & education - by exposing bfcache as something configurable, we highlight that it’s a caching layer developers must keep in mind.

(Imho, the latter point is easily overlooked, but as pointed out in the other thread I see it as crucial. Many devs are not aware of bfcache and how it works - and we already see enough “.vcl-only bfcache” setups breaking stores because of that.)

In summary:
Even if the exclusion list is rarely used, we believe it adds value by providing flexibility and robustness by being ready for edge cases while also raising awareness of bfcache’s implications.

That said, I’m open to other views. If the consensus is that this adds more complexity than value, I won’t object to removing it if it’s not a good fit for the Mage-OS implementation. What do others think?

[rhoerr]

@jissereitsma Any follow up thoughts on this? Thanks!

[jissereitsma]

Thanks @JaJuMa for the thorough answer. And sorry for not getting back earlier - was sick most of last week.

Don't get me wrong - I think the exclude list might very well be a very handy feature. But again, if it is not needed, it is not needed and would just clutter things.

Let me dive into a couple of things that you are saying. You mention the login state on cached pages might cause issues. But as I understand, cachable pages are pages that are cached with the Magento Full Page Cache on (either built-in or via Varnish). And this automatically forces login-related content on those pages to be depersonalized by the depersonalization plugin. In short, server-side HTML content on those pages will not be an issue. In JavaScript, things should be dealt with via private content (aka localStorage), so if the login state changes, the localStorage changes and all JavaScript follows. In the module code, the HTTP context is already used to determine things. But the rest should be dealt in JavaScript right? What would - in your opinion - still get stuck when the bfcache is at play?

Perhaps we can come up first with a real-life coding sample that requires this exclude list, because otherwise the theory is just theory. You sound very convincing in your statements that this occurs, but yet there is no code yet to actually test for this. And I have to be honest, I'm not getting your point for such a edge case either, so unable to think of an actual code sample that causes this edge case to happen. Do you have any ideas on what code could be causing this? I'm happy to start coding to proof that you are right, but as of yet do not know where to begin.

As I see it, the edge cases are actually concerning badly written edge cases that should be avoided. Similarly to just disabled the layout cacheable flag, I'm not sure whether allowing excludes to option so easily is encouraging people to create the right solution.

I personally hate myself for bringing this up anyway: Open source contributions that go to waste is a pity. But I - just like you - just want to make sure Mage-OS contributions kick-ass. If the discussion goes nowhere, I would rather just have the exclude list to be kept in this module, just for the sake of happy campers.

[JaJuMa]

@jissereitsma Thanks for the thoughtful and detailed follow-up, and no worries about the delay.

To make sure we’re on the same page, let’s quickly recap:

Fresh Page Load vs. bfcache Restore
On a fresh page load, a well-architected Magento site handles personalization correctly:

• Server-side: FPC/varnish depersonalizes cached pages.
• Client-side: Private content (localStorage) fetches the right data.

The tricky part (and the reason this module exists) is that a bfcache restore is not a fresh page load.

• FPC/varnish is server-side: it builds and delivers a new page.
• bfcache is client-side: it takes a literal snapshot of the fully rendered DOM and JS state and freezes it in memory.
• On back/forward navigation, the browser just restores that snapshot,no initial load logic runs and no server checks happen.

This means bfcache is completely unaware of depersonalization, login state changes, or localStorage updates that may have happened since the page was first rendered.

Taking above into account, lets look at the current bfcache implementation as a series of layers, each handling a different type of content:

Layer 1: Private / Uncacheable Pages

Pages like Customer Account, Cart, and Checkout are already excluded from both FPC and bfcache by default. No issues here.

Layer 2: Public Pages with Dynamic Content

This is the critical layer. Because bfcache restores a frozen snapshot, certain dynamic client-side features break unless they’re explicitly handled.
Currently we have safeguards for the most common Magento cases:

• Minicart updates
• Menu/Cart Drawer toggle state
• Global messages
• Login state changes

Layer 3: Custom & Third-Party Edge Cases (Where Exclusions Help)

The exclusion list is designed for scenarios outside of Layer 2.
As an extension provider, we’ve seen several in the wild
(Using our own modules as examples here, since honestly I’m not aware of others actively handling bfcache. Most advice I see simply suggests removing the no-store header without additional safeguards, which comes with obvious risks.):

• A module that “remembers” customer configurable/custom option selections (via local storage).
  → We made our Auto Select Options extension bfcache-ready, but any similar client-side customization can easily break/stop working/display stale selections on bfcache restore
• Analytics and reporting integrations:
  → We made our Matomo Analytics & RUM implementations bfcache ready, but if not bfcache-aware (as most such integrations), reports will be skewed by missing restores.

Fair enough, those cases affect too many pages to use the exclude list as a solution.
So lets look at other examples:

• With our Category Grid Callout extension we allow adding callouts and banners into the category grid, incl. an option to display random banners.
  → This we haven't made bfcache ready yet. On bfcache restore, it will show the previous banner.
  If we wanted to have it truly random on bfcache restore we would need to add the pageshow restore logic to refresh the banner as the "right"/best-practice solution.

But in practice, not every merchant wants to invest in that level of development for low-impact or temporary cases (like a randomized banner or other uses cases mentioned above in last message)
Excluding a URL (the category with the random banner, JS-driven personalizations, the flash sale page...) can be the pragmatic, cost-effective alternative.

Summary
The exclusion list is not about encouraging sloppy code. It’s about providing a safety valve for Layer 3, reducing headaches for merchants while keeping the defaults clean for most stores.
We believe it adds robustness and flexibility, but we also see this as a matter of preference.
If the Mage-OS consensus is to drop it and require custom handling everywhere, we’re fine with that too.
Even without exclusions, the current implementation is - as far as I can see - the most advanced bfcache handling in Magento-space. The exclusion option just provides an extra layer of flexibility for (edgy) real-world scenarios.
And if someone comes up with an even better solution for Layer 3, we’d be the first to support it.

[CyberSecutor]

Hi @jissereitsma and @JaJuMa ,

maybe I am not interpreting it correctly but I think the exclusion list is less useful for another reason.
The only time it gets checked and processed is in the HTTP plugin.

However the plugin (both the before and after) checks the current cache type.
If it is anything other then 'built-in', i.e. varnish, fastly, it does nothing with the exclusion list. (As well as the enable/disable option).

I believe most if not all Magento sites concerned with performance will use some sort of caching back-end like Varnish or Fastly.

module-theme-optimization/Plugin/Framework/App/Response/Http.php

Lines 42 to 46 in 4cf2065

	public function beforeSetNoCacheHeaders(\Magento\Framework\App\Response\Http $subject): void
	{
	if ($this->config->getType() !== Config::BUILT_IN || !$this->isEnabled()) {
	return;
	}

This will skip all exclusion list checking done afterwards if the caching method is not 'buit-in'.

I understand why this was added, since when the VCL snippets are implemented in the caching application, the cache headers sent by Magento don't matter anymore since they get checked and then overridden.