Better mechanism of ordering global transformations

[davesnx]

When users specify multiple global ppx transformations via preprocessors in a dune file, the order they apply is alphabetically by library name and described like this on the ppxlib documentation https://ocaml-ppx.github.io/ppxlib/ppxlib/driver.html#global-transfo-phase (Thanks for the ref @sim642)

This is problematic because ppx ordering is semantically significant: the output of one ppx often serves as input to another, and many ppxes don't commute. Using the name as a order is problematic since users can't change it (and they make authors responsible for the ppx name)

Current Behavior

Given this dune file:

(library
 (name mylib)
 (preprocess (pps ppx_jane ppx_deriving ppx_blob)))

Expected: ppxes are applied in the order written: ppx_jane → ppx_deriving → ppx_blob
Actual: Dune sorts them alphabetically and applies them as: ppx_blob → ppx_deriving → ppx_jane

Cross-reference: ocaml/dune#13551

[NathanReb]

The whole idea behind ppxlib is to encourage people to write context-free transformation so that the problem you mention does not have to be taken into consideration.

Whole AST transforms are slow and pose a semantic and interaction problem. We still support them because some transformations can't be expressed with context-free rules but they are not the prefered form and should not be encouraged. Global transformation are not meant to be ordered in a specific way and as such, they should not depend on the result of each other.

What you suggest requires changes in both dune and ppxlib and I'm not sure we want to go down that road.

Can you give a bit of context on what you're trying to do and why you need this? Which ppx-es are you using and which ones of them use global transformations?

Have you tried building your own driver ? This would require the ppx-es you use to provide the right functions in a library without the registration side-effect but this could make for a good work around in the meantime.

I'd be willing to consider this if and only if to there is no other possible way and a very good reason to do it. My default assumption here is that it is note the case so I'd need a concrete example to be convinced otherwise.

Side note: as @sim642 mentioned in the dune counterpart, ppx_blob and ppx_deriving are context-free transformations ppx-es and ppx_jane is a compilation of multiple ppx-es, some of which might be global transformations, I don't remember the full list. So what happens is that global transformations from ppx_jane would be applied in alphabetical order, before or after the context-free pass where all other ppx-es from ppx_jane (the majority of them I believe), ppx_blob and ppx_deriving would be applied recursively until there is nothing left for them to rewrite.

[davesnx]

Totally, context-free transformations are clearly superior, but there are cases that are not possible.

We have this set of ppxes:

• server-reason-react-ppx (global transformation): turns JSX attributes into the right nodes, does some static analysis, and operates on structure_items
• browser_only_ppx (mostly context free, but 2 cases of global transformation): https://ml-in-barcelona.github.io/server-reason-react/server-reason-react/browser_ppx.html#[@platform]-attribute. Discards some ast nodes, based on being melange or native. Operates on structure_items
• styled-ppx (most context-free, but 1 case that needs the traversal). let x = [%cx "asdfasd"] and the V3 allows

adding any other global transformation is really challenging now, and a way in dune to order them would benefit this a lot. I agree I could create a Driver, but If I want those ppxes to be used by users that aren't me, I can't recommend have a driver

[NathanReb]

From experience I know that sometimes, it's tempting to write global transformations for a smoother syntax so I'd advise making sure that's not the case for any of the global transformations here. I trust you've done that already but it's always a good reminder. I'll take a look when I get the time to see if I can suggest slight changes that allow writing some of those as context-free rules, a fresh pair of eyes can be useful in those situations.

I don't doubt there are cases where you have no choice. I would argue that ppxlib might not be the right tool for those transformations which usually have to implement complex logic on top of the AST, sometimes closer to static analysis than to source preprocessing.

All this being said, I agree that the lack of alternatives makes ppxlib the de facto choice for this kind of transformations and that we do need to provide some support, even if very limited and low level.

I discussed this with @patricoferris and we agreed on a simple solution. We'll expose as semi-private API the reference to the list of registered transformation along with the abstract type for transformations and the minimum required accessors (likely just a get_name : Transformation.t -> string). This should let advanced users build a ppxlib driver as follows:

let reorder_transformations : transformation list -> transformation list = ....

let () =
  Ppxlib.Driver.Private.Dangerous.Do_not_use.(transformations := reorder_transformations !transformations);
  Ppxlib.Driver.standalone ()

Whether the dune folks want to provide support for setting this order or not will be up to them, I personally don't think they should and that this feature should remain as secret as possible. I do not want to encourage order dependent global transformations. Advanced users with specific use cases can always define their own driver this way and pass it as a "custom" preprocessing command in their dune files.

Does that sound like it would work for your personal usecase?

[davesnx]

I like the thought put into this issue, but I disagree with the angle: it's not an advanced use case, nor is it for advanced users. Regular users will use ppxes that may cause global transformations to collide and be completely blocked (and using Private.Dangerous.Do_not_use... isn't even a decent workaround).

I agree that global transformations are bad, but they are necessary for a few use-cases, if those didn't exist, I would happily close this issue and fix it in my code.

This issue is open to allow users to configure the order of transformations by the order on dune (or any other method really), so at least users could unblock themselves or we can add some documentation to mitigate this particular issue.

[NathanReb]

You're right and I phrased this poorly. I do want to provide a workaround for users impacted by this but thankfully there aren't too many because the vast majority of ppx-es are expressed as context-free rules and allow fast preprocessing with a clear semantic, which global transformations don't.

My point is that I do not want to encourage ppx authors to add more of those to the ecosystem than there already is as I'm not convinced ppxlib is or will be the right tool for stacking up such complex language extensions. I think if there's a need for such extensions, maybe we, as a community, should discuss how to handle them efficiently.

As I said earlier though, I am eager to solve the existing problem. I think exposing the list of transformations to allow either dune or users to explicitly set the order of non context-free transformations should be enough here. The choice of submodule was mostly a joke though I still want this to be documented as something that should be kept as last resort and that we encourage writing context-free transformations.

[NathanReb]

Out of curiosity, could you illustrate with a concrete example the kind of issues that arise when the ppx-es you mention aren't correcty ordered?