introducing dexpr - #8649
Conversation
b29cf1e to
886afbb
Compare
peterdschwartz
left a comment
There was a problem hiding this comment.
coded by geniuses
bartgol
left a comment
There was a problem hiding this comment.
I am a bit concerned this will become "ask Peter cause he wrote it" kind of code. There is little documentation, and it's a bit hard to understand why each part of the code is needed and how they play together.
Would it be possible to enhance the in-code docs? I've seen a few lines of code in my life, but this seems hard to digest, and very "dense". I don't question its cleverness/efficacy, but, for maintainability reasons, can we make it more legible/documented?
| tok = make_token(TokenTypes::Slash); | ||
| break; | ||
| case '*': | ||
| if (peek_char() == '*') { |
There was a problem hiding this comment.
Not sure how I feel about ** for exponentiation... I would prefer ^, since it's closer to math notation... But it may just be my personal preference (not a fan of py or fortran's **).
There was a problem hiding this comment.
We want this to be closest to python... in fact, that's our main compass here... that's the main reason for keeping it...
|
|
||
| } | ||
|
|
||
| Precedence cur_precedence(TokenTypes type) { |
There was a problem hiding this comment.
I don't really get what this fcn is supposed to be used for.
There was a problem hiding this comment.
just to deal with the one exception below...
// '**' is the only right-associative operator: 2**3**2 is 2**(3**2).
case TokenTypes::Exp:
return static_cast<Precedence>(static_cast<int>(prec) - 1);There was a problem hiding this comment.
Note that i renamed this function to right_binding_precedence.
There was a problem hiding this comment.
It's a bit obscure though. I'm interpreting the name as "current precedence". Why wouldn't the Precedence returned by "token_precedence" be already the correct precendence for the current token? My thought was "if this prec is wrong, why not changing what token_precedence returns for Exp?".
That's a good point. I think we should do as you suggest. I will defer to @peterdschwartz to decide if he wants to do it or if he wants me to do it. I will be happy to walk through the key parts and document them for the reader. My strategy here was pretty simple: I took Peter's code in 41a83d5 and I worked inside the tests by basically coming up with test cases I wanted to pass in human language and I told bots to implement them (tests + corresponding edits). Part of that was me having to make several calls (including the one you dislike about |
|
I can add descriptions tomorrow |
|
pushed new commit responding to some of the review comments. |
jgfouca
left a comment
There was a problem hiding this comment.
Since this is its own independent package, I only care that it works, is tested, and has some documentation. It looks like this is the case, so I approve. I will let copilot handle the implementation review.
|
Oh, I should ask, is this intended to be a sharedlib or built concurrently with E3SM? |
There was a problem hiding this comment.
Pull request overview
Introduces dexpr, a standalone C++20 lexer and Pratt parser for diagnostic
expressions, producing ASTs for future component integration.
Changes:
- Adds lexer, parser, AST, precedence, and function metadata APIs.
- Adds Catch2 coverage, CLI tooling, documentation, and build scripts.
- Adds dedicated GitHub Actions testing across compilers and build modes.
Reviewed changes
Copilot reviewed 22 out of 23 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/dexpr-testing.yml |
Adds compiler/build CI matrix. |
share/README |
Documents the new library. |
share/dexpr/.gitignore |
Ignores local build artifacts. |
share/dexpr/CMakeLists.txt |
Configures library, CLI, and tests. |
share/dexpr/LICENSE |
Adds vendored MIT license. |
share/dexpr/README.md |
Documents usage, architecture, and plans. |
share/dexpr/include/dexpr/ast.hpp |
Defines public AST nodes and visitors. |
share/dexpr/include/dexpr/lexer.hpp |
Declares the lexer API. |
share/dexpr/include/dexpr/parser.hpp |
Declares parser and error APIs. |
share/dexpr/include/dexpr/precedences.hpp |
Defines precedence levels. |
share/dexpr/include/dexpr/supported_functions.hpp |
Lists diagnostic functions. |
share/dexpr/include/dexpr/tokens.hpp |
Defines tokens and keywords. |
share/dexpr/run_tests.sh |
Adds local build/test runner. |
share/dexpr/src/ast_print.cpp |
Serializes AST expressions. |
share/dexpr/src/lexer.cpp |
Implements lexical analysis. |
share/dexpr/src/parser.cpp |
Implements Pratt parsing. |
share/dexpr/src/precedences.cpp |
Maps operators to precedence. |
share/dexpr/src/tokens.cpp |
Implements token utilities. |
share/dexpr/tests/CMakeLists.txt |
Configures Catch2 tests. |
share/dexpr/tests/test_lexer.cpp |
Tests lexer behavior and errors. |
share/dexpr/tests/test_list_supported_functions.cpp |
Adds an empty test placeholder. |
share/dexpr/tests/test_parser.cpp |
Tests parsing and AST output. |
share/dexpr/tools/dexpr.cpp |
Adds the command-line helper. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
@peterdschwartz / @mahf708 : Any reason you chose not use tools like ANTLR/Bison? |
I had already written a Pratt parser in python and this is essentially a trimmed down version of that ported to C++. The handwritten parser is only a small amount of code, and avoids introducing a parser-generator/code-generation dependency into the build. If the language were expected to grow substantially or we needed more flexibility I think a generator such as ANTLR would become more compelling. |
You should be able to check in the generated code. |
I am all for reusing existing tools. But I don't think autogenerated code should be added to the repo. I have not seen antlr/bison output, but I'm guessing it will be not very readable, and lack substantial comments. Sure, we can put a header "do not touch, this was autogenerated", but I still dislike having code nobody understands in the repo. Btw, Peter, bison is relatively common on linux systems, so it would not be a big dependency to add... FWIW, gemini seems to strongly believe that hand-rolling our own lexer/parser is a superior choice over on-the-fly generation (via antlr/bison or similar), as well as over stuff like boost::spirit. |
|
My $0.02: ANTLR and Bison are great, but you really have to buy into their way of doing things to use them (and deal with annoyances like generated code). I'd prefer a hand-rolled solution if we can get away with it (i.e. if we can avoid falling down one or more parsing-related rabbit holes). |
|
How much technical debt do we take on with a roll-our-own solution vs. bison/ANTR? Will future developers curse us for having to maintain this thing? Checking in generated code is fine if its only re-generated once in a while. We already do that with the Physics Constants Dictionary. |
|
Beyond the generated code vs not --- I think adding a dependency is dangerous. We are trying to trim our sprawling dependencies and submods. What's more, I think we view this solution (a few files of cpp) as a generally static library that will live largely unchanged. We prefer an in-house solution because we will want to attach a custom interface and validator. Consider the following, a future component decides it wants to introduce new vocabulary/grammar to the mix. In a soon-to-be-submitted PR, we will have share/dexpr/include/dexpr/supported_functions.hpp become an inteface with an additional validator. That way, the component can add its new grammar (say Thus, in a specific way, this little library will be much more malleable than relying on external vendors with their specific ways of doing things. |
|
I invite anyone who has actually used ANTLR and/or Bison to weigh in on the tech debt question . I've used them both enough to know that they're not something you just drop into your code. There's a lot to know about parser generators that isn't necessarily stuff you need to know about parsing, and unless we want to develop a wing of parser generator experts in E3SM, I suggest we at least try to get by without them. If this is clearly the wrong approach, we'll find out and then we can reassess. :-) |
|
Last minute comment, but...I think the name "dexpr" is a bit cryptic. What do people think about a different or at least more spelled out name? Maybe |
I think cryptic is ok here (it's unique and doesn't need to be as descriptive since it is specialized). We will want something to convey "expression" in the name as well. I would strongly vote for "dexpr" (I considered dxpr, but I think dexpr is better) |
|
According to the README.md "dexpr produces an abstract syntax tree. It does not evaluate anything, and it knows nothing about fields, grids or timesteps -- turning an AST into an actual diagnostic is the caller's job." Can someone point me to an example/sketch of how this all is supposed to work? |
here: #8623. In very broad strokes:
Also, see these internal notes here: https://e3sm.atlassian.net/wiki/spaces/p3ai/pages/6548127761/2026-08-13+ai+group+infrastructure+e3sm+dsl (there are two additional pieces for this to become functional (as in applied): 1) attaching the words to ops and 2) replacing the regexes in EAMxx. These will be done in a follow-up PR.) |
|
|
||
| inline constexpr std::array<std::string_view, 0> tend_args{}; | ||
|
|
||
| inline constexpr std::array supported{ |
There was a problem hiding this comment.
This is the initial list of functions which will grow?
There was a problem hiding this comment.
This part will become an 'interface' in the next PR with some standard functions built-in, but the key is that components can/will define their own. We talked about this with the omega team already. In the next PR, you will see two movements to address this, first, the supported function stuff will become an interface, but second (and importantly) we will add a validating utility so that components can test if their definitions actually make sense/pass.
There was a problem hiding this comment.
When this is all done, where will the various code pieces live?
There was a problem hiding this comment.
it doesn't matter because they're modular pieces. This part is the parser and will here. We may have one other related tool that will live alongside in share, but everything else will belong to the components initially (in the distant future, we can consolidate more, but we don't want to overengineer at first)
|
@rljacob A general remark that may not be readily clear yet: the goal here is we want to have an xarray-like snytax for the eventual "language" --- so users won't need to learn new ways of doing anything, it will very closely mimick (where possible) the classic dataframe in our domain, which is xarray. |
|
This is PR 1 of a few PRs that will implement the capability. Please write down the full plan somewhere. Include where the various code pieces will live and a "before" and "after" example. (what does a diagnostic currently look like in EAMxx and what it will look like after this is fully implemented). That will help alot. Since you want lots of people to use this, start a discussion in https://github.com/E3SM-Project/E3SM/discussions/categories/development-discussion |
|
We had a few meetings; people who expressed interest in this attended (Xylar, Brian, Jeff, Peter, Aaron, Luca, ...) For people interest, here's the link to meeting notes internally: click. They will continue until Oct 1, when we will submit a manuscript about this work. Everyone is invited to be co-author (opt-in). |
|
That page is pretty sparse. The big picture is obviously clear in your heads. You could expand those notes in Confluence if you prefer. |
|
Any more feedback before we merge this? @jeff-cohere, @xylar, and @bartgol --- any comments? Since this doesn't affect anything in E3SM, let's aim to merge to next today and to the main branch tomorrow. |
|
@mahf708, please proceed. No feedback from me right now. I haven't had time to look at it this week and was at a conference last week. |
|
With the last two commits, everything related to this PR should have been resolved. |
rljacob
left a comment
There was a problem hiding this comment.
Hope to get some documentation in a future PR.
|
@peterdschwartz can you quickly fix the md linter complaints? Then I will integrate. |
Import the diagnostics expression lexer/parser from the upstream
standalone repo, unmodified apart from its location. The code is a
hand-written lexer plus a Pratt parser producing a std::variant-based
AST; it is plain C++20 with no Kokkos, MPI, netCDF or EKAT dependency.
It lands under share/ rather than a single component because nothing in
it is component-specific, and it is deliberately not wired into any
build yet -- this commit only parks the source.
Upstream is MIT licensed; LICENSE is carried over verbatim.
Co-authored-by: peterdschwartz <peterdschwartz83@gmail.com>
build the parser standalone with its own ctest suite
Split the test setup into tests/CMakeLists.txt and give the project real
options, so the library, the tool and the tests can each be turned off
independently.
Catch2 is now found before it is fetched. A machine that already has
Catch2 v3 installed uses it; anything else falls back to FetchContent, so
a bare `cmake -S . -B build` still works from a clean checkout.
Turn on -Wall -Wextra -Wpedantic. Nothing external is in the include path,
so there are no third-party headers to fight and the library can simply
stay warning-clean. Fix the one signed/unsigned comparison that exposed.
upgrade the lexer, and test it properly
Bring the lexer to the point where it handles the input the diagnostics
DSL actually contains, and cover it with tests: the suite goes from 2
cases to 26.
Identifiers may now contain digits after the first character, which still
may not be one. Field names routinely carry them -- T_2m, qv_850, bc_a1,
ne30pg2 -- so a lexer that stops at the first digit is unusable here.
Numeric literals no longer parse to a wrong answer in silence:
".5.3" lexed as one Float("0.5.3"). The '.' branch consumed the
leading dot and then called read_number(), which started over
and folded in the second one. Downstream parsing read the
"0.5" prefix, stopped, and did not complain. Meanwhile "1.2.3"
was correctly rejected, so the two spellings disagreed.
read_number() now accounts for a dot the caller already took.
"1E5" lexed as Integer("1E5"), because the float/integer split
tested ".e" and missed the uppercase exponent marker that
read_number() happily accepts. Integer parsing then stopped at
the 'E' and yielded 1. Classification now tests ".eE".
The input is no longer case-folded wholesale. Folding the buffer made
keywords case-insensitive but also rewrote string literals, which are
data rather than syntax, so 'MyVar' silently became 'myvar'. Case
insensitivity now applies where it belongs: keyword lookup folds before
matching, and the exponent marker accepts either case.
An unterminated string literal is reported as Illegal instead of being
accepted as a well-formed string.
The unused Newline token is gone and token_precedence gets an explicit
default, which together clear the -Wswitch warning the build has been
emitting since the parser was vendored.
changed alphanumeric identifier
finish the parser, and test it properly
Bring the parser to the same standard as the lexer and cover it: the
suite goes from 26 cases to 57.
The new tests exercise precedence and associativity, prefix operators,
grouping, function calls, member access, array literals, equality
operators, and printing an AST back to a form that lexes again. Several
fixes to precedences and to ast_print fall out of what they exposed, and
exponentiation is now right-associative, so 2**3**2 is 2**(3**2) as it is
everywhere else.
Numeric literals are held at double precision. As float, a threshold did
not survive being parsed:
273.15 as float = 273.14999389648438
273.15 as double = 273.14999999999998
and a legal value like 1e40 was rejected outright as out of range. For a
language whose main job is expressing thresholds over climate fields,
narrowing in the AST is the wrong place to do it; a consumer that wants
single precision can narrow when it knows that.
Literals are read with std::from_chars rather than std::stoi/std::stof.
It does not depend on the locale, where std::stof reads "1.5" as 1
wherever ',' is the decimal separator; it does not throw; and it reports
where it stopped. Requiring it to stop at the end of the literal rejects
a malformed literal outright instead of quietly accepting its leading
prefix, which is the same failure the lexer fixes guard against from the
other side.
A missing ')' now throws where it is detected instead of returning null.
parse() caught that at the end anyway, but in between the null travelled
through the tree builders, and every visitor dereferences its children
unguarded.
Parser::cur_token_is is gone. It was declared and defined but never
called from anywhere, while its sibling peek_token_is has seven call
sites.
settle on the name dexpr, enforce it, and document it
Three things that together fix the library's identity, done at once
because the rename touches every file and splitting it would mean
reviewing the same mechanical diff twice.
The e3sm prefix was redundant inside the E3SM repo, and the edp acronym
expanded to "E3SM Diags Parser", so it went stale the moment that name
did. One word is now used everywhere: directory, CMake target, namespace,
include prefix and CLI binary are all dexpr.
The word expression is what earns the name. Without it, "diags parser"
reads as the thing that parses diagnostics config -- output YAML field
lists, legacy diag name strings, namelist entries -- which is a real and
separate job in EAMxx. dexpr parses expressions.
include/edp/ -> include/dexpr/ EDP_*_HPP -> DEXPR_*_HPP
namespace edp -> namespace dexpr EDP_ENABLE_* -> DEXPR_ENABLE_*
tools/edp.cpp -> tools/dexpr.cpp
The library now owns the name dexpr, so the tool's CMake target is
dexpr_cli with OUTPUT_NAME dexpr; the binary is unchanged.
Warnings become errors in CI. The code has been warning-clean since the
switch statements were completed, so this makes it a property the build
enforces rather than one that happens to hold. DEXPR_WERROR is off by
default, leaving a developer build unaffected, and on in CI. The flags
are set once at the top level so the tests compile under exactly the
same set as the library. CI also runs the tool now, which covers the one
path the unit tests do not: main().
Finally, a README covering what the library does, how it is laid out, how
to build and test it, and where it came from, plus its entry in
share/README alongside the other subdirectories.
tidy up the token table, error messages, and float printing
Four fixes that share no code but do share a shape: each is a place where
the library was doing something that happened to work rather than
something it could rely on.
The keyword table had one definition per translation unit. `const` at
namespace scope has internal linkage, so the std::unordered_map in
tokens.hpp was constructed separately in every unit that included the
header -- three heap-allocated string keys plus a static initializer each
-- including in ast_print.cpp and precedences.cpp, which never look a
keyword up. Three entries do not need a hash table. A constexpr array of
string_view searched linearly is one definition and no runtime
construction at all. Measured on the built objects, static-init routines
went from one in each of five units to none.
Four token types were declared that the lexer never produces. Percent,
Semicolon, DoubleColon and Concat had to_string arms, so they looked
supported from outside, but nothing ever emitted one:
% -> Illegal(%) ; -> Illegal(;)
:: -> Colon(:) Colon(:) a//b -> a Slash Slash b
None is referenced anywhere, including on the diag-integration branches,
so they are gone rather than given syntax to justify them. Colon stays:
it is lexed, tested, and holds the Bounds precedence for the slicing
that ast.hpp still has stubbed out. Dropping the default arm from
to_string means -Wswitch now names the next token type someone adds
instead of letting it print as UNKNOWN. The unused ostream operator for
Token goes too, along with the "Identifer" spelling that reached users
through parser messages, and a `Token tok;` whose type was left
indeterminate.
Parse errors say where they happened. They named the offending token but
not its location, so a typo in a long expression sent the reader back
through the whole thing to find what the parser had already pinpointed:
before: Unexpected Prefix Token {Type: Plus, Literal: +}
after: Unexpected Prefix Token {Type: Plus, Literal: +} at line 1, column 5
Token gains line and column, both defaulted so the two-argument aggregate
form still works. The lexer maintains them in read_char(), and
next_token() captures the position before scanning and stamps the result,
so the scanner's many early returns cannot forget to.
Floats print with std::to_chars instead of std::format. <format> was the
only thing here requiring GCC 13, and it bought nothing: the standard
defines std::format("{}", d) for floating point in terms of
std::to_chars(first, last, d), so the output is identical by
specification rather than by luck -- every existing float printing test
passes unmodified. <charconv> was already a hard dependency, since
literals are parsed with from_chars, so this removes a requirement rather
than adding one, and the floor drops to GCC 11. Parsing and printing are
now inverses over one facility, which a round-trip test asserts directly.
note what the parser does not do yet
Record the extension points that are deliberately absent, so the next
person to want one starts from the reasoning rather than rediscovering
it: component-supplied functions, slicing, and the fact that operator
syntax is fixed even if functions stop being.
The first of these is the one that matters. supported_functions.hpp is a
fixed table that nothing consults, so an unknown call parses and is never
rejected; replacing it with a registry a component fills in is a separate
change with its own tests, and does not belong in the commit that only
parks the parser.
Adjustments due to reviewer comments.
- Added DOxygen style comment docs to major headers and functions
- renamed `cur_precedence` to reflect its role in associativity of
operators
- Switched unordered_map in Parser with switch table lookup
- Removed un-used code portions
respond to reviewer comments
5687341 to
fc946e4
Compare
|
ok -- just installed |
dexpr (said like crispr) is the expression parser for in-situ e3sm diagnostics. We will expand it to allow for a component interface and other neat nifty items like trees and such in future PRs This PR only implements the parsing layer to generate an Abstract Syntax Tree from user input. This defines the expected grammar for e3sm diagnostics and serves as a shared stable foundation between components.
dexpr is said like crispr, it's the expression parser for in-situ e3sm diagnostics, we will expand it to allow for a component interface and other neat nifty items like trees and such in future PRs
This PR only implements the parsing layer to generate an Abstract Syntax Tree from user input. This defines the expected grammar for e3sm diagnostics and serves as a shared stable foundation between components.