Skip to content

introducing dexpr - #8649

Merged
bartgol merged 1 commit into
masterfrom
mahf708/share/edp
Aug 27, 2026
Merged

introducing dexpr#8649
bartgol merged 1 commit into
masterfrom
mahf708/share/edp

Conversation

@mahf708

@mahf708 mahf708 commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

dexpr is said like crispr, it's the expression parser for in-situ e3sm diagnostics, we will expand it to allow for a component interface and other neat nifty items like trees and such in future PRs

This PR only implements the parsing layer to generate an Abstract Syntax Tree from user input. This defines the expected grammar for e3sm diagnostics and serves as a shared stable foundation between components.

@peterdschwartz
peterdschwartz self-requested a review August 20, 2026 16:17
@mahf708
mahf708 force-pushed the mahf708/share/edp branch from b29cf1e to 886afbb Compare August 20, 2026 17:49

@peterdschwartz peterdschwartz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

coded by geniuses

@mahf708
mahf708 marked this pull request as ready for review August 20, 2026 21:13

@bartgol bartgol left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a bit concerned this will become "ask Peter cause he wrote it" kind of code. There is little documentation, and it's a bit hard to understand why each part of the code is needed and how they play together.

Would it be possible to enhance the in-code docs? I've seen a few lines of code in my life, but this seems hard to digest, and very "dense". I don't question its cleverness/efficacy, but, for maintainability reasons, can we make it more legible/documented?

Comment thread share/dexpr/include/dexpr/ast.hpp Outdated
Comment thread share/dexpr/include/dexpr/ast.hpp Outdated
Comment thread share/dexpr/src/lexer.cpp
tok = make_token(TokenTypes::Slash);
break;
case '*':
if (peek_char() == '*') {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how I feel about ** for exponentiation... I would prefer ^, since it's closer to math notation... But it may just be my personal preference (not a fan of py or fortran's **).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want this to be closest to python... in fact, that's our main compass here... that's the main reason for keeping it...

Comment thread share/dexpr/include/dexpr/precedences.hpp
Comment thread share/dexpr/include/dexpr/precedences.hpp Outdated
Comment thread share/dexpr/src/precedences.cpp Outdated

}

Precedence cur_precedence(TokenTypes type) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really get what this fcn is supposed to be used for.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just to deal with the one exception below...

  // '**' is the only right-associative operator: 2**3**2 is 2**(3**2).
  case TokenTypes::Exp:
    return static_cast<Precedence>(static_cast<int>(prec) - 1);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that i renamed this function to right_binding_precedence.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit obscure though. I'm interpreting the name as "current precedence". Why wouldn't the Precedence returned by "token_precedence" be already the correct precendence for the current token? My thought was "if this prec is wrong, why not changing what token_precedence returns for Exp?".

Comment thread share/dexpr/src/parser.cpp Outdated
@mahf708

mahf708 commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

I am a bit concerned this will become "ask Peter cause he wrote it" kind of code. There is little documentation, and it's a bit hard to understand why each part of the code is needed and how they play together.

Would it be possible to enhance the in-code docs? I've seen a few lines of code in my life, but this seems hard to digest, and very "dense". I don't question its cleverness/efficacy, but, for maintainability reasons, can we make it more legible/documented?

That's a good point. I think we should do as you suggest. I will defer to @peterdschwartz to decide if he wants to do it or if he wants me to do it. I will be happy to walk through the key parts and document them for the reader.

My strategy here was pretty simple: I took Peter's code in 41a83d5 and I worked inside the tests by basically coming up with test cases I wanted to pass in human language and I told bots to implement them (tests + corresponding edits). Part of that was me having to make several calls (including the one you dislike about **). There are several issues that came up in the DEMO PRs which required some revision in the parser itself (xref #8623 & #8624)

@peterdschwartz

Copy link
Copy Markdown
Contributor

I can add descriptions tomorrow

@peterdschwartz

peterdschwartz commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

pushed new commit responding to some of the review comments.

@jgfouca jgfouca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is its own independent package, I only care that it works, is tested, and has some documentation. It looks like this is the case, so I approve. I will let copilot handle the implementation review.

@jgfouca

jgfouca commented Aug 21, 2026

Copy link
Copy Markdown
Member

Oh, I should ask, is this intended to be a sharedlib or built concurrently with E3SM?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces dexpr, a standalone C++20 lexer and Pratt parser for diagnostic
expressions, producing ASTs for future component integration.

Changes:

  • Adds lexer, parser, AST, precedence, and function metadata APIs.
  • Adds Catch2 coverage, CLI tooling, documentation, and build scripts.
  • Adds dedicated GitHub Actions testing across compilers and build modes.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
.github/workflows/dexpr-testing.yml Adds compiler/build CI matrix.
share/README Documents the new library.
share/dexpr/.gitignore Ignores local build artifacts.
share/dexpr/CMakeLists.txt Configures library, CLI, and tests.
share/dexpr/LICENSE Adds vendored MIT license.
share/dexpr/README.md Documents usage, architecture, and plans.
share/dexpr/include/dexpr/ast.hpp Defines public AST nodes and visitors.
share/dexpr/include/dexpr/lexer.hpp Declares the lexer API.
share/dexpr/include/dexpr/parser.hpp Declares parser and error APIs.
share/dexpr/include/dexpr/precedences.hpp Defines precedence levels.
share/dexpr/include/dexpr/supported_functions.hpp Lists diagnostic functions.
share/dexpr/include/dexpr/tokens.hpp Defines tokens and keywords.
share/dexpr/run_tests.sh Adds local build/test runner.
share/dexpr/src/ast_print.cpp Serializes AST expressions.
share/dexpr/src/lexer.cpp Implements lexical analysis.
share/dexpr/src/parser.cpp Implements Pratt parsing.
share/dexpr/src/precedences.cpp Maps operators to precedence.
share/dexpr/src/tokens.cpp Implements token utilities.
share/dexpr/tests/CMakeLists.txt Configures Catch2 tests.
share/dexpr/tests/test_lexer.cpp Tests lexer behavior and errors.
share/dexpr/tests/test_list_supported_functions.cpp Adds an empty test placeholder.
share/dexpr/tests/test_parser.cpp Tests parsing and AST output.
share/dexpr/tools/dexpr.cpp Adds the command-line helper.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread share/dexpr/src/precedences.cpp Outdated
Comment thread share/dexpr/src/ast_print.cpp
Comment thread share/dexpr/include/dexpr/supported_functions.hpp
Comment thread share/dexpr/README.md Outdated
Comment thread share/dexpr/include/dexpr/ast.hpp
@jayeshkrishna

jayeshkrishna commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

@peterdschwartz / @mahf708 : Any reason you chose not use tools like ANTLR/Bison?

@peterdschwartz

Copy link
Copy Markdown
Contributor

@peterdschwartz / @mahf708 : Any reason you chose not use tools like ANTLR/Bison?

I had already written a Pratt parser in python and this is essentially a trimmed down version of that ported to C++. The handwritten parser is only a small amount of code, and avoids introducing a parser-generator/code-generation dependency into the build. If the language were expected to grow substantially or we needed more flexibility I think a generator such as ANTLR would become more compelling.

@jayeshkrishna

Copy link
Copy Markdown
Contributor

@peterdschwartz / @mahf708 : Any reason you chose not use tools like ANTLR/Bison?

I had already written a Pratt parser in python and this is essentially a trimmed down version of that ported to C++. The handwritten parser is only a small amount of code, and avoids introducing a parser-generator/code-generation dependency into the build. If the language were expected to grow substantially or we needed more flexibility I think a generator such as ANTLR would become more compelling.

You should be able to check in the generated code.

@bartgol

bartgol commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

@peterdschwartz / @mahf708 : Any reason you chose not use tools like ANTLR/Bison?

I had already written a Pratt parser in python and this is essentially a trimmed down version of that ported to C++. The handwritten parser is only a small amount of code, and avoids introducing a parser-generator/code-generation dependency into the build. If the language were expected to grow substantially or we needed more flexibility I think a generator such as ANTLR would become more compelling.

You should be able to check in the generated code.

I am all for reusing existing tools. But I don't think autogenerated code should be added to the repo. I have not seen antlr/bison output, but I'm guessing it will be not very readable, and lack substantial comments. Sure, we can put a header "do not touch, this was autogenerated", but I still dislike having code nobody understands in the repo.

Btw, Peter, bison is relatively common on linux systems, so it would not be a big dependency to add...

FWIW, gemini seems to strongly believe that hand-rolling our own lexer/parser is a superior choice over on-the-fly generation (via antlr/bison or similar), as well as over stuff like boost::spirit.

@jeff-cohere

Copy link
Copy Markdown
Contributor

My $0.02: ANTLR and Bison are great, but you really have to buy into their way of doing things to use them (and deal with annoyances like generated code). I'd prefer a hand-rolled solution if we can get away with it (i.e. if we can avoid falling down one or more parsing-related rabbit holes).

@rljacob

rljacob commented Aug 24, 2026

Copy link
Copy Markdown
Member

How much technical debt do we take on with a roll-our-own solution vs. bison/ANTR? Will future developers curse us for having to maintain this thing?

Checking in generated code is fine if its only re-generated once in a while. We already do that with the Physics Constants Dictionary.

@mahf708

mahf708 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

Beyond the generated code vs not --- I think adding a dependency is dangerous. We are trying to trim our sprawling dependencies and submods.

What's more, I think we view this solution (a few files of cpp) as a generally static library that will live largely unchanged. We prefer an in-house solution because we will want to attach a custom interface and validator. Consider the following, a future component decides it wants to introduce new vocabulary/grammar to the mix. In a soon-to-be-submitted PR, we will have share/dexpr/include/dexpr/supported_functions.hpp become an inteface with an additional validator. That way, the component can add its new grammar (say .histogram(bins) with order and options, etc.), with a new validating interface that gives the developers feedback that newly added grammar will be accepted and parsed correctly.

Thus, in a specific way, this little library will be much more malleable than relying on external vendors with their specific ways of doing things.

@jeff-cohere

Copy link
Copy Markdown
Contributor

I invite anyone who has actually used ANTLR and/or Bison to weigh in on the tech debt question . I've used them both enough to know that they're not something you just drop into your code. There's a lot to know about parser generators that isn't necessarily stuff you need to know about parsing, and unless we want to develop a wing of parser generator experts in E3SM, I suggest we at least try to get by without them.

If this is clearly the wrong approach, we'll find out and then we can reassess. :-)

@bartgol

bartgol commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Last minute comment, but...I think the name "dexpr" is a bit cryptic. What do people think about a different or at least more spelled out name? Maybe diagnostic_parser? Or something else? I think we can spend more chars here.

@mahf708

mahf708 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Last minute comment, but...I think the name "dexpr" is a bit cryptic. What do people think about a different or at least more spelled out name? Maybe diagnostic_parser? Or something else? I think we can spend more chars here.

I think cryptic is ok here (it's unique and doesn't need to be as descriptive since it is specialized). We will want something to convey "expression" in the name as well. I would strongly vote for "dexpr" (I considered dxpr, but I think dexpr is better)

@rljacob

rljacob commented Aug 25, 2026

Copy link
Copy Markdown
Member

According to the README.md "dexpr produces an abstract syntax tree. It does not evaluate anything, and it knows nothing about fields, grids or timesteps -- turning an AST into an actual diagnostic is the caller's job." Can someone point me to an example/sketch of how this all is supposed to work?

@mahf708

mahf708 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

According to the README.md "dexpr produces an abstract syntax tree. It does not evaluate anything, and it knows nothing about fields, grids or timesteps -- turning an AST into an actual diagnostic is the caller's job." Can someone point me to an example/sketch of how this all is supposed to work?

here: #8623. In very broad strokes:

  1. Users asks EAMxx for something like T_mid.where(omega>100).horiz_avg()
  2. EAMxx puts term (T_mid.where(omega>100).horiz_avg()) into the parser (i.e., dexpr)
  3. dexpr returns a tree that decomposes T_mid.where(omega>100).horiz_avg() into T_mid -> operation where on operation omega > 100 followed by operation horiz_avg
  4. EAMxx takes that tree and attaches the impl/math to the words (i.e., finds field T_mid (exists in eamxx) then does conditional sampling (a diag in eamxx) on the condition omega>100 and then pipes the result through a horiz average (another diag in eamxx))

Also, see these internal notes here: https://e3sm.atlassian.net/wiki/spaces/p3ai/pages/6548127761/2026-08-13+ai+group+infrastructure+e3sm+dsl

(there are two additional pieces for this to become functional (as in applied): 1) attaching the words to ops and 2) replacing the regexes in EAMxx. These will be done in a follow-up PR.)


inline constexpr std::array<std::string_view, 0> tend_args{};

inline constexpr std::array supported{

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the initial list of functions which will grow?

@mahf708 mahf708 Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part will become an 'interface' in the next PR with some standard functions built-in, but the key is that components can/will define their own. We talked about this with the omega team already. In the next PR, you will see two movements to address this, first, the supported function stuff will become an interface, but second (and importantly) we will add a validating utility so that components can test if their definitions actually make sense/pass.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When this is all done, where will the various code pieces live?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it doesn't matter because they're modular pieces. This part is the parser and will here. We may have one other related tool that will live alongside in share, but everything else will belong to the components initially (in the distant future, we can consolidate more, but we don't want to overengineer at first)

Comment thread share/dexpr/README.md
@mahf708

mahf708 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

@rljacob A general remark that may not be readily clear yet: the goal here is we want to have an xarray-like snytax for the eventual "language" --- so users won't need to learn new ways of doing anything, it will very closely mimick (where possible) the classic dataframe in our domain, which is xarray.

@rljacob

rljacob commented Aug 26, 2026

Copy link
Copy Markdown
Member

This is PR 1 of a few PRs that will implement the capability. Please write down the full plan somewhere. Include where the various code pieces will live and a "before" and "after" example. (what does a diagnostic currently look like in EAMxx and what it will look like after this is fully implemented). That will help alot. Since you want lots of people to use this, start a discussion in https://github.com/E3SM-Project/E3SM/discussions/categories/development-discussion

@mahf708

mahf708 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

We had a few meetings; people who expressed interest in this attended (Xylar, Brian, Jeff, Peter, Aaron, Luca, ...)

For people interest, here's the link to meeting notes internally: click. They will continue until Oct 1, when we will submit a manuscript about this work. Everyone is invited to be co-author (opt-in).

@rljacob

rljacob commented Aug 26, 2026

Copy link
Copy Markdown
Member

That page is pretty sparse. The big picture is obviously clear in your heads. You could expand those notes in Confluence if you prefer.

@mahf708

mahf708 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Any more feedback before we merge this?

@jeff-cohere, @xylar, and @bartgol --- any comments? Since this doesn't affect anything in E3SM, let's aim to merge to next today and to the main branch tomorrow.

@xylar

xylar commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

@mahf708, please proceed. No feedback from me right now. I haven't had time to look at it this week and was at a conference last week.

@xylar
xylar removed their request for review August 26, 2026 14:15
@peterdschwartz

Copy link
Copy Markdown
Contributor

With the last two commits, everything related to this PR should have been resolved.

@rljacob rljacob left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hope to get some documentation in a future PR.

@bartgol

bartgol commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

@peterdschwartz can you quickly fix the md linter complaints? Then I will integrate.

Import the diagnostics expression lexer/parser from the upstream
standalone repo, unmodified apart from its location. The code is a
hand-written lexer plus a Pratt parser producing a std::variant-based
AST; it is plain C++20 with no Kokkos, MPI, netCDF or EKAT dependency.

It lands under share/ rather than a single component because nothing in
it is component-specific, and it is deliberately not wired into any
build yet -- this commit only parks the source.

Upstream is MIT licensed; LICENSE is carried over verbatim.

Co-authored-by: peterdschwartz <peterdschwartz83@gmail.com>

build the parser standalone with its own ctest suite

Split the test setup into tests/CMakeLists.txt and give the project real
options, so the library, the tool and the tests can each be turned off
independently.

Catch2 is now found before it is fetched. A machine that already has
Catch2 v3 installed uses it; anything else falls back to FetchContent, so
a bare `cmake -S . -B build` still works from a clean checkout.

Turn on -Wall -Wextra -Wpedantic. Nothing external is in the include path,
so there are no third-party headers to fight and the library can simply
stay warning-clean. Fix the one signed/unsigned comparison that exposed.

upgrade the lexer, and test it properly

Bring the lexer to the point where it handles the input the diagnostics
DSL actually contains, and cover it with tests: the suite goes from 2
cases to 26.

Identifiers may now contain digits after the first character, which still
may not be one. Field names routinely carry them -- T_2m, qv_850, bc_a1,
ne30pg2 -- so a lexer that stops at the first digit is unusable here.

Numeric literals no longer parse to a wrong answer in silence:

  ".5.3"  lexed as one Float("0.5.3"). The '.' branch consumed the
          leading dot and then called read_number(), which started over
          and folded in the second one. Downstream parsing read the
          "0.5" prefix, stopped, and did not complain. Meanwhile "1.2.3"
          was correctly rejected, so the two spellings disagreed.
          read_number() now accounts for a dot the caller already took.

  "1E5"   lexed as Integer("1E5"), because the float/integer split
          tested ".e" and missed the uppercase exponent marker that
          read_number() happily accepts. Integer parsing then stopped at
          the 'E' and yielded 1. Classification now tests ".eE".

The input is no longer case-folded wholesale. Folding the buffer made
keywords case-insensitive but also rewrote string literals, which are
data rather than syntax, so 'MyVar' silently became 'myvar'. Case
insensitivity now applies where it belongs: keyword lookup folds before
matching, and the exponent marker accepts either case.

An unterminated string literal is reported as Illegal instead of being
accepted as a well-formed string.

The unused Newline token is gone and token_precedence gets an explicit
default, which together clear the -Wswitch warning the build has been
emitting since the parser was vendored.

changed alphanumeric identifier

finish the parser, and test it properly

Bring the parser to the same standard as the lexer and cover it: the
suite goes from 26 cases to 57.

The new tests exercise precedence and associativity, prefix operators,
grouping, function calls, member access, array literals, equality
operators, and printing an AST back to a form that lexes again. Several
fixes to precedences and to ast_print fall out of what they exposed, and
exponentiation is now right-associative, so 2**3**2 is 2**(3**2) as it is
everywhere else.

Numeric literals are held at double precision. As float, a threshold did
not survive being parsed:

  273.15 as float  = 273.14999389648438
  273.15 as double = 273.14999999999998

and a legal value like 1e40 was rejected outright as out of range. For a
language whose main job is expressing thresholds over climate fields,
narrowing in the AST is the wrong place to do it; a consumer that wants
single precision can narrow when it knows that.

Literals are read with std::from_chars rather than std::stoi/std::stof.
It does not depend on the locale, where std::stof reads "1.5" as 1
wherever ',' is the decimal separator; it does not throw; and it reports
where it stopped. Requiring it to stop at the end of the literal rejects
a malformed literal outright instead of quietly accepting its leading
prefix, which is the same failure the lexer fixes guard against from the
other side.

A missing ')' now throws where it is detected instead of returning null.
parse() caught that at the end anyway, but in between the null travelled
through the tree builders, and every visitor dereferences its children
unguarded.

Parser::cur_token_is is gone. It was declared and defined but never
called from anywhere, while its sibling peek_token_is has seven call
sites.

settle on the name dexpr, enforce it, and document it

Three things that together fix the library's identity, done at once
because the rename touches every file and splitting it would mean
reviewing the same mechanical diff twice.

The e3sm prefix was redundant inside the E3SM repo, and the edp acronym
expanded to "E3SM Diags Parser", so it went stale the moment that name
did. One word is now used everywhere: directory, CMake target, namespace,
include prefix and CLI binary are all dexpr.

The word expression is what earns the name. Without it, "diags parser"
reads as the thing that parses diagnostics config -- output YAML field
lists, legacy diag name strings, namelist entries -- which is a real and
separate job in EAMxx. dexpr parses expressions.

  include/edp/   -> include/dexpr/     EDP_*_HPP    -> DEXPR_*_HPP
  namespace edp  -> namespace dexpr    EDP_ENABLE_* -> DEXPR_ENABLE_*
  tools/edp.cpp  -> tools/dexpr.cpp

The library now owns the name dexpr, so the tool's CMake target is
dexpr_cli with OUTPUT_NAME dexpr; the binary is unchanged.

Warnings become errors in CI. The code has been warning-clean since the
switch statements were completed, so this makes it a property the build
enforces rather than one that happens to hold. DEXPR_WERROR is off by
default, leaving a developer build unaffected, and on in CI. The flags
are set once at the top level so the tests compile under exactly the
same set as the library. CI also runs the tool now, which covers the one
path the unit tests do not: main().

Finally, a README covering what the library does, how it is laid out, how
to build and test it, and where it came from, plus its entry in
share/README alongside the other subdirectories.

tidy up the token table, error messages, and float printing

Four fixes that share no code but do share a shape: each is a place where
the library was doing something that happened to work rather than
something it could rely on.

The keyword table had one definition per translation unit. `const` at
namespace scope has internal linkage, so the std::unordered_map in
tokens.hpp was constructed separately in every unit that included the
header -- three heap-allocated string keys plus a static initializer each
-- including in ast_print.cpp and precedences.cpp, which never look a
keyword up. Three entries do not need a hash table. A constexpr array of
string_view searched linearly is one definition and no runtime
construction at all. Measured on the built objects, static-init routines
went from one in each of five units to none.

Four token types were declared that the lexer never produces. Percent,
Semicolon, DoubleColon and Concat had to_string arms, so they looked
supported from outside, but nothing ever emitted one:

  %    -> Illegal(%)          ;     -> Illegal(;)
  ::   -> Colon(:) Colon(:)   a//b  -> a Slash Slash b

None is referenced anywhere, including on the diag-integration branches,
so they are gone rather than given syntax to justify them. Colon stays:
it is lexed, tested, and holds the Bounds precedence for the slicing
that ast.hpp still has stubbed out. Dropping the default arm from
to_string means -Wswitch now names the next token type someone adds
instead of letting it print as UNKNOWN. The unused ostream operator for
Token goes too, along with the "Identifer" spelling that reached users
through parser messages, and a `Token tok;` whose type was left
indeterminate.

Parse errors say where they happened. They named the offending token but
not its location, so a typo in a long expression sent the reader back
through the whole thing to find what the parser had already pinpointed:

  before: Unexpected Prefix Token {Type: Plus, Literal: +}
  after:  Unexpected Prefix Token {Type: Plus, Literal: +} at line 1, column 5

Token gains line and column, both defaulted so the two-argument aggregate
form still works. The lexer maintains them in read_char(), and
next_token() captures the position before scanning and stamps the result,
so the scanner's many early returns cannot forget to.

Floats print with std::to_chars instead of std::format. <format> was the
only thing here requiring GCC 13, and it bought nothing: the standard
defines std::format("{}", d) for floating point in terms of
std::to_chars(first, last, d), so the output is identical by
specification rather than by luck -- every existing float printing test
passes unmodified. <charconv> was already a hard dependency, since
literals are parsed with from_chars, so this removes a requirement rather
than adding one, and the floor drops to GCC 11. Parsing and printing are
now inverses over one facility, which a round-trip test asserts directly.

note what the parser does not do yet

Record the extension points that are deliberately absent, so the next
person to want one starts from the reasoning rather than rediscovering
it: component-supplied functions, slicing, and the fact that operator
syntax is fixed even if functions stop being.

The first of these is the one that matters. supported_functions.hpp is a
fixed table that nothing consults, so an unknown call parses and is never
rejected; replacing it with a registry a component fills in is a separate
change with its own tests, and does not belong in the commit that only
parks the parser.

Adjustments due to reviewer comments.

- Added DOxygen style comment docs to major headers and functions
- renamed `cur_precedence` to reflect its role in associativity of
  operators
- Switched unordered_map in Parser with switch table lookup
- Removed un-used code portions

respond to reviewer comments
@peterdschwartz

Copy link
Copy Markdown
Contributor

ok -- just installed markdownlint-cli2 and had it fix it. I also squashed all the commits

@bartgol bartgol added Utils BFB PR leaves answers BFB labels Aug 27, 2026
bartgol added a commit that referenced this pull request Aug 27, 2026
dexpr (said like crispr) is the expression parser for in-situ e3sm diagnostics.
We will expand it to allow for a component interface and other neat nifty items
like trees and such in future PRs

This PR only implements the parsing layer to generate an Abstract Syntax Tree
from user input. This defines the expected grammar for e3sm diagnostics
and serves as a shared stable foundation between components.
@bartgol
bartgol merged commit b3f0093 into master Aug 27, 2026
5 checks passed
@bartgol
bartgol deleted the mahf708/share/edp branch August 27, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

BFB PR leaves answers BFB Utils

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants