[taras]

Motivation

Configliere's phased parsing model depends on a phase-1 parser being able to extract its known flags from anywhere in argv, passing unrecognized tokens through as remainder. The object() parser's tokenization loop stops when it encounters a token that no field matcher claims, breaking this contract: a bootstrap parser defining --config but not <suite> fails to extract --config from run ./suite --config app.json. Consumers are forced to duplicate positional definitions between phases or manually re-parse remainder — workarounds that the architecture is designed to prevent.

This causes two user-visible problems:

• Issue Support parsing options after unknown positional arguments (permissive/interspersed mode) #11: Known options after an unknown positional are left in remainder unparsed.
• Issue Consistent help behavior regardless of --help position #12: --help only triggers reliably when it appears before any unrecognized token. The program() architecture already treats help as a boolean flag alongside a populated config: T — the tokenizer just doesn't reach --help when an unrecognized token precedes it.

Approach

Decision A (this PR)

object() parses known options interspersed with unrecognized tokens. Unrecognized tokens go to remainder in original order. -- terminates option processing. --help/-h detection becomes position-insensitive as a consequence of the general fix, with targeted changes to command() and program() to handle the token at the correct scope.

Core change: scopeInput in lib/object.ts

When no matcher claims the current token, the loop now moves that token to a skipped list and continues rather than halting. Skipped tokens become remainder in original order. The loop terminates when args is empty. Existing matcher priority and positional field matching rules are preserved — only the scope over which they operate is extended.

Help routing: command() in lib/commands.ts

command() runs the inner parser first with all args, then checks if --help/-h survived in the inner parser's result remainder. If a descendant commands() parser dispatched a subcommand and its command() handled --help, the token is consumed before the parent ever looks — preserving nested help routing (outer inner --help → help for inner).

Program-level fallback: program() in lib/program.ts

program() retains its existing args[0] early check for --help/--version. After delegating to the config parser, it also scans the config parser's remainder for help/version tokens that no sub-parser handled. Handled tokens are stripped from result.remainder.args.

Decision B (provisional)

Whether help intent should suppress required-field validation is documented and tested in this PR but not resolved. The current behavior (validation proceeds normally; consumer checks help: true) is preserved. command() returns ok: true with help: true regardless of whether the inner parse succeeded, so help: true is reachable even when required fields are missing. A provisional spec note marks this as subject to revision.

Spec amendments

• Interspersed parsing as the normative object() tokenization rule (§8.2)
• Remainder contract tightened: "did not match" vs "did not reach" (§3)
• --help/-h and --version/-v position-insensitivity stated as normative (§8.4)
• Subcommand help routing and help-validation interaction documented (§8.4)
• Phased parsing extraction guarantee made explicit (§12.1.1)
• Normative test agenda addition (§18.11)

Tests (150 steps across 10 suites)

• Interspersed parsing (23 tests): flags before, after, and interleaved with unknown positionals; -- sentinel; full-consume and no-consume cases; phased parsing; regression (equals syntax, negative switches, array fields, remainder order); spec-locking
• Help position-insensitivity (25 tests): --help in every position; -h shorthand; --help after --; command-scoped help; nested command help routing (outer inner --help on both commands() and program(commands(...))); path comparison; Decision B observation; version position-insensitivity; spec-locking

Risks

This is a correctness fix that changes externally visible argv behavior. Remainder will contain fewer tokens (recognized flags are now extracted instead of left in). No previously parsed token stops being parsed. Existing consumers should be checked for assumptions about early termination.

Fixes #11, partially addresses #12

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/configliere@15

commit: 069f8bc

[cowboyd]

Configliere's phased parsing model depends on a phase-1 parser being able to extract its known flags from anywhere in argv, passing unrecognized tokens through as remainder

This is a circular justification. Given that it is the first statement of the PR, I'm going to truncate reading the rest. I am still not convinced that the two issues are indeed issues.

I would want to see a theory of the parsing that was both hierarchical and compositional; that means that there are clearly defined rules about consumption precedence as pertains to depth of the combined parser and conflict resolution in the event that an arbitrarily nested parsers contain similar options. eg. --help vs --subcommand.help So that conflicts are rare and there always exists a way to disambiguate.

[taras]

Baseline architecture

The following are facts about the current Configliere codebase.

Parsers are compositional

Configliere is a parser-combinator system. Small parsers (field(),
constant()) compose into larger ones (object(), commands()) via
structural nesting. The composition is the parser tree.

Two distinct parsing concerns exist

There are two separate jobs in parsing, handled by two separate parsers:

Tokenization is object()'s job. It receives the args array,
iterates through it trying each child field's matchers, converts matched
tokens into scoped value entries, and returns whatever it did not match
as remainder.

Resolution is field()'s job. For a single field, it collects
candidate values from all sources (none, default, values, envs),
validates each against a Standard Schema, and selects a winner by
priority. It does not touch the token stream.

The boundary: object() transforms args into scoped value entries per
child, then delegates to field(). field() merges those values with
env and object-value sources, validates, and picks a winner.

The matcher system

object() uses a chain of matchers from parse-args.ts to claim tokens.
Matchers are tried in priority order: parseSwitch, parseNegativeSwitch,
parseLongOption, parseLongOptionEql, parseArgument. When a matcher
claims a token, it returns the value and the shortened args. Non-array
fields are skipped after their first match. Array fields accumulate.

Commands establish dispatch scope

commands() matches the first positional token (by name, alias, config
values, or default), strips it from args, scopes envs by prefix and
values by key, then passes scoped input to the matched command's inner
parser. The command's inner parser never sees the selector token.

Every parser returns remainder

ParseResult always carries remainder: Input. Parsers that do not
touch args return remainder equal to their input. Parsers that consume
tokens return remainder with those tokens removed.

Inspection derives from structure

inspect() walks the parser tree and returns typed ParserInfo that
includes help metadata and the parse result. Help text is rendered from
the info, not from a separate system.

Parsing is pure

parse() does not perform IO, mutate shared state, or depend on
ambient context. All effects (file reads, plugin discovery) happen in
application code between parse phases.

Phased parsing exists via sequence() and step

sequence() composes parsers into phases. The application drives IO
between phases. Remainder threads from phase-1 to phase-2.

Program wraps help as a boolean flag

program() treats help and version as boolean flags on the parse
result, alongside a populated config: T. The consumer checks
result.help and decides whether to act on the config or render help.
This design was an explicit choice — Charles moved away from a
discriminated-union model that short-circuited on --help.

---

3. Clarified parsing theory

The following rules are not new features. They are the explicit
articulation of what the baseline architecture already implies. Stating
them makes it possible to adjudicate whether a behavior is correct or
incorrect without arguing from consequences ("phased parsing needs
this") — instead, arguing from the parsing model itself.

Token claiming is scope-local

A parser can only claim tokens within the scope it was given. object()
receives args from its parent context. It attempts to claim tokens from
that array using its child matchers. It has no visibility into tokens
that a parent already consumed, and no authority over tokens that belong
to a sibling or ancestor.

This is already true in the code. commands() strips the command-name
token before scoping args to the matched command. The command's inner
object() receives scoped args and can only claim from those. Nothing
in the architecture gives a child parser access to the parent's consumed
tokens.

Parent parsers establish scope; child parsers claim within scope

The relationship between parent and child is:

1. The parent receives the full input.
2. The parent narrows the input to establish scope for each child — by
   dispatching (commands), scoping values by key (object), scoping envs
   by prefix (commands), or stripping consumed tokens.
3. The child receives the narrowed scope and claims tokens within it.
4. Whatever the child does not claim remains in the child's remainder.
5. The parent collects remainders from children and produces its own
   remainder.

This is the compositional structure. It is already true — commands()
scopes envs and values before passing to command(), and object()
scopes values per field before passing to field().

Remainder means "not claimed by this parser"

Remainder is defined by what the parser did not claim. It is not defined
by where the parser stopped iterating. This distinction matters:

• "Not claimed" means the parser tried every token in its scope and
  these tokens did not match any of its defined fields.
• "Where the parser stopped" is an implementation detail of the
  iteration strategy, not a semantic property of remainder.

The architecture already implies this. Remainder is returned as
Input — a first-class value that threads between parsers and between
phases. Its content is supposed to represent "what this parser left for
the next parser." If remainder contains tokens that the parser could
have claimed but did not reach, that is an implementation deficiency,
not a correct semantic outcome.

Unclaimed tokens belong in remainder

If a parser was given a token and has no matcher that claims it, that
token belongs in remainder. There is no third category. Every token in
the input ends up in exactly one place: consumed as a claimed value, or
present in remainder.

This is the conservation invariant. It is already implied by the type
system — ParseResult always has remainder: Input, and the only way
a token leaves the input is by being matched and converted to a scoped
value. PR #15 does not introduce this invariant; it fixes a violation
of it.

Hierarchical parsing correctness

Correct parsing means:

1. Every parent correctly establishes scope for its children.
2. Every child correctly claims all tokens in its scope that match its
   defined fields.
3. Every unclaimed token appears in remainder in original relative order.
4. No token is dropped or duplicated.
5. The composition of these behaviors across the parser tree produces a
   correct global result.

This is the theory that lets you judge a parsing behavior as correct or
incorrect without reference to phased parsing. If a parser fails to
claim a token it should have claimed, the parse result is wrong
regardless of whether phased parsing is involved.

---

4. Token claim model

What it means to claim a token

A token is claimed when a matcher in object() matches it and converts
it into a scoped value entry ({ name: "cli", value }) for one of
object()'s child fields. After claiming, the token is removed from the
working args array and does not appear in remainder.

Claiming happens at the object() level, not at the field() level.
Fields do not touch the token stream. They receive pre-scoped values
from object() and resolve them.

What can be claimed

A token can be claimed if:

• It matches a child field's switch pattern (--flag / -f for booleans)
• It matches a negative switch pattern (--no-flag)
• It matches a long option pattern (--name value / --name=value)
• It matches a positional argument pattern (non-dash token, field marked
  as argument, not already matched)

The matchers are tried in the priority order listed in section 2. The
first matcher that succeeds claims the token.

What cannot be claimed

A token cannot be claimed if no child field's matcher matches it. In
that case, the token is unclaimed. The question is what happens to
unclaimed tokens — this is where the baseline implementation and the
clarified theory diverge (section 7).

Parent claim vs. child claim

commands() claims the command-name token. This is a parent-level claim.
The token is consumed and never appears in any child's scope or in
remainder.

object() claims option and positional tokens for its children. These
are delegated claims — object() does the matching, but the resulting
value is scoped to the specific child field.

field() does not claim tokens. It resolves values from whatever
sources object() scoped for it.

---

5. Remainder model

Definition

Remainder is the set of tokens from the parser's input that the parser
did not claim, preserved in their original relative order.

Properties

1. Conservation: input tokens = claimed tokens + remainder tokens.
   No tokens are created or destroyed.

2. Order preservation: unclaimed tokens appear in remainder in the
   same relative order as in the original input.

3. Always present: remainder appears on both branches of
   ParseResult (ok and error). A parser that claims nothing returns
   remainder equal to its input.

4. Scope-local: a parser's remainder refers only to its own scope.
   Tokens that a parent consumed before scoping are not in the child's
   input and therefore not in the child's remainder.

What remainder is not

Remainder is not "everything after the point where parsing stopped." It
is not "everything the parser chose not to look at." It is not a
debugging artifact. It is the mechanism by which tokens flow between
parsers in a compositional system.

---

6. Consumption and precedence model

Parent scope establishment

When a container parser receives input, it establishes scope for its
children before any child claims anything:

• commands(): dispatches on the first positional token (consuming it),
  scopes envs by prefix, scopes values by key. The child sees args minus
  the selector token, filtered envs, and filtered values.
• object(): passes args to its tokenization loop, scopes values by key
  for each child field. The child sees scoped values and the full env
  list.
• program(): intercepts --help / --version via preamble, scopes
  remaining args to the main config parser.

Child claim priority

Within object()'s tokenization loop, claims are resolved per-token:

1. Matchers are tried in priority order (switch > negative switch >
   long option > equals option > positional argument).
2. The first matcher that succeeds claims the token.
3. Once claimed, the token is removed and the loop advances.
4. A non-array field is skipped after its first match.

There is no conflict between children for the same token — the iteration
tries all children's matchers for each token, and the first child whose
matcher succeeds claims it.

Source precedence within a field

After tokenization, field() resolves from multiple sources. The
intended precedence (lowest to highest) is: none, default, config-file
values, env, cli. The mechanism is findLast(source => !source.issues)
on an ordered source array.

Ambiguity

Ambiguity in token claiming is limited to positional fields: if
object() defines a positional and there are multiple non-dash tokens,
the positional claims the first one encountered during iteration. The
rest go to remainder. This is the existing first-match rule.

For named options, ambiguity does not arise because option names are
unique within an object() and matchers check exact string matches.

---

7. Additive behavior introduced by PR #15

Everything above is baseline or clarified theory. The following are the
actual behavioral changes this PR introduces.

The tokenization loop no longer halts on unclaimed tokens

Before: when the scopeInput loop in object() encounters a token
that no child matcher claims, the loop's progress guard fires and the
loop terminates. All tokens from the unclaimed token onward stay in
remainder — including tokens that matchers would have claimed if the
loop had continued.

After: when no matcher claims the current head token, the loop moves
it to a skipped list and continues with the next token. The loop
terminates only when the args array is empty. Skipped tokens become
remainder in original order.

This is the core change. Everything else follows from it.

Known options can be consumed after unknown positionals

As a direct consequence of the loop continuing, --config foo
appearing after ./suite is now correctly claimed by the --config
matcher. Previously, ./suite would stall the loop and --config
would be left unclaimed in remainder.

-- terminates option processing

If the tokenization loop encounters --, it stops processing and moves
-- plus all subsequent tokens to the skipped list. If this sentinel
was already handled, this PR preserves that behavior. If it was not
handled, this PR adds it as part of the tokenizer change.

Help detection becomes position-insensitive at the tokenizer level

--help and -h are boolean flags parsed by the same matcher system as
any other flag. Because the tokenizer now processes the full argv, it
will claim --help wherever it appears — before or after unknown
positionals. This is not a help-specific change. It is a consequence of
the general tokenizer fix.

Targeted spec amendment included

The PR includes a narrow spec amendment codifying: interspersed parsing
as the tokenization rule, the tightened remainder contract, position-
insensitive help detection, the phased parsing extraction guarantee, and
a provisional note about help-validation interaction (Decision B).

Decision B documented as provisional

The PR documents, tests for, and adds a provisional spec note about
whether help intent should suppress required-field validation. It does
not resolve this question.

[cowboyd]

The circularity was swept under the rug, but it is still there:

Correct parsing means:

1. Every parent correctly establishes scope for its children.
2. Every child correctly claims all tokens in its scope that match its
defined fields.
3. Every unclaimed token appears in remainder in original relative order.
4. No token is dropped or duplicated.
5. The composition of these behaviors across the parser tree produces a
correct global result.

The tell is in (2) where "correct" parsing requires "correct" scoping and "correct" claiming of matching tokens. This implies that there is a universal model of "correct scoping" and "correct matching" for any parser P so that the correctness of P can be determined. But no such models exist. But how a parser matches is up to the parser, and is externally opaque. How a parser narrows scope, and how it matches tokens are at the sole discretion of the parser. In other words, the circularity is "A correct parser must match and claim tokens in the way a correct parser would."

The theory of input and remainder is not without merit, but to make it coherent, we would need some way to externally declare (and therefore validate) not only parsed values, but also matched tokens.

This has me thinking how we can get around this within our current situation. Part of the problem is the gymnastics that object() requires to match both arguments and fields and that it does not really align with the concept of the "object" as we know it (a set of named properties). It has made me think that perhaps we can side-step the issue for now and resolve a lot of internal consistency by making the object() parser more "normal" by decomposing it into separates parsers: the object() parser in its current form, and an argument() parser, and an and() parser

argument() consume a single token. CLI sources only
object() consume options. All sources
and() consume the tokens of both constituent parsers. Sources of constituent parsers

The key here is that argument() marks a parser as only coming from the CLI which resolves the fundamental tension that ENV and CLI sources can only be represented as nested key:value pairs.

app argument --foo bar

and([
  argument(field(z.number()))),
  object({ foo: field(z.string())),
])

But this is a digression. A fundamental problem we have is no way to delimit scope inside arguments:

1. values type naturally delimit scope: { serve: { port: 8000 }} --> { serve: { port: 8000 }}
2. env type delimited with prefix: SERVE_PORT=8000 --> { serve: { port: 8000 }}
3. args: ['serve', '--port', '8000'] -> ???

(1) the scope of each value is clear and independent
(2) the scope is parser dependent but clear-ish because the key:value pairs allow prefixing for encoding structure.
(3) the scope is parser dependent, totally opaque and also a mess. you have a flat array of strings that could represent a very large number of structures

What I really want to develop is a theory for (1) (2) and (3) that given the same parser all unambiguously generate the same tree.

[cowboyd]

fixed by #20