Report an error instead of asserting when the token stream ends early (#386)

[1820893135-pixel]

next() advanced the token pointer and, on reaching the end of the token
stream, hit assert(0) before the switch that was written to handle
exactly that case (src/compiler/parser.c:357):

if (P->token == P->te) {
    --P->token;
    assert(0);
    switch (P->token->id) {
    case LEX_TOK_EOS: case LEX_TOK_EOB: case LEX_TOK_EOF:
        P->token->id = LEX_TOK_EOF;
        return P->token;
    }
    error_tok(P, P->token, "unexpected end of input");
}

An embedded NUL in the schema truncates the token stream at the lexer level, so
a parser that has not yet consumed everything it expects reaches this branch and
aborts the whole compiler instead of reporting a syntax error.

Reproducer

57-byte schema (the \0 is a real NUL byte, 0x00 - not the two characters
x00):

table DoubleVec {
  a: [double];
}

root_type<0x00>DoubleVec;

$ xxd schema.fbs | tail -2
00000020: 0a7d 0a0a 726f 6f74 5f74 7970 6500 446f  .}..root_type.Do
00000030: 7562 6c65 5665 633b 0a                   ubleVec;.

$ bin/flatcc_d schema.fbs
flatcc_d: src/compiler/parser.c:357: next: Assertion `0' failed.
Aborted (core dumped)          # rc=134

In a release build (NDEBUG) the assert is compiled out and the compiler
instead walks into the switch below with the reduced token pointer, so the
debug build is what makes the bug visible.

Fix

Remove the assert(0) so the handling directly below it runs. The same schema
now fails with a normal error and exit status 255:

schema.fbs:5:10: error: '': missing ';' expected by root_type at: schema.fbs:5:10: ''
output failed

Testing

• The reproducer above: SIGABRT before, clean error after.
• Other NUL positions (root_type<0x00>T;, table T {<0x00>}, root_type T<0x00>;)
  all report errors rather than aborting.
• Schemas without an embedded NUL are unaffected - the 9 .fbs files under
  test/ still compile.

Fixes #386.

[mikkelfj]

OK, I assume the assert was a stray debug assert during development.

Can you please remove the comment from the PR, it refers to past history that would make no sense to a new reader, and the code is not special enough to warrant a comment here at all. Note that when injecting a null terminator, this is captured by the token case above the error, allowing the parser to produce a context specific error rather than unexpected end of file.

[1820893135-pixel]

Done — removed the comment, and the diff is now a one-line deletion:

@@ -354,7 +354,6 @@ again:
     if (P->token == P->te) {
         /* We keep returning end of token to help binary operators etc., if any. */
         --P->token;
-        assert(0);
         switch (P->token->id) {

Thanks for pointing out that the null terminator is caught by the token case
above the error — I've reworded the commit message to say that the switch
already produces a context-specific error for an exhausted token stream and
that the assert was only turning those cases into an abort, rather than
framing the NUL as the root cause.

Re-checked after the edit: root_type\0DoubleVec; still fails with
error: '': missing ';' expected by root_type and exit status 255, no abort.