Added a full SAX-style parser with bounded memory usage.

[TkTech]

Closes #33. This is used by tktech/py_yyjson and is 2.66x to 4.22x faster than existing options like ijson (built on YAJL) while using fixed memory - parsing through a 4GB JSON file with the default window will use a flat ~33mb (in py_yyjson).

Should have no performance impact on the regular DOM-style parsing.

[codecov]

Codecov Report

❌ Patch coverage is 88.00000% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 98.02%. Comparing base (19980a6) to head (71ade14).
⚠️ Report is 9 commits behind head on master.

Files with missing lines	Patch %	Lines
src/yyjson.c	87.96%	42 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #289      +/-   ##
==========================================
- Coverage   98.47%   98.02%   -0.46%     
==========================================
  Files           2        2              
  Lines        7739     8089     +350     
==========================================
+ Hits         7621     7929     +308     
- Misses        118      160      +42     

Flag	Coverage Δ
unittests	98.02% <88.00%> (-0.46%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ibireme]

Thanks for the PR. This is a big change, so I'll need some time to go through it carefully.

I'm still unsure about the streaming API design and performance, and I may try a few other styles first. I'll leave more feedback once I've taken a closer look.

[TkTech]

Thanks @ibireme, definitely needs a close look. I kept it self-contained, re-using as much as possible. Makes the surface area very low but definitely means it's not as optimal as it could be, especially within the parser loop itself. ijson with the yacjl-c backend is the "state of the art" for Python, so that was my performance target and we're about 2-4x faster than ijson while using 18% less memory (a flat 33mb for a 4GB document). But yajl itself isn't terribly fast.

If it helps to have an example implementation of the API on the client side, https://github.com/TkTech/py_yyjson/pull/18/changes#diff-bddb87ab36706d7c7a056c5d52dd01cab842cfcb2b7d7350a40642030da1687fR257

[ibireme]

I'm working on another streaming read/write API. It is still at a very early stage, so both the implementation and the API naming need more work.

The reader currently looks roughly like this:

yyjson_read_err err;
yyjson_sr sr;
yyjson_sr_init_stream(&sr, ...);

yyjson_sr_obj_begin(&sr);

yyjson_sr_obj_find_lit(&sr, "values");
yyjson_sr_arr_begin(&sr);

while (yyjson_sr_arr_next(&sr)) {
    yyjson_type type = yyjson_sr_peek(&sr);

    if (type == YYJSON_TYPE_NUM) {
        int64_t value = yyjson_sr_read_sint(&sr);
    } else if (type == YYJSON_TYPE_STR) {
        yyjson_sv value = yyjson_sr_read_str(&sr);
    } else {
        yyjson_sr_skip(&sr);
    }

    if (yyjson_sr_get_error(&sr, &err)) {...}
}

yyjson_sr_obj_end(&sr);

The main idea is single-pass parsing over a small, fixed-size memory window. So far, it performs about as well as the DOM API in my benchmarks.

I think it should also be possible to build a SAX API as a thin wrapper around this pull API. I'm curious what you think, and whether this could work as a base for the SAX API.

[TkTech]

Is the eventual goal something similar to simdjson's ondemand API? I don't see any issue with the proposed API, so long as yyjson_sr_peek works at all times - e.g we peek, see it's a list, and only then call yyjson_sr_arr_begin.