Add Streaming Bulk Import/Export APIs for Large Document Sets

[cferrys]

Summary

The PHP client currently buffers full import payloads and export responses in memory, which can make large hlquery collections impractical to ingest, migrate, or back up from PHP applications.

Context

lib/Documents.php exposes import() and export(), but both route through lib/HttpClient.php, which JSON-encodes the entire payload before sending and decodes the entire response body after receiving it. For a high-performance RocksDB-backed search engine, the client should support high-volume document flows without requiring the caller to hold the complete dataset in memory.

This is especially risky in Documents::import(): if /documents/import returns 404, the fallback inserts documents one-by-one via add(), which can turn a bulk operation into thousands or millions of HTTP requests. Response also only represents fully buffered bodies, so there is no clean way to stream exports directly to a file or consumer.

Proposed Implementation

Add explicit streaming/chunked document transfer APIs:

• Introduce Documents::importStream($collectionName, iterable $documents, array $params = [], array $options = []).
• Support configurable batch size, max bytes per request, and optional progress callback.
• Validate each document lazily as it is read, preserving recent document-field validation behavior.
• Avoid one-document-at-a-time fallback for large imports; surface a clear unsupported-endpoint response or require an explicit allow_slow_fallback option.
• Add Documents::exportStream($collectionName, array $params, $sink) where $sink can be a file path, resource, or callback.
• Extend HttpClient with streaming request/response support using curl read/write callbacks.
• Preserve existing import() and export() behavior for backward compatibility, but document them as buffered convenience methods.
• Add offline tests for chunk sizing, validation failures mid-stream, fallback behavior, and response streaming.

Impact

This would make the PHP client viable for production-scale indexing, reindexing, migrations, and backups. It reduces memory pressure, avoids catastrophic request amplification, and better matches hlquery’s role as a high-performance search engine rather than a small-document REST wrapper.