tracing file Formats

[youngsofun]

Summary

refacors

• new input pipeline
  • new framework & replace copy feat(query): unify pipeline for all inputs with format. #7613
  • replace all
    • streaming load
      • refactor(query): streaming load use planner v2. #7756
      • refactor(query): streaming load use new InputFormat. #7769
    • clickhouse refactor(clickhouse_handler): use new InputFormat. #7843
    • rm unused code chore(format): remove unused code about old input formats #7854
• refactor2
  • feat(file format): unify format settings/options. #8566
  • refactor: refactor output format with FieldEncoders #8700
  • refactor input refactor(input format): refactor with FieldEncoder. #8778
  • refactor JsonValue encoder/decoder
  • remove FormatSettings.

speed up

• Parallel read

  • TSV/ndjson (read beyond split boundary) feat(query): parallel read of ndjson in copy. #8199
  • parquet (make big file loadable) feat(parquet): read in parallel. #7903

• refactor NestedBufferReader refactor: use cursor instead of BufferReader for input format #8486

  • refactor: replace NestedCheckpointReader with Cursor. #8716
  • refactor(handler): use FieldEncoder to encode data. #8733

distributed copy

• allow executing COPY INTO in a cluster #6395 @zhang2014 @RinChanNOWWW

streaming copy

• Streaming COPY #7889 @Xuanwo

compact

• try avoid compactor for copy into/streaming load with row-based format #7760
  • feat(intpu_format): check memory_size() for building data block. #7927
• feat(intpu_format): check memory_size() for building data block. #7927
• feat(format): track memory size of VariantDeserializer. #7948
• feat(compact): optimize compact for data load. #8644
• bug: unload -> max_file_size not take effect in parquet type #8488

features

• support None/Default for FileFormatOptions and format_xxx settings #8541
• result
  • feat(input format): scan progress of copy into and streaming load. #8375
• format settings/options
  • csv quote
  • skip
  • null default/ bool
  • limit
  • error tolerate

copy

• copy return status as SQL results
• per file progress/affect
• feat: Support ON_ERROR = CONTINUE | ABORT_STATEMENT in the CopyOptions when do COPY INTO #8642

streaming load

• Feature: support format (type=ndjson ...) #8604
• Feature: streaming load use http query parameters instead of header to pass settings #8243
• [ ]
  https://github.com/datafuselabs/databend/issues/assigned/youngsofun

error handling

• limit size of csv field/row when loading #6936

create format

• feat: support CREATE FILE FORMAT #8100

parquet

• can not load variant columns from parquet file. #8661

TSV

• feat(TSV): allow nested field in TSV. #8606
• bug: copy TSV file errors #8579

CSV

• feat(csv): support setting escape. #8527
• fix(test): rm the tailling comma of ontime.csv. #8532
• feat(csv): not trim spaces by default. #8459
• feat(csv): allow no new line at the file end #8698
• Feature: used format option empty_as_default #8088

other format

• orc/arrow/
  • Feature: Allow COPY FROM Feather file #7655
  • Feature: Allow COPY FROM ORC file #8016
• avro Feature: Allow COPY FROM AVRO file #8017
• execl Feature: Allow COPY FROM excel #7654

test

• more test for alinger #8101
• add unit tests for new impls
  • can refer to the deleted cases for old impls chore(format): remove unused code about old input formats #7854)