Use arrow format

[HelenCEBM]

Feather format is smaller than CSV, i.e. more efficient on space/processing, and stores dtypes, helping to avoid some problems when loading the data for further processing.

We initially moved to .csv.gz, which was an improvement on uncompressed CSVs. However, it uses a significant amount of CPU. We believe that moving to Arrow/Feather would use much less CPU and be an overall improvement.

To do:

• Update project.yaml and code sample
• Update gitignore to ignore .feather/.arrow files
• Update docs, including Getting Started Guide and ehrql tutorials (Remove csv.gz from ehrql tutorials and how-tos documentation#1610)
• Ensure that arrow files can be viewed using Codespaces
• Provide researchers with instructions about how to view arrow files during local development, for researchers using VS Code, R Studio and the Stata IDE (Add fallback support for viewing arrow files locally. opensafely-core/opensafely-cli#267) (Added by Lucy)

[StevenMaude]

This issue was opened almost two years ago.

What's changed since then:

• We moved to .csv.gz by default in the documentation examples and this template. It's maybe not as small as .feather/.arrow, but smaller than CSV, and doesn't require much special handling.
• ehrQL supports .arrow files.
• The OpenSAFELY Stata image includes a library that can load .arrow files.

What hasn't been done:

• updating .gitignore
• explaining somewhere in the ehrQL output formats why you might want to use .arrow (as mentioned above)

[bloodearnest]

Updating this old issue, as discussed in a recent team meeting, I think we should make arrow the default, and not use or mention csv.gz in this template.

We may also want to test and include an extension like this in the vscode setup for viewing arrow files also.

https://marketplace.visualstudio.com/items?itemName=w568w.datasets-viewer

[lucyb]

I'm going to remove it from the team's board, as it's something that's on Eli's list to investigate and I don't think we should do this work until they've investigated.