JGI: Provide Data in Ingest Directory and Organize ETL Scripts

[vchendrix]

Implement JGI data ingest per standardized requirements:

• Create an ingest/jgi subfolder.
• Place all JGI data files in this directory, formatted as JSON lists (enclosed in brackets) and named using the convention jgi_00001.json, jgi_00002.json, etc.
• Ensure each file is limited to ~25 MB and contains only complete records.
• All files must conform to the latest release schema.
• Document and implement a file splitting strategy if necessary.
• All ETL scripts for JGI should be placed in contrib/jgi.
• Ensure independent file validation is possible.
• Document the JGI ingest process, folder structure, file format, and splitting strategy.

[valerie-autumn-skye]

From this commit:

• Cleaned up GOLD database import, views, and exports.
• Restricted map data exports to records associated with JGI sequencing records.
  • Reduced number of rows exported from 208,443 to 54,366 distinct biosamples and from 62,139 down to 9,503 distinct organisms.
  • See also bertron issue 89.

Original plan was to switch to JGI Data Warehouse, but that will contain stale data or be partially unavailable through November 2025.

In the meantime, here is the plan for supporting the ingest script:

• Extend view to data relevant to new schema version. See commit.
• Develop Python script to construct schema-compliant JSON from new views.
• Extend Python script to export JSON records in bulk files compliant with file naming and size conventions.
• Automate GOLD download, extract, import, header translation for one-click updates.
• Potentially automate entire process via GitHub action.