Process journal papers and add content to MTE

[wkiri]

The first step is to try parsing the journal documents @stevenlujpl already downloaded.
For some documents, we may need to process them multiple times for each mission whose targets are mentioned (see issue #22).

• Generate initial annotations (w/Contains and HasProperty from jSRE) using the MTE pipeline
• Perform expert review of Target, Contains, and HasProperty annotations. Guidelines: https://docs.google.com/document/d/1KnkVtxkKb9kcRVKZqPWwuXSi_6zK8Scay1M6qiz1Khc/edit#
• For each mission, update aliases table (if any updates are needed)
• For each mission, concatenate LPSC + journal .jsonl files and use them for ingest_sqlite.py
• For each mission, run update_sqlite.py twice: once with LPSC annotations and once with journal paper annotations
• Check contents via MTE mission-specific websites
• Generate MTE bundle v3.0 with MER-B and journal content added

[wkiri]

Note: the MTE schema currently has an abstract column in the documents table. Journal papers do not have an abstract number, so we should decide how to handle this.

We will also need to decide how to generate a doc_id which currently is year_abstract. Perhaps it should be year_venue_paperid in which case venue would be a short form like lpsc or jgr and paperid would be the same as abstract for LPSC publications and something like volume-number-page for journal papers. Then maybe this paperid would take the place of the abstract column in documents (we'd want to rename the column as needed).

[stevenlujpl]

• 35 journal papers were processed, and it took about 30 minutes to process them using paper_parser.py
• ADS title search failed for 11 papers
  • ADS doesn't have the papers
  • Grobid failed to extract titles
  • Extracted titles were wrong
• jSRE relation extraction failed for 6 papers (due to the jSRE "out of memory" error)
• 18 papers passed initial filter

[stevenlujpl]

I categorized 18 papers that passed the initial filtering process by missions to ensure that we at least have one paper for each MERA, MERB, MPF, and PHX mission. Some papers may appear in more than one mission list. For example, the paper 2003JE002125.pdf talks about the MER landing sites, so it has been categorized within both MERA and MERB mission lists.

MERA:

• 2003JE002125.pdf
• 2010JE003633.pdf
• 2016JE005079.pdf
• Niles_SpaceSciRev_v174_2013.pdf

MERB:

• 2003JE002125.pdf
• 2006JE002728.pdf
• 2010JE003746.pdf
• 2014JE004686.pdf
• 2016JE005079.pdf
• 1248097.full.pdf

MPF:

• 2016JE005079.pdf

PHX:

• carrier_GRL_v42_2015.pdf
• Cull_GRL_v37_2010.pdf
• Heet_JGR_v114_2009.pdf
• Mellon_JGR_v114_2009.pdf
• Renno_JGR_v114_2009.pdf
• Toner_GeochimCosmochimActa_v236_2014.pdf

[wkiri]

Thanks, Steven! It looks like there are 14 unique papers here. Are the other 4 that passed the filter worth including?

[stevenlujpl]

@wkiri The other 4 papers are MSL papers. Sorry that I forgot to mention them.

MSL:

• jackson-drillholes-16.pdf
• mezzacappa-dist-corr-16.pdf
• rapin-sulfate-16.pdf
• schwenzer-diagenesis-16.pdf

[wkiri]

@stevenlujpl Great, thanks for the clarification!

[stevenlujpl]

@wkiri I tested the changes I made to the MTE codebase, and it seems it is working fine. Please see the commits above for details about the code changes. The changes are currently checked into the issue45-journal branch. Please see the summary of the changes below:

1. doc_id field: The doc_id field is now used to store filenames (without extension) for journal papers. The doc_id field can also be made to store document indices. If you think document indices make more sense than filenames, please let me know and I will update the code to store document indices in the doc_id field.
2. abstract field: The abstract field is ignored for journal papers. With the changes I made, the abstract field will be empty strings in the DB, and it will not be included as a column in the exported CSV file of the PDS4 bundle.
3. doc_url and year fields: The doc_url and year fields are currently handled in the same way as the abstract field.
4. I added a new CLI argument venue to the bundle generation script. The venue argument is used to distinguish LPSC v.s. other documents. If the input documents are from LPSC, the documents.csv of the PDS4 bundle will have 8 columns (i.e., all the 8 fields from the documents table of the DB are exported to the CSV file). If the input documents are from a venue other than LPSC, then the documtnes.csv will only have 5 columns (the abstract, doc_url, and year fields are skipped).

I tested this approach using the 18 journal papers that passed the initial filtering process. the MPF jsonl, DB, and PDS4 bundle files can be found at the following locations in my /home dir. The bundle validate tool reported 0 errors.

/home/youlu/MTE/working_dir/process_journals/mpf/mpf_init_filter.jsonl
/home/youlu/MTE/working_dir/process_journals/mpf/mpf_init_filter.db
/home/youlu/MTE/working_dir/process_journals/mpf/mars_target_encyclopedia

This approach requires only a few minor changes (as shown in the commits above) in the codebase because it doesn't require changes in DB schema. The drawback is that the abstract field of the documents table in the current DB schema doesn't really apply to journal papers. I think this may be okay because I consider DB files as the intermediate products. The PDS4 bundles as the final delivered products don't have the abstract field in the CSV file. Please let me know what you think.

The current MTE website code won't work with the DB files generated from journal papers primarily because of the lack of the doc_url field in the documents table. I will update the website code if we are planning to use it for journal DBs. Please let me know. Thanks.

[wkiri]

@stevenlujpl This is great progress! Thank you!

Can you place the generated .jsonl files under /proj/mte/? That way I can generate brat review pages for them.

The changes seem fine in general. I have two questions:

1. I think that it would be good to retain the year field. Is this possible?
2. Can we populate doc_url with the journal paper DOI (which can be formatted as a URL if it isn't already)? I think this is included in the ADS results. While not available for LPSC, it should be available for journal papers.

It seems I overlooked that "abstract" is not included in the final .csv files that are delivered. I should correct this in the schema diagram and in the README.

I agree that the sqlite DB is an intermediate product so it's ok for it to have more information even if not used later. As you note, however, the website does use the DB directly. It makes sense to prioritize getting the journal paper content into PDS4 bundles first, and if time remains, then update the website (but it's not on the critical path for the time remaining).

[stevenlujpl]

1. I think that it would be good to retain the year field. Is this possible?

It should be possible to retain the year field for documents indexed in the ADS database.

2. Can we populate doc_url with the journal paper DOI (which can be formatted as a URL if it isn't already)? I think this is included in the ADS results. While not available for LPSC, it should be available for journal papers.

It seems from the ADS website search results, the DOI fields are already formatted as URLs. I will double-check the format of the DOIs returned by directly querying the ADS database. These are great suggestions. I will work on them now.

I copied the .jsonl files to the following locations in /proj/mte/. Please let me know if you run into any problems generating the brat review pages.

/proj/mte/data/steven_working_dir/process_journals/mera/mera_init_filter.jsonl
/proj/mte/data/steven_working_dir/process_journals/merb/merb_init_filter.jsonl
/proj/mte/data/steven_working_dir/process_journals/mpf/mpf_init_filter.jsonl
/proj/mte/data/steven_working_dir/process_journals/phx/phx_init_filter.jsonl

[stevenlujpl]

@wkiri I couldn't test the update_sqlite.py step to update the DB with human-reviewed brat annotations yet, but I don't foresee any problems because the DB schema isn't changed.