Alpha testing

[brindakv]

Feedback from alpha testing (July 13th code freeze)

• File upload dynamic chunk sizes (@j-s-135)
  • https://github.com/wwPDB/onedep_lib/tree/feat/chunk
• Test checksum after upload (@wmorellato)
  • https://github.com/wwPDB/py-wwpdb_apps_deposit/pull/678 (DepUI)
  • Sending md5 checksum with uploaded files #6
• Authentication tokens: what happens if access token is refreshed during an upload session (@wmorellato)
  • It fails... Refreshing token between chunks #7
• Modularize schema (@j-s-135)
  • https://github.com/wwPDB/onedep_lib/tree/modularize-schema
• Add human readable errors/warnings to schema (@j-s-135)
  • Suppress manual file check error/warning messages
  • https://github.com/wwPDB/onedep_lib/tree/schema
• Add entry point functions for retrieving deposition metadata (@j-s-135)
• Instead of copying both access and refresh keys, use one key for both (@wmorellato) (Required refresh token only for first use #13)
  • Remove access token from UI
• Expose check_api_keys() directly somewhere so that the user's keys can be checked in advance without creating any depositions (@wmorellato) (Required refresh token only for first use #13)
• Add functionality in onedep_lib to construct the URL for the deposition that was just uploaded using the DSP (@wmorellato)
  • Storing the deposition view url in the session store #17
  • The URL is constructed on the server and returned in the deposition creation response
• Make onedep_lib available on pypi (@epeisach)
  • https://pypi.org/project/onedep-lib/
• Retain resume sessions? (Testing @KingAlejandro)
  • List sessions by session id, get metadata associated with each session
  • Resume deposition using session id?
  • Add simple entry points?
  • Fix examples/resume_deposition.py
  • Scope for July 13: Retain or suppress
  • June 22 update: Retain the code in onedep_lib but will not be supported in adapters
• Fix tests (lower priority)

[KingAlejandro]

I think the current resume session functionality makes sense, as long as we add some convenient entry point functions for retrieving sessions and their metadata, while the adapter controls where the session files are stored

I propose #8, as an example of what this could look like, we are going to discuss it in today's meeting

[KingAlejandro]

On Resume Sessions, Oleg (@olegsobolev) and I just had a meeting, here's where we landed. We can keep the discussion going on Monday, but these are our thoughts so far, correct me if I am wrong Oleg.

We suggest we shelf the sessions and deposit_resume() functionality till after the first release, it can be potentially useful but would require a lot of work in the adapter and in the UI layer in CCP4/Doppio/Phenix for a functionality the benefits of which we do not currently fully understand.

I suggest we keep the functionality in the code and keep working on it, but we shouldn't expose it to the user for now.

[KingAlejandro]

From what I can see uploading new files with deposit_resume(), adds files to OneDep and presumably initiates some merge mechanism, it would be good to understand how this mechanism functions so that we can explain it to our users, before we fully implement it in the software suites.

[CV-GPhL]

There is still the bug on the OneDep backend: it doesn't fully understand the difference between "test set flags", "cross-validation" and "Freer set". We've had a lot of discussions about that over the last weeks/months (mainly via PDBe): this needs urgent fixing (maybe it is only the test server that is behind?) otherwise multi-datablock reflection data deposition is blocked.

In short:

• test set flags (often 0/1 or 0-19 integer values) are optional information that can be used for cross-validation purposes
• computing a Rfree value is one of those cross-validation methods
• Rfree is computed during refinement to validate model parametrisation and refinement procedure
• at the refinement stage, reflections are given a status flag (work or test set) based on a particular test set flag value
• the first datablock within a reflection data file is by convention related to the refinement of the model - the other blocks are not
• what should be checked is the presence of that status flag within the first datablock ... nothing else
• additional datablocks can contain purely map coefficients (e.g. radiation-damage maps, ligand screening event maps etc) or data from various upstream data processing stages (prior to cut-off or correction), right up to scaled+unmerged data: these have nothing to do with refinement at that stage and therefore no refinement-specific Rfree status flag is required within these datablocks.

The message

is incorrect and causes our users a lot of headache.

[CV-GPhL]

Another weird set of warnings from OneDep - this time e.g. about "unwanted CIF item". What makes a CIF item "unwanted"? It is either valid or invalid as far as I can see ... Anyway:

Is a user required to remove those unwanted CIF items beforehand? Or will the OneDep system drop them (because they are unwanted)?

The other warning about missing mandatory items is an incorrect message: amplitude/intensity data is only mandatory for the first datablock, not subsequent datablocks that might contain map coefficients only.

I have no idea what "I_avg/sigI_avg = 1.87" is supposed to mean - is it the <I/sig(I)> for a particular (merged?) datablock? Or over all datablocks? Or is it really the average of something (as opposed to inverse-variance weighted mean) of something? Or maybe not a re-computation based on deposited data, but a specific data metrics (which category.item?)?

(I'll keep mentioning it: reflection data != structure factors).

[CV-GPhL]

Item 1:

After I get a message back from our adapter with the Deposition_ID, I do the following:

• go to https://onedep-depui-test.wwpdb.org/
• login via Occid ID
• see the list of deposition IDs
• select the relevant one

I see a red exclamation mark next to the "File upload" menu item. When I then click on it and continue to https://onedep-depui-test.wwpdb.org/deposition/pageview/uploads, that exclamation mark is then gone it seems. Am I just too fast and between my initial view and the second one something happened at the backend that updated this? Or is it my interaction with the frontend that causes something?

Item 2:

How can I have two tabs/windows in my browser pointing to two different deposition IDs? The URL doesn't contain the deposition ID and if I initially open two separate tabs things look fine (they report the two different deposition IDs), but as soon as I do a reload on either I end up with two tabs showing me the same ID - namely the latest I clicked on.

That's very confusing - and since we are now allowing programatically depositing models the likely hood of people using on several at the same time is very high ;-)

[epeisach]

With regards to the reflections file:
a) audit records belong only in the first block
b) The others are warnings for multiple block data...

[wmorellato]

Now we get the complete URL, which is returned by the server

[epeisach]

Believe everything on the list handled - and this issue can be close.