Add Euclid landing pages and flatten TOC

[troyraen]

This is a draft of what I understood in our meeting this afternoon. I decided to do it separately from #193 to make things easier to review. This contains dummy content for the HATS notebooks so we can see what it will look like, but I'll remove that before merging.

The Q1 landing page currently contains lists of notebooks because I don't know how to make a gallery or something else that's prettier. But whenever we figure that out, hopefully the text in the list items can be used without much change.

Questions I have and problems I'm seeing:

• The main Euclid landing page is very short. Is that what we wanted? (merged the Q1 landing page in, so now there's just one)
• Is the Q1 Euclid landing page too long? If so, how to shorten it?
• At the very bottom of all of the Euclid pages, the buttons pointing to the previous and next page all say 'SPHEREx' instead of 'Euclid' (screenshot). I suspect this is happening because I changed Euclid title -> file in toc.yml in order to add the landing pages. I tried using both title and file but that didn't make any difference. Those files have short_title frontmatter but it's not being used for this. Any ideas @bsipocz?
  

To do before merging:

• Remove the two temp commits
  - [ ] Remove the label html rendering / skip testing. (maybe this isn't necessary?) [edit by @bsipocz] - keep the label, there won't be content change in here so no need for pytesting
• Maybe make landing pages and flatten TOC for the other missions/sections as well?

[bsipocz]

The main Euclid landing page is very short. Is that what we wanted?
Is the Q1 landing page too long? If so, how to shorten it?

Merge them together, so Q1 will be a section with a heading link on the Euclid landing page. So it will be possible to link to it, but it flattens the structure a little bit (and we can still have nested toc on the side.

[bsipocz]

At the very bottom of all of the Euclid pages, the buttons pointing to the previous and next page all say 'SPHEREx'

That's a wow. I mean I think it should honour the - start at the blocks, but apparently it inherits the title. I'll copy this example and report on it, but for now just keep adding titles in the TOC, too. I think the page title/short-title will override it anyway.

Edit: this is a known bug: jupyter-book/mystmd#2589

[bsipocz]

Remove the label html rendering / skip testing. (maybe this isn't necessary?)

Since this PR won't change the notebook content, it can stay on

[troyraen]

Merge them together

Done, but let me know if it is as you envisioned.

[bsipocz]

Merge them together
Done, but let me know if it is as you envisioned.

yes. well, I envision cards, but landing page wise yes, one top level landing and then no more narrative nesting, even if there are notebook nestings.

[troyraen]

I keep wondering about these anchors. They're great because we can change file names and heading text without breaking the links, and that's super helpful. But on the other hand, they make the links totally useless outside of the website. When looking at the markdown files, even on github where the markdown is rendered, the links don't work at all and the reader is lost. I don't know what to do about that.

[bsipocz]

hmm, that's a good point, I totally was focusing on the rendered outputs and was not even thinking about the notebooks themselves.

If we don't yet track this issue upstream, let me add it to the "papercut with notebooks" list.

[jaladh-singhal]

That's a valid complaint but isn't that expected?! I mean a renderer decides how to render anchor - this syntax doesn't mean anything to Github markdown same as how frontmatter doesn't mean anything. So Github markdown just renders it as text and rather creates heading-text-based anchors. Whereas it has a meaning for the JB website renderer so it shows up as expected.

Do you expect people to linking anchors or other items from the Github markdown doc as opposed to from our documentation website?

[jaladh-singhal]

Thinking more about it...I think we can get best of both worlds by refraining from using these special header targets for H1 headings because then you can just link the document path.

[bsipocz]

Just to rephrase, if we need to drop one of the 3 access points, I would totally drop the Github rendering. It doesn't matter how things look on github as our end user products are the website and the notebooks. (and we know a bunch of myst stuff is already broken in github markdown rendering as it doesn't speak the myst flavour of the format)

[troyraen]

The github rendering is just one example of how users interact with the markdown files, which we definitely expect them to do. I don't think it's a big deal that the frontmatter and admonitions look as they do, but links not working is significant. Not saying it's more important than the other 2 access points, just that there is tension here that is not insignificant.

Are there good reasons not to use standard markdown syntax to link to sections? For example, when I want to put a link in the euclid hats notebooks to the hats section of this landing page, standard markdown syntax would look something like [link name](../euclid_access/euclid.md#merged-objects-hats-catalog) and no anchor (or myst label or header target) would be needed. That will be more work for us to maintain, but if it means the links will work everywhere then we should do it.

[bsipocz]

apparently I started a review, but now I cannot leave single comments, so just release the whole review as is

[bsipocz]

hmm, that's a good point, I totally was focusing on the rendered outputs and was not even thinking about the notebooks themselves.

If we don't yet track this issue upstream, let me add it to the "papercut with notebooks" list.

[jaladh-singhal]

@troyraen thanks for quickly putting this together! Landing page and TOC looks exactly like I envisioned it in the meeting, except more emphasis on notebooks in the landing page by making them gallery-like boxes (we can chase that in another issue and I'm open to take a dig at it).

I feel Header Targets are great for landing page headings (>=H2) but they are probably an overkill for the notebook H1 headings. We can instead just link the document (see "Link to documents using relative links from the markdown" syntax at https://mystmd.org/guide/cross-references#link-references)

[jaladh-singhal]

Ooh custom anchor links - that's great!

[jaladh-singhal]

That's a valid complaint but isn't that expected?! I mean a renderer decides how to render anchor - this syntax doesn't mean anything to Github markdown same as how frontmatter doesn't mean anything. So Github markdown just renders it as text and rather creates heading-text-based anchors. Whereas it has a meaning for the JB website renderer so it shows up as expected.

Do you expect people to linking anchors or other items from the Github markdown doc as opposed to from our documentation website?

[jaladh-singhal]

is there a specific reason for linking the custom anchors for H1 heading for a notebook rather than linking the notebook document path itself (assuming that is possible)?

Suggested change

	- [Introduction](#euclid-q1-hats-intro) — Understand the content and format of the Euclid Q1 Merged Objects HATS Catalog, then perform a basic query.
	- [Introduction](../parquet-catalog-demos/euclid-q1-hats/1-euclid-q1-hats-intro) — Understand the content and format of the Euclid Q1 Merged Objects HATS Catalog, then perform a basic query.

I'm not certain but this may resolve several complaints with Github markdown rendering, because Github recognizes path-based linking IIRC.

[bsipocz]

it may not create hovers though and I suspect neither of these would work in a jupyterlab setting?

[jaladh-singhal]

We won't need hovers when we do the gallery-like view of notebooks. Jupyterlab won't work with this special header directives anyways - I think it's worth trying.

[troyraen]

is there a specific reason for linking the custom anchors for H1 heading for a notebook rather than linking the notebook document path itself (assuming that is possible)?

Good point. In repos like fornax-documentation I would argue yes, and that's why I got in the habit of doing this. But now thinking through it for these notebooks repos, no it seems better all around to just link to the path.

[bsipocz]

eventually we will want to cross link between repos, too. At least I want to use that functionality to expose the euclid notebooks in the euclid archive docs, and spherex notebook in the spherex archive docs. I think file path won't work for those use cases.

[troyraen]

We could have anchors in the markdown files in this repo but only use them when creating links externally in other myst projects. For docs repos, we don't expect users to ever be looking at the markdown files, only the website, so we don't need to worry about links working in those markdown files. If when we want to cross link between notebook repos we'll have to decide what to do. Maybe it will be better to link to website urls there.

[jaladh-singhal]

This flatter TOC renders nicely - thanks!

[jaladh-singhal]

Is the Q1 Euclid landing page too long? If so, how to shorten it?

@troyraen it looks fine to me since we condensed all other landing pages into here. I'd say we can experiment with gallery-like rendering which will help drawing attention of reader to the notebooks more. Feel free to open an issue and assign to me if you want me to take a dig at what JB offers and what custom stuff I can hack up with my UI knowledge.

[jkrick]

General comment: I don't think the organization is clear, specifically how we differentiate between the currently divided sets of notebooks. In the drop downs on the left column they are differentiated by "VO" and "HATS", with one lone "cloud access".

These two words "VO" and "HATS" mean nothing to users, so I would suggest we brainstorm other ideas. Is the main difference between them the end database? ie/, one is in parquet format and one is not? So maybe the main difference that is a valuable distinction to users is that one does large scale searches faster, and one is more appropriate for small scale searches? Looking at this from a user perspective, we need to be able to answer why we are showing users more than one way to do the same task.

Maybe this is heretical, but maybe the merged catalog gets its own level because it is its own data product. So I am suggesting we have Euclid as the main landing page. That subdivides into ERO, Q1, and Merged Objects Catalog (Q1). We'll figure out what to do with Q2 when that hits in 6 months and we decide to reorganize this anyway. Yes, I like this. Then recommend removing "HATS" and "VO" from the front of the short names of the notebooks.

In my suggested scheme, Jaladh's "cloud access" would go under Q1

Probably this means my individual comments below aren't relevant anymore because the overall structure will change, so I won't go back and mess with those if you do decided to adopt this change.

[vandesai1]

Thanks for making this! Some comments:

• I think we should put ERO at the bottom instead of the top. It's really not that important anymore, and the fact that there's only 1 notebook there makes the side menu feel weird to me. Every new release should go on the top, because it will be the one everyone wants.

[vandesai1]

To answer your specific question, I do think that the Euclid landing page is too long. I think it will help to have the gallery, because it will be easier to pick out which text is a notebook and which text isn't. Here are suggestions for things that can be removed and additions to consider:

1. The opening pargraph says "The notebooks in this section introduce the Euclid data releases and products, and demonstrate access using various methods. They are organized by data release." I feel like this could be cut. It's all pretty obvious by the title and the table of contents. I wonder if it would be more useful to say what Euclid is and add a link to the Euclid mission page.

2. Under "Quick Release 1 (Q1)", the opening paragraph is "These notebooks describe the Euclid Q1 data products and demonstrate access using various methods." Again, this isn't very useful information. Maybe say what Q1 is and when it was released and put a link to the Q1 page that contains the documentation.

3. I wonder if we should rethink "VO Access to Images, Spectra, and Catalogs" versus "Cloud Access to Images, Catalogs, and Spectra". The Cloud notebook uses SIA, after all. This is an implementation-focused organization, and we may want to shift a user-driven organization.

4. I wonder if we should also rethink the organization by data release. Anahita says that Q1 will be entirely inside DR1, so it might not make sense to maintain separate Q1 notebooks once DR1 is released. We should probably just update them to point to DR1. But I think that DR2 will be disjoint spatially from DR1, so we'll have to figure out if the data services are going to take care of that for us. But again, I don't think we'll necessarily need a DR2 section. The same data services will apply for the most part.

One possible user-centric organization structure that we can grow into is:

- Basic Data Access

• cloud access notebook
• ERO data access (deprecated)
• could add a basic on-prem counterpart

- Images:

• MER mosaics
• could add notebooks about the VIS and NIR calibrated frames

- Catalogs:

• MER catalogs
• SPE catalogs
• PHZ catalogs
• HATS series

- Spectra:

• SIR Spectra
• Could add one about the 2D spectra, so we can see if we trust the 1D spectra

[troyraen]

Thanks everyone! I had a lot of questions/thoughts/clarifications, but in the process of typing them out they all became resolved well enough or just obsolete. I like the idea to organize the page by product type. There's a couple things I'm not totally sure about but I'll give it a try and put up another draft.

[troyraen]

I pushed a new draft organized by data product. Please look and comment if you have any suggestions. Personally, my main concerns/questions are about the ‘Basic Data Access’ section and the HATS organization.

Putting a ‘Basic Data Access’ section at top is tricky without an on-prem notebook to include there. Instead, I named it ‘Special Topics’ and put it at the bottom. Maybe someone else has a better idea.

I collapsed the HATS notebooks in the TOC. If we want this flattened, I need to help coming up with short names that make sense.

On the landing page, the HATS notebooks are in the same list as the other catalogs, just indented to give structure. I don't love this for two reasons:

1. When we switch to gallery-style cards we won’t have the list indentation, so we’ll need a different way to identify them.
2. I would really like to be able to link to this HATS tutorial series in other places. With the current list, that's not possible.

Alternatives that would solve those two things including giving the HATS notebooks their own subsection (under Catalogs) or giving them their own landing page. I don't love those solutions for other reasons, but maybe one of them (or something else) is better than the current list.

[jaladh-singhal]

IMO it's looking good enough for this iteration. With the gallery view of notebooks, we can experiment more with all different cross-linkings

[jaladh-singhal]

this looks clean and works in both readme and website!

[jaladh-singhal]

On the landing page, the HATS notebooks are in the same list as the other catalogs, just indented to give structure. I don't love this for two reasons:

When we switch to gallery-style cards we won’t have the list indentation, so we’ll need a different way to identify them.
I would really like to be able to link to this HATS tutorial series in other places. With the current list, that's not possible.

@troyraen IMO we can make HATS merged catalog a dedicated section right after catalogs section. It's technically still catalogs but has lot of items within it to justify being an independent section

[troyraen]

I think we've converged on the major points here and can go ahead and merge this. I'll leave the other sections (SPHEREx, etc.) to do separately.