Collection API

[cmelone]

This PR implements a webhook for the status change of Gitlab jobs. Build jobs are selected for processing, where a series of Prometheus queries are issued and data about the build and resource utilization are made available.

Adds: POST endpoint under /collect.

Priorities:

• Minimize the amount of requests needed to collect the necessary data. At the moment, for builds: 1 request to Gitlab + 5 to Prometheus. For VMs: 2 requests to Prometheus.
• Handling the potential for duplicates and multiple/erroneous requests for collection. Protections are implemented for race conditions.
• Composability: API should be readable and potentially reusable for other purposes
• Speed: while this application is written with async functionality, there is still a requirement to respond to the webhook in under 10 seconds.

All main functionality is implemented and I have removed the draft status from the PR. There are just a couple more things to do:

• if/when spack-infrastructure!733 is merged, make sure that stack is being properly collected in the data

Also, I realize that this PR is quite big, so I'm happy to break it up into smaller pieces.

[alecbcs]

Super nice job! This PR looks great. Just a couple points of feedback.

I also might suggest creating the sqlite db in the root of the repo and actually putting db/ under gantry/ instead of leaving it as its own top level component. This might fit a bit better depending on how we structure the components. I'd probably go for something like the following, but feel free to push back if you don't find that as easy to work with.

gantry/
  db/
     get.py
     insert.py
  clients/
    prometheus.py
    gitlab.py
  models/
    jobs.py
    vms.py
  routes/
    collection.py
  __main__.py

[cmelone]

I also might suggest creating the sqlite db in the root of the repo and actually putting db/ under gantry/ instead of leaving it as its own top level component.

I like this! Should I stick the .sql schema file in there as well?

[cmelone]

@alecbcs 11d05fc addresses your comments regarding the Prometheus client

lmk it that works

rewritten

[cmelone]

Going to tackle testing in another PR to avoid adding too much more to this. marking this one as ready

[alecbcs]

Big fan of the rewrite! Just a few small nitpicks neither of which are real blockers if you prefer keeping as is. Otherwise looks good to me!

[alecbcs]

Looks good to me. @cmelone thank you for all of your work on this!