TRT-2564: Cache Risk Analysis responses

[sosiouxme]

Risk Analysis queries are fairly heavy and can impact Sippy DB performance. During regression periods we are likely to see an increase in repetitive calls for the same job / test failures. So this adds a caching mechanism.

Both RA and Sippy classic test pages use the same underlying query mechanism, so the same cache is introduced for both (replacing the existing request-level cache for the test API). This caches the test queries for an hour by default, but cache entries are invalidated if the underlying matview is refreshed, so new data is surfaced faster.

1. Sippy's matview refresh code is updated to write a timestamp to redis whenever a matview refresh completes.
2. Query results are written to the cache with a timestamp.
3. On a cache hit, if the underlying matview timestamp is newer than the result timestamp, it is treated as a miss and regenerated. (In the case where the matview timestamp is missing, cached data is used until it expires normally.)

Individual commits separate the concepts that built this change, so reviewing them serially may be easiest.

openshift/origin#30965 proposes one simple way to benchmark risk analysis requests.

[openshift-ci-robot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

[openshift-ci-robot]

@sosiouxme: This pull request references TRT-2564 which is a valid jira issue.

Details

In response to this:

Risk Analysis queries are fairly heavy and can impact Sippy DB performance. During regression periods we are likely to see an increase in repetitive calls for the same job / test failures. So this adds a caching mechanism.

Both RA and Sippy classic test pages use the same underlying query mechanism, so the same cache is introduced for both (replacing the existing request-level cache for the test API). This caches the test queries for an hour by default, but cache entries are invalidated if the underlying matview is refreshed, so new data is surfaced faster.

1. Sippy's matview refresh code is updated to write a timestamp to redis whenever a matview refresh completes.
2. Query results are written to the cache with a timestamp.
3. On a cache hit, if the underlying matview timestamp is newer than the result timestamp, it is treated as a miss and regenerated. (In the case where the matview timestamp is missing, cached data is used until it expires normally.)

Individual commits separate the concepts that built this change, so reviewing them serially may be easiest.

openshift/origin#30965 proposes one simple way to benchmark risk analysis requests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sosiouxme

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [sosiouxme]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sosiouxme]

/retest-required

[openshift-ci-robot]

Scheduling required tests:
/test e2e

[openshift-ci]

@sosiouxme: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.