ImageStreamTags are not updated

[olaf-meyer]

Hello,

I have a question regarding the cluster-sample-operator: I understand the cluster-sample-operator is maintaining the ImageStreams and Templates in the project openshift. I had a look at an older test OpenShift cluster and noticed that the ImageStreamTag were created but are never been updated since. From my point of view, the ImageStreamTags are created when a tag attribute is "added" in an ImageStream. If a tag is removed from an ImageStream, the Adminstration Consol will display ImageStreamTag in the ImageStream with the text "pushed image" in the from column. From my point of view, ImageStreamTags are containing no reference to ImageStreams. Any modification (besides the creation) of tags in the ImageStreams seem to be not reflected in ImageStreamTags.

Is this on purpose that ImageStreamTags are not updated (replaced with a new version), if the associated ImageStream is modified?

Thank you very much in advance,

Olaf

[gabemontero]

Hello,

I have a question regarding the cluster-sample-operator: I understand the cluster-sample-operator is maintaining the ImageStreams and Templates in the project openshift. I had a look at an older test OpenShift cluster and noticed that the ImageStreamTag were created but are never been updated since. From my point of view, the ImageStreamTags are created when a tag attribute is "added" in an ImageStream. If a tag is removed from an ImageStream, the Adminstration Consol will display ImageStreamTag in the ImageStream with the text "pushed image" in the from column. From my point of view, ImageStreamTags are containing no reference to ImageStreams. Any modification (besides the creation) of tags in the ImageStreams seem to be not reflected in ImageStreamTags.

Is this on purpose that ImageStreamTags are not updated (replaced with a new version), if the associated ImageStream is modified?

Hey @olaf-meyer

Short answer: yes, this is on purpose. At least the imagestream internals. I cannot speak to what the console does.

Longer answer: As RHEL SCL and RH Middleware update their imagestreams with new tags, remove old tags, or change which images a tag points to, we only update in the current release of OCP when we receive that update. Customers pick up those updates by upgrading their clusters to the newer 4.x. Where as you probably have seen with OCP 4.x, we only support N-2 dot versions. i.e. if 4.6 is the current release, only 4.5 and 4.4 are still supported.

Thank you very much in advance,

Olaf

[olaf-meyer]

Hallo Gabe,

thank you very much for the answer. Did I understood correct that the Images are only updated, if the OpenShift version itself is updated (i.e. from 4.5 to 4.6) or by adding new tags to the ImageStreams?

Olaf

[gabemontero]

Hallo Gabe,

thank you very much for the answer.

My pleasure @olaf-meyer

Did I understood correct that the Images are only updated, if the OpenShift version itself is updated (i.e. from 4.5 to 4.6) or by adding new tags to the ImageStreams?

I think you have it right @olaf-meyer though allow be to clarify your verbiage a bit:

• We only pull in updates to the ImageStreams provided by the RH SCL team (things like python, ruby, mysql, etc.) and the RH Middleware teams (anything under the broad JBoss umbrella) on an initial major release ... i.e. 4.2.0, 4.3.0, 4.4.0, 4.5.0, 4.6.0, 4.7.0, 4.8.0, etc.
• on maintenance on those ".0" releases, i.e. 4.4.1, 4.4.2, 4.5.3, 4.6.4, etc. we do not pull in later versions of those imagestreams ... for samples operator, only code fixes to the operator itself are pulled in
• cluster administrators can edit the imagestreamtags of those imagestreams as they see fit in the interim between say 4.5.0 and 4.6.0; if only the imagestreamtags are change, samples operator will not interfere

Olaf

[olaf-meyer]

Hallo Gabe,

what do you mean by verbiage?

Anyhow, are there plans to annotate image tags with the schedules attribute to keep the tag up to date? I have seen that productive OpenShift clusters run for quite a long time. Certain images i.e. dotnet-core:3.1 will currently not receive updates till the base images changes from rhel7 to rhel8. Before a new tag is available, most of the images get security updates and it is hard to explain to customers, why the updates from the container catalog are not reflected to the ImageStream tags in an OpenShift cluster.

Kind regards,

Olaf

[gabemontero]

Hallo Gabe,

what do you mean by verbiage?

verbiage == the words used

Sorry, I was just trying to convey I think we were on the same page, but I would have worded it slightly differently.

Anyhow, are there plans to annotate image tags with the schedules attribute to keep the tag up to date?

That would be the responsibility of the image providers, not the samples operator. I know those image providers provide details on their images in the containers catalog, but have not heard about them annotating their images as well.

One thing we will be doing as part of samples operator is that in the release notes we'll be documenting which images the image providers removed in a given release. We are even doing this for releases which have shipped, back to 4.4, in addition to future releases moving forward.

But with the current source of requirements we have for samples, we are not going beyond that.

I have seen that productive OpenShift clusters run for quite a long time. Certain images i.e. dotnet-core:3.1 will currently not receive updates till the base images changes from rhel7 to rhel8. Before a new tag is available, most of the images get security updates and it is hard to explain to customers, why the updates from the container catalog are not reflected to the ImageStream tags in an OpenShift cluster.

Yeah, for what it is worth, we have some competing concerns here. Before we digress too much more on all that, I want to get you on a path for surfacing your wishes to the right place. Which to summarize, is updating the sample imagestreams and associated templates, assuming they do not need certain kubernetes level features, in the dot releases of older 4.x releases, in addition to the latest current release.

Among other things, we'd have to get agreement from the image providers for that, as well as take on some additional release plumbing changes and testing.

The best way to pursue this is opening an RFE and get our planning involved. I believe the URL for that is https://issues.redhat.com/projects/RFE/summary

Let me know though if you run into difficulties with that, and I'll reach out to project management here for further guidance.

@siamaksade @adambkaplan @sbose78 @pedjak FYI ^^

Kind regards,

Olaf

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[X-dark]

Anyhow, are there plans to annotate image tags with the schedules attribute to keep the tag up to date?

That would be the responsibility of the image providers, not the samples operator. I know those image providers provide details on their images in the containers catalog, but have not heard about them annotating their images as well.

@olaf-meyer is speaking about the scheduled=true you can add on the ImageStream to have OpenShift automatically pull the last image for a given tag. This cannot be done by the image provider as it provides an image, not an ImageStream: https://docs.openshift.com/container-platform/4.7/openshift_images/managing_images/tagging-images.html#images-add-tags-to-imagestreams_tagging-images

One thing we will be doing as part of samples operator is that in the release notes we'll be documenting which images the image providers removed in a given release. We are even doing this for releases which have shipped, back to 4.4, in addition to future releases moving forward.

Are you speaking here about new tags or new images within a tag?

But with the current source of requirements we have for samples, we are not going beyond that.

Not wanting to provides latest patch releases is one thing. Not updating images within a tag leading to having images present with major CVE (as the image present will always be the one that a given tag was pointing to at install time) is quite another thing...

/remove-lifecycle rotten

[gabemontero]

Hey @X-dark - wanted to let you know that addressing your points / concerns is "on the books". But one of the items going on with samples is that we are transferring ownership of it to @dperaza4dustbit and his team over the course of this summer. As such, I've asked him and his team to go over the training materials I've given them so far and see what they can come up with.

So if you can bear with us for a little bit, we'll make progress on this, it just may be a bit "slower".

All that said, I'll embed some "starting responses" to tee things up for @dperaza4dustbit and team

Anyhow, are there plans to annotate image tags with the schedules attribute to keep the tag up to date?

That would be the responsibility of the image providers, not the samples operator. I know those image providers provide details on their images in the containers catalog, but have not heard about them annotating their images as well.

@olaf-meyer is speaking about the scheduled=true you can add on the ImageStream to have OpenShift automatically pull the last image for a given tag. This cannot be done by the image provider as it provides an image, not an ImageStream: https://docs.openshift.com/container-platform/4.7/openshift_images/managing_images/tagging-images.html#images-add-tags-to-imagestreams_tagging-images

Yes it is fair to say I glossed over a few details and possibly merged roles that might actually be assigned to different people.

@dperaza4dustbit and team will provide the additional detail I was referring to that connects the dots between a given image, the non-openshift repository it is produced from, and how it ultimately lands in samples operator, and where "scheduled=true" could come into play wrt that.

One thing we will be doing as part of samples operator is that in the release notes we'll be documenting which images the image providers removed in a given release. We are even doing this for releases which have shipped, back to 4.4, in addition to future releases moving forward.

Are you speaking here about new tags or new images within a tag?

But with the current source of requirements we have for samples, we are not going beyond that.

Not wanting to provides latest patch releases is one thing. Not updating images within a tag leading to having images present with major CVE (as the image present will always be the one that a given tag was pointing to at install time) is quite another thing...

Addressing CVEs absolutely falls under the type of things wrt updating samples content in z-stream / released OCP versions. But their is a very specific process we have to follow, that involves more than responding to a github issue.

@dperaza4dustbit and team will detail that, and then work with you all to gather the necessary details on any specific images/CVEs you all might have in mind.

/remove-lifecycle rotten

[X-dark]

Hi @gabemontero, thanks for your answer.

We are currently in the process of testing RHACS and we were quite surprised to see all those images flashing red despite provided by the sample operator.
I understand the difficulty of handling such issues while going through a transition process.

I have a support request open anyway, so I will see with them if they have any suggestion to at least do a one shot refresh of those tags to lower the number of CVE while we wait for a more automated solution.

[gabemontero]

OK are skill transfer is still in progress, but we've received another issue related to this, again with the vulnerability warnings in the console with respect to sample images (see #383 (comment) ) so I'm stepping in with an update while @dperaza4dustbit and team get up to speed.

1. I noted previously about how updates are only pulled into our current release, and that to backport updates to the z stream, we would need support cases / bugzillas.

Do we want to entertain adjustments to that policy? Dev / engineering is not there yet.

2. One item dev/eng is considering of providing samples outside of the core OCP install, replaced by either by an OLM based operator or via the OLM related OpenShift Helm catalog. Among other things, this would give us greater flexibility and provide updated samples to OCP release that are already GA'ed. So resources may be applied to address this situation in that manner instead.

3. Another item I do not believe I covered here precisly previously is that deprecated imagestreamtags which are no longer present upstream and at https://github.com/openshift/library are explicitly left in imagestreams by the samples operator during upgrade. See https://docs.openshift.com/container-platform/4.8/openshift_images/configuring-samples-operator.html#images-samples-operator-deprecated-image-stream_configuring-samples-operator

This is done so that dependent deployment and builds are not immediately broken. It is currently left to cluster admins to ensure there are no dependencies on such items, and then they can manually update the imagestreams to remove those tags.

Note: on new installs those old imagestreamtags are not installed.

We did originally consider more automated methods of dealing with this situation. There are possibilities, but not without cost nor complexities to consider when trying to address all the angles.

Could we revisit this: certainly. Where and how to make changes in this space: while engineering's collective understanding of what OCP operators can/should do, and should not / cannot do in this space, has evolved, and I can think of a few approaches to this situation, I would not claim there is a slam dunk solution.

And effort in this space would required discussion in OCP planning, as well and technical iteration via the OCP enhancement process at https://github.com/openshift/enhancements/. And there is the consideration that we are considering samples management to be something outside of OCP, and balancing resources spent on this problem there vs. in the current OCP samples operator.

As I type, Jira is unfortunately down :-) so I can't cite current epics arounds samples, open new items, etc., but between @dperaza4dustbit and myself, we will take on at least getting this requirement registered so the due diligence I mentioned above occurs.

@sbose78 @bparees @pedjak @adambkaplan @openshift/openshift-team-build-api FYI