Cloned-Restore Fails on OpenShift Clusters with ImagePull Error
Oops! Something went wrong.
Please, try again later.
Challenge
Cloned-restore jobs time-out and eventually fail due to an ImagePullError. Upon further investigation, the root cause is that the default SA (in the target namespace) does not have ImageStreams for the pod to pull an image from a private registry.
Failed to pull image "image-registry.openshift-image-registry.svc:5000/testify-ps/runner@sha256:9317d61bb7943fb2b3f0d62c37aab233a07a3e0ccc67b104a7e3e69cb778dea0": rpc error: code = Unknown desc = Error reading manifest sha256:9317d61bb7943fb2b3f0d62c37aab233a07a3e0ccc67b104a7e3e69cb778dea0 in image-registry.openshift-image-registry.svc:5000/testify-ps/runner: unauthorized: authentication required
Cause
In Openshift 4.x, pods can pull images with the use of ImageStreams. ImageStreamTag represents an image that is retrieved by tag name from an ImageStream. If these are hard-coded in a service account, pods in the target namespace will fail to come up with an ImagePullError.
Example
Namespace = testify-ps SA = default Target namespace = testify-ps-restore
Solution
Identify missing ImageStreamTag(s)
Compare information about the source and target namespaces to identify the missing ImageStreamTag(s), then use one of the three options described in the next section to resolve this issue.
Use the following command to get show namespace information:
Example:
#Source Namespace $ oc get istag -n testify-ps NAME IMAGE REFERENCE UPDATED mgmt:develop image-registry.openshift-image-registry.svc:5000/testify-ps/mgmt@sha256:092577fb9c72f7631b08c1f96658b34a5f928953f59ff8e1d38e91a5a64ef918 6 days ago mongo:4.2 mongo@sha256:8c48baa1571469d7f5ae6d603b92b8027ada5eb39826c009cb33a13b46864908 6 days ago runner:develop image-registry.openshift-image-registry.svc:5000/testify-ps/runner@
#Target Namespace $ oc get istag -n testify-ps-restore NAME IMAGE REFERENCE UPDATED mongo:4.2 mongo@sha256:8c48baa1571469d7f5ae6d603b92b8027ada5eb39826c009cb33a13b46864908 2 days ago
Option 1: Manually Add Missing ImageStreamTag
Manually add them to the default SA in the target namespace and restart pods.
Example
$ oc project testify-ps-restore $ oc edit serviceaccount default <Add the missing mongo:4.2 secret> $ oc delete pod <pod name>
Option 2: Use Script to Pull ImageStreams Between NameSpaces
Run the script below to pull ImageStreams from one namespace to another.
For more information review: Allowing Pods to Reference Images Across Projects
$ oc policy add-role-to-user \ system:image-puller system:serviceaccuont:testify-ps:default \ -namespace=testify-ps-restore
Option 3: Use Veeam Kasten for Kubernetes Transforms Feature
Alternatively, the Veeam Kasten for Kubernetes transforms feature can be used to add missing secrets to the default SA as part of the restore process.
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.
Spelling error in text
Thank you!
Your feedback has been received and will be reviewed.
KB Feedback/Suggestion
This form is only for KB Feedback/Suggestions, if you need help with the software open a support case
Thank you!
Your feedback has been received and will be reviewed.
Oops! Something went wrong.
Please, try again later.