{"id":332,"date":"2023-08-21T16:09:46","date_gmt":"2023-08-21T21:09:46","guid":{"rendered":"http:\/\/www.p14nd4.com\/blog\/?p=332"},"modified":"2023-08-21T16:11:02","modified_gmt":"2023-08-21T21:11:02","slug":"gitlab-ci-cd-container-registry-case-sensitive-hostname","status":"publish","type":"post","link":"http:\/\/www.p14nd4.com\/blog\/2023\/08\/21\/gitlab-ci-cd-container-registry-case-sensitive-hostname\/","title":{"rendered":"Gitlab CI\/CD container registry: case-sensitive hostname"},"content":{"rendered":"

I’ve recently been using self-hosted Gitlab<\/a> at work, particularly for its CI\/CD<\/abbr> features. We have a few projects using it already, and I’m trying to expand that utilization. It all went pretty well until I tried building with a new Docker image, and got an error:<\/p>\n

ERROR: Job failed: failed to pull image \"[..]:latest\" with specified policies [always]: Error response from daemon: Head \"https:\/\/[..]:5050\/v2\/[..]\/manifests\/latest\": denied: access forbidden (manager.go:237:0s)<\/pre>\n
<\/p>\n
The project I most recently set my sights on requires some external tools\/resources for the build. While they’re publicly available, the prospect of adding a build script step to download and extract them was unpalatable (waste of bandwidth and time), and while I toyed with the idea of trying to hack them into build cache restoration, I finally relented and just built a new Dockerfile \/ docker image, and pushed the image to the project’s container repository. (Incidentally, I would’ve loved if .gitlab-ci.yml<\/span> understood Docker Compose, and I could just mount a volume containing these resources on top of an upstream-maintained\/updated Docker image, so please let me know if I missed that or it gets added.)<\/p>\n
The Problem<\/h3>\n
All of that got me to the point of trying to run the build for the first time, which failed quickly with a log like this:<\/p>\n
Running with gitlab-runner 16.2.0 (782e15da)\n  on gitlab-runner02 Ua6jigiQ, system ID: r_tRd8aLibbXym\nPreparing the \"docker\" executor 00:03\nUsing Docker executor with image [gitlabHostName]:5050\/[project\/image path]:latest ...\nPulling docker image [gitlabHostName]:5050\/[project\/image path]:latest ...\nWARNING: Failed to pull image with policy \"always\": Error response from daemon: Head \"https:\/\/[gitlabHostName]:5050\/v2\/[project\/image path]\/manifests\/latest\": denied: access forbidden (manager.go:237:0s)\nERROR: Job failed: failed to pull image \"[gitlabHostName]:5050\/[project\/image path]:latest\" with specified policies [always]: Error response from daemon: Head \"https:\/\/[gitlabHostName]:5050\/v2\/[project\/image path]\/manifests\/latest\": denied: access forbidden (manager.go:237:0s)<\/pre>\n
After trying several things and banging my head on the wall a bit, I eventually spotted a difference between this and a working build: the presence of this line before ‘pulling docker image’:<\/p>\n
Authenticating with credentials from job payload (GitLab Registry)<\/pre>\n
The Solution<\/h3>\n
Additional searching and scrutinizing the config revealed that in a working build, the image repository path hostname was lowercase, and in the new\/broken build, it had capitalization. Sure enough, when I replaced the “gitlabHostName” with “gitlabhostname” (equivalent), the new build started “Authenticating with credentials from job payload (GitLab Registry)” as well, and was able to complete the build.<\/p>\n
So, if you’re butting your head against Failed to pull image<\/q> … denied: access forbidden<\/q> errors in Gitlab while trying to pull an image from your container registry, check to make sure the hostname specified in the .gitlab-ci.yml<\/span> image path matches your actual Gitlab container repository and is lower-case.<\/strong> (After seeing this only work with the lower-cased hostname, I was surprised to find that the value in my gitlab.rb registry_external_url<\/span> does<\/em> have the capitalization found in the broken build, so it would seem that it’s not even a matter of matching that value, but necessarily being lower-case.)<\/p>\n","protected":false},"excerpt":{"rendered":"
I’ve recently been using self-hosted Gitlab at work, particularly for its CI\/CD features. We have a few projects using it already, and I’m trying to expand that utilization. It all went pretty well until I tried building with a new Docker image, and got an error: ERROR: Job failed: failed to pull image “[..]:latest” with […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[35,34,19],"class_list":["post-332","post","type-post","status-publish","format-standard","hentry","category-technical","tag-docker","tag-gitlab","tag-solved"],"_links":{"self":[{"href":"http:\/\/www.p14nd4.com\/blog\/wp-json\/wp\/v2\/posts\/332","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.p14nd4.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.p14nd4.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.p14nd4.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.p14nd4.com\/blog\/wp-json\/wp\/v2\/comments?post=332"}],"version-history":[{"count":5,"href":"http:\/\/www.p14nd4.com\/blog\/wp-json\/wp\/v2\/posts\/332\/revisions"}],"predecessor-version":[{"id":337,"href":"http:\/\/www.p14nd4.com\/blog\/wp-json\/wp\/v2\/posts\/332\/revisions\/337"}],"wp:attachment":[{"href":"http:\/\/www.p14nd4.com\/blog\/wp-json\/wp\/v2\/media?parent=332"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.p14nd4.com\/blog\/wp-json\/wp\/v2\/categories?post=332"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.p14nd4.com\/blog\/wp-json\/wp\/v2\/tags?post=332"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}