Docker registry cache or internal use? Por que no los dos?

After configuring an internal Docker registry to be pull-through cache pushing images became "unsupported":

docker_registry-registry-1  | 172.18.0.16 - - [25/Aug/2023:12:56:07 +0000] "POST /v2/my_custom_image/blobs/uploads/ HTTP/1.0" 405 78 "" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.5 \\(linux\\))"

docker_registry-registry-1  | time="2023-08-25T12:56:07.81997852Z" level=error msg="response completed with error" err.code=unsupported err.message="The operation is unsupported." go.version=go1.19.9 http.request.host=registry.internal.network http.request.id=foo http.request.method=POST http.request.remoteaddr=1.2.3.4 http.request.uri="/v2/my_custom_image/blobs/uploads/" http.request.useragent="docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.5 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.869256ms http.response.status=405 http.response.written=78 vars.name="my_custom_image"

docker compose logs

Pushing to a registry configured as a pull-through cache is unsupported.

Ref: Docker registry configuration: proxy

It's currently not possible to mirror another private registry. Only the central Hub can be mirrored.

Ref: Docker registry recipe: pull through cache

Docker compose stack

When presented with these constraints a few solutions and variations come to mind:

• Keep the pull-through cache and push built images to docker hub
• Disable pull-through cache and use the registry for private images only
• Run two registries in the compose stack

Two registries

• docker-build.internal.network
• docker-cache.internal.network

Configure the site to use docker-cache by default and any Dockerfile or Compose files can point to the docker-build registry as needed.

One or both?  Enter Ansible

Managing services on a system is hard, #hugops.

Doing it manually is a major pain and usually resulted in a repository of scripts to automate tasks.

Ansible is one of many Config As Code solutions that aimed to remediate this pain.

There are widely available roles to accomplish many tasks, like Jeff Geerling's Docker role for installing docker and docker compose on systems in your inventory or the Nginx Inc role for installing and configuring multiple versions of Nginx.

Ansible can interact with most services, thanks to the many modules available and the portability of Python.  In the context of this task it'll be used to:

• Configure host system for the new service (create directories, files, users, etc ...)
• Write a custom Docker Compose file based on the site and system config
• Integrate the new service with Systemd

Main.yml

Ansible has an expected directory structure for its roles (Ansible role directory structure).  Glancing at the Variable Precedence can illustrate how complicated the inheritance can get.

 Example main.yml for basic service role

---
- name: User
  ansible.builtin.include_tasks: user.yml
  tags:
    - setup

- name: Files
  ansible.builtin.include_tasks: files.yml
  tags:
    - setup
    - update

- name: Service
  ansible.builtin.include_tasks: service.yml
  tags:
    - setup
    - update

roles/my_role/tasks/main.yml

Tasks get divided into dedicated sub-tasks and everything is tagged.  This is technically optional as all tasks can be put into a single monolithic file and run all at once.  Which might work for some tasks, but as we move more toward containerization it helps to break everything down into its smallest component.  This also serves the eventual goal of moving into full Container Orchestration.

Two registries?

This new role for the docker registry should configure the end system in one of three ways:

• Pull-through cache registry only
• Internal build registry only
• One cache registry and one build registry

Easy enough to facilitate with a couple new default variables to set the desired config.

# Enable internal build registry
registry_enable_build: true
# Enable site pull-through cache
registry_enable_cache: true

defaults/main.yml

Add dedicated data directories for each registry (build_data and cache_data), create a new env file for the build registry, and update the compose file template.

---
version: '3'

services:
{% if registry_enable_build %}
  # https://hub.docker.com/_/registry
  build:
    env_file:
        - docker.build.env
    hostname: docker-build
    image: "registry:2"
    networks:
      - docker_internal
    restart: unless-stopped
    volumes:
      - {{ registry_base_path }}/build_data:/data
{% endif %}

{% if registry_enable_cache %}
  # https://hub.docker.com/_/registry
  cache:
    env_file:
        - docker.cache.env
    hostname: docker-cache
    image: "registry:2"
    networks:
      - docker_internal
    restart: unless-stopped
    volumes:
      - {{ registry_base_path }}/cache_data:/data
{% endif %}

networks:
  docker_internal:
    external: true

docker-compose.yml

This will result in a compose file that's customized to the site's configuration, allowing any of the three aforementioned configurations.

Neither?

Ok, yes.  The above actually provides for four possibilities, the last being neither registry defined.

This gets handled one level up (at the site playbook) with a Conditional based on variables:

- name: Docker Registry
  hosts: "docker_registry"
  gather_facts: false
  become: true

  roles:
    - role: docker-registry
      when: registry_enable_build or registry_enable_cache | bool

Other updates

Configuring a site in code using Ansible, or any of the Config Management systems really, allows for a tightly integrated environment that's easy to maintain, replicate, and update.

Other Ansible tasks not detailed in this devlog include:

• DNS
  • Depending on implementation (e.x., Cloudflare, Route53, DigitalOcean)
• Docker installation
• Web server configuration
  • SSL Generation
    • Dev: self-signed
    • Stage: Let's Encrypt (Staging)
    • Prod: Let's Encrypt (Prod)
  • Authentication
• Update docker hosts' mirror URL
  • Discussed in /devlog/20230824
• Building images

Validation

Docker Registry (or two) running, web server configured with SSL and basic auth (optional), and the system's docker daemon configured to use the pull through cache we're ready to run a few tests.

Pull an image through the cache

If the daemon's mirror configuration is correct (and the daemon has been restarted since the last change) pulling an image through the cache should work seamlessly

$ docker pull ubuntu

Check the registry's catalog

Return and parse the list of images in the pull-through cache

$ curl -ks https://docker-cache.dev.internal.network/v2/_catalog | jq .
{
  "repositories": [
    "library/alpine",
    "library/ubuntu"
  ]
}

Check tags associated with an image

Show a list of tags associated with an image

curl -ks https://docker-cache.dev.internal.network/v2/library/alpine/tags/list | jq . | head
{
  "name": "library/alpine",
  "tags": [
    "2.6",
    "2.7",
    "20190228",
    "20190408",
    "20190508",
    "20190707",
    "20190809",

Check a tag manifest

Pull manifest data from a tag

$ curl -ks https://docker-cache.dev.internal.network/v2/library/alpine/manifests/3.18 | jq '[.name, .tag, .architecture]'
[
  "library/alpine",
  "3.18",
  "amd64"
]