Docker registry and pull-through cache

By cyberpuffin, 24 August, 2023

If you wish to make an apple pie from scratch, you must first invent the universe.
-- Carl Sagan, Cosmos

Services

Managing services on a system is hard, #hugops.

When installed directly to a system (i.e., bare metal) that system's dependencies become intwined with the service's dependencies.

The more services you add the greater the likelihood of dependencies being deprecated, unavailable, or conflicting.

Virtualization

One of the early solutions for slicing up a system's hardware between dedicated services is Virtualization.

Instead of installing services directly to the base system, install a single service to manage virtualization. Next create system copies that can be dedicated to individual services.

The base layer, normally called the hypervisor, would remain untainted by service dependencies and is all the more stable for it.

This segregation comes at the cost of overhead and resource duplication.

After the hypervisor defines the virtual machine it's handed to Ops to install an OS and apply the base configuration. Which is fine for a few virtual machines, but when you start getting into the dozens or hundreds the duplication of system files and resources takes its toll.

Containerization

Enter containers, thanks in large part to cgroups.

Containers are similar to Virtual Machines, in as much as creating pseudo-isolated environments for services to run, but they rely on cgroups for isolation and share resources with the hypervisor / OS layer.

Virtualization vs Containers (src: Red Hat) — Image src: https://www.redhat.com/en/topics/virtualization#arent-vms-just-containers

So Docker?

Well yes, but also no.

Docker is to containers as Linux is to operating systems.

There are other container / isolation / virtualization solutions: Bhyve, BSD Jails, KVM, LXD, Qemu, VMWare, Xen ... the list goes on.

Point is that Docker is one of the container technologies available, albeit probably the most popular.

Docker vs LXD (src: Ubuntu) — Image source: https://ubuntu.com/blog/lxd-vs-docker

Run the service in a container

Service providers build images that contain their service and any requesite libraries or dependencies. These images get distributed in layers via a Docker Registry, the most popular one being Docker Hub.

These images / layers get pulled from the registry to the system that will run the service. The images then get executed in their isolated environment, creating the running container.

Self-host Docker Registry

There's no shortage of available options for hosting a Docker Registry. That being said, running a private registry is a great way to test changes and manage internal resources.

There's not much to Deploying a registry server, especially with the compose plugin installed.

docker compose vs docker-compose

Somewhere in 2022 docker-compose got merged into the main docker package (Docker blog). Support for docker-compose ended June 2023. If muscle memory can be adjusted the drop-in replacement, docker compose is easy enough. Otherwise, alias docker-compose='docker compose' should suffice.

Compose file

Docker Compose files are a quick and simple way to define a software stack to run in isolation. For the internal / testing Docker Registry the following can be used:

---
version: '3'
services:
  # https://hub.docker.com/_/registry
  registry:
    env_file:
        - docker-compose.env
    hostname: registry
    image: "registry:2"
    networks:
      - docker_internal
    restart: unless-stopped
    volumes:
      - /opt/docker_registry/data:/data
networks:
  docker_internal:
    external: true

Ref: version, services, networks

This is a very simple version with a few expectations.

Docker network "docker_internal" has been created (handled in another Ansible role or manually).

An environment file sits next to docker-compose.yml, named docker-compose.env:

REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY="/data"
REGISTRY_PROXY_REMOTEURL="https://registry-1.docker.io"
REGISTRY_PROXY_USERNAME="<login user>"
REGISTRY_PROXY_PASSWORD="<login password>"
REGISTRY_DELETE_ENABLED="true"

Ref: Configuring a registry

A web server sits on the same docker network and handles the connection to the registry.
- Can be in the same compose stack as the registry or separate.
- Ref: Authenticate proxy with nginx

Configure Docker daemon

With the service running it's time finalize the configuration and set the docker daemon to use the new Registry as a pull through cache service for docker hub.

/etc/docker/daemon.json

{
  "registry-mirrors": ["https://my.internal.registry"]
}

Restart the daemon service and monitor the resolver logs (docker compose logs -f). In a separate terminal run a docker pull to see the registry in action.

Pull through cache mode or internal dev registry

Edit: This section added 2023-08-25 after trying to push a custom image to the internal repository set in cached mode.

docker_registry-registry-1  | 172.18.0.16 - - [25/Aug/2023:12:56:07 +0000] "POST /v2/my_custom_image/blobs/uploads/ HTTP/1.0" 405 78 "" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.5 \\(linux\\))"

docker_registry-registry-1  | time="2023-08-25T12:56:07.81997852Z" level=error msg="response completed with error" err.code=unsupported err.message="The operation is unsupported." go.version=go1.19.9 http.request.host=registry.internal.network http.request.id=foo http.request.method=POST http.request.remoteaddr=1.2.3.4 http.request.uri="/v2/my_custom_image/blobs/uploads/" http.request.useragent="docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.5 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.869256ms http.response.status=405 http.response.written=78 vars.name="my_custom_image"