Private container registry

Using private registry

Using a registry besides docker hub, requires us to create a file on each node that should access it. (Usually every node)
rancher docs

Since the registry will require inputting values into a file on each server, it could be a good idea for it to have its own credentials.

Install htpasswd if you do not have it

apt install apache2-utils -y

Create the secret

# You should replace the username and password with more secure ones
export REGISTRY_USERNAME=username
export REGISTRY_PASSWORD=password
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: registry-basic-auth-secret
  namespace: default
data:
  users: |2
    $(htpasswd -Bbn $REGISTRY_USERNAME $REGISTRY_PASSWORD | base64)
EOF

Configure the nodes to use the local registry

**Note: ** This has to be done on every node that will pull containers

cat <<EOF > /etc/rancher/k3s/registries.yaml
mirrors:
  registry.${DOMAIN}:
    endpoint:
      - "https://registry.${DOMAIN}"
configs:
  "registry.${DOMAIN}":
    auth:
      username: ${REGISTRY_USERNAME}
      password: ${REGISTRY_PASSWORD}
EOF
# check to see
cat /etc/rancher/k3s/registries.yaml

Restart K3S for this to take effect

systemctl restart k3s

Alternatively

You can add a secret like this and add it to the deployment

kubectl create secret docker-registry my-private-registry \
--docker-server=registry.${DOMAIN} \
--docker-username="username" \
--docker-password="password" \
--docker-email=something@randomhere.com

apiVersion: apps/v1
kind: Deployment
metadata:
  name: system-info-web
spec:
  selector:
    matchLabels:
      app: system-info-web
  replicas: 1
  template:
    metadata:
      labels:
        app: system-info-web
    spec:
      containers:
      - name: system-info-web
        image: registry.${DOMAIN}/system-info-web:latest
        ports:
        - containerPort: 8000
      imagePullSecrets:
      - name: secret-docker

Then create the middleware

cat basic-auth-registry-middleware.yaml | envsubst | kubectl apply -f -

We are planning to use a simple registry-ui for the registry. That requires us to use a cors middleware. This could be done in the registry settings as well, but its convenient to use the traefik ingress to solve this.

cat registry-cors-middleware.yaml | envsubst | kubectl apply -f -

Then we create the registry service with cors middleware. Authentication is handled by the registry auth. (We could optionally add basic auth on the UI itself, but it would require us to enter two sets of credentials. One for the UI and one for the registry)

cat registry-ephemeral.yaml | envsubst | kubectl apply -f -

Finally the registry-ui (optional)

cat registry-ui.yaml | envsubst | kubectl apply -f -

For the registry UI, we will use the same basic auth middleware as for the registry, as the registry-ui forwards the credentials. If you use a different basic auth middleware here then you will best case have to enter two sets of credentials. So its better to just have the same as the registry.

Test the registry

On any machine with docker:

docker login registry.${DOMAIN}
# Username: username
# Password: password
# (The registry credentials)

docker pull nginx:latest

docker tag nginx:latest registry.dog.example.com/nginy:later

docker push registry.dog.example.com/nginy:later

The push refers to repository [registry.mydomain.com/nginy]
f0f30197ccf94: Pushed
e23424ff930d4: Pushed
c9732d7567184: Pushed
4b8db2d43325a: Pushed
4312341684c4a: Pushed
02c055fgh42f5: Pushed
latest: digest: sha256:eba373a0620f68ffdc3f4327041ad25e2342175b8feb35b992574cd83698e9e3c size: 1570

Add persistance

If everything looks allright you might want to consider adding some persistance to that registry.

Delete the ephememeral deployment/service/ingress

cat registry-ephemeral.yaml | envsubst | kubectl delete -f -

Create the longhorn (or localhost) PVC

cat pvc-registry-longhorn.yaml | envsubst | kubectl apply -f -

Create the longhorn (or localhost) deployment/service/ingress

cat registry-longhorn.yaml | envsubst | kubectl apply -f -

Build, push and deploy to the registry

Install docker

Install Docker following the official guide...
...or alternatively, run the official convenience script:

# Download Docker
curl -fsSL get.docker.com -o get-docker.sh
# Install Docker using the stable channel (instead of the default "edge")
CHANNEL=stable sh get-docker.sh
# Remove Docker install script
rm get-docker.sh

Build the container

Go the the /apps/system-info-web folder

cd ../apps/system-info-web/

Build the container

docker build -t registry.${DOMAIN}/system-info-web .

Push the container

docker push registry.${DOMAIN}/system-info-web

Deploy it
Go back to the manifests folder if neccesary

cd ../../manifests

Deploy

cat system-info-web.yaml | envsubst | kubectl apply -f -

Check

Go to http://sysinfo.dog.example To see if it has worked.

system-info-web

Note: See the value under "hostname" which is the name of the pod running the service. In the deployment yaml it has been specified with 2 replicas. Try refreshing and see that this value will change as the ingress is now load balancing between the two replicas.