Docker

Build - Dockerfile

  • For each instruction in Dockerfile, Docker looks to see if it already has an image layer for that instruction in its cache. If it does, this is a cache hit and it uses that layer. If it doesn’t, this is a cache miss and it builds a new layer from the instruction. Getting cache hits can hugely speed up the build process.
  • Use multi-stage builds for smaller images and better security
  • As soon as any instruction results in a cache-miss (no layer was found for that instruction), the cache is no longer used for the rest of the entire build.
  • COPY and ADD instructions use file checksums to ensure that the content being copied into the image has not changed since the last build.

Dockerfile - Commands

FROM

  • The first instruction must be FROM, which sets the base image to start from.

  • Multi-stage build can have multiple FROM statements.

ARG

  • Define variables to be used in Dockerfile.

  • Variable values can be supplied at build time via CLI:

    # e.g.
dbl --build-arg JAVA=jdk

WORKDIR

  • Performs mkdir and cd implicitly in the image

COPY

ADD

  • Officially discouraged to use
  • Source can be a local file or directory, and a directory will be copied recursively.
  • Source can also be a URL.
  • A compressed archive as source can be automatically extracted in the given destination.
  • The src path must be inside the context of the build; you cannot ADD ../something /something, because the first step of a docker build is to send the context directory (and subdirectories) to the docker daemon.

CMD

  • Default command for when running a container
  • If you list more than one CMD then only the last CMD will take effect.
  • exec form is preferred over shell form.
  • Typically used to supply default parameters to ENTRYPOINT, which can be overridden from CLI parameters
  • If ENTRYPOINT is absent, CMD will be used as a command.

ENTRYPOINT

  • The ENTRYPOINT instruction sets the executable to run at container startup.
  • Docker has a default ENTRYPOINT which is /bin/sh -c
  • exec form is preferred over shell form, but it does not invoke a command shell. This means that normal shell processing such as variable and command substitution does not happen.
  • Usually not overriden at runtime, can only be overridden with --entrypoint flag

RUN

  • The RUN instruction will execute any commands in a new layer on top of the current image and commit the results. The resulting committed image will be used for the next step in the Dockerfile.

  • exec form: RUN ["/bin/bash", "-c", "echo hello"]

    The exec form is parsed as a JSON array, which means that you must use double-quotes (") around words not single-quotes (').

LABEL

  • The LABEL instruction adds key/value pairs to the image’s metadata.

EXPOSE

  • The port on which the container listens on at runtime.
  • Informational purpose, does not actually publish the port

VOLUME

Dockerfile - Debug build

  • Comment out the step that fails.
  • Build the Dockerfile again.
  • You will get an image with all the layers before the failed one.
  • Use dive to examine any step in question or use docker run -it to navigate the container.

Build - CLI

Build an image

d build -t $repo_name/$image_name:$image_tag $relative_dockerfile_path

Build an image with BuildKit

  • One-off

    • Run the build command with environment variable DOCKER_BUILDKIT=1

      e.g. $ DOCKER_BUILDKIT=1 docker build .

      Use DOCKER_BUILDKIT=0 will disable BuildKit.

    • Alternatively, use Docker Buildx CLI plugin

      e.g. $ docker buildx build .

  • Permanent

    • Linux: /etc/docker/daemon.json

      {
  "features": {
    "buildkit": true
  }
}

    • Windows / macOS: Use Docker Desktop settings

  • Resources

Display build logs with BuildKit

  • Run the build command with environment variable BUILDKIT_PROGRESS=plain

    e.g. BUILDKIT_PROGRESS=plain docker-compose build

Tag a local image

d tag $SOURCE_IMAGE $TARGET_IMAGE

To push a new tag to a repository

d push $user_name/$repo_name:$image_tag

Remove build cache

d builder prune

Build - OCI Image

Docker in Docker

Cloud Native Buildpacks

Suggest builder based on existing code base

pack builder suggest

Build an image with pack CLI

pack build $image_name --builder $builder_name

Jib (Java)

Client - CLI

List images

d images / d image ls

  • Options

    • -a

      List all images

    • -q

      Display image ID only

  • Ascending order by image size

    dils | sort -k7 -h

  • Ascending order by image name

    dils | sort -k1

List images with filter(s)

  • Example 1

    d images -f reference='confluentinc/*:*'

    If filtering with multiple reference, output would give, either match A or B.

    REPOSITORY                                  TAG       IMAGE ID       CREATED        SIZE
confluentinc/confluent-cli                  latest    734463176a4d   2 weeks ago    275MB
confluentinc/cp-ksqldb-server               7.2.1     140d2ac32177   3 months ago   1.36GB
confluentinc/cp-ksqldb-cli                  7.2.1     287039530a46   3 months ago   857MB
confluentinc/cp-schema-registry             7.2.1     afaac043dcc1   3 months ago   1.86GB
confluentinc/cp-enterprise-control-center   7.2.1     3ef895a26b4e   3 months ago   1.23GB
confluentinc/cp-server                      7.2.1     2fa77493d25b   3 months ago   1.67GB
confluentinc/cp-zookeeper                   7.2.1     3f28db6a433d   3 months ago   782MB
confluentinc/cp-kafka-rest                  7.2.1     784b8061ad0c   3 months ago   1.76GB
confluentinc/ksqldb-server                  0.8.0     e67283e9949c   2 years ago    667MB

  • Example 2

    d images -f reference='docker.elastic.co/*/*:*'

    Asterisk cannot represent slash.

    REPOSITORY                                      TAG       IMAGE ID       CREATED       SIZE
docker.elastic.co/elasticsearch/elasticsearch   8.4.3     ce2b9dc7fe85   2 weeks ago   1.26GB
docker.elastic.co/kibana/kibana                 8.4.3     b14d91e49f3f   3 weeks ago   800MB
docker.elastic.co/apm/apm-server                8.4.3     81b0ed053a8a   3 weeks ago   230MB
docker.elastic.co/elasticsearch/elasticsearch   6.3.0     56d3dc08212d   4 years ago   783MB

Remove image(s)

d rmi $IMAGE_ID

Remove multiple images

d rmi $image1_id $image2_id ...

List all containers

d ps -a / d container ls -a
CONTAINER ID   IMAGE           COMMAND                  CREATED         STATUS         PORTS                  NAMES
63cd9c728e0f   nginx:1.17.10   "nginx -g 'daemon of…"   4 seconds ago   Up 3 seconds   0.0.0.0:9332->80/tcp   sharp_hofstadter
 
PORTS:
<HOST_IP:HOST_PORT>-><CONTAINER_PORT>/<PROTOCOL>

Only display container ID

dps -q

Run a container

  • If a command is supplied, the specified command will be invoked.

  • If no command is supplied, the command specified by CMD instruction will be invoked.

  • Image name and tag must follow the options

  • Options

    • -d

      Run container in background and print container ID

    • -p <HOST_IP>:<HOST_PORT>:<CONTAINER_PORT>/<PROTOCOL>

      Expose container port to host, eg: -p 127.0.0.1:80:8080/tcp

    • -e

      Set environment variables

    • --rm

      Automatically remove the container when it exits

docker run -d -p <[HOST_IP:]HOST_PORT>:<CONTAINER_PORT> --name <CONTAINER_NAME> -e <KEY>=<VALUE> <IMAGE>:<TAG> <COMMAND> <ARGS...>

Start a container

d start $container_name_or_id

Stop a container

d stop $container_name_or_id

Remove container(s)

# Some running containers need to be stopped before they can be removed.
d rm $container_name_or_id ...

Log into a container

d exec -it $container_name_or_id bash

Make permanent changes to a container

  1. docker exec -it $CONTAINER_NAME / $CONTAINER_ID bash
  2. apt install vim
  3. docker container commit

Stop a container from auto restarting

d update --restart=no $container_name_or_id

List ports of a container

d port $container_name

Display command line of a container

d container inspect $container_name | jq '.[].Config'

Copy files between a container and host

  • Docker volumes cannot be accessed directly from host. They can only be mounted to containers. Alternatively, use docker cp to copy files in and out.

  • d cp <local_path> <CONTAINER>:<path>

    From host to container

  • d cp <CONTAINER>:<path> <local_path>

    From container to host

  • Files can only be copied from container and host, if there's no container, create a temp one by d create --name <container-name> <IMAGE>

Hosting a local repository (opens in a new tab)

Start the registry automatically by using --restart=always

dr -d -p 5000:5000 --restart=always --name registry registry:2

Push an image to the local registry running at localhost:5000

d tag ubuntu:16.04 localhost:5000/my-ubuntu
 
d push localhost:5000/my-ubuntu

Registry authentication

  • Authenticated registries are listed in $HOME/.docker/config.json

  • To reauthenticate, docker login <registry_URL>

Export an image to a TAR file

d save $IMAGE_ID > $IMAGE_TAR_FILE

Save and load image(s) to and from a TAR file, optionally with compression

  • Note: image Tar only contains image layers, to examine files in image, need to use d export with an existing container.

  • Supported compression types:

    • gzip
    • bzip2
    • xz
# Export image(s) as a TAR file
> d save $image > ${file.tar}
 
# Export image(s) as a Gzipped TAR file
> d save $image | gzip > ${file}.tar.gz
 
# Load image(s) from a TAR file
> d load < ${file.tar}
 
# Load image(s) from a Gzipped TAR file
> d load < ${file.tar.gz}

docker save command with different compression types

docker load command with a Gzipped TAR archive

  • Note: Use repository:tag to reference Image, because using Image ID will make loaded image lose repository and tag.

Save and load images to STDOUT and from STDIN

d save $image | bzip2 | ssh $user@$host docker load

Save and load multiple images

# Retrieve all target image names (repo:tag)
> export IMAGES=$(d images \
-f reference='openapitools/*:*' \
-f reference='openzipkin/*:*' \
-f reference='pack.local/*/*:*' \
-f reference='jupyter/*:*' \
-f reference='paketobuild*/*:*' \
-f reference='jenkins/*:*' \
-f reference='grafana/*:*' \
-f reference='docker*/*/*:*' \
--format="{{.Repository}}:{{.Tag}}")
 
# Save images to TAR archive
> d save $IMAGES > ${file.tar}

Examine file system in a image/container

  • Prepare the container of the image, if doesn't exist already

  • Export a container’s filesystem as a TAR archive

    d export $container_name > ${file.tar}

Docker contexts - List available contexts

# The current one has asterisk after its name.
d context ls

Docker contexts - Use another context

d context use $context_name

or

# This will override the current context
export DOCKER_CONTEXT=$context_name

Docker contexts - Create a new context

d context create ${context_name} \
  --description ${description} \
  --docker "host=$docker_endpoint"

Docker contexts - Update a context

d context update $context_name --description "$description"

Connect to a specified Docker daemon with different protocols

d -H $docker_endpoint $command

or

Use DOCKER_HOST environment variable

  • Protocols

    • Unix domain socket: unix:///var/run/docker.sock
    • Systemd socket activation: fd://
    • TCP: tcp://192.168.59.106
    • SSH: ssh://me@example.com:22

List BOM of an image

d sbom $image:$tag

Check if a container is running

d ps -q -f name=$container_name

If the container is running, container ID will be printed, otherwise no output.

Check the current storage driver

d info | grep -i storage

Display the logs of a container

  • Display logs in stdout

    d logs $container_name

    or

    dlo $container_name

  • Save the logs to a file

    d inspect --format='{{.LogPath}}' $container_name

    or

    dcin --format='{{.LogPath}}' $container_name

Display disk usage by Docker daemon

d system df

Display disk usage of different components in details

d system df -v

Display disk usage of all volumes sorted by size

d system df --verbose --format '{{ range .Volumes }}{{ .Name }} {{ .Size }}\n{{ end }}' | sort -k2 -h | column -t

Display volumes and mount points

d volume ls -q | xargs docker volume inspect --format '{{.Name}} {{.Mountpoint}}' | sort -k2 | column -t

Clean up build cache

d builder prune

docker - Send a SIGHUP to a container

d kill -s HUP $container_name

docker - Run curl in a container for troubleshooting

drit --rm curlimages/curl $arguments
# e.g. drit --rm curlimages/curl -s -v -X GET http://$host:$port

Client - Docker Compose

Add DNS entries for containers

kafka-http-cli-mvn:
  image: cq-playground/pm-kafka-http-cli:latest
  build:
    extra_hosts:
      host.docker.internal: 172.17.0.1

Update image of a container with minimal downtime

  1. Update image tag

  2. Restart container

    dcup -d --no-deps --build $service_name

Externalize environment variables

Server - Docker Daemon

  • One daemon can expose multiple unix sockets
  • Can be protected with TLS authentication

Daemon - Storage

Daemon - Check which Daemon is active

# List all contexts, including the active one with **asterisk**
$ d context ls
 
NAME            DESCRIPTION                               DOCKER ENDPOINT                             ERROR
default *       Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
desktop-linux   Docker Desktop                            npipe:////./pipe/dockerDesktopLinuxEngine
docker-ce       Docker CE                                 unix:///run/docker.sock
# Inspect the active context
# In this case, assuming it is `default`
$ d context inspect default | jq
 
[
  {
    "Name": "default",
    "Metadata": {},
    "Endpoints": {
      "docker": {
        "Host": "unix:///var/run/docker.sock",
        "SkipTLSVerify": false
      }
    },
    "TLSMaterial": {},
    "Storage": {
      "MetadataPath": "<IN MEMORY>",
      "TLSPath": "<IN MEMORY>"
    }
  }
]
  • DOCKER_HOST can be used to override the active context. DOCKER_HOST must be set with an end point of the daemon.

Daemon - Get the current active daemon socket

d context ls --format json | jq -r 'select(.Current == true) | .DockerEndpoint'
 
> unix:///var/run/docker.sock

Daemon - Test if a Daemon socket is accessible

# Suppose daemon socket is `/var/run/docker.sock`
$ curl -s -G --unix-socket /var/run/docker.sock http://localhost/version | jq
{
  "Platform": {
    "Name": "Docker Desktop 4.0.0 ()"
  },
  "Components": [
    {
      "Name": "Engine",
      "Version": "27.4.0",
      "Details": {
        "ApiVersion": "1.47",
        "Arch": "amd64",
        "BuildTime": "2024-12-07T10:38:57.000000000+00:00",
        "Experimental": "false",
        "GitCommit": "92a8393",
        "GoVersion": "go1.22.10",
        "KernelVersion": "5.15.167.4-microsoft-standard-WSL2",
        "MinAPIVersion": "1.24",
        "Os": "linux"
      }
    },
    {
      "Name": "containerd",
      "Version": "1.7.21",
      "Details": {
        "GitCommit": "472731909fa34bd7bc9c087e4c27943f9835f111"
      }
    },
    {
      "Name": "runc",
      "Version": "1.1.13",
      "Details": {
        "GitCommit": "v1.1.13-0-g58aa920"
      }
    },
    {
      "Name": "docker-init",
      "Version": "0.19.0",
      "Details": {
        "GitCommit": "de40ad0"
      }
    }
  ],
  "Version": "27.4.0",
  "ApiVersion": "1.47",
  "MinAPIVersion": "1.24",
  "GitCommit": "92a8393",
  "GoVersion": "go1.22.10",
  "Os": "linux",
  "Arch": "amd64",
  "KernelVersion": "5.15.167.4-microsoft-standard-WSL2",
  "BuildTime": "2024-12-07T10:38:57.000000000+00:00"
}

Daemon - Access Docker Engine HTTP API

curl -s -G --unix-socket $docker_daemon_socket $docker_engine_api_endpoint | jq
 
# e.g.
curl -s -G --unix-socket /var/run/docker.sock http://localhost/version | jq

Daemon - Display daemon details

curl -s -G --unix-socket $docker_daemon_socket http://localhost/info | jq

Access host from a container

Configure host gateway IP

Installation

Docker Engine

Docker Desktop

WSL

Tools

Tools - dive (opens in a new tab)

  • Key Bindings

    Key BindingDescription
    Ctrl + C or QExit
    TabSwitch between sections
    Ctrl + FFilter files
    ESCClose filter files
    Page Up or UScroll up a page
    Page Down or DScroll down a page
    Arrow Up or KMove up one line within a page
    Arrow Down or JMove down one line within a page
    Ctrl + ALayer view: see aggregated image modifications
    Ctrl + LLayer view: see current layer modifications
    SpaceFiletree view: collapse/uncollapse a directory
    Ctrl + SpaceFiletree view: collapse/uncollapse all directories
    Ctrl + AFiletree view: show/hide added files
    Ctrl + RFiletree view: show/hide removed files
    Ctrl + MFiletree view: show/hide modified files
    Ctrl + UFiletree view: show/hide unmodified files
    Ctrl + BFiletree view: show/hide file attributes
    Page Up or UFiletree view: scroll up a page
    Page Down or DFiletree view: scroll down a page

dive - Run dive Docker image

docker run --rm -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    docker.io/wagoodman/dive $dive_arguments

dive - Inspect a Docker image with tag

dive $image_name:$tag

dive - Inspect a Docker image with ID

dive $image_id

dive - Inspect a Docker image with an image TAR archive

Image TAR archive can be created by docker save command.

dive --source docker-archive $path_to_image_tar_archive

Troubleshooting

Integration

Testcontainers

  • For integration with Java projects, you can achieve similar effects to Docker Compose from Java code.
  • Use methods of org.testcontainers.containers.ContainerState to confirm container state.
  • Wait.forHealthcheck() refers to Dokcer's Healthcheck support. Image must implement Docker HEALTHCHECK (opens in a new tab) for ContainerState.isHealthy method to work.
  • Exposed ports will be mapped to random ports, and those random ports can be retrieved by ContainerState.getMappedPort. Mapped random ports are the ports via which your other services would be communicating with this container. Therefore the config of port numbers must be specified dynamically to align with the randomness.

Testcontainers - Cheatsheet

Start containers in parallel
  • Using Startables.deepStart(container1, container2, ...).join() will start all containers in parallel.
Expose specific ports of a container
Get the mapped host port for the exposed container port
Explicitly map a host port to a container port

Testcontainers - Troubleshoting

Docker Compose integration
  • Not enough documentation
  • Containers end before the test starts
  • Vanilla docker image works fine but not with Docker Compose

Testcontainers - Spring Boot

Apache Maven

  • Use Maven Docker image (opens in a new tab)

    • Pros

      • settings.xml can be deployed together in the image, therefore simplifying repo config.

    • Cons

      • Caching artifacts

        • Options

          • Create a Docker volume with .m2 directory in it and share the volume between containers

Resources

Fast Chicken and Slaw TacosPodman