Ryuk the Resource Reaper

Peter Steiner
4 Min To Read
04 Jan, 2023
- Development and programming

Ryuk

When you don’t know who (or what) Ryuk is, then you’re not alone. I first encountered this name when I perused the Testcontainers documentation (configuration and lifecycle control ), but the documention doesn’t really introduce it.

Only when I googled about a problem I had with the Ryuk container I noticed results about a manga series Death Note , where Ryuk is a “Death God” (shinigami), having a “Death Note”, a notebook where writing the name of somebody into it has the effect of causing the death of this person.

This is quite an accurate description of the Ryuk container of the Testcontainers project. You send a label to Ryuk’s notebook (example from the README.md: label=something), and at the end of a Testcontainers test suite, when containers/networks/volumes/images are not needed anymore, Ryuk removes all these resources carrying the labels written in its notebook (the Death Note).

The Problem of an old Docker Version with Ryuk

And here is the link to my problem. For removing container images Ryuk uses the filters parameter of the Docker ImagePrune API to delete all images with the label in the Death Note. Unfortunately though the CI/CD build server I’m using (no, it’s not in a Worldline data center…) has a very old Docker version (1.13 from 2017), with API v1.26 where the label parameter doesn’t exist yet. The absence of the label parameter makes Docker remove ALL images, a little bit like the population of a whole city was noted in Ryuk’s notebook.

Of course I didn’t know all this when the problem first made my Jenkins pipeline fail. In a nutshell my pipeline creates a docker image of my service and then uses Testcontainers to run tests against this image (temporarily deployed together with Kafka and database containers). Only when all tests pass the docker image can travel down the pipeline and ultimately be put into production. This sequence guarantees that the same image is deployed that passed the tests.

So when all tests passed on my local PC and I started the Jenkins pipeline, I didn’t understand why the pipeline failed to deploy my service on the test environment even though all the tests were green. It took quite some time and reading too many Jenkins logs to understand why the deployment failed: the docker image was missing! A couple of internet searches later I found this stackoverflow question that pointed at Ryuk, removing this image with some old versions of Docker. My immediate workaround to fix my pipeline was to disable Ryuk.

A Workaround for the Problem

Trying to be a good open source citizen, I asked if Ryuk should be made configurable with respect to image pruning. Basically the maintainers answered that it makes not much sense to support a workaround for a five-year old Docker version. To be honest I would have written the same if I was a maintainer…

The advice from the Testcontainers maintainers was to use a patched Ryuk image instead of disabling Ryuk:

We strongly advise against disabling Ryuk as Testcontainers maintainers. This is often done out of cargo-cult reasons and consequences, especially on CI, are not always well understood.

So here is my patched Ryuk: you can use my pre-built image from Docker Hub or build it yourself (replace pesche with your docker hub user name):

# clone my fork of the Ryuk repo
git clone -b feature/no-image-pruning https://github.com/pe-st/moby-ryuk.git
cd moby-ryuk
docker build -t pesche/ryuk-patched:0.3.4 .
docker tag pesche/ryuk-patched:0.3.4 pesche/ryuk-patched:latest
docker login
docker push pesche/ryuk-patched:0.3.4
docker push pesche/ryuk-patched:latest

Note the current official image is testcontainers/ryuk:0.3.4 (released 2022-07-28).

To use a different Ryuk image in your Java project than the official one the endorsed way is to write a Java class. Here’s the one I use (if you don’t use Slf4j and/or Lombok, you can remove the @Slf4j annotation and the log.xxx() statements):

@Slf4j
public class RyukImageNameSubstitutor extends ImageNameSubstitutor {

    private final TestcontainersConfiguration configuration = TestcontainersConfiguration.getInstance();

    @Override
    public DockerImageName apply(DockerImageName original) {
        final String configuredRyukImage = configuration.getEnvVarOrProperty("substitutor.ryuk.container.image", "");

        if ("testcontainers/ryuk".equals(original.getRepository()) && "".equals(original.getRegistry())) {
            if (configuredRyukImage.isEmpty()) {
                // if you have configured the RyukImageNameSubstitutor class you probably want also a substitute configured
                log.info("No ryuk substitute is configured");
                return original;
            }

            log.debug("Substituting testcontainers/ryuk with {}", configuredRyukImage);
            return DockerImageName.parse(configuredRyukImage);
        }
        return original;
    }

    @Override
    protected String getDescription() {
        return getClass().getSimpleName();
    }
}

Finally you have to configure Testcontainers to use this class, by adding a testcontainers.properties file to the classpath (for my Maven project I have put it into src/test/resources), with this content:

# use a patched Ryuk because of the very old Docker on our Jenkinses
image.substitutor=com.company.myproject.RyukImageNameSubstitutor
# from Docker Hub: https://hub.docker.com/r/pesche/ryuk-patched
substitutor.ryuk.container.image = pesche/ryuk-patched:latest

Conclusion

If the tests for your service/application need external resources like a database or a messaging system, use these resources from a (Docker) container and control them with Testcontainers, it is a very flexible solution. It is supple enough to adapt to a situation where you’re stuck with a very old Docker version: just configure which Ryuk image to use; a Ryuk image for this situation is freely available (see above) but you can also build your own very easily.