I recently found myself with only 59GB free on a 1TB drive. That’s not great. I had a suspicion about where it all went, I’ve been here before.

Finding the culprit

As I said - I’ve been through this before. Last time I found that Docker’s build cache had ballooned to 284GB. Just the build cache. I cleaned that up and compacted the virtual hard disks and got a decent chunk back. But here I was again, running low.

A quick check:

docker system df

My results:

93GB in images, 53GB in container layers, 140GB in volumes. That’s kind of a lot. But the build cache was already at 0B this time - I had pruned that previously.

The containers

40 containers, none running. Most are stopped devcontainers from various projects I work on. Each one accumulates writable data over time - installed packages, build artifacts, caches. Some had several gigabytes of writable data.

I listed them with their last-used dates and image names to figure out which ones I still needed:

docker ps -a \
    --format '' \
  | while read name; do
    docker inspect "$name" \
    | jq -r '.[0] |
      ([.State.FinishedAt,
        .State.StartedAt]
       | max
       | split("T")[0])
      + "|" +
      (.Created
       | split("T")[0])
      + "|" + "'"$name"'"
      + "|" +
      .Config.Image'
  done | sort

Devcontainer images are named vsc-[projectname]-[hash], so it’s easy to tell what’s what. I had containers from 8 months ago that I’d completely forgotten about.

I went through and removed the ones I didn’t need anymore. Important: don’t just blindly delete everything. If you use devcontainers, your stopped containers hold state you might want. I kept anything I’d used in the last couple of weeks and anything for active projects.

Shared volumes

Something I learned during this cleanup: many devcontainers share volumes. Things like shell-history, minikube-config, and the vscode volume itself are often shared across multiple containers. This is actually probably bugs in how we’ve set up our devcontainers with named volumes to have things survive rebuilds (hint: make the project-name part of the volume-name).

This does mean that if you have three iterations of a python project as separate containers, they might all share the same shell history and config volumes. Deleting the older containers won’t lose that data - the volumes persist as long as at least one container still references them.

You can check what volumes each container uses:

docker ps -a \
    --format '' \
  | while read c; do
    echo "=== $c ==="
    docker inspect "$c" \
    | jq -r '
      .[0].Mounts[] |
      "    \(.Type)
        \(.Name // "-")
        \(.Destination)"'
    echo
  done

Pruning carefully

After removing old containers, I pruned in stages:

# images not referenced by containers
docker image prune -a -f

# volumes not referenced by containers
docker volume prune -f

# Remove build cache
# do this LAST, it tends to re-appear
docker builder prune -a -f

One thing that surprised me: build cache keeps re-appearing. After pruning images, suddenly there’s build cache again. Is this the build-cache from the image that’s being kept aroudn? I don’t know. I ended up running the builder prune multiple times. Run docker system df between rounds to see if more has surfaced.

After all the pruning I’d freed about 144GB inside Docker. Images went from 93GB to 53GB, containers from 53GB to 24GB, volumes from 140GB to 119GB, and build cache gave up about 52GB across several rounds.

The vhdx file you don’t know about

Here’s the thing that tripped me up last time, and that I only discovered properly this time: Docker on Windows uses multiple virtual hard disk files, and you need to compact all of them.

I knew about these two:

• C:\Users\...\AppData\Local\Docker\wsl\main\ext4.vhdx - the Docker engine
• C:\Users\...\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu_...\LocalState\ext4.vhdx - Ubuntu WSL

But there was a third one:

• C:\Users\...\AppData\Local\Docker\wsl\disk\docker_data.vhdx - the actual Docker data

That third file was 555GB. Yes, really. The main/ext4.vhdx that I’d been diligently compacting was tiny in comparison. All the container layers, images, volumes - it all lived in docker_data.vhdx.

The names and locations of the virtual hard-drives may vary by setup. Find all of them with:

find /c/Users/<YourUsername>/AppData \
  -name "*.vhdx"

Compacting the vhdx files

The virtual hard disks grow as Docker writes data, but they do not automagically shrink. Even after you delete data inside Docker, the vhdx file stays the same size on your Windows drive. You have to compact them manually.

Here’s the process, and the order matters:

1. Kill Docker Desktop first

docker desktop stop tended to hang forever on my machine. If it works for you - great! If it doesn’t, instead:

1. Open Task Manager, find Docker Desktop, end the process tree
2. Then run wsl --shutdown

The order matters. Docker Desktop keeps a file handle on the vhdx files even after WSL shuts down. If you don’t kill Docker Desktop first, diskpart won’t be able to open the files. Then again, that may only have been for my never-ending docker desktop process. Not sure.

2. Compact in diskpart

Open diskpart.exe (it needs admin rights, and will open a new terminal): For each of the virtual hard-drive -files you’ve found do:

select vdisk file="path_to_the_vhdx"
compact vdisk

The docker_data.vhdx took the longest - mine was at 555GB and took a while (time for lunch). If it gets stuck partway, leave it for a while, if it doesn’t recover - Ctrl+C to cancel that (you can also restart the compaction if you want).

3. Start back up

wsl
docker desktop start

Verify everything is still there with docker system df.

Was it worth it?

In my case I recovered about 380GB of space on my C-drive. Pretty good, and definitely necessary when I was down to just over 10GB before I started.

Without compaction all the space freed inside Docker will be reused by Docker instead of it allocating new blocks and growing the vhdx further, so this will buy you time before you need to do it again or buy a bigger drive. With compaction you get the space back on your drive, but it will be used up again as you use Docker.

If you have the time to clean up and compact it, it’s worth it.

The checklist

For my future self, and for anyone else running lots of devcontainers on Windows:

1. Run docker system df to see where the space is
2. List containers by last used date, remove ones you don’t need
3. Check for shared volumes before deleting containers
4. docker image prune -a -f
5. docker volume prune -f
6. docker builder prune -a -f (run this last, and maybe twice)
7. Kill Docker Desktop process tree, then wsl --shutdown
8. Find all vhdx files - especially docker_data.vhdx
9. Compact all of them in diskpart.exe
10. Start WSL and Docker Desktop back up
11. Verify with docker system df

Docker’s disk usage is hidden behind virtual hard disks that only grow and never shrink. And the biggest one might not be the one you’d expect, go looking through your disk for others!

Ironic then, that the devcontainer CLI tool needs Node, NPM, Python and C/C++ (in particular versions). Yes, I’m not kidding. This is a travesty, and should not be allowed to stand! It is, in fact, registered as an issue, which has been open since summer 2022. That is creeping up on 4 years ago, at the time of writing. Safe to say - it’s not going to be fixed.

Personally, I don’t really care all that much - I use devcontainers through VS Code and that handles all interactions with them for me. But, it is wrong, dammit. Just plain wrong!

There should be only one dependency for devcontainers, and that is Docker! Well, Docker and Linux, but that’s a given. And a computer. And the internet. Oh, gods, we’re depending on the internet again. Well, let’s get this exercise in futility on the road:

The Devcontainer CLI

Well, how do we get the devcontainer CLI without installing it? Like any programmer I’m a fan of recursion, so immediately I thought - we run it in a devcontainer! Silly, right? Without a devcontainer CLI we cannot run a devcontainer. True, but we can step down one level and run it in a docker container. In fact, one of the common devcontainers already contain most of what we need to run the devcontainer CLI: mcr.microsoft.com/vscode/devcontainers/typescript-node.

Let’s just try it out - and see if we can run the devcontainer CLI from there:

host $ docker run --rm -it \
    --entrypoint /bin/bash \
    mcr.microsoft.com/vscode/devcontainers/typescript-node
root ➜ / $ npm install -g @devcontainers/cli
# installation logging...
root ➜ / $ devcontainer --version
0.83.0

Good, that worked! We can indeed install the devcontainer CLI in the container and run it. Now, let’s make a docker image out of this, so we don’t have to install the devcontainer CLI every time, so we make this Dockerfile:

FROM mcr.microsoft.com/vscode/devcontainers/typescript-node
RUN npm install -g @devcontainers/cli
ENTRYPOINT ["devcontainer"]

And build it:

host $ docker build -t devcontainer-cli .

Great - now we have a docker image with the devcontainer CLI installed. Let’s run it:

host $ docker run --rm -it devcontainer-cli --version
0.83.0

That’s cool, but we want to run it as if it was a tool on our host machine (without installing), so we need to give the running container access to our host’s Docker daemon and file-system. To make it easier to call, we also put it inside a bash-script instead of typing it out every time (we are lazy, after all). Here’s the script (called dc.sh):

#!/bin/bash
docker run --rm -it \  # run, but remove when done
    -v /var/run/docker.sock:/var/run/docker.sock \ # give access host docker
    -v "$(pwd)":"$(pwd)" \ # access current directory (and subdirectories)
    -w "$(pwd)" \ # set working directory
    devcontainer-cli "$@" # pass all arguments to container

This script mounts the docker socket, so the container can talk to the host’s Docker daemon. It also mounts the current directory (and subdirectories) into the container, and sets the working directory to the current directory. Finally, it passes all arguments to the devcontainer command inside the container. With it we should be able to run the devcontainer CLI as if it was installed on our host machine, without actually installing it.

host $ chmod +x dc.sh
host $ ./dc.sh --version
0.83.0

Excellent!

One small snag

Very cool so far, let’s try it out:

host $ mkdir test-devcontainer
host $ cd test-devcontainer
host $ ../dc.sh up
# unhappy logs about not being able to find a devcontainer.json file

Of course! We need to tell the devcontainer CLI what do start!

host $ mkdir .devcontainer
host $ echo '{
    "name": "Test Devcontainer",
    "image": "mcr.microsoft.com/devcontainers/base:ubuntu"
}' > .devcontainer/devcontainer.json
host $ ../dc.sh up
[4 ms] @devcontainers/cli 0.83.0. Node.js v24.13.0. linux 6.6.87.2-microsoft-standard-WSL2 x64.
Error: spawn docker ENOENT
    at ChildProcess._handle.onexit (node:internal/child_process:286:19)
    at onErrorNT (node:internal/child_process:484:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:89:21)
{"outcome":"error","message":"spawn docker ENOENT","description":"An error occurred setting up the container."}

And this is where we hit a snag. Turns out the container we based off of doesn’t have Docker installed, so it can’t talk to the Docker daemon. Easily fixed - we just modify our Dockerfile to install Docker:

FROM mcr.microsoft.com/vscode/devcontainers/typescript-node

# Install docker and clean up after
RUN apt-get update && apt-get install -y docker.io && rm -rf /var/lib/apt/lists/*

# Install devcontainer CLI
RUN npm install -g @devcontainers/cli

ENTRYPOINT ["devcontainer"]

Now we build the image again, and try to run it:

host $ docker build -t devcontainer-cli .
host $ ../dc.sh up
[2 ms] @devcontainers/cli 0.83.0. Node.js v24.13.0. linux 6.6.87.2-microsoft-standard-WSL2 x64.
[+] Building 0.8s (6/6) FINISHED
# chatty logs omitted..
Container started
{"outcome":"success","containerId":"c55f3e53963bcc08f8e6484e79334d219b47757d9ed3daf05841e487e516fc3f","remoteUser":"vscode","remoteWorkspaceFolder":"/workspaces/setup"}
host $ docker ps
# list of containers running - one of which should have the containerId from above

And there we have it! The devcontainer CLI is running, and it can talk to the Docker daemon on our host machine, and it can see our files. We can run all devcontainer CLI commands as if it was installed on our host machine, without actually installing it.

Conclusion

This was a fun little exercise showing how we can use containers to run tools without installing them on our host machine. The devcontainer CLI is just one example, but this approach can be used for any tool that can run in a container. We just need to make sure the container has access to the necessary resources (like the Docker daemon and the file system) to do its job.

Personally I haven’t really used the devcontainer CLI outside of VS Code, but it’s good to know that we can run it without installing it, and that we can even run it in a container if we want to.

For developers who want to use devcontainers without VS Code, maybe directly from the command line this could be helpful. Or for CI/CD pipelines that want to use devcontainers without installing the CLI on the build agents. There must be dozens of you out there!

For me it just scratches the itch of the devcontainer CLI (a tool that I use to avoid installing things on my host machine) requiring me to install things on my host machine.

Until next time!

Addendum: Docker socket permissions on Linux

If you’re trying this on a pure Linux host (as opposed to Docker Desktop on macOS or Windows), you might get a permission-problem when the container tries to talk to the Docker socket. This is because the socket is owned by the docker group on your host, and the user inside the container isn’t a member of that group.

The fix is to tell Docker to add the host’s docker group to the container process. Update dc.sh to:

#!/bin/bash
docker run --rm -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v "$(pwd)":"$(pwd)" \
    -w "$(pwd)" \
    --group-add $(stat -c '%g' /var/run/docker.sock) \
    devcontainer-cli "$@"

The stat -c '%g' command gets the group-ID of the socket file, and --group-add grants that group to the container’s process. Now the container should have access to write to the socket without running as root or changing permissions on the host.

I spent my Sunday trying to replace Docker Desktop with Podman. It didn’t work out. Here’s what happened, what I learned, and why I’m back on Docker Desktop.

The motivation

Docker Desktop’s licensing changed a while back, and while it’s not expensive, I was curious about alternatives. Podman keeps coming up as the obvious choice: it’s open source, daemonless, rootless by default, and claims Docker CLI compatibility. My C: drive was also running extremely low on space, and I figured a fresh start might help with that out too.

What worked

Installation was straightforward:

winget install RedHat.Podman
podman machine init
podman machine start

Basic container operations work fine. The CLI is compatible enough that you can alias docker to podman and most commands just work:

Set-Alias -Name docker -Value podman
docker run hello-world  # works
docker ps               # works

I moved the Podman machine to my E: drive to save space on C:, which was easy enough with WSL’s export/import:

wsl --export podman-machine-default E:\podman\backup.tar
wsl --unregister podman-machine-default
wsl --import podman-machine-default E:\podman\machine E:\podman\backup.tar

Podman Desktop exists too, if you want a nice GUI similar to Docker Desktop.

Hitting a wall

I use devcontainers a lot. They’re how I keep my development environments reproducible and contained. I wrote about them back in 2023, and they’ve become central to how I work.

When I tried to open a project in VS Code and launch the devcontainer, things quickly fell apart.

The first problem: VS Code’s Dev Containers extension runs inside my Ubuntu WSL distro, where it looks for a Docker socket at /var/run/docker.sock. But Podman runs in its own separate WSL distro (podman-machine-default), exposing its API through a Windows named pipe. These two can’t talk to each other directly.

Tying things together

I spent hours setting up a relay between the Windows named pipe and a Unix socket in Ubuntu WSL. This involved:

• Installing npiperelay on Windows to bridge named pipes to stdin/stdout
• Installing socat in Ubuntu WSL to listen on a Unix socket
• Writing a script to wire them together

sudo socat UNIX-LISTEN:/var/run/docker.sock,fork,mode=666 \
    EXEC:"npiperelay.exe -ep //./pipe/podman-machine-default",nofork

After some debugging (npiperelay wasn’t in PATH when running under sudo, the socat process was messing with terminal settings, etc.), the relay actually worked. docker ps from Ubuntu WSL talked to Podman successfully.

But the devcontainer builds still failed.

Unsurmountable filesystem issues

The devcontainer build got further, but then failed with an error about an invalid symlink in a devcontainer feature. I worked around that by removing the offending feature. Then it failed because the workspace folder was empty.

The bind mount: the thing that maps my actual project folder (in the Ubuntu WSL distro filesystem) into the container: wasn’t working. The container was running, the mount was configured, but the folder was empty inside the container.

This is a fundamental problem: Podman’s WSL distro cannot see the filesystem of my Ubuntu WSL distro.

Docker Desktop solves this transparently. It has special filesystem sharing between its VM and all WSL distros. Podman doesn’t have that. This is good for security, but bad for my workflow. Its WSL distro is isolated.

My options at this point were:

1. Move all my code to a Windows drive (C: or E:): that’s slow, especially for git operations from within a Linux container
2. Clone repos inside the Podman machine directly: then I lose all my tooling, git credentials, SSH keys, and dotfiles
3. Install Podman natively inside Ubuntu WSL instead of using the Windows Podman: this is its own can of worms and might not work seamlessly with VS Code either
4. Go back to Docker Desktop

The trade-offs

Podman’s security-first approach is genuinely good. Rootless by default, no daemon, better isolation. But this security isolates the running container from the filesystem I need, and that broke my workflow.

There’s also value in using the standard approach. When you use mainstream tooling:

• Sources like Stack Overflow are easier to find
• Blog tutorials apply
• Colleagues can help
• VS Code extensions are tested against it

This is a general point: Every deviation from standard adds friction. I can absorb a Sunday of troubleshooting. But if I recommended this to my teams of developers, that’s many work-days lost, plus ongoing support burden, plus “it works on my machine” debugging when setups drift. One of the major points of using containers is to reduce such friction, not add to it.

The real cost of Docker Desktop licensing: $5/user/month or whatever it is now: suddenly looks cheap compared to developer time.

Back to Docker Desktop

I cleaned up Podman:

podman machine stop
podman machine rm
winget uninstall RedHat.Podman
wsl --unregister podman-machine-default

And reinstalled Docker Desktop:

winget install Docker.DockerDesktop

One thing I made sure to do: move Docker’s data to E: immediately. In Docker Desktop’s settings, under Resources → Advanced → Disk image location, you can point it at another drive. No more filling up C:.

The devcontainer built and started on the first try. My files were there. Everything just worked.

Conclusion

Podman is a good tool. It works well for many things. Its security model is genuinely better than Docker’s. But for using Dev Containers on Windows with WSL, it fails.

For my workflow: devcontainers with code living in WSL’s native filesystem: Docker Desktop is still the pragmatic choice. The “magic” it does behind the scenes to make filesystem sharing work across WSL distros is exactly what I need.

Maybe Podman will solve this eventually. Maybe there’s a configuration I missed. But after a full Sunday of trying, I’m back where I started, with a bit more appreciation for what Docker Desktop actually does.

Sometimes the boring, mainstream choice is the right one.

And, I know - why do I still use Windows for my host OS?

HTTP Status codes are not enough

When we implement HTTP APIs we can deal with errors in many ways. The HTTP protocol has a set of status codes that we can use to indicate what went wrong, and we should certainly use those!

Aside: there’s a special place reserved for developers who use 200 OK for everything, including errors. Don’t be that developer. No, it is not OK. Never. Stop it!

The most basic was is to return a 200 if everything is OK, and a 500 if something went wrong. This is, however, not very informative. A 500 means “Internal Server Error”, which is not very helpful to the client. It does not tell the client what went wrong, or how to fix it.

It is slightly better to use more specific status codes. The simple act of dividing errors into 4xx (client errors) and 5xx (server errors) is a good start. Using 400 Bad Request for invalid input, 401 Unauthorized for missing or invalid authentication, 403 Forbidden for lack of permissions, 404 Not Found for missing resources, and so on, is even better.

There are a lot of status codes to choose from, but even so you’re likely to find yourself with a case that doesn’t quite fit - or you would like to be more specific about what went wrong.

When a client submits a form with validation-errors, for example. It is good to return a 400 Bad Request, but it would be even better to tell the client which fields were invalid, and why.

Problem Details to the rescue

RFC 9457: Problem Details for HTTP APIs defines a standard way to return more information about errors in HTTP APIs. The RFC defines a JSON format for the response body that can be used to provide more information about the error. It replaces the older RFC 7807 which defined a similar format, but was less flexible. These RFCs were published in July 2023 and March 2016 respectively. That’s right, this has been around for nearly 10 years at the time of writing!

These RFCs don’t change the status codes, you still return them as normal, but they talk about what you should include in the response body to provide more information about the error.

Sure, you can just return an empty body, or come up with your own format, but using a standard format has many advantages:

• Clients can be written to understand the standard format, and can handle errors in a consistent way.
• It is easier to document your API, as you can refer to the standard format.
• It is easier to test your API, as you can use standard tools to generate and validate requests and responses.
• It is much easier to integrate with other systems, as they can understand the standard format.
• You don’t have to invent your own format.
  • And these RFCs have already considered all the things your format would forget
  • Seriously - you’re not that special. Use the standard.

The Problem Details format

The Problem Details format is a JSON object that looks like this (example from the spec):

{
  "type": "https://example.com/probs/out-of-credit",
  "title": "You do not have enough credit.",
  "status": 403,
  "detail": "Your current balance is 30, but that costs 50.",
  "instance": "/account/12345/msgs/abc",
  "balance": 30,
  "accounts": [
    "/account/12345",
    "/account/67890"
  ]
}

This includes type, title, status, detail and instance properties, which are defined in the RFC. It also includes additional properties, like balance and accounts, which are not defined in the RFC, but can be used to provide more information about the error specific to the case.

A common extension to the Problem Details format is to include a list of errors, (e.g. validation-errors for multiple fields) like this:

{
  "type": "https://tools.ietf.org/html/rfc9110#section-15.5.1",
  "title": "One or more validation errors occurred.",
  "detail": "The submitted form was not valid.",
  "status": 400,
  "errors": {
    "Language": [
      "The Language field is required.",
      "'Language' must not be empty."
    ],
    "Tags": [
      "The Tags field is required."
    ],
    "CreatedBy": [
      "The CreatedBy field is required.",
      "'Created By' must not be empty."
    ],
    "Reference": [
      "The Reference field is required.",
      "'Reference' must not be empty."
    ]
  },
  "traceId": "00-84c9d447edf8d4bcba82187356336e76-ca92c022b7da8472-01"
}

With this information developers using you API can more easily understand what went wrong, and how to fix it. They can also use the errors to decorate the user-interface with errors, helping the user fix their input.

Note that this particular style of errors is not defined in the RFC, and I’ve seen many different variations of this. Since it is not defined in the RFC, you can should work with your clients to agree on a format that works for you.

Implementing Problem Details in ASP.NET

ASP.NET has built-in support for Problem Details, so you can use it out of the box. You can return a ProblemDetails object from your controller actions, and ASP.NET will automatically serialize it to JSON and set the appropriate status code.

You have to add the Microsoft.AspNetCore.Mvc.ProblemDetails package to your project.

dotnet add package Microsoft.AspNetCore.Mvc.ProblemDetails

This is pretty easy to do - you just add the following code in your application startup (usually in Program.cs):

builder.Services.AddProblemDetails();

This makes any problems with Asp.NET automatically return Problem Details responses, and also any error responses that do not have a body. With this a 404 Not Found will return a Problem Details response with the appropriate status code and a default message.

But, that middleware does not know what actually went wrong, and cannot provide any additional information. For that you have to do it yourself.

I like Milan Jovanović’s approach, but I have an alternative approach that I prefer.

The ProblemDetailer

What I want is for my controller to be able to return a ProblemDetails object with the appropriate status code and additional information, without having to write a lot of boilerplate code.

Here’s a small example of how I want a controller action to look (you can do this with minimal APIs as well):

using Microsoft.AspNetCore.Mvc;
using OneOf;
using OneOf.Types;

namespace ProblemDetailsExample;

[ApiController]
[Route("api/example")]
public partial class ExampleController(
    IProblemDetailer _problems,
    IExampleHandler _handler
) : ControllerBase
{
    [HttpGet("{id:guid}")]
    public async Task<IActionResult> GetExample(
        Guid id,
        CancellationToken ct = default)
    {
        var maybeExample = await _handler.Get(id, ct);

        return maybeExample.Match(
            example => Ok(example),
            notFound => _problems.NotFound(
                detail: "Example not found",
                errors: new()
                {
                    ["id"] = id.ToString()
                }
            ),
            error => _problems.Error(
                detail: "Failed to get example",
                errors: new()
                {
                    ["message"] = error.Value,
                    ["id"] = id.ToString()
                }
            )
        );
    }
}

public record Example(
    Guid Id
);

public interface IExampleHandler
{
    Task<OneOf<Example, NotFound, Error<string>>> Get(
        Guid id,
        CancellationToken ct
    );
}

// just a silly example showing returning different results
public class ExampleHandler : IExampleHandler
{
    static readonly Guid _illegal = Guid.Parse(
        "00000000-0000-0000-0000-000000000001"
    );

    public Task<OneOf<Example, NotFound, Error<string>>> Get(
        Guid id,
        CancellationToken ct
    ) =>
        Task.FromResult<OneOf<Example, NotFound, Error<string>>>(
            id switch
            {
                var empty when empty == Guid.Empty =>
                    new NotFound(),
                var special when special == _illegal =>
                    new Error<string>(
                        "That ID is forbidden"
                    ),
                _ => new Example(id)
            }
        );
}

I am using OneOf that I have written of earlier to represent the possible outcomes of the GetExample method. It can return an Example, a NotFound, or an Error<string>. I find this very useful for representing the possible outcomes of a method, and it works well with the Problem Details approach.

I feel this works well together and ends up with pretty clean code.

Here’s my IProblemDetailer interface:

using Microsoft.AspNetCore.Mvc;

namespace ProblemDetailsExample;

public interface IProblemDetailer
{
    BadRequestObjectResult BadRequest(
        string title = "Bad Request",
        string detail = "The request is invalid.",
        Dictionary<string, string>? errors = null,
        string? instance = null
    );

    NotFoundObjectResult NotFound(
        string title = "Not Found",
        string detail = "The requested resource was not found.",
        Dictionary<string, string>? errors = null,
        string? instance = null
    );

    ConflictObjectResult Conflict(
        string title = "Conflict",
        string detail = "The request could not be completed due to a conflict.",
        Dictionary<string, string>? errors = null,
        string? instance = null
    );

    ObjectResult Error(
        string title = "Error",
        string detail = "An unexpected error occurred.",
        Dictionary<string, string>? errors = null,
        string? instance = null,
        int statusCode = StatusCodes.Status500InternalServerError
    );

    UnprocessableEntityObjectResult UnprocessableEntity(
        string title = "Unprocessable Entity",
        string detail = "The request was well-formed but could not be processed.",
        Dictionary<string, string>? errors = null,
        string? instance = null
    );

    ObjectResult TooManyRequests(
        string title = "Too Many Requests",
        string detail = "Unable to process the request due to rate limiting.",
        Dictionary<string, string>? errors = null,
        string? instance = null,
        string? retryAfter = null
    );

    ObjectResult Gone(
        string title = "Gone",
        string detail = "The requested resource is no longer available.",
        Dictionary<string, string>? errors = null,
        string? instance = null
    );

    ObjectResult MethodNotAllowed(
        string title = "Method Not Allowed",
        string detail = "The HTTP method is not allowed for the requested resource.",
        Dictionary<string, string>? errors = null,
        string? instance = null
    );
}

As you can see the caller can include a title, detail, errors, and instance, or nothing at all, and get a reasonable default response.

The implementation of the IProblemDetailer interface is straightforward, and populates the type based on the status code. The implementation is a bit repetitive to read, but is included here for reference:

using System.Collections.Frozen;
using System.Diagnostics;
using Microsoft.AspNetCore.Mvc;
using static Microsoft.AspNetCore.Http.StatusCodes;

namespace ProblemDetailsExample;

public class ProblemDetailer(
    IHttpContextAccessor _httpContextAccessor
) : IProblemDetailer
{
    public BadRequestObjectResult BadRequest(
        string title = "Bad Request",
        string detail = "The request is invalid.",
        Dictionary<string, string>? errors = null,
        string? instance = null
    )
    {
        var (inst, ext) = GetContext(
            _httpContextAccessor.HttpContext,
            errors,
            instance
        );

        var (code, type) = GetType(
            Status400BadRequest
        );

        return new(
            new ProblemDetails
            {
                Title = title,
                Detail = detail,
                Status = code,
                Type = type,
                Instance = inst,
                Extensions = ext
            }
        );
    }

    public NotFoundObjectResult NotFound(
        string title = "Not Found",
        string detail = "The requested resource was not found.",
        Dictionary<string, string>? errors = null,
        string? instance = null
    )
    {
        var (inst, ext) = GetContext(
            _httpContextAccessor.HttpContext,
            errors,
            instance
        );

        var (code, type) = GetType(
            Status404NotFound
        );

        return new(
            new ProblemDetails
            {
                Title = title,
                Detail = detail,
                Status = code,
                Type = type,
                Instance = inst,
                Extensions = ext
            }
        );
    }

    public ConflictObjectResult Conflict(
        string title = "Conflict",
        string detail = "The request could not be completed due to a conflict.",
        Dictionary<string, string>? errors = null,
        string? instance = null
    )
    {
        var (inst, ext) = GetContext(
            _httpContextAccessor.HttpContext,
            errors,
            instance
        );
        var (code, type) = GetType(
            Status409Conflict
        );

        return new(
            new ProblemDetails
            {
                Title = title,
                Detail = detail,
                Status = code,
                Type = type,
                Instance = inst,
                Extensions = ext
            }
        );
    }

    public ObjectResult Error(
        string title = "Error",
        string detail = "An unexpected error occurred.",
        Dictionary<string, string>? errors = null,
        string? instance = null,
        int statusCode = Status500InternalServerError
    )
    {
        var (inst, ext) = GetContext(
            _httpContextAccessor.HttpContext,
            errors,
            instance
        );
        var (code, type) = GetType(statusCode);

        return new(
            new ProblemDetails
            {
                Title = title,
                Detail = detail,
                Status = code,
                Type = type,
                Instance = inst,
                Extensions = ext
            }
        )
        {
            StatusCode = statusCode
        };
    }

    public UnprocessableEntityObjectResult UnprocessableEntity(
        string title = "Unprocessable Entity",
        string detail = "The request was well-formed but could not be processed.",
        Dictionary<string, string>? errors = null,
        string? instance = null)
    {
        var (inst, ext) = GetContext(
            _httpContextAccessor.HttpContext,
            errors,
            instance
        );
        var (code, type) = GetType(
            Status422UnprocessableEntity
        );

        return new(
            new ProblemDetails
            {
                Title = title,
                Detail = detail,
                Status = code,
                Type = type,
                Instance = inst,
                Extensions = ext
            }
        );
    }

    public ObjectResult TooManyRequests(
        string title = "Too Many Requests",
        string detail = "Unable to process the request due to rate limiting.",
        Dictionary<string, string>? errors = null,
        string? instance = null,
        string? retryAfter = null
    )
    {
        var (inst, ext) = GetContext(
            _httpContextAccessor.HttpContext,
            errors,
            instance
        );

        if (retryAfter is not null)
        {
            ext["retryAfter"] = retryAfter;
            _httpContextAccessor
                .HttpContext
                ?.Response
                .Headers
                .Append(
                    "Retry-After",
                    retryAfter
                );
        }

        var (code, type) = GetType(
            Status429TooManyRequests
        );

        return new(
            new ProblemDetails
            {
                Title = title,
                Detail = detail,
                Status = code,
                Type = type,
                Instance = inst,
                Extensions = ext
            }
        )
        {
            StatusCode = Status429TooManyRequests
        };
    }

    public ObjectResult Gone(
        string title = "Gone",
        string detail = "The requested resource is no longer available.",
        Dictionary<string, string>? errors = null,
        string? instance = null
    )
    {
        var (inst, ext) = GetContext(
            _httpContextAccessor.HttpContext,
            errors,
            instance
        );
        var (code, type) = GetType(
            Status410Gone
        );

        return new(
            new ProblemDetails
            {
                Title = title,
                Detail = detail,
                Status = code,
                Type = type,
                Instance = inst,
                Extensions = ext
            }
        )
        {
            StatusCode = Status410Gone
        };
    }

    public ObjectResult MethodNotAllowed(
        string title = "Method Not Allowed",
        string detail = "The HTTP method is not allowed for the requested resource.",
        Dictionary<string, string>? errors = null,
        string? instance = null
    )
    {
        var (inst, ext) = GetContext(
            _httpContextAccessor.HttpContext,
            errors,
            instance
        );
        var (code, type) = GetType(
            Status405MethodNotAllowed
        );

        return new(
            new ProblemDetails
            {
                Title = title,
                Detail = detail,
                Status = code,
                Type = type,
                Instance = inst,
                Extensions = ext
            }
        )
        {
            StatusCode = Status405MethodNotAllowed
        };
    }

    static (int, string) GetType(int statusCode)
    {
        if (StatusLinks.TryGetValue(statusCode, out var type))
        {
            return (statusCode, type);
        }

        return (
            Status500InternalServerError,
            StatusLinks[Status500InternalServerError]
        );
    }

    static (string, Dictionary<string, object?>) GetContext(
        HttpContext? context,
        Dictionary<string, string>? errors,
        string? possibleInstance = null
    )
    {
        var instance = possibleInstance
            ?? context?.Request.Path;
        var traceId = Activity.Current?.Id
            ?? context?.TraceIdentifier;

        return (
            instance ?? string.Empty,
            new()
            {
                ["traceId"] = traceId ?? string.Empty,
                ["errors"] = errors ?? []
            }
        );
    }

    static readonly FrozenDictionary<int, string> StatusLinks = new Dictionary<int, string>(24)
    {
        [Status400BadRequest]           = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.1",
        [Status401Unauthorized]         = "https://datatracker.ietf.org/doc/html/rfc7235#section-3.1",
        [Status402PaymentRequired]      = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.2",
        [Status403Forbidden]            = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.3",
        [Status404NotFound]             = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.4",
        [Status405MethodNotAllowed]     = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.5",
        [Status406NotAcceptable]        = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.6",
        [Status407ProxyAuthenticationRequired] = "https://datatracker.ietf.org/doc/html/rfc7235#section-3.2",
        [Status408RequestTimeout]       = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.7",
        [Status409Conflict]             = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.8",
        [Status410Gone]                 = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.9",
        [Status411LengthRequired]       = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.10",
        [Status412PreconditionFailed]   = "https://datatracker.ietf.org/doc/html/rfc7232#section-4.2",
        [Status413PayloadTooLarge]      = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.11",
        [Status414UriTooLong]           = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.12",
        [Status415UnsupportedMediaType] = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.13",
        [Status416RangeNotSatisfiable]  = "https://datatracker.ietf.org/doc/html/rfc7233#section-4.4",
        [Status417ExpectationFailed]    = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.14",
        [Status418ImATeapot]            = "https://datatracker.ietf.org/doc/html/rfc7168#section-2.3.3",
        [Status422UnprocessableEntity]  = "https://datatracker.ietf.org/doc/html/rfc4918#section-11.2",
        [Status429TooManyRequests]      = "https://datatracker.ietf.org/doc/html/rfc6585#section-4",
        [Status500InternalServerError]  = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.6.1",
        [Status501NotImplemented]       = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.6.2",
        [Status502BadGateway]           = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.6.3",
        [Status503ServiceUnavailable]   = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.6.4",
        [Status504GatewayTimeout]       = "https://datatracker.ietf.org/doc/html/rfc7231#section-6.6.5",
    }.ToFrozenDictionary();
}

I hope this helps you implement Problem Details in your ASP.NET applications!

Our AuthenticationHandler

Let us say we have an AuthenticationHandler that looks like this to handle authentication. Here is a silly example randomly allowing or denying authentication:

public partial class RandomAuthenticationHandler
: AuthenticationHandler<AuthenticationSchemeOptions>
{
    public const string SchemaName = "Random";

    readonly bool _enabled;
    readonly double _chance;
    readonly ILogger<RandomAuthenticationHandler> _logger;

    public RandomAuthenticationHandler(
        IOptionsMonitor<AuthenticationSchemeOptions> options,
        ILoggerFactory loggerFactory,
        UrlEncoder encoder,
        IOptions<RandomSettings> randomSettings
        ) : base(
            options,
            loggerFactory,
            encoder
        )
    {
        _logger = loggerFactory
            .CreateLogger<RandomAuthenticationHandler>();

        if (randomSettings?.Value is null)
        {
            LogMissingConfiguration(_logger);
            throw new InvalidOperationException(
                "Application misconfigured: No RandomSettings"
            )
        }
        _randomSettings = randomSettings.Value;
    }

    protected override Task<AuthenticateResult> HandleAuthenticateAsync()
    {
        if (!_randomSettings.Enabled)
        {
            LogDisabled(_logger);
            return Task.FromResult(
                AuthenticateResult.NoResult()
            );
        }

        if (Random.Shared.NextDouble() < _randomSettings.Chance)
        {
            LogRandomlyFailing(_logger);
            return Task.FromResult(
                AuthenticateResult.Fail("Not authenticated")
            );
        }

        LogRandomlySucceeding(_logger);
        return Task.FromResult(
            AuthenticateResult.Success(
                new AuthenticationTicket(
                    principal: new ClaimsPrincipal(
                        new ClaimsIdentity(
                            authenticationType: SchemaName,
                            claims:
                            [
                                new Claim(
                                    ClaimTypes.NameIdentifier,
                                    Guid.NewGuid().ToString()
                                ),
                                new Claim(
                                    ClaimTypes.Name,
                                    "Random User"
                                ),
                            ]
                        )
                    ),
                    authenticationScheme: SchemaName
                )
            )
        );
    }

    // logging omitted for brevity
}

/**
 * Bind this in your IoC setup
 **/
public class RandomSettings
{
    public bool Enabled { get; set; }

    [Range(0.0, 1.0)]
    public double Chance { get; set; }
}

How would we test such a thing? Set aside that this is a silly example, and randomness it inherently hard to test. How do we set up the AuthenticationHandler in a test?

Setting up the test

We need to instantiate our RandomAuthenticationHandler in a test. We can do that fine.

In the following example I’m using NSubstitute for mocking, and xUnit for the test framework. The logging is set up in a base class called Test_with_logs that I use in many tests, you can read about that in this post.

public class Given_a_RandomAuthenticationHandler
: Test_with_logs
{
    protected IOptionsMonitor<AuthenticationSchemeOptions> _options_monitor;
    protected IOptions<RandomSettings> _random_settings;
    protected UrlEncoder _url_encoder;

    protected RandomAuthenticationHandler _handler;

    protected Given_a_RandomAuthenticationHandler()
    {
        _options_monitor = Substitute
            .For<IOptionsMonitor<AuthenticationSchemeOptions>>();

        _options_monitor
            .Get(Arg.Any<string>())
            .Returns(new AuthOptions());

        _url_encoder = Substitute.For<UrlEncoder>();

        _random_settings = Options.Create(
            new RandomSettings
            {
                Enabled = true,
                Chance = 0.2
            }
        );

        _handler = new RandomAuthenticationHandler(
            _options_monitor,
            TestLoggerFactory,
            _url_encoder,
            _random_settings
        );
    }
}

public class When_authenticating_randomly
: Given_a_RandomAuthenticationHandler
{
    [Fact]
    public async Task It_should_not_crash()
    {
        await _random_authentication_handler.AuthenticateAsync();

        true.Should().BeTrue();
    }
}

Our simple test fails, since the AuthenticationHandler must be initialised before use. We can do that by calling the InitializeAsync -method. This is the thing that aspnet calls before our HandleAuthenticationAsync method is called. Let’s add that to our test base-class (the Given -class):

public class Given_a_RandomAuthenticationHandler
: Test_with_logs
{
    // ...previously shown code omitted for brevity...

    protected async Task Initialize_with(HttpContext context)
    {
        await _handler.InitializeAsync(
            new AuthenticationScheme(
                RandomAuthenticationHandler.SchemaName,
                null,
                typeof(RandomAuthenticationHandler)
            ),
            context
        );
    }
}

Now we can call that in our test, and pass in a HttpContext that we set up for the test. Let’s do that:

public class When_authenticating_randomly
: Given_a_RandomAuthenticationHandler
{
    readonly DefaultHttpContext _context = new();

    [Fact]
    public async Task It_should_return_a_result()
    {
        await Initialize_with(_context);
        var result = await _handler.AuthenticateAsync();

        result.Should().NotBeNull();
    }

    [Fact]
    public async Task It_should_log_something()
    {
        await Initialize_with(_context);
        await _handler.AuthenticateAsync();

        Logs
            .Should()
            .NotBeEmpty();
    }

    // more tests here...
}

And there you have it! Now you can manipulate the HttpContext as you like before you initialize, and test your AuthenticationHandler in isolation.

Exercise for the reader: write a test wherein the _random_settings.Value is null verifying that the InvalidOperationException is thrown.

Happy testing!

2023-12-29: There’s a update in the end of this post

Use records for value semantics

Records are a great thing in C#. They are immutable, and give you “value semantics” for comparison. This means that two records with the same values are considered equal, even if they are different instances. This saves you a lot of time and troublesome code!

public record Person(
    string FirstName,
    string LastName);

var person1 = new Person("Tomas", "Ekeli");
var person2 = new Person("Tomas", "Ekeli");

// will print "true"
Console.WriteLine(person1 == person2);

There are some caveats to this, but they are not the focus of this post. You can read more about them here.

The problem with collections

Wonderful, but it all falls down if you have a collection of some sort as a property in your record. Then the comparison will only compare the references of the collections, not the contents. This is because the default implementation of Equals and GetHashCode for collections only compares the references.

public record PersonWithNickname(
    string FirstName,
    string LastName,
    List<string> Nicknames);

var person1 = new PersonWithNickname(
    "Tomas",
    "Ekeli",
    [ "Tommy" ]
);

var person2 = new PersonWithNickname(
    "Tomas",
    "Ekeli",
    [ "Tommy" ]
);

// will print "false"
Console.WriteLine(person1 == person2);

// referencing person1's nicknames
var person3 = new PersonWithNickname(
    "Tomas",
    "Ekeli",
    person1.Nicknames
);

// will print "true"
Console.WriteLine(person1 == person3);

This is (in my experience) usually not what I want. When I use a record I am almost always using it for its value semantics. I want to compare the contents of the collections, not the references. So how do we do that?

Solution: Nemesis.Essentials

We could implement our own collections that override Equals and GetHashCode to compare the contents. But that is a lot of work, and it is easy to get wrong, and will be a chore to maintain.

But, there is such a library already out there that we can just use, it is called Nemesis.Essentials and you can install it from NuGet. It is MIT licensed, so you can use it in your projects without any worries about licensing.

dotnet add package Nemesis.Essentials

You change your record to use the ValueCollection<T> from the library instead of List<T>. Then you get the value semantics you want. With the new collection expressions in C# 12, it is even easier to use.

using Nemesis.Essentials.Design;

public record PersonWithNickname(
    string FirstName,
    string LastName,
    ValueCollection<string> Nicknames);

var person1 = new PersonWithNickname(
    "Tomas",
    "Ekeli",
    new([ "Tommy" ])
);

var person2 = new PersonWithNickname(
    "Tomas",
    "Ekeli",
    new([ "Tommy" ])
);

There are a lot of other things in that library as well, but this one seemed very helpful. I hope you find it useful as well!

Update 2023-12-29 Danger! Caveats!

As has been pointed out to me by Leszek Ciesielski - there are some caveats to this approach. While it does provide the equality semantics I am after, it comes with consequences.

ValueCollection inherits System.Collections.ObjectModel.Collection, a mutable data-structure.

This is not what we want in a record, as their immutability is why they can have value semantics in the first place. And, since the collection can be part of how the record is compared and hashed - this means that the record’s hash-code can change. This is a problem. It can also affect serialization -performance, which is bad.

Whether or not these problems are deal-breakers for you depends on your use-case. Do you need to serialize and deserialize your records? Do you need to use them as keys in a dictionary? Will you never mutate the collection? If the answer to any of these questions is “yes”, then you should probably not use this approach.

I still think that the library is useful, but it is not a silver bullet. If anyone ever finds one of those fabled silver bullets - please tell me! I will probably use the Nemesis -library in some cases, but not in others.

An alternative solution is to use System.Collections.Immutable.ImmutableList instead of ValueCollection<T>. This is an immutable data-structure, and it has the equality semantics we want. But, it is not a drop-in replacement for List<T>, so you will have to change your code to use it. It also has some performance implications, so you need to consider those carefully before you use it.

There is also the System.Collections.Frozen -namespace in dotnet8, which spends more time when constructing the collection, but is immutable from then on and offers better performance after creation. They only have Dictionaries and Sets, though - no Lists.

When you write your code a sometimes disregarded part of the functionality is what gets logged and when. As you write code that needs to log to some specific level, or where logs are an important part of the functionality, you should also test that you are logging correctly.

Verifying logging can be tricky

The problem is that logging is often done through static methods or extension methods in C#, and these are hard to mock. Your actual code might look something like this, logging with the Microsoft.Extensions.Logging -library:

using Microsoft.Extensions.Logging;
/* ... other usings to get IService
 and Result */

public class Thing(
  ILogger _logger,
  IService _dependency)
{
  public async Task<Result> DoTheThing()
  {
    _logger.LogInformation(
      "Doing something");
    try
    {
      var something = await
        _dependency.DoSomething();
      return Result
        .Success(something);
    }
    catch (Exception ex)
    {
      _logger.LogError(
        ex,
        "Something went wrong");
      return Result.Failure;
    }
  }
}

If you want to have a test that verifies that the logging is done correctly, you need to mock the logger. This is not hard, you just mock the ILogger -interface, but the LogInformation and LogError -methods are extension methods, and you can’t mock extension methods. You can’t mock static methods either, so what do you do?

Mocking and extension methods

You can find lots of suggestions on the internet to mock out the methods on the actual ILogger -interface that the extension methods call, but that is not a good idea. Mocking like that means you are tying your tests to the inner details of the extension methods. You should not need to know how the extension methods work to test your code! Depending on inner workings leaves you open to breaking changes in the extension methods, and it makes your tests harder to read.

Instead - take a step back and think about what you’re actually trying to test. You don’t care about how the message is written, you just want to test that the correct log message is written, and that it is written at the correct log level. What you really need is the log-output in some way.

MELT

This is just what MELT gives you! MELT is a library that lets you work with the standard Microsoft.Extensions.Logging -library, but lets you run your tests and capture the logged output. Then you verify that the output (which is what you actually care about) is correct. This frees you up from knowing how the logging actually happens under-the-hood, and you can assert on the output instead. Here’s a great post on it.

To install it, add Microsoft.Extensions.Logging (if not already there) and the MELT NuGet package to your test project:

dotnet add package Microsoft.Extensions.Logging
dotnet add package MELT

In this is an example XUnit test that verifies that the correct log message is written, with NSubstitute and Shouldly for mocking and assertions:

using MELT;
using Microsoft.Extensions.Logging;
using NSubstitute;
using NSubstitute.ExceptionExtensions;
using Shouldly;
using Xunit;

public class ThingLogsAsExpected
{
  [Fact]
  public async Task LogsCorrectMessage()
  {
    // Arrange
    var factory = TestLoggerFactory
      .Create();
    var logger = factory
      .CreateLogger<Thing>();

    var dependency = Substitute
      .For<IService>();
    dependency
      .DoSomething()
      .Throws(
        new Exception("Boom!")
      );

    var system_under_test = new Thing(
      logger,
      dependency);

    // Act
    await system_under_test
      .DoTheThing();

    // Assert
    factory
      .Sink
      .LogEntries
      .ShouldContain(entry =>
        entry.LogLevel ==
          LogLevel.Error
        && entry.Exception != null
        && entry.Exception.Message
          == "Boom!"
        && entry.Message ==
          "Something went wrong"
      );
  }
}

A handy base class for your tests

The access to the TestLoggerFactory and its Sink -property is what gives you access to the logged output. Personally I find this direct use a bit too verbose - so I usually create a super-class for my tests that let me access it in a more convenient way:

using MELT;
using Microsoft.Extensions.Logging;
using NSubstitute;
using NSubstitute.ExceptionExtensions;
using Shouldly;
using Xunit;

public abstract class TestsWithLogging
{
  protected ITestLoggerFactory Factory =
     TestLoggerFactory.Create();

  protected IEnumerable<LogEntry> Logs =>
    Factory.Sink.LogEntries;

  public TestsWithLogging() =>
    Factory.Sink.Clear();
}

public class WhenTheDependencyThrows
  : TestsWithLogging
{
  readonly IService _dependency;
  readonly ILogger<Thing> _logger;
  readonly Thing _system_under_test;
  readonly Result _result;

  public WhenTheDependencyThrows()
  {
    // Arrange
    _dependency = Substitute
      .For<IService>();
    _logger = Factory
      .CreateLogger<Thing>();

    _system_under_test = new Thing(
      _logger,
      _dependency);

    _dependency
      .DoSomething()
      .Throws(
        new Exception("Boom!")
      );

    // Act
    _result = _system_under_test
      .DoTheThing()
      .Result;
  }

  // each Fact is an assertion
  [Fact]
  public void LogsError() =>
    Logs
      .ShouldContain(logEntry =>
        logEntry.LogLevel ==
          LogLevel.Error
        && logEntry.Exception !=
          null
        && logEntry
          .Exception
          .Message == "Boom!"
        && logEntry.Message ==
          "Something went wrong"
      );

  [Fact]
  public void LogsInformation() =>
    Logs
      .ShouldContain(logEntry =>
        logEntry.LogLevel ==
          LogLevel.Information
        && logEntry.Message ==
          "Doing something"
      );

  [Fact]
  public void LogsTwoMessages() =>
    Logs
      .Count()
      .ShouldBe(2);
}

This way I can access the Logs -property directly in the test, and verify that it contains what I expect. By clearing the logs before each test each run gets their own log and the tests won’t affect each other.

This way of writing tests with the assertion and the act in the constructor is a bit unusual, but I find it very convenient. It makes the tests very readable, and it makes it easy to add or remove assertions. As you can see here I’ve added two more assertions, and the test still reads well.

Some readers may not agree with my rather aggressive column-limit. I don’t actually break my code at 43 characters in real life, but I do try to keep it under 80 characters. I do it here to keep the code within the screen for readers on mobile devices (there have been complaints).

If you’re using NUnit you can do the same thing with a OneTimeSetUp -method, and if you’re using MSTest you can do the same thing with a [TestInitialize] -attribute. I prefer XUnit, as each Fact is inherently separate, but you can do this with any test framework. Just be careful with any statics.

Conclusion

When you want to test that you are logging correctly it can be tricky - as extension methods and statics are hard to mock. By using MELT you can test that the correct log message is written, and that it is written at the correct log level. This frees you up from knowing how the logging actually happens under-the-hood, and you can specify just what you are interested in.

You may not need or want to test your logging, but if you do - MELT is a great way to do it!

P.S.: Performance and memory when logging

The example Thing here uses the extension-methods from the Microsoft.Extensions.Logging -library directly in the code. This works, and is often what we do when we are not massively concerned with performance.

If you are you should generate log-methods using the LoggerMessage -attribute. This will generate a static method that you can call instead of the extension methods on ILogger. It is much faster and uses less memory (particularly when you log values).

Note that for this to work your class must be partial (to allow the generated code to “take over” for the log-methods). Example:

public partial class Thing(
  ILogger _logger,
  IService _dependency)
{
  public async Task<Result> DoTheThing()
  {
    Entered(_logger);
    try
    {
      var something = await _dependency
        .DoSomething();
      return Result.Success(something);
    }
    catch (Exception ex)
    {
      Failed(_logger, ex);
      return Result.Failure;
    }
  }

  [LoggerMessage(
    EventId = 1,
    Level = LogLevel.Information,
    Message = "Doing something")]
  public static partial void Entered(
    ILogger logger);

  [LoggerMessage(
    EventId = 2,
    Level = LogLevel.Error,
    Message = "Something went wrong")]
  public static partial void Failed(
    ILogger logger,
    Exception ex);
}

Hmm, maybe I’ve finally found a real use for #region? In fact you may pull those partial methods out to a different file, perhaps called Thing.Logging.cs or something like that. That way you can keep the code that does the actual work separate from the logging code. Personally I prefer to have it all in one file.

Some history

I used to post a lot on Twitter, when that still was a good place to be. I met lots of interesting people, learnt and taught, helped and received help. I have never been, nor wanted to be, any kind leader or influencer, but I felt I was part of a community. I was a part of something bigger than myself. I was a part of the Internet.

Then Twitter started to change. Maybe it did not change, maybe I just started to notice it more.

Still, Twitter was where the people were - it was where I needed to be to keep interacting with so many good people. I was caught by the network effect, and did not want to leave.

I made small forays into the Fediverse, but kept coming back to Twitter. Maybe it was just inertia, maybe it was Metcalfe’s Law, but I stayed. Then, some years ago, I decided to try out something else.

I did not delete my Twitter account yet, but I made a Mastodon account, and I made a Pixelfed account. I did not use them much. I had few followers and followed few people. It felt quiet and lonely, I did not feel like I was part of a community.

Slowly I built up a group of people to follow. It takes a little effort, but it’s not complex. Here, without further ado, is the way to build yourself an interesting timeline on the Fediverse:

1. Find someone interesting. This is the first and somewhat hard thing to do. I found people by looking at mastodon servers that seemed interesting and following some likely people there. I also found people by looking at the “local timeline” on the server I joined.
2. Find out who the interesting person is following. This is the easy part. Just click on the “following” link on their profile.
3. Follow some of those people.
4. Look at your timeline and interact with people you find interesting.
5. Mute or block assholes. There are going to be assholes, but you don’t have to listen to them. Don’t be offended if people think you’re an asshole and blocks you. It’s fine.
6. Repeat.

Don’t worry about whether people follow you - just post things that interest you and be a nice person. If you seem interesting, people will follow you. If you do not, they won’t. It’s fine.

Nobody gets a prize for having followers, that’s not what the Fediverse is about. This is probably my favourite thing about the Fediverse - it does not play by the “eternal growth at any cost” rules of commercial social media. It’s not about the numbers, it’s about the people.

I had interesting interactions on the Fediverse, and Twitter, at the same time. It was a little awkward, maintaining two “presences”, but it kind of worked.

Things really changed, however, when the current owner of Twitter cum X bought the company and started changing things. I’d had enough, and wanted no part in supporting what that company was turning into, and I no longer needed to be there to interact with people. I left.

The Fediverse

There are many explanations on the internet about what the Fediverse is, and how it works. I will not repeat them here, but I will give a short summary:

When you’re using “traditional” (commercial) social media, you are using a service that is owned by a company. You sign up to Facebook and get an account in their system. Then you tell them who you want to connect with (that’s in their system), and they give you a timeline and ways of communicating with those people.

With the Fediverse it’s slightly different. You still use a service (let’s say Mastodon Social), which may be controlled by a company but usually is a group of friends or a community. You sign up to that service, and get an account in their system. Then you tell them who you want to connect with that’s on the Fediverse, and they give you a timeline and ways of communicating with those people.

The important difference is that the people you can connect with on the Fediverse are not just people on the service you’ve signed up to (Mastodon Social in our example). You can connect with anyone on any number of other services that participate in the Fediverse.

There are many kinds of services in the Fediverse, that provide many different services. Micro-blogging (like Twitter) is the most common, but there are also services for photo sharing (like Instagram), video sharing (like YouTube), and many others.

You’re probably going to set up an account on a Mastodon -server first. Mastodon is a type of server, and there are many different instances of that server (like Mastodon Social). Just like WordPress is a kind of server, and many people and organisations run their own Wordpress -sites using that kind of server.

The magic comes from all of these different kinds of servers communicating with each other. A user on one server can communicate with a user on a different server (just that is kind of cool), even if the two servers are of totally different kinds! This is possible because, whilst the servers are different, they all speak the same language. They all use the same protocol to communicate with each other. This is what makes the Fediverse so powerful.

An example

I have an account on a server called plud.re. My username there is @tomasekeli, so my full address is @tomasekeli@plud.re. My friends over on Mastodon Social can enter that address in their search bar, and find me, follow me and interact with me. And plud.re is a totally different kind of server from Mastodon! If Facebook, X and, TikTok and Instagram did this you would be able to follow TikTok users from Instagram and share their videos with your friends on Facebook from your X timeline!

My own server

Another neat part of the Fediverse is that you are not expected to stay on one server forever. You can migrate to a new home on the internet that you like better. I have done this several times: I started on mastodon.social (most people do) before I went to mastodon.tech. That server was closed down, for personal reasons of the owner and maintainer, and I moved on to snabelen.no, which is a Norwegian Mastodon server. And now, this weekend, I moved over to plud.re - which is my own server.

Yes, I am now running my own server, and am in control of what that server does and how it behaves. This means that I can set it up how I want it to be, but it also means that I am no longer using resources on a server that other people pay for. I did donate to the upkeep of the other servers I’ve been on, but now I pay my own way.

This server is not a Mastodon server, but runs something called Firefish. It has a very different user-interface than Mastodon, and has some other features that I quite like.

It is also a very small server with only one user (me), and an admin-bot. This is important to me, as I don’t think I’d make a good moderator of others’ behaviour and speech. If I do let others join plud.re it will be restricted to people I trust and know, personally.

And, that’s just it! The Fediverse lets me to do this - I can run my own server and decide on the policies of that server. If I notice that certain other servers contain a lot of assholes (there are neo-nazi servers out there) I can tell my server not to communicate with those servers. I can do this and remain a part of the community in the Fediverse. All the people who followed me had their “follow” automatically moved to my new server, and we can still interact! I am still a part of the Internet. Feels good, man.

Conclusion

I think the Fediverse is a great place to be, and I would like you to join in! I have had enough of free-loading on huge companies that make money by being the funnel i must talk through. These companies are not good companies, and they do not have good intentions - they cannot have, as they are beholden to their shareholders. I want to be a part of the community of interesting people that is the Fediverse. A virtual place where the people are the important thing, not the “engagement” or how many you can reach. And, I want you to join me there.

So, go to joinmastodon.org and find a server that looks interesting! It’s not massively important to pick the right one, but I’d recommend something other than mastodon.social. Join up and find me - just search for @tomasekeli@plud.re and I’ll follow you back and we can have a conversation.

Find your new home in the Fediverse, and I’ll meet you there!

Setting up a new Windows PC can be quite the task, but it doesn’t have to be daunting. Here I’m sharing my own setup routine, covering everything from the must-have utilities to customizing browsers and setting up development environments. If you’re curious about how a seasoned developer and tech enthusiast approaches this task, or looking for ideas to optimize your own setup, this guide’s for you.

Starting with the Browser: Firefox

The Gateway to the Web: Firefox

My setup journey begins with Firefox, my preferred internet browser. It’s known for its speed, privacy, and customization options. Other options are out there, but I try to stay away from Google and Microsoft’s browsers. And, since they started going into the crypto bullshit I no longer want to touch Brave. For me, it should be some sort of a Firefox. There are other variants of Firefox out there, but I prefer the original on Windows.

There are two ways to install Firefox

• Use the command line for a swift installation: winget install firefox.firefox.
• Or, use Edge to download it from https://getfirefox.com then set it as the default browser without importing old data.

Essential Utilities

After setting up the browser, I focus on essential utilities to streamline my workflow. The first things I need to even be able to install other things and connect to my data are:

• 7-Zip: A file archiver known for its high compression ratio. Install command: winget install 7zip.7zip.
• CopyQ: An advanced clipboard manager that stores more than the default number of clipboard entries. I do not want to compute without something like this. Install command: winget install hluk.copyq.
• KeePassXC: A secure password manager to keep all credentials safe. Its database syncs across various cloud services for accessibility. Installation and setup involve:
  • winget install keepassxcteam.keepassxc.
  • Open my database-file with it - that’s stored on several cloud providers and on my phone. This database is encrypted with the one password I must remember - it is a pretty long and gnarly one.
  • Custom settings for startup behavior and browser integration.
    • Activate auto-launch on startup, minimised to tray
    • Minimise-on-exit (don’t close)
    • Hide to system-tray when minimised
    • Activate browser integration
      • For firefox, install the KeePassXC-Browser.
• Joplin: A note-taking application that synchronizes with cloud storage for easy access to notes. Installation and setup involve syncing with the cloud storage for note retrieval, but all my notes are then available on all my devices, encrypted and safe - so the cloud-provider cannot read them. I use Joplin extensively, and find it to be a great tool.
  • winget install joplin.joplin

Customizing Firefox

Firefox isn’t just a browser for me; it’s a customized tool that enhances my browsing experience.

• Sync: I use the sync feature with my email for seamless access across devices.
• Extensions: I use add-ons to give Firefox new capabilities, and redirect it to privacy-friendly alternatives. A few of the most important extensions are:
  • KeePassXc-Browser makes logging into the myriad of sites easy and secure. I don’t think I remember any site passwords other than my master password anymore, and neither should you.
  • FirefoxPWA lets me run web-apps as native applications with Firefox.
    • winget install -e Microsoft.VcRedist.2015+.x64
    • winget install mozillafirefoxpwa.
  • LibRedirect lets me redirect mainstream sites to privacy-friendly alternatives. I use it to redirect YouTube to my own Invidious instance, Google Maps to OpenStreetMap, etc.
  • NoScript: I have sites I trust, for everything else I want to decide if I want to run scripts or not. This is a bit of a hassle, but it’s worth it, removes all cookie-popups and sign-up pop-overs.
  • uBlock Origin: I use this to block ads and trackers. I am against being sold by ad-networks, and it’s getting downright dangerous to browse the web without an ad-blocker.
  • Privacy Badger: I use this to block trackers and fingerprinting. Worth it.
  • Joplin Web Clipper: I use this to save web-pages to my Joplin notes. Don’t do it a lot, but sometimes it’s a time-saver.
  • Dark Reader: I use this to make sites dark. I like dark sites, and this is a great way to get most pages to not blind me.
  • Omnivore: It’s like Pocket or ReadItLater, but it’s open-source and I can host it myself. I use it to save articles for later reading. I use this a lot, for when I find interesting things I don’t have time to read right now.
  • User-Agent Switcher and Manager: I use this to change my user-agent, when bad web-devs (there, I said it) block Firefox by the user-agent. I don’t like it, but it’s a fact of life (bad devs exist).
  • Modify Header Value: I use this to change my request headers. A bit of a developer-tool for when I use APIs that require a specific header. I don’t use it a lot, but it’s nice to have.
  • Open in Reader View: Reader view is a wonderful feature that strips away all the cruft and lets me read the article without distractions. This add-on lets me open links in reader view with a right-click.
  • Right-Click Borescope: This is a true gem - it lets you find the images you clicked on even if they are hidden under layers of divs. I never have to go into the dev-tools to find the image I want to save again.
  • Add custom search engine: Makes it real easy to add new search-engines to Firefox. I use this to add my own Searx-instance as a search-engine, and others. That so many people just use Google is sad, I don’t even think it’s a very good search engine anymore. I use Searx, but there are lots of others out there.
  • SimpleLogin: I use this to create disposable email-addresses for sites that require an email-address. I don’t want to give my email-address to every site out there, and I don’t want to use my work-email for personal stuff. This is a great way to get around that. With the integrated password manager I don’t even need to remember the account I set up for a site, and since I’m a paying customer of ProtonMail I get SimpleLogin with my account there.
• Search Engine: Adding a privacy-focused search engine as the default one in Firefox enhances my browsing privacy. I have my own instance of Searx that I run for me and my family. So, I set that as my default search engine in Firefox.

Cloud Services for Seamless Sync

Cloud storage is integral to accessing files and databases on any device.

• Nextcloud: A secure cloud storage solution that I use to access files and the password database. It’s also where my personal calendar and address-book lives. You can use this for a lot of things, and you can host it yourself if you want. Install command for the client-software: winget install nextcloud.nextclouddesktop.
• Proton Services: For secure email, file-storage and VPN services, I use and pay for Proton. There are other good providers out there, but this is among the better ones. Installation can be done through winget commands or directly from their website.
  • winget install proton.protondrive - like Dropbox, but secure.
  • winget install protontechnologies.protonmailbridge - a Bridge that lets Thunderbird connect to ProtonMail through a small, local IMAP-server.
  • winget install protontechnologies.protonvpn - a VPN-service that I use when I’m on public networks, or when I want to access content that is blocked in my country.

Email and Calendar Management: Thunderbird

Thunderbird is my go-to for managing emails and calendars. I don’t absolutely adore it, although it’s gotten much better, but it’s far better than Outlook.

• Install Thunderbird using winget install mozilla.thunderbird.
• Add-ons like Dark Reader and calendar integrations enhance functionality.
• As my various places of work have tended to use Microsoft Exchange, and that has a spotty record of supporting IMAP and CALDav, I have used DavMail to connect to Exchange. But, in recent years I have switched over to the (paid) extension Owl for Exchange. It’s extremely easy to set up and works flawlessly. I highly recommend it! Well worth the money.
• I have my personal calendar on my NextCloud-instance, and I use the Lightning extension to connect to it. Works great, and I can share my calendar with my family. I tried using Proton Calendar for a while, but it doesn’t support anything but their web-interface and their own apps. I want to use my own calendar-app, so I switched back to my NextCloud-calendar.

Development: VSCode

As a developer, an efficient coding environment is crucial. I use VSCode for my development needs. People tell me I should use VSCodium, to get rid of the telemetry. I agree, but I’m not willing to give up the extensions.

• VSCode: A versatile code editor with extensions for various programming languages. Install command: winget install microsoft.visualstudiocode. I really should make a post on the extensions I use, but that’s for another day.
• Fira Code: A monospaced font with programming ligatures for a better coding experience. Install command: winget install nerdfonts.firacode.
• These days I do all my development in Docker-containers - so I keep my computer (the host-machine) as clean as possible. I don’t install any SDKs, or development tools on it, and never have conflicts between them anymore.

Command-Line Efficiency: Git and PowerShell

For version control and scripting, I rely on Git and PowerShell on the Windows -machine.

• Git: Essential for version control, installed using winget install git.git.
  • When I still did development on my host-machine, I used GitExtensions to get a nice GUI for Git. If you’re developing directly on Windows I would recommend it: winget install gitextensions.gitextensions.
• PowerShell: Enhanced with Oh My Posh and other modules for a powerful scripting experience. Customized with a profile script and an oh-my-posh configuration-file for a personalized touch.
  • Also needs a font that supports the glyphs used by oh-my-posh, I use CascadyiaCove NF for this. Remember to change the WinTerm default font to this one, or the shell will look garbled.
  • This is based on and slightly modified from the setup shared by Scott Hanselman in his blog post.

Integrating Linux: Windows Subsystem for Linux (WSL)

Bringing Linux into Windows adds versatility to my setup. I do all my development, and much of my other work from Linux under windows. For many things it’s just a nicer experience.

• Activate Hyper-V and install a preferred Linux distro with WSL for Linux environment support on Windows.
  • wsl --install - this will install the latest Ubuntu LTS, and set it up for you.

Final Touches: Office, Docker, and Communication Tools

To round up the setup:

• LibreOffice: For handling office documents, installed via winget install libreoffice. I know most people prefer Microsoft Office, but I don’t.
• Docker Desktop: Essential for containerization projects. I keep meaning to try out alternatives like PodMan or Rancher, but Docker just keeps on working for me. Requires WSL to be installed and activated. Install command: winget install docker.dockerdesktop
• ReMarkable: For integration with my paper-tablet. I use it for note-taking and sketching. I have a ReMarkable 2, and I love it. I use it for all my note-taking, and for sketching out ideas, and reading. Installed via winget install remarkable.remarkable
• Communication tools for staying connected with teams and communities.
  • Teams: For work-related communication. Not very good for chat, but excellent for video-meetings. Installed via winget install microsoft.teams.
  • Slack: For work and community-related communication. Much better than Teams for chats and channels as well as informal video-meetings, but not nearly as good for “real meetings”. Installed via winget install slacktechnologies.slack.
  • Discord For more community-related communication. Installed via winget install discord.discord.
  • Element: For community-related communication over the Matrix protocol. There are lots of other cool clients for Matrix, but I like Element. I wish more people would use Matrix. Installed via winget install element.element.

By following these steps, I ensure that my new Windows PC is not just a machine, but a personalized workspace that aligns with my workflow and preferences. It’s all about creating a space where productivity is effortless and secure.

Got your own Windows setup that you’re proud of, or some cool tips to share? I’m keen to hear about how others tackle their tech setups. There are no comments enabled here, but feel free to reach out and connect with me on the social platforms listed in the footer. Maybe send me a link to your own post describing how you set up a new PC? Let’s swap stories and maybe pick up some new tricks along the way!

From big ideas to small wins - a story about a hackathon.

Hackathing

We had a hackathon this last week, working on a real challenge that’s been bugging us for a while. I was part of a team of five programmers. We spent the first day of two talking about the problem, exploring it and coming up with possible approaches. We knew we’d have a short time to make a presentation, and if possible a demo, so we had to be careful not to overreach. We had to go small.

We came up with four different approaches. I won’t go into the details of the problem or solutions, but our approaches differed in complexity and scope.

First off we had an approach that boiled down to creating a new “box” that would attend to the challenge. It would solve the problem, and it had the potential to be a very powerful tool. But it would take time to build, and would lead to a new thing in our landscape that we would need to connect, support, maintain, document and monitor. It was, however, a pretty sexy solution - so it had a lot going for it.

Our second approach was extremely minimal. It amounted to adding some new things to a database, and changing around some things. It played with our existing tools, but it also risked introducing bugs in our current offering. It would mean taking advantage of some capabilities in our existing stack that we haven’t yet used, but it would also mean that we’d have to be careful not to break anything. It was a very small solution, but it was also a bit risky. Being such a small solution it would also not be very impressive in a demo, it was “not the stuff of legends”.

Our third approach leaned heavily on using some very hyped technology. It would let us do some very cool things that we cannot do today, and would probably give us a cool point from the marketing people. It was not technology we were familiar with, though, so we couldn’t be specific on how long it would take to implement. It was also a bit risky, since we didn’t know how well it would work in our environment.

Our fourth approach was a bit of a wild-card - implementing a fresh new algorithm we came up with to solve our need. The algorithm was conceptually simple, but implementing it in a way that was fast enough and actually solved our problem might get tricky. We also weren’t sure that it would solve all edge-cases, so we’d have to dig deeper into it and test it thoroughly. It was a bit risky, but it was also a very small solution. It’s main advantage was that it would require no new components added and could be a pure software solution.

With all these, and keeping in mind that we wanted a demo that really impressed our co-workers, we decided to go for the first solution. It was unlikely that we’d have a fully working box, but we felt we could create a prototype that proved the concept, and it could even lead to an entirely new product. It did not hurt that it was the brainchild of the most senior developer on the team! We were excited about it, and left for the day feeling good about our choice.

Unexpected challenges

We met on the second and final day, expecting to start creating a small new service with a very limited and specific scope. We had a plan, and we were ready to go.

As they say, life happens. And this Friday had a lot of life happening - particularly with some of our customers and a gnarly database-issue that needed our best people’s attention. Our group was down from five to three, and our small-scoped service suddenly seemed like a very big task. Then we got the message that our demo had been brought forward from 15:00 to 13:00. What’s a few hours? Well, for us it was about a third of our remaining time. We had to rethink our approach.

We put our heads together and looked at our options. We could concede defeat and throw in the towel, we could crunch for a few hours and see what we got to, or we could go for one of the other approaches we had discussed. With such a tight deadline we decided to drop our plans for a cool new service, and go for the second approach, the one that was a little bit risky and not very sexy. We had to go small.

We spent some time working on presentation and story, as our demo was going to be mainly a presentation. We had a few slides, we had a some code and we had a demo. The code-changes were so small that they fit on a single slide. We included a sequence-diagram to explain where the new code would have to fit into our current flow. We even identified a way to make this change with a minimal impact - it would change nothing about the existing tables in our database and include just one new line of code in our server. Our solution was no longer risky.

We had a story that we could tell, and we had a solution that we could show. It was a very short story, and a small change, but it had an impact that solved our challenge. We had to go small, but we had something to show for it.

The demo

Our team of three were two remote people and just one in the offices. And, our demo was up against people present in the office. The other two of us were cheering on remotely, and active in chat, but not our presence was small. We presented our solution with a Miro-board presentation enriched with animations by our remote members moving things around on cue. Delightfully low-tech and home-spun, we thought, at least we’d have a chance at the “best presentation” award.

In the end we actually won the “best solution” -group by popular vote. The voting was anonymous and no reasons were given, but we got an incredible 60% of the votes. This was quite surprising to us, as we had not expected to win anything. We had to go small, but we had something to show for it, and it resonated.

And our hopes of winning “best presentation” - nope. We got smashed in that vote, and came in last. We had to go small, and we had something to show for it, but our presentation obviously did not win the crowd over.

Learning

As I sit and think about it - I think we should have gone for the small solution from the start. Anything that has a minimal impact, but solves a real need is inherently preferable to something that is bigger and more involved that solves that same need. We should have gone small, but we did not until we were forced to. Losing much of our team and time brought the need into a clearer focus - and we got to deliver on something focused and small.

I hope I am wise enough to remember this lesson the next time I am faced with a challenge. I hope I am wise enough to go small.