A brief introduction to DiagnosticSource

In this post I talk a little about the DiagnosticSource infrastructure, how it compares to the other logging infrastructure in ASP.NET Core, and give a brief example of how to use it to listen to framework events.

I wrote an introductory blog post looking at DiagnosticSource back in the ASP.NET Core 1.x days, but a lot has changed (and improved) since then!

ASP.NET Core logging systems

There are, broadly speaking, three logging systems in ASP.NET Core:

1. EventSource - Fast and strongly typed. Designed to interface with OS logging systems.
2. ILogger - An extensible logging system designed to allow you to plug in additional consumers of logging events.
3. DiagnosticSource - Similar in design to EventSource, but does not require the logged data be serialisable.

EventSource has been available since the .NET Framework 4.5 era and is used extensively by the framework to instrument itself. The data that gets logged is strongly typed, but must be serialisable as the data is sent out of the process to be logged. Ultimately, EventSource is designed to interface with the underlying operating system's logging infrastructure, e.g. Event Tracing for Windows (ETW), LTTng on Linux, as well as to EventPipes (generally the prefered approach with modern .NET).

The ILogger infrastructure is the most commonly used logging ASP.NET Core infrastructure in application code. You can log to the ILogger infrastructure by injecting an instance of ILogger into your classes, and calling, for example, ILogger.LogInformation(). The infrastructure is generally designed for logging strings, but it allows you to pass objects as additional parameters which can be used for structured logging (such as that provided by SeriLog). Generally speaking, the ILogger implementation will be the infrastructure you want to use in your applications, so check out the documentation if you are not familiar with it.

The DiagnosticSource infrastructure is very similar to the EventSource infrastructure, but the data being logged does not leave the process, so it does not need to be serialisable. There is also an adapter to allow converting DiagnosticSource events to ETW events which can be useful in some cases.

There are other logging APIs too, such as the legacy System.Diagnostics.Trace API, and the Windows-only EventLog APIs, but the three listed above are the main ones to think about in modern .NET.

Now we have a vague overview of DiagnosticSource works, let's look at an example of how you you can use it to listen to events in an ASP.NET Core application.

Subscribing to DiagnosticListener events

The framework libraries in ASP.NET Core emit a large number of diagnostic events. These have very low overhead when not enabled, but allow you to tap into them on demand. As an example, we'll look at some of the hosting events emitted in .NET 6.

The following shows a basic .NET 6 application, created with dotnet new web:

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
WebApplication app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

Now we'll configure a DiagnosticListener to listen for the "hosting" events, record the name of the event, and write the type of the object we receive.

DiagnosticListener derives from the abstract DiagnosticSource and is the type you typically interact with, as you'll see.

First, we need an IObserver<DiagnosticListener> implementation. We'll use this to subscribe to "categories" of events. In the following example, we'll listen for the "Microsoft.Extensions.Hosting" events (emitted by HostBuilder):

using System.Diagnostics;

public class TestDiagnosticObserver : IObserver<DiagnosticListener>
{
    public void OnNext(DiagnosticListener value)
    {
        if (value.Name == "Microsoft.Extensions.Hosting")
        {
            value.Subscribe(new TestKeyValueObserver());
        }
    }
    public void OnCompleted() {}
    public void OnError(Exception error) {}
}

Each "component" in the .NET framework will emit a DiagnosticListener with a given name. The IObserver<DiagnosticListener> is notified each time a new listener is emitted, giving you the option to subscribe to it (or ignore it).

The important method for our purposes is OnNext. It's here we subscribe to a DiagnosticListener's stream of events. In this example we pass in another custom type, TestKeyValueObserver, which is shown below. This is the class that will actually receive the events emitted by the DiagnosticListener instance.

The event is emitted as a KeyValuePair<string, object?>, synchronously, inline with the framework code, so you can manipulate the real objects! In the example below I'm simply writing the name of the event and the type of the provided object, but you could do anything here!

using System.Diagnostics;

public class TestKeyValueObserver : IObserver<KeyValuePair<string, object?>>
{
    public void OnNext(KeyValuePair<string, object?> value)
    {
        Console.WriteLine($"Received event: {value.Key} with value {value.Value?.GetType()}");
    }
    public void OnCompleted() {}
    public void OnError(Exception error) {}
}

Note that the object provided is object?, and it could be anything. However, if you know the name of the event (from value.Key) then you will also know the type of the object provided. For example, if we look into the source code of HostBuilder, we can see that the "HostBuilding" event passes in a HostBuilder object.

The last step is to register our TestDiagnosticObserver in the application, right at the start of Program.cs. This ensures your TestDiagnosticObserver receives the stream of DiagnosticListeners:

DiagnosticListener.AllListeners.Subscribe(new TestDiagnosticObserver());

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
// ... etc

Sure enough, if we now run the app, we'll see the following logged to the console:

Received event: HostBuilding with value Microsoft.Extensions.Hosting.HostBuilder
Received event: HostBuilt with value Microsoft.Extensions.Hosting.Internal.Host

This is all very nice, but the real power of the DiagnosticListener events comes from the fact that you get strongly-typed objects.

Manipulating strongly-typed objects in an observer

In the previous example, I showed that you're receiving strongly-typed objects, but I didn't really take advantage of that. Just to prove it's possible, lets manipulate the HostBuilder object passed with the "HostBuilding" event!

public class ProductionisingObserver : IObserver<KeyValuePair<string, object?>>
{
    public void OnCompleted() {}
    public void OnError(Exception error) {}

    public void OnNext(KeyValuePair<string, object?> value)
    {
        if(value.Key == "HostBuilding")
        {
            var hostBuilder = (HostBuilder)value.Value;
            hostBuilder.UseEnvironment("Production");
        }
    }
}

The ProductionisingObserver above listens for a single event, "HostBuilding", and then casts the event's object value to the HostBuilder type. This is important, as we can now manipulate the object directly. In the example above I am forcing the application Environment to be Production. Sure enough, if you wire this up to the TestDiagnosticObserver and dotnet run the application, you'll see that the Environment has switched to Production:

info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production

This is a silly little example, but DiagnosticListeners can provide hooks deep into the framework. In fact, some of the .NET tools use these events for just this: in a previous post I described how the EF Core tools were updated to work with .NET 6's minimal hosting by using the DiagnosticSource infrastructure.

Creating the various IObserver implementations requires a bit of boilerplate, and the only good way to know which events exist and their associated object Types is to look through the source code (or experiment!).

Some parts of the framework are easier to work with than others of course, EF Core, for example, centralises the names of its DiagnosticListener instances and provides strongly typed event names. This makes it easier to listen to events of interest:

public void OnNext(KeyValuePair<string, object> value)
{
    // listens for the ConnectionOpening event in the Microsoft.EntityFrameworkCore category
    if (value.Key == RelationalEventId.ConnectionOpening.Name)
    {
        var payload = (ConnectionEventData)value.Value;
        Console.WriteLine($"EF is opening a connection to {payload.Connection.ConnectionString} ");
    }
}

Once you have figured out the name of the event you need, and the Type of the object it supplies, creating an IObserver to use the value is pretty easy, as you can just cast the provided object to the concrete Type.

So what if you can't cast the provided object, because it's an anonymous Type, created with the new { } syntax? In the next post I'll describe why anyone would do that, as well as how to work with the provided `object.

Summary

In this post I described the DiagnosticSource infrastructure, and how it differs from the ILogger and EventSource logging APIs. I showed how you could create a simple IObserver<> to listen for DiagnosticListener events, and to use those to log to the console. I also showed that you can manipulate the provided object, because your code is executed synchronously from the calling site. In this post I only dealt with events that emit named Types; in the next post I'll look at events that use anonymous Types.