Enriching logs with [TagProvider] and ILogEnricher

In my previous post I looked at the Microsoft.Extensions.Telemetry.Abstractions NuGet package, the new logging source generator it contains, and its support for [LogProperties] to log whole objects as State with your logs.

In this post I continue to look at the features in the NuGet package, as well as associated implementations in Microsoft.Extensions.Telemetry. I'll show another approach to control how [LogProperties] objects are logged using [TagProvider], and then we'll look at automatic log enrichment.

This post assumes you've already read my previous post on the new logging source generator, but I'll give a quick recap just to set the stage.

The new Microsoft.Extensions.Telemetry source generator

ASP.NET Core has had a source generator built-in to support high-performance logging since .NET 6, but Microsoft.Extensions.Telemetry.Abstractions includes a new source generator that supports more features. The feature I'm looking at in this post is support for [LogProperties] objects.

To use the logging source generator, you create a method that looks something like the following, and decorate it with the [LoggerMessage] attribute:

// 👇 The `[LoggerMessage]` attribute triggers the source generator
[LoggerMessage(Level = LogLevel.Debug, Message = "Generating {ForecastCount} forecasts")]
private static partial void GeneratingForecasts(ILogger logger, int forecastCount);

You have a bit of flexibility in how you define the method. You can make it static or an instance method, and if there's an "ambient" ILogger available in the enclosing class, you can remove the ILogger argument entirely. Alternatively, as Matt Kotsenas pointed out on Mastodon, you can make the method an extension method instead.

The above feature works with both the built-in logging source generator and the Microsoft.Extensions.Telemetry.Abstractions source generator. But the latter also allows you to pass objects decorated with a [LogProperties] object and they'll be logged along with your message.

For example, if you define a method like this:

// 👇 The `[LoggerMessage]` attribute triggers the source generator
[LoggerMessage(Level = LogLevel.Debug, Message = "Generating {ForecastCount} forecasts")]
private static partial void GeneratingForecasts(
    ILogger logger, 
    int forecastCount,
    [LogProperties] WeatherForecast forecast); // 👈 [LogProperties]

Then all the properties of WeatherForecast are added to the State properties logged with your messages:

{
  "EventId": 0,
  "LogLevel": "Debug",
  "Category": "Handler",
  "Message": "Generating 3 forecasts",
  "State": {
    "Message": "{OriginalFormat}=Generating {ForecastCount} forecasts,forecast.TemperatureF=125,forecast.TemperatureC=52,forecast.Date=11/27/2023,ForecastCount=3",
    "{OriginalFormat}": "Generating {ForecastCount} forecasts",
    "ForecastCount": 3,
    "forecast.TemperatureF": 125,
    "forecast.TemperatureC": 52,
    "forecast.Date": "11/27/2023"
  }
}

And if you don't want to include a particular property of your [LogProperties] object in the logs, you can decorate them with [LogPropertyIgnore]:

internal record WeatherForecast(DateOnly Date, int TemperatureC)
{
    [LogPropertyIgnore] // 👈 Add this attribute to exclude TemperatureF from logs
    public int TemperatureF => 32 + (int)(TemperatureC / 0.5556);
}

That's a whistle-stop tour of what I covered in the previous post, so it's time to look at some new features.

Customising [LogProperties] objects with [TagProvider]

The [LogPropertyIgnore] property is likely fine if what you're logging is specific to the method you're working in, for example if it's a DTO of some sort. But what if you're trying to log an object that's part of your domain, or something that's more widely used in your app?

In those cases, adding attributes to the properties feels a bit messy. You probably don't want to be cluttering up your domain models with logging concerns. Similarly, maybe you want to log different things in different locations, perhaps based on the state of the object. That's not possible to define using static attributes.

For these cases, you can use a different approach, the [TagProvider] attribute.

We'll start with our "reduced" WeatherForecast record again. This time it has no [LogPropertyIgnore] (or any other) attributes:

internal record WeatherForecast(DateOnly Date, int TemperatureC)
{
    public int TemperatureF => 32 + (int)(TemperatureC / 0.5556);
}

Instead, we define a dedicated "tag provider" implementation. There's no interface or anything to implement, your implementation just needs the correct "shape". For example, a tag provider that can log a WeatherForecast object might look something like the following:

internal static class WeatherForecastTagProvider
{
    // 👇 Has the required signature 'void RecordTags(ITagCollector, T)'
    public static void RecordTags(ITagCollector collector, WeatherForecast forecast)
    {
        // You can add aribrtrary objects to be logged. 
        // You provide a key (first arg) and a value.
        collector.Add(nameof(forecast.Date), forecast?.Date);
        collector.Add(nameof(forecast.TemperatureC), forecast?.TemperatureC);
    }
}

As you can see in the above code, you can log arbitrary data, based on the WeatherForecast object. So we could, for example, conditionally log TemperatureF, only if the value is < 0, or whatever we want.

With the tag provider defined we can now use it in our logging method. Replace the [LogProperties] attribute with the [TagProvider] attribute as shown below:

// 👇 The `[LoggerMessage]` attribute triggers the source generator
[LoggerMessage(Level = LogLevel.Debug, Message = "Generating {ForecastCount} forecasts")]
private static partial void GeneratingForecasts(
    ILogger logger, 
    int forecastCount,
    [TagProvider(typeof(WeatherForecastTagProvider), nameof(WeatherForecastTagProvider.RecordTags))] WeatherForecast forecast);

The [TagProvider] attribute has two required parameters and one optional parameter:

• The Type to use (required).
• The string name of the method to run on the tag provider (required).
• The OmitReferenceName property (optional). Controls whether the name of the decorated property (forecast in the example above) is included in the log. Defaults to false (so the prefix is included).

As a tangent, I was thinking it might be nice if this attribute could be tidied up a bit. For example, you could use generic attributes to save a small bit of typing: [TagProvider<WeatherForecastTagProvider>(nameof(WeatherForecastTagProvider.RecordTags))]. Or even better, provide a single-parameter overload that looks for the method in the current type [TagProvider(nameof(WeatherForecastTagProvider.RecordTags))]. I'm not sure, maybe it's not worth it 🤷‍♂️

With the [TagProvider] attribute applied to the log message, we can see that TemperatureF is excluded:

{
  "EventId": 0,
  "LogLevel": "Debug",
  "Category": "Handler",
  "Message": "Generating 3 forecasts",
  "State": {
    "Message": "{OriginalFormat}=Generating {ForecastCount} forecasts,forecast.TemperatureC=52,forecast.Date=11/27/2023,ForecastCount=3",
    "{OriginalFormat}": "Generating {ForecastCount} forecasts",
    "ForecastCount": 3,
    "forecast.TemperatureC": 52,
    "forecast.Date": "11/27/2023"
  }
}

Note that in the above example the tags we added are logged under State, and the forecast. prefix is added to the entries (because I didn't specify OmitReferenceName=false in the [TagProvider] attribute).

Implementation-wise, supporting the [TagProvider] in the source generated code is a trivial update. I explained all this generated code in the previous post, so I've just highlighted the important line in which our tag provider is called.

private static partial void GeneratedForecast(ILogger logger, int forecastCount, WeatherForecast forecast)
{
    if (!logger.IsEnabled(LogLevel.Debug))
    {
        return;
    }

    LoggerMessageState state = LoggerMessageHelper.ThreadLocalState;

    _ = state.ReserveTagSpace(2);
    state.TagArray[1] = new("ForecastCount", forecastCount);
    state.TagArray[0] = new("{OriginalFormat}", "Generating {ForecastCount} forecasts");
    state.TagNamePrefix = nameof(forecast);
    
    // 👇 Call our custom tag provider, passing in the LoggerMessageState
    // as the ICollector instance
    WeatherForecastTagProvider.RecordTags(state, forecast);

    logger.Log(
        LogLevel.Debug,
        new(0, nameof(GeneratedForecast)),
        state,
        null,
        static (s, _) =>
        {
            var ForecastCount = s.TagArray[1].Value;
            return System.FormattableString.Invariant($"Generating {ForecastCount} forecasts");
        });

    state.Clear();
}

By using [TagProvider] you could log different properties of your objects in different places, or based on the state of the object itself. You could also support "proper" logging of collections of objects, for example:

internal static class WeatherForecastTagProvider
{
    public static void RecordTags(ITagCollector collector, WeatherForecast[] forecasts)
    {
        for (int i = 0; i < forecasts.Length; i++)
        {
            var forecast = forecasts[i];
            collector.Add($"[{i}]{nameof(forecast.Date)}", forecast?.Date);
            collector.Add($"[{i}]{nameof(forecast.TemperatureC)}", forecast?.TemperatureC);

        }
    }
}

This will log each of the objects similar to the following:

{
  "State": {
    "forecasts.[0]Date": "11/29/2023",
    "forecasts.[0]TemperatureC": -3,
    "forecasts.[1]Date": "11/30/2023",
    "forecasts.[1]TemperatureC": -4
  }
}

Obviously you need to be careful that you're not going to log 10,000 objects if you do this! 🙈

We'll leave [LogProperties] and [TagProvider] for now to take a look at a related feature, where you always want to log values to your logs.

Adding properties to all logs using enrichment

For a long time, Serilog has had the concept of "enrichment". In its most basic form, enrichment adds one or more values to all log messages. For example, you might want to always add your application's environment name to all logs. Alternatively, enrichment could be context specific, so you might want to ensure you always add the selected route to logs written by an ASP.NET Core app.

Enrichment is very closely related to the concept of scopes support by ILogger. The main difference is that you define a new scope "inline" in your code, in an imperative way, whereas enrichment happens in a more declarative way, as you'll see shortly. There may also be differences in how different LoggingProviders treat scopes compared to enriched state.

So far we've been looking at the Microsoft.Extensions.Telemetry.Abstractions NuGet package, but for enrichment we'll need to pull in the Microsoft.Extensions.Telemetry package instead. Add it to your project using

dotnet add package Microsoft.Extensions.Telemetry

Or by adding the package directly to your .csproj file:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <InvariantGlobalization>true</InvariantGlobalization>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Telemetry" Version="8.0.0" />
  </ItemGroup>

</Project>

We'll start by exploring some the built-in enricher support, namely the AddProcessLogEnricher() method. To use enrichment, you must:

• Enable enrichment globally by calling ILoggingBuilder.EnableEnrichment().
• Add one or more enrichers to the service collection.

In the following example, I've added one of the built-in enrichers: the process enricher.

var builder = WebApplication.CreateBuilder(args);
builder.Logging.EnableEnrichment(); // Enable log enrichment
builder.Services.AddProcessLogEnricher(x =>
{
    x.ProcessId = true; // Add the process ID (true by default)
    x.ThreadId = true; // Add the managed thread ID (false by default)
});

var app = builder.Build();

app.MapGet("/weatherforecast", Handler.GetForecasts);

app.Run();

With this setup, all your logs will have process.pid and thread.id added to the State object:

{
  "EventId": 0,
  "LogLevel": "Debug",
  "Category": "Handler",
  "Message": "Generating 2 forecasts",
  "State": {
    "Message": "Microsoft.Extensions.Logging.ExtendedLogger\u002BModernTagJoiner",
    "{OriginalFormat}": "Generating {ForecastCount} forecasts",
    "ForecastCount": 2,
    "thread.id": "4",      // 👈 Added by the enricher
    "process.pid": "22412" // 👈 Added by the enricher
  }
}

When you call builder.Logging.EnableEnrichment() you're actually replacing the default ILoggerFactory with a new implementation, ExtendedLoggerFactory. This is responsible for the enrichment feature, but also other features that I'll cover in subsequent posts, such as redaction.

One oddity in the implementation is that the "State.Message" key is no longer a "useful" value, instead it reveals the underlying type name. This feels like a bug to me (though I haven't dug into it properly yet).

There's one other enricher included in the Microsoft.Extensions.Telemetry package: the service log enricher. You can add it to your application by calling AddServiceLogEnricher()

var builder = WebApplication.CreateBuilder(args);
builder.Logging.EnableEnrichment();
builder.Services.AddServiceLogEnricher(options =>
{
    options.ApplicationName = true; // Choose which values to add to the logs
    options.BuildVersion = true;
    options.DeploymentRing = true;
    options.EnvironmentName = true;
});

After making this change, you might be surprised to find that your logs are enriched, but not with anything useful:

{
  "EventId": 0,
  "LogLevel": "Debug",
  "Category": "Handler",
  "Message": "Generating 4 forecasts",
  "State": {
    "Message": "Microsoft.Extensions.Logging.ExtendedLogger\u002BModernTagJoiner",
    "{OriginalFormat}": "Generating {ForecastCount} forecasts",
    "ForecastCount": 4,
    "service.name": "",          // 👈 Enriched, but
    "deployment.environment": "" // 👈 not very useful
  }
}

After a bit of digging, I realised that you have to also define these values in an IOptions object, for example:

builder.Services.AddApplicationMetadata(x =>
{
    x.ApplicationName = "My App";
    x.BuildVersion = "1.2.3";
    x.EnvironmentName = "Development";
    x.DeploymentRing = "Canary";
});

In case its not apparent, the behaviour of all of these enrichers, including the values provided above can be configured using the standard IConfiguration and IOptions system. So you can define them in environment variables, appSettings.json, or a Key Vault—whatever you need.

With these values added, the enricher now has something to add!

{
  "EventId": 0,
  "LogLevel": "Debug",
  "Category": "Handler",
  "Message": "Generating 1 forecasts",
  "State": {
    "Message": "Microsoft.Extensions.Logging.ExtendedLogger\u002BModernTagJoiner",
    "{OriginalFormat}": "Generating {ForecastCount} forecasts",
    "ForecastCount": 1,
    "service.name": "My App", // 👈 Enriched logs
    "deployment.environment": "Development", // 👈 Enriched logs
    "DeploymentRing":"Canary",  // 👈 Enriched logs
    "service.version": "1.2.3" // 👈 Enriched logs
  }
}

These built-in enrichers are handy, but you can also create your own implementations.

Creating a custom enricher

You can create your own enrichers by deriving from the ILogEnricher or IStaticLogEnricher interfaces:

• IStaticLogEnricher—enrichers are created once on app startup, and their tags added to the logger. Use if the values you're adding won't change for the app lifetime.
• IStaticLogEnricher—enrichers are invoked every time you write a log, so good for values that may change.

For our example, we'll create a simple IStaticLogEnricher that adds the current machine name to the log. You need to implement a single method, Enrich(IEnrichmentTagCollector) and add any tags you need to it:

internal class MachineNameEnricher : IStaticLogEnricher
{
    public void Enrich(IEnrichmentTagCollector collector)
    {
        collector.Add("MachineName", Environment.MachineName);
    }
}

You can register your enricher with the AddStaticLogEnricher<T> extension method (there's also an AddLogEnricher<T> version for ILogEnricher):

var builder = WebApplication.CreateBuilder(args);
builder.Logging.EnableEnrichment();
// register the custom enricher 👇
builder.Services.AddStaticLogEnricher<MachineNameEnricher>(); 

And sure enough, the machine name is added to the logs:

{
  "EventId": 0,
  "LogLevel": "Debug",
  "Category": "Handler",
  "Message": "Generated Forecast",
  "State": {
    "Message": "Microsoft.Extensions.Logging.ExtendedLogger\u002BModernTagJoiner",
    "{OriginalFormat}": "Generated Forecast",
    "forecast.Date": "11/30/2023",
    "forecast.TemperatureC": 15,
    "MachineName": "MYLAPTOP" // 👈 Adding the machine name
  }
}

And that brings us to the end of this post. In the next post, we'll look at the redaction features that are part of the telemetry and compliance packages.

Summary

In this post I showed some of the new features included in Microsoft.Extensions.Telemetry.Abstractions and Microsoft.Extensions.Telemetry. This includes the [TagProperties] attribute as an alternative to [LogProperties] and [LogPropertyIgnore] for controlling which properties of your objects are logged. I also demonstrated the enrichers feature, which lets you add values to all log messages. Finally, I created a custom enricher and showed how to register it in your app.