Behind [LogProperties] and the new telemetry logging source generator

In this post I take a look at an apparently simple enhancement to logging introduced along with .NET 8: the [LogProperties] attribute. This attribute can make it easier to log additional state in your log messages as you'll see later. While looking into the specifics of how this new attribute works, I discovered bigger changes to the logging source generator.

This post digs into those source generator changes associated with the Microsoft.Extensions.Telemetry.Abstractions NuGet package, looks at the generated code and how it works, and checks how [LogProperties] fits in.

The built-in logging source-generator

Source generators were a headline feature way back in .NET 5, and each subsequent release of .NET has included more and more source generators built-in to the framework. One of the first such source generators was the logging source generator, driven by the [LoggerMessage] attribute.

I wrote a big post about the logging source generator way back in .NET 6. That post discusses the "evolution" of logging approaches, shows why the source generator is so useful, and adds benchmarks for the various approaches. All of this is still applicable in .NET 8, so I suggest reading that post first if you haven't already, as I'm not going to duplicate it here!

For this post I'm going to assume that you're already using (or considering using) the logging source generator to create your log messages. This can be particularly useful if you're writing high-performance code where every ms or byte of allocation matters.

I don't know if I'd recommend the source generator in all circumstances, as it does add ceremony to your code. But if you're writing a library that uses the ILogger abstractions I think it makes a lot of sense.

In the following example, I start with the simple weather forecast example app. I've then refactored it a little, making the following changes:

• Move the minimal API handler to a separate Handler class.
• Update the implementation to generate a random number of WeatherForecast objects
• Log the number of objects generated using the [LoggerMessage] attribute source generator.

The code below shows these changes, and highlights the important points

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

// Minimal API using the static handler pattern 👇
app.MapGet("/weatherforecast", Handler.GetForecasts);

app.Run();

// Note  👇 The handler must be partial when you're using the source generator
internal partial class Handler
{ 
    // Inject the ILogger instance using DI     👇
    public static WeatherForecast[] GetForecasts(ILogger<Handler> logger)
    {
        var entriesToGenerate = Random.Shared.Next(5); // Generate a number between 0-4
        GeneratingForecasts(logger, entriesToGenerate); // 👈 Log the number of forecasts

        // Generate the instances
        var entries = new WeatherForecast[entriesToGenerate];
        for (int i = 0; i < entriesToGenerate; i++)
        {
            var forecast = new WeatherForecast
            (
                Date: DateOnly.FromDateTime(DateTime.Now.AddDays(i)),
                TemperatureC: Random.Shared.Next(-20, 55)
            );

            entries[i] = forecast;
        }

        return entries;

    }

    // Use the source generator to log the number of forecasts
    [LoggerMessage(
        Level = LogLevel.Debug,
        Message = "Generating {ForecastCount} forecasts")]
    private static partial void GeneratingForecasts(ILogger logger, int forecastCount);
}

// The slightly simplified WeatherForecast record
internal record WeatherForecast(DateOnly Date, int TemperatureC)
{
    public int TemperatureF => 32 + (int)(TemperatureC / 0.5556);
}

The [LoggerMessage] attribute applied to the GeneratingForecasts() method is the trigger for the logging source-generator to do it's thing. If we navigate to the generated definition for this method in the IDE, we find code that looks like the snippet below. I've heavily annotated it to point out the interesting features of this code.

// Generated as a partial class
partial class Handler
{
    // Callback which efficiently writes the log message to the ILogger
    private static readonly Action<ILogger, int, .Exception?> __GeneratingForecastsCallback =
        // 👇 Using LoggerMessage.Define means the template is only parsed once, 
        // and is cached here, instead of being re-parsed on every invocation
        LoggerMessage.Define<int>( 
            LogLevel.Debug, new EventId(1759284123, nameof(GeneratingForecasts)), 
            "Generating {ForecastCount} forecasts", 
            new LogDefineOptions()
            {                           // The enabled check can be skipped in the 
                SkipEnabledCheck = true // generated code, as it's already performed
            });                         // in the GeneratingForecasts method

    private static partial void GeneratingForecasts(ILogger logger, int forecastCount)
    {
        // Ensures a fast path with no-allocation if logging is not enabled for this message
        if (logger.IsEnabled(LogLevel.Debug))
        {
            __GeneratingForecastsCallback(logger, forecastCount, null);
        }
    }
}

So that's what you get out-of-the-box. The [LoggerMessage] attribute and generator is included as part of the Microsoft.Extensions.Logging.Abstractions NuGet package, which is included in all ASP.NET Core apps, so it's basically always available. Now let's look at the new bits.

Logging whole objects with [LogProperties]

As part of the .NET 8 launch at .NET Conf 2023, Microsoft announced several new packages based on adding telemetry, observability, and resilience to your applications. The [LogProperties] attribute was a relatively minor announcement as part of general additional ILogger features added in .NET 8.

I'll discuss most of those new features, such as Personal Identifiable Information (PII) redaction and enrichment in a separate post. In this post I focus solely on the basic source generator changes and [LogProperties].

The [LogProperties] attribute is designed to be used with the existing [LoggerMessage] source generation. You pass additional parameters to your logging message, decorate them with [LogProperties], and the logger will include all the object's properties in the log message.

For example, we could add a log message that's called with each of the WeatherForecast objects we generate. Even though the message template doesn't have any parameters, each of the public properties of WeatherForecast are passed to the logger (we'll look into the details of what that means shortly).

[LoggerMessage(
    Level = LogLevel.Debug,
    Message = "Generated Forecast")] // Log message without parameters
private static partial void GeneratedForecast(
  ILogger logger, 
  [LogProperties] WeatherForecast forecast); // 👈 [LogProperties]

Before we can try this out, we need to add the Microsoft.Extensions.Telemetry.Abstractions NuGet package so we can use [LogProperties].

The Microsoft.Extensions.Telemetry.Abstractions package

You can add the abstractions library to your application by running

dotnet add package Microsoft.Extensions.Telemetry.Abstractions

Or by adding the package directly to your .csproj file:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <InvariantGlobalization>true</InvariantGlobalization>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Telemetry.Abstractions" Version="8.0.0" />
  </ItemGroup>

</Project>

One point of interest, Microsoft.Extensions.Telemetry.Abstractions is not tied to .NET 8; the package supports .NET Framework and .NET 6+!

One thing I was surprised to find is that the [LoggerMessage] attribute source generator has been completely rewritten for this package! Even if you don't use the [LogProperties] attribute or any other features, the generated code is completely different! One of the most obvious differences in the generated code is that it doesn't use Logger.Define any more.

We'll look in detail at the generated code in the next section.

I was interested to see exactly how it was achieving that: was it tying into the existing source generator somehow, or replacing it completely. Digging into the NuGet package, it was clear that it was latter:

The Microsoft.Extensions.Telemetry.Abstractions package contains its own logging source generator (the source code for which is here) as well as .props and .targets files that explicitly disable the Microsoft.Extensions.Logging.Abstractions generator:

The .props file contains:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <!-- This package should replace the Microsoft.Extensions.Logging.Abstractions source generator, so we set the property to remove the source generator from the project. -->
  <PropertyGroup>
    <DisableMicrosoftExtensionsLoggingSourceGenerator>true</DisableMicrosoftExtensionsLoggingSourceGenerator>
  </PropertyGroup>
</Project>

And the .targets file contains:

<Project>
  <!-- This package should replace the Microsoft.Extensions.Logging.Abstractions source generator. -->
  <Target Name="_Microsoft_Extensions_Logging_AbstractionsRemoveAnalyzers" 
          Condition="'$(DisableMicrosoftExtensionsLoggingSourceGenerator)' == 'true'"
          AfterTargets="ResolveReferences">
    <ItemGroup>
      <_Microsoft_Extensions_Logging_AbstractionsAnalyzer Include="@(Analyzer)" 
        Condition="'%(Analyzer.AssemblyName)' == 'Microsoft.Extensions.Logging.Generators' 
                Or '%(Analyzer.NuGetPackageId)' == 'Microsoft.Extensions.Logging.Abstractions'" />
    </ItemGroup>

    <!-- Remove Microsoft.Extensions.Logging.Abstractions Analyzer -->
    <ItemGroup>
      <Analyzer Remove="@(_Microsoft_Extensions_Logging_AbstractionsAnalyzer)" />
    </ItemGroup>
  </Target>
</Project>

That explains exactly what the package is doing, so now let's look at the generated code!

Looking at the generated code from Microsoft.Extensions.Telemetry.Abstractions

Before we consider the [LogProperties] attribute, lets look first at how the existing generated code changes after adding the Microsoft.Extensions.Telemetry.Abstractions package. Remember, that this was the original logging method definition:

[LoggerMessage(
    Level = LogLevel.Debug,
    Message = "Generating {ForecastCount} forecasts")]
private static partial void GeneratingForecasts(ILogger logger, int forecastCount);

Previously the generated code used LoggerMessage.Define<T>() but you can see in the following snippet that the generated code is now very different! I've annotated the code to explain how it works in general:

// As always, the generated code is in a partial class
partial class Handler
{
    private static partial void GeneratingForecasts(ILogger logger, int forecastCount)
    {
        // If logging isn't enabled for this logger, bail out early
        if (!logger.IsEnabled(LogLevel.Debug))
        {
            return;
        }

        // LoggerMessageState is a "wrapper" for a KeyValuePair<string, object>[]
        // Together with LoggerMessageHelper it uses pooling to reduce allocations
        LoggerMessageState state = LoggerMessageHelper.ThreadLocalState;

        // Ensure the state has the correct size, and record two parameters
        _ = state.ReserveTagSpace(2);
        state.TagArray[1] = new("ForecastCount", forecastCount);
        state.TagArray[0] = new("{OriginalFormat}", "Generating {ForecastCount} forecasts");

        // This is the "standard" `ILogger.Log` message
        logger.Log(
            LogLevel.Debug,
            new EventId(0, nameof(GeneratingForecasts)),
            state, // The LoggerMessageState is passed as the state
            exception: null,
            formatter: static (s, _) => // s is the LoggerMessageState we passed in
            {
                // Extracts the value we passed in to the method
                var ForecastCount = s.TagArray[1].Value;

                // Use FormattableString to efficiently generate the final message
                // For example, this returns "Generating 3 forecasts" 
                return FormattableString.Invariant($"Generating {ForecastCount} forecasts");
            });

        // Clear the state (effectively return it to the "pool")
        state.Clear();
    }
}

As you can see, this is very different to the original generated code. Of particular interest:

• The library includes a LoggerMessageState type that acts as a "wrapper" around an array of KeyValuePair<string, object> which are available when writing the log messages
• A cache/pool of LoggerMessageState is created by using ThreadLocal state.
• The original format message and parameters are stored in LoggerMessageState, which can then be accessed by the LoggingProvider implementation and written to the final log.
• The formatter uses the FormattableString type to efficiently generate the final message without requiring an explicit "parsing" step.

The LoggerMessageState and LoggerMessageHelper types are part of the Microsoft.Extensions.Telemetry.Abstractions library. This post is already getting a little long, so I won't explore their implementation in this post, but you can find them on GitHub here. For the most part, they're not doing anything magical, they're mostly a mechanism for reusing arrays to avoid allocations.

Interestingly, you might see some small changes to your log messages just from adding the package. For example, if you're using the built-in JSON console logger, then without the package, the generated log will look something like this:

{
  "EventId": 823449399,
  "LogLevel": "Debug",
  "Category": "Handler",
  "Message": "Generating 4 forecasts",
  "State": {
    "Message": "Generating 4 forecasts",
    "ForecastCount": 4,
    "{OriginalFormat}": "Generating {ForecastCount} forecasts"
  }
}

Whereas with the package it looks like this:

{
  "EventId": 0,
  "LogLevel": "Debug",
  "Category": "Handler",
  "Message": "Generating 4 forecasts",
  "State": {
    "Message": "{OriginalFormat}=Generating {ForecastCount} forecasts,ForecastCount=4",
    "{OriginalFormat}": "Generating {ForecastCount} forecasts",
    "ForecastCount": 4
  }
}

Note the difference in the State.Message property. This difference occurs because of the way the JSON console logger calls ToString() on the state object. In the former case, the state is of type LoggerMessage.LogValues<int>, whereas in the latter case it's the LoggerMessageState object. The difference in ToString() behaviour describes the differences here. Other logging providers won't necessarily handle the provided state object in the same way.

Right, it's finally time to look at the new [LogProperties] attribute and how it works with the new source generator.

Logging whole objects with the [LogProperties] Attribute

At the start of this post I said my investigation of the new source generator was driven by the [LogProperties] attribute. We now (finally!) have all the background we need to understand this feature!

First, lets define our logging message, and pass in an object to log using [LogProperties]

[LoggerMessage(Level = LogLevel.Debug,Message = "Generated Forecast")]
private static partial void GeneratedForecast(
  ILogger logger, 
  [LogProperties] WeatherForecast forecast); // 👈 [LogProperties]

If we look at the generated code for this method, we can quickly see the impact of [LogProperties]:

// As always, the generated code is in a partial class
partial class Handler
{
     private static partial void GeneratedForecast(ILogger logger, WeatherForecast forecast)
    {
        if (!logger.IsEnabled(LogLevel.Debug))
        {
            return;
        }

        LoggerMessageState state = LoggerMessageHelper.ThreadLocalState;
        _ = state.ReserveTagSpace(4);

        // 👇Each of the object's propeties are added to the state
        state.TagArray[3] = new("forecast.Date", forecast?.Date);
        state.TagArray[2] = new("forecast.TemperatureC", forecast?.TemperatureC);
        state.TagArray[1] = new("forecast.TemperatureF", forecast?.TemperatureF);
        state.TagArray[0] = new("{OriginalFormat}", "Generated Forecast");

        logger.Log(
            LogLevel.Debug,
            new EventId(0, nameof(GeneratingForecasts)),
            state, // The LoggerMessageState is passed as the state
            exception: null,
            formatter: static (s, _) =>
            {
                return "Generated Forecast";
            });

        state.Clear();
    }
}

There's only really one difference in the generated code: each of the WeatherForecast public properties are included in the State object. If we look at the resulting log message written by the JSON console formatter we can see these properties all written out:

{
  "EventId": 0,
  "LogLevel": "Debug",
  "Category": "Handler",
  "Message": "Generated Forecast",
  "State": {
    "Message": "{OriginalFormat}=Generated Forecast,forecast.TemperatureF=125,forecast.TemperatureC=52,forecast.Date=11/27/2023",
    "{OriginalFormat}": "Generated Forecast",
    "forecast.TemperatureF": 125,
    "forecast.TemperatureC": 52,
    "forecast.Date": "11/27/2023"
  }
}

Customizing the generated code

By default, [LogProperties] adds the parameter name as a prefix to the key and writes values whether they are null or not. You can customise this behaviour by setting properties on the [LogProperties] attribute itself (they default to false if not set). For example:

[LoggerMessage(Level = LogLevel.Debug, Message = "Generated Forecast")]
private static partial void GeneratedForecast(
    ILogger logger, 
    [LogProperties(OmitReferenceName = true, SkipNullProperties = true)] WeatherForecast forecast);

With this change, the generated state logging code changes to the following:

LoggerMessageState state = LoggerMessageHelper.ThreadLocalState;
_ = state.ReserveTagSpace(1);

state.TagArray[0] = new("{OriginalFormat}", "Generated Forecast");
state.AddTag("Date", forecast?.Date);
state.AddTag("TemperatureC", forecast?.TemperatureC);
state.AddTag("TemperatureF", forecast?.TemperatureF);

There's a few things to note here:

• The properties are missing the "forecast." prefix as requested
• Only 1 entry is reserved in the state instead of the 4 we had previously
• AddTag() is used to conditionally add to the state. With each call, it first checks the value to log is not null, and then expands the array by 1 (if necessary) before recording the value.

Controlling what gets logged

Using [LogProperties] and logging the whole object is convenient, as it means you don't have to manually list out all the properties you want to log with each message. However, maybe there are some fields that you don't want to log at all, perhaps because they're not interesting or they're sensitive.

You can tell the [LogProperties] generator to ignore a property by adding the [LogPropertyIgnore] attribute to it. For example, maybe we don't want to log TemperatureF because it's trivially calculated from TemperatureC. In which case, we can add LogPropertyIgnore to it:

internal record WeatherForecast(DateOnly Date, int TemperatureC)
{
    [LogPropertyIgnore] // 👈 Add this attribute
    public int TemperatureF => 32 + (int)(TemperatureC / 0.5556);
}

and the generated code is updated accordingly:

LoggerMessageState state = LoggerMessageHelper.ThreadLocalState;
_ = state.ReserveTagSpace(3);

state.TagArray[2] = new("forecast.Date", forecast?.Date);
state.TagArray[1] = new("forecast.TemperatureC", forecast?.TemperatureC);
state.TagArray[0] = new("{OriginalFormat}", "Generated Forecast");
// No forecast?.TemperatureF is logged

Logging multiple objects

One thing I decided to try was whether I could pass an IEnumerable or IDictionary of objects, and have the generator log all the objects. For example, can we pass all the generated WeatherForecast objects to our original log message definition?

[LoggerMessage(Level = LogLevel.Debug, Message = "Generating {ForecastCount} forecasts")]
private static partial void GeneratingForecasts(
  ILogger logger,
  int forecastCount,
  [LogProperties] WeatherForecast[] forecasts); // 👈 Array/IEnumerable of forecasts

When I first tried this approach, I expected/hoped that each object would have all of its properties explicitly written out, using a key like "forecasts[0].Date" for example, but that's not what happens.

I didn't notice originally, but in the above example, the source generator adds a warning to the forecasts parameter, LOGGEN018, warning that the value can't be logged.

In the generated code, the whole array is "stringified" by the LoggerMessageHelper into a single property, "forecasts":

LoggerMessageState state = LoggerMessageHelper.ThreadLocalState;
_ = state.ReserveTagSpace(3);

state.TagArray[2] = new("ForecastCount", forecastCount);
var forecastsToLog = forecasts != null 
    ? LoggerMessageHelper.Stringify(forecasts)
    : null;
state.TagArray[1] = new("forecasts", forecastsToLog);
state.TagArray[0] = new("{OriginalFormat}", "Generating {ForecastCount} forecasts");

Which gives a log message that looks something like this:

{
  "EventId": 0,
  "LogLevel": "Debug",
  "Category": "Handler",
  "Message": "Generating 4 forecasts",
  "State": {
    "Message": "{OriginalFormat}=Generating {ForecastCount} forecasts,forecasts=[\"WeatherForecast { Date = 11/25/2023, TemperatureC = 16, TemperatureF = 60 }\",\"WeatherForecast { Date = 11/26/2023, TemperatureC = 0, TemperatureF = 32 }\",\"WeatherForecast { Date = 11/27/2023, TemperatureC = 25, TemperatureF = 76 }\",\"WeatherForecast { Date = 11/28/2023, TemperatureC = 37, TemperatureF = 98 }\"],ForecastCount=4",
    "{OriginalFormat}": "Generating {ForecastCount} forecasts",
    "forecasts": "[\"WeatherForecast { Date = 11/25/2023, TemperatureC = 16, TemperatureF = 60 }\",\"WeatherForecast { Date = 11/26/2023, TemperatureC = 0, TemperatureF = 32 }\",\"WeatherForecast { Date = 11/27/2023, TemperatureC = 25, TemperatureF = 76 }\",\"WeatherForecast { Date = 11/28/2023, TemperatureC = 37, TemperatureF = 98 }\"]",
    "ForecastCount": 4
  }
}

I can understand why they took this approach, as it ensures that state has a bounded (fixed in this case) size, but it could be a little surprising to find this renders so differently.

It's important to be aware that the generator literally calls ToString() on each object, so decorating your properties with [LogPropertyIgnore] does nothing. You can see TemperatureF appears in the log message above, despite being excluded.

And that's it for this post. In a subsequent post I'll look at some of the additional features of the source generator, such as data classification and tag providers!

Summary

In this post I described how the new Microsoft.Extensions.Telemetry.Abstractions package replaces the Microsoft.Extensions.Logging.Abstractions source generator. I showed the differences in the generated code when you add the new package, and how it no longer uses LoggerMessage.Define<T>. Finally, I showed how you can use the [LogProperties] attribute to log all the properties of an object along with your log message.