For apps running in Kubernetes, it's particularly important to be storing log messages in a central location. I'd argue that this is important for all apps, whether or not you're using Kubernetes or docker, but the ephemeral nature of pods and containers make the latter cases particularly important.
If you're not storing logs from your containers centrally, then if a container crashes and is restarted, the logs may be lost forever.
There are lots of ways you can achieve this. You could log to Elasticsearch or Seq directly from your apps, or to an external service like Elmah.io for example. One common approach is to use Fluentd to collect logs from the Console output of your container, and to pipe these to an Elasticsearch cluster.
By default, Console log output in ASP.NET Core is formatted in a human readable format. If you take the Fluentd/Elasticsearch approach, you'll need to make sure your console output is in a structured format that Elasticsearch can understand, i.e. JSON.
In this post, I describe how you can add Serilog to your ASP.NET Core app, and how to customise the output format of the Serilog Console sink so that you can pipe your console output to Elasticsearch using Fluentd.
Note that it's also possible to configure Serilog to write directly to Elasticsearch using the Elasticsearch sink. If you're not using Fluentd, or aren't containerising your apps, that's a great option.
Writing logs to the console output
When you create a new ASP.NET Core application from a template, your program file will looks something like this (in .NET Core 2.1 at least):
public class Program
{
public static void Main(string[] args)
{
CreateWebHostBuilder(args).Build().Run();
}
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.UseStartup<Startup>();
}
The static helper method WebHost.CreateDefaultBuilder(args) creates a WebHostBuilder and wires up a number of standard configuration options. By default, it configures the Console and Debug logger providers:
.ConfigureLogging((hostingContext, logging) =>
{
logging.AddConfiguration(hostingContext.Configuration.GetSection("Logging"));
logging.AddConsole();
logging.AddDebug();
})
If you run your application from the command line using dotnet run, you'll see logs appear in the console for each request. The following shows the logs generated by two requests from a browser - one for the home page, and one for the favicon.ico.
Unfortunately, the Console logger doesn't provider much flexibility in how the logs are written. You can optionally include scopes, or disable the colours, but that's about it.
An alternative to the default Microsoft.Extensions.Logging infrastructure in ASP.NET Core is to use Serilog for your logging, and connect it as a standard ASP.NET Core logger.
Adding Serilog to an ASP.NET Core app
Serilog is a mature open source project, that predates all the logging infrastructure in ASP.NET Core. In many ways, the ASP.NET Core logging infrastructure seems modelled after Serilog: Serilog has similar configuration options and pluggable "sinks" to control where logs are written.
The easiest way to get started with Serilog is with the Serilog.AspNetCore NuGet package. Add it to your application with:
dotnet add package Serilog.AspNetCore
You'll also need to add one or more "sink" packages, to control where logs are written. In this case, I'm going to install the Console sink, but you could add others too, if you want to write to multiple destinations at once.
dotnet add package Serilog.Sinks.Console
The Serilog.AspNetCore package provides an extension method, UseSerilog() on the WebHostBuilder instance. This replaces the default ILoggerFactory with an implementation for Serilog. You can pass in an existing Serilog.ILogger instance, or you can configure a logger inline. For example, the following code configures the minimum log level that will be written (info) and registers the console sink:
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.UseSerilog((ctx, config) =>
{
config
.MinimumLevel.Information()
.Enrich.FromLogContext()
.WriteTo.Console();
})
.UseStartup<Startup>();
}
Running the app again when you're using Serilog instead of the default loggers gives the following console output:
The output is similar to the default logger, but importantly it's very configurable. You can change the output template however you like. For example, you could show the name of the class that generated the log by including the SourceContext parameter.
For more details and samples for the Serilog.AspNetCore package, see the GitHub repository. For console formatting options, see the Serilog.Sinks.Console repository.
As well as simple changes to the output template, the Console sink allows complete control over how the message is rendered. We'll use that capability to render the logs as JSON for Fluentd, instead of a human-friendly format.
Customising the output format of the Serilog Console Sink to write JSON
To change how the data is rendered, you can add a custom ITextFormatter. Serilog includes a JsonFormatter you can use, but it's suggested that you consider the Serilog.Formatting.Compact package instead:
“
CompactJsonFormattersignificantly reduces the byte count of small log events when compared with Serilog's defaultJsonFormatter, while remaining human-readable. It achieves this through shorter built-in property names, a leaner format, and by excluding redundant information.”
We're not going to use this package for our Fluentd/Elasticsearch use case, but I'll show how to plug it in here in any case. Add the package using dotnet add package Serilog.Formatting.Compact, create a new instance of the formatter, and pass it to the WriteTo.Console() method in your UseSerilog() call:
.UseSerilog((ctx, config) =>
{
config
.MinimumLevel.Information()
.Enrich.FromLogContext()
.WriteTo.Console(new CompactJsonFormatter());
})
Now if you run your application, you'll see the logs written to the console as JSON:
This formatter may be useful to you, but in my case, I wanted the JSON to be written so that Elasticsearch could understand it. You can see that the compact JSON format (pretty-printed below) uses, as promised, compact names for the timestamp (@t), message template (@mt) and the rendered message (@r):
{
"@t": "2018-05-17T10:23:47.0727764Z",
"@mt": "{HostingRequestStartingLog:l}",
"@r": [
"Request starting HTTP\/1.1 GET http:\/\/localhost:5000\/ "
],
"Protocol": "HTTP\/1.1",
"Method": "GET",
"ContentType": null,
"ContentLength": null,
"Scheme": "http",
"Host": "localhost:5000",
"PathBase": "",
"Path": "\/",
"QueryString": "",
"HostingRequestStartingLog": "Request starting HTTP\/1.1 GET http:\/\/localhost:5000\/ ",
"EventId": {
"Id": 1
},
"SourceContext": "Microsoft.AspNetCore.Hosting.Internal.WebHost",
"RequestId": "0HLDRS135F8A6:00000001",
"RequestPath": "\/",
"CorrelationId": null,
"ConnectionId": "0HLDRS135F8A6"
}
For the simplest Fluentd/Elasticsearch integration, I wanted the JSON to be output using standard Elasticsearch names such as @timestamp for the timestamp. Luckily, all that's required is to replace the formatter.
Using an Elasticsearch compatible JSON formatter
The Serilog.Sinks.Elasticsearch package contains exactly the formatter we need, the ElasticsearchJsonFormatter. This renders data using standard Elasticsearch fields like @timestamp and fields.
Unfortunately, currently the only way to add the formatter to your project short of copying and pasting the source code (check the license first!) is to install the whole Serilog.Sinks.Elasticsearch package, which has quite a few dependencies.
Ideally, I'd like to see the formatter as its own independent package, like Serilog.Formatting.Compact is. I've raised an issue and will update this post if there's movement.
If that's not a problem for you (it wasn't for me, as I already had a dependency on Elasticsearch.Net, then adding the Elasticsearch Sink to access the formatter is the easiest solution. Add the sink using dotnet add package Serilog.Sinks.ElasticSearch, and update your Serilog configuration to use the ElasticsearchJsonFormatter:
.UseSerilog((ctx, config) =>
{
config
.MinimumLevel.Information()
.Enrich.FromLogContext()
.WriteTo.Console(new ElasticsearchJsonFormatter();
})
Once you've connected this formatter, the console output will contain the common Elasticsearch fields like @timestamp, as shown in the following (pretty-printed) output:
{
"@timestamp": "2018-05-17T22:31:43.9143984+12:00",
"level": "Information",
"messageTemplate": "{HostingRequestStartingLog:l}",
"message": "Request starting HTTP\/1.1 GET http:\/\/localhost:5000\/ ",
"fields": {
"Protocol": "HTTP\/1.1",
"Method": "GET",
"ContentType": null,
"ContentLength": null,
"Scheme": "http",
"Host": "localhost:5000",
"PathBase": "",
"Path": "\/",
"QueryString": "",
"HostingRequestStartingLog": "Request starting HTTP\/1.1 GET http:\/\/localhost:5000\/ ",
"EventId": {
"Id": 1
},
"SourceContext": "Microsoft.AspNetCore.Hosting.Internal.WebHost",
"RequestId": "0HLDRS5H8TSM4:00000001",
"RequestPath": "\/",
"CorrelationId": null,
"ConnectionId": "0HLDRS5H8TSM4"
},
"renderings": {
"HostingRequestStartingLog": [
{
"Format": "l",
"Rendering": "Request starting HTTP\/1.1 GET http:\/\/localhost:5000\/ "
}
]
}
}
Now logs are being rendered in a format that can be piped straight from Fluentd into Elasticsearch. We can just write to the console.
Switching between output formatters based on hosting environment
A final tip. What if you want to have human readable console output when developing locally, and only use the JSON formatter in Staging or Production?
This is easy to achieve as the UseSerilog extension provides access to the IHostingEnvironment via the WebHostBuilderContext. For example, in the following snippet I configure the app to use the human-readable console in development, and to use the JSON formatter in other environments.
.UseSerilog((ctx, config) =>
{
config
.MinimumLevel.Information()
.Enrich.FromLogContext();
if (ctx.HostingEnvironment.IsDevelopment())
{
config.WriteTo.Console();
}
else
{
config.WriteTo.Console(new ElasticsearchJsonFormatter());
}
})
Instead of environment, you could also switch based on configuration values available via the IConfiguration object at ctx.Configuration.
Summary
Storing logs in a central location is important, especially if you're building containerised apps. One possible solution to this is to output your logs to the console, have Fluentd monitor the console, and pipe the output to an Elasticsearch cluster. In this post I described how to add Serilog logging to your ASP.NET Core application and configure it to write logs to the console in the JSON format that Elasticsearch expects.