In this post I describe how and why I created a .NET Core global tool to easily convert configuration stored in web.config files to the JSON format more commonly used for configuration in ASP.NET Core.
tl;dr; You can install the tool by running
dotnet tool install --global dotnet-config2json. Note that you need to have the .NET Core 2.1 SDK installed.
Background - Converting ASP.NET apps to ASP.NET Core
I've been going through the process of converting a number of ASP.NET projects to ASP.NET Core recently. As these projects are entirely Web APIs and OWIN pipelines, there's a reasonable upgrade path there (I'm not trying to port WebForms to ASP.NET Core)! Some parts are definitely easier to port than others: the MVC controllers work almost out of the box, and where third-party libraries have been upgraded to supported to .NET Standard, everything moves across pretty easily.
One area that's seen a significant change moving from ASP.NET to ASP.NET Core is the configuration system. Whereas ASP.NET largely relied on the static ConfigurationManager reading key-value-pairs from web.config, ASP.NET Core adopts a layered approach that lets you read configuration values from a wide range of sources.
As part of the migrations, I wanted to convert our old web.config-based configuration files to use the more idiomatic appsettings.json and appsettings.Development.json files commonly found in ASP.NET Core projects. I find the JSON files easier to understand, and given that JSON is the defacto standard for this stuff now it made sense to me.
Note: If you really want to, you can continue to store configuration in your .config files, and load the XML directly. There's a sample in the ASP.NET repositories of how to do this.
Before I get into the conversion tool I wrote itself, I'll give an overview of the config files I was working with.
The old config file formats
One of the bonuses with how we were using the .config in our ASP.NET projects, was that it pretty closely matched the concepts in ASP.NET Core. We were using both configuration layers and strongly typed settings.
Layered configuration with .config files
Each configuration file, e.g. global.config, had a sibling file for staging and testing environments, e.g. global.staging.config and global.prod.config. Those files used XML Document transforms which would be applied during deployment to overwrite the earlier values.
For example, the global.config file might look something like this:
<Platform>
<add key="SlackApi_WebhookUrl" value="https://hooks.slack.com/services/Some/Url" />
<add key="SlackApi_DisplayName" value="Slack bot" />
</Platform>
That would set the SlackApi_WebhookUrl and SlackApi_DisplayName values when running locally. The global.prod.config file might look something like this:
<Platform xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<add key="SlackApi_WebhookUrl" value="https://example.com/something"
xdt:Transform="Replace" xdt:Locator="Match(key)" />
</Platform>
On deployment, the global.prod.config file would be used to transform the global.config file. Where a key matched (as specified by the xdt:Locator attribute), the configuration value would be replaced by the prod equivalent.
While this isn't quite the same as for ASP.NET Core, where the ASP.NET Core app itself overwrites settings with the environment-specific values, it achieves a similar end result.
Note: There should only be non-sensitive settings in these config files. Sensitive values or secrets should be stored externally to the app.
Strongly typed configuration
In ASP.NET Core, the recommended approach to consuming configuration values in code is through strongly typed configuration and the Options pattern. This saves you from using magic strings throughout your code, favouring the ability to inject POCO objects into your services.
We were using a similar pattern in our ASP.NET apps, by using the DictionaryAdapter from Castle. This technique is very similar to the binding operations used to create strongly typed settings in ASP.NET Core. There's a nice write-up of the approach here.
We were also conveniently using a naming convention to map settings from our config files, to their strongly typed equivalent, by using the _ separator in our configuration keys.
For example, the keys SlackApi_WebhookUrl and SlackApi_DisplayName would be mapped to an interface:
public interface ISlackApi
{
string WebhookUrl { get; set; }
string DisplayName { get; set; }
}
This is very close to the way ASP.NET Core works. The main difference is that ASP.NET Core requires concrete types (as it instantiates the actual type), rather than the interface required by Castle for generating proxies.
Now you've seen the source material, I'll dive into the requirements of why I wrote a global tool, and what I was trying to achieve.
Requirements for the conversion tool
As you've seen, the config files I've been working with translate well to the new appsettings.json paradigm in ASP.NET Core. But some of our apps have a lot of configuration. I didn't want to manually be copying and pasting, and while I probably could have eventually scripted a conversion with bash or PowerShell, I'm a .NET developer, so I thought I'd write a .NET tool 🙂. Even better, with .NET Core 2.1 I could make a global tool, and use it from any folder.
The tool I was making had just a few requirements:
- Read the standard
<appSettings>and<connectionString>sections of web.config files, as well as the generic<add>style environment-specific .config files shown earlier. - Generate nested JSON objects for "configuration sections" demarcated by
_, such as theSlackApishow earlier. - Be quick to develop - this was just a tool to get other stuff done!
Building the tool
If you're new to global tools, I suggest reading my previous post on building a .NET Core global tool, as well as Nate McMasters blog for a variety of getting started guides and tips. In this post I'm just going to describe the approach I took for solving the problem, rather than focusing on the global tool itself.
.NET Core global tools are just Console apps with
<PackAsTool>true</PackAsTool>set in the .csproj. It really is as simple as that!
Parsing the config files
My first task was to read the config files into memory. I didn't want to have to faff with XML parsing myself, so I cribbed judiciously from the sample project in the aspnet/entropy repo. This sample shows how to create a custom ASP.NET Core configuration provider to read from web.config files. Perfect!
I pulled in 4 files from this project (you can view them in the repo):
ConfigFileConfigurationProviderConfigurationActionIConfigurationParserKeyValueParser
If you were going to be using the configuration provider in your app, you'd also need to create an IConfigurationSource, as well as adding some convenience extension methods. For my tool, I manually created a ConfigFileConfigurationProvider instance and passed in the path to the file and the required KeyValueParsers. These two parsers would handle all my use cases, by looking for <add> and <remove> elements with key-value or name-connectionString attributes.
var file = "path/to/config/file/web.config";
var parsersToUse = new List<IConfigurationParser> {
new KeyValueParser(),
new KeyValueParser("name", "connectionString")
};
// create the provider
var provider = new ConfigFileConfigurationProvider(
file, loadFromFile: true, optional: false, parsersToUse);
// Read and parse the file
provider.Load();
After calling Load(), the provider contains all the key-value pairs in an internal dictionary, so we need to get to them. Unfortunately, that's not as easy as we might like: we can only enumerate all the keys in the Dictionary. To get the KeyValuePairs we need to enumerate the keys and then fetch them from the dictionary one at a time. Obviously this is rubbish performance wise, but it really doesn't matter for this tool! 🙂
const string SectionDelimiter = "_";
// Get all the keys
var keyValues = provider.GetChildKeys(Enumerable.Empty<string>(), null)
.Select(key =>
{
// Get the value for the current key
provider.TryGet(key, out var value);
// Replace the section delimiter in the key value
var newKey = string.IsNullOrEmpty(SectionDelimiter)
? key
: key.Replace(SectionDelimiter, ":", StringComparison.OrdinalIgnoreCase);
// Return the key-value pair
return new KeyValuePair<string, string>(newKey, value);
});
We use GetChildKeys to get all the keys from the provider, and then fetch the corresponding values. I'm also transforming the keys if we have a SectionDelimiter string. This will replace all the _ previously used to denote sections, with the ASP.NET Core approach of using :. Why we do this will become clear very shortly!
After this code has run, we'll have a dictionary of values looking something like this:
{
{ "IdentityServer:Host", "local.example.com" },
{ "IdentityServer:ClientId", "MyClientId" },
{ "SlackApi:WebhookUrl", "https://hooks.slack.com/services/Some/Url" },
{ "SlackApi:DisplayName", "Slack bot" }
}
Converting the flat list into a nested object
At this point we've successfully extracted the files from the the config files into a dictionary. But at the moment it's still a flat dictionary. We want to create a nested JSON structure, something like
{
"IdentityServer": {
"Host": "local.example.com",
"ClientId": "MyClientId
},
"SlackApi": {
"WebhookUrl": "https://hooks.slack.com/services/Some/Url",
"DisplayName": "Slack bot"
}
}
I thought about reconstructing this structure myself, but why bother when somebody has already done the work for you? The IConfigurationRoot in ASP.NET Core uses Sections that encapsulate this concept, and allows you to enumerate a section's child keys. By generating an IConfigurationRoot using the keys parsed from the .config files, I could let it generate the internal structure for me, which I could subsequently convert to JSON.
I used the InMemoryConfigurationProvider to pass in my keys to an instance of ConfigurationBuilder, and called Build to get the IConfigurationRoot.
var config = new ConfigurationBuilder()
.AddInMemoryCollection(keyValues)
.Build();
The config object contains all the information about the JSON structure we need, but getting it out in a useful format is not as simple. You can get all the child keys of the configuration, or of a specific section, using GetChildren(), but that includes both the top level sections names (which don't have an associated value), and key-value pairs. Effectively, you have the following key pairs (note the null values)
{ "IdentityServer", null },
{ "IdentityServer:Host", "local.example.com" },
{ "IdentityServer:ClientId", "MyClientId" },
{ "SlackApi", null },
{ "SlackApi:WebhookUrl", "https://hooks.slack.com/services/Some/Url" },
{ "SlackApi:DisplayName", "Slack bot" }
The solution I came up with is this little recursive function to convert the configuration into a JObject
static JObject GetConfigAsJObject(IConfiguration config)
{
var root = new JObject();
foreach (var child in config.GetChildren())
{
//not strictly correct, but we'll go with it.
var isSection = (child.Value == null);
if (isSection)
{
// call the function again, passing in the section-children only
root.Add(child.Key, GetConfigAsJObject(child));
}
else
{
root.Add(child.Key, child.Value);
}
}
return root;
}
Calling this function with the config parameter produces the JSON structure we're after. All that remains is to serialize the contents to a file, and the conversion is complete!
var newPath = Path.ChangeExtension(file, "json");
var contents = JsonConvert.SerializeObject(jsonObject, Formatting.Indented);
await File.WriteAllTextAsync(newPath, contents);
I'm sure there must be a function to serialize the
JObjectdirectly to the file, rather than in memory first, but I couldn't find it. As I said before, performance isn't something I'm worried about but it's bugging me nevertheless. If you know what I'm after, please let me know in the comments, or send a PR!
Using the global tool
With the console project working as expected, I converted the project to a global tool as described in my previous post and in Nate's posts. If you want to use the tool yourself, first install the .NET Core 2.1 SDK, and then install the tool using
> dotnet tool install --global dotnet-config2json
You can then run the tool and see all the available options using
> dotnet config2json --help
dotnet-config2json
Converts a web.config file to an appsettings.json file
Usage: dotnet config2json [arguments] [options]
Arguments:
path Path to the file or directory to migrate
delimiter The character in keys to replace with the section delimiter (:)
prefix If provided, an additional namespace to prefix on generated keys
Options:
-?|-h|--help Show help information
Performs basic migration of an xml .config file to
a JSON file. Uses the 'key' value as the key, and the
'value' as the value. Can optionally replace a given
character with the section marker (':').
I hope you find it useful!
Summary
In this post I described how I went about creating a tool to convert web.config files to .json files when converting ASP.NET apps to ASP.NET Core. I used an existing configuration file parser from the aspnet/entropy repo to load the web.config files into an IConfiguration object, and then used a small recursive function to turn the keys into a JObject. Finally, I turned the tool into a .NET Core 2.1 global tool.