This post is part of the third annual C# Advent. Check out the home page for up to 50 C# blog posts in December 2019!
In this post I describe an issue I found when porting an ASP.NET Windows application to ASP.NET Core on Linux. It took me several attempts to get to the bottom of the issue, and rather than jump straight to the answer, I've detailed my failed attempts to fix it on the way!
A little while ago Steve Gordon wrote about solving a similar issue. I've had this post in draft for quite a while so unfortunately that was too late for me! 😆
Background: Porting an ASP.NET Windows app to ASP.NET Core on Linux
Recently I've been working on porting a large, old (~10 years), ASP.NET app to .NET Core. On the face of it, that sounds like a fools errand, but luckily the app has moved with the times. It uses Web API 2 and OWIN/Katana, with no Razor or WebForms dependencies, or anything like that. Generally speaking, the port actually hasn't been too bad!
Disclaimer: porting from ASP.NET to ASP.NET Core may not be worth it for you. In this case, we're pretty confident it's the right choice!
As well as moving to .NET Core, we're also switching OS from Windows to Linux. We've already ported various smaller applications in a similar way, which again, has been surprisingly hassle-free in most cases.
I initially focused on porting the app on Windows, and subsequently configured builds to use multi-stage Dockerfiles (using Cake). After some inevitable trial-and-error, I eventually got a port of the app running for testing. I fixed some obvious bugs/typos I had introduced, and most things appeared to be working well. 🎉
However, on checking the local log files, I found the following error, hundreds of times:
Culture 4096 (0x1000) is an invalid culture identifier - CultureNotFoundException
The rest of this post details the saga of me trying to understand and fix this error.
The culprit: creating RegionInfo from CultureInfo
I traced the source of the error to the following code:
public static IEnumerable<RegionInfo> AllRegionInfo { get; } =
CultureInfo.GetCultures(CultureTypes.SpecificCultures)
.Select(culture => new RegionInfo(culture.LCID))
.ToList();
I don't find myself working with globalization constructs like cultures and regions very often, so it took me a little while to figure out exactly what the code was doing, or why it should be failing. Breaking it down:
var cultures = CultureInfo.GetCultures(CultureTypes.SpecificCultures)
The CultureInfo class contains information about various locales, such as language details, formatting for dates and numbers, currency symbols, and so on. CultureInfo.GetCultures() returns all the cultures known to .NET given the current operating system and version, filtered by the provided CultureTypes enum. (We'll be coming back to that emphasised section shortly.)
The CultureTypes enum is a sign of how much legacy cruft is in .NET - of the 8 values described in the docs, 5 of them are deprecated! The remaining three values are:
- SpecificCultures: Cultures that are specific to a country/region (e.g. en-GB, en-US, es-ES)
- NeutralCultures: Cultures that are associated with a language but are not specific to a country/region. (e.g. es, en). This also includes the Invariant Culture.
- AllCultures: All the cultures.
So the code shown above should fetch all the specific cultures, i.e. cultures associated with a country/region. This brings us to the next line:
.Select(culture => new RegionInfo(culture.LCID))
This LINQ expression uses the CultureInfo.LCID property of the specific cultures returned by GetCultures(). This is the culture identifier, which apparently maps to Windows NLS locale identifier. Essentially an integer ID for the culture.
The RegionInfo constructor takes an LCID, and creates the appropriate RegionInfo associated with that culture. The RegionInfo object contains details about the country/region like the region name, the two and three letter ISO names, and ISO currency symbols, for example.
So this code is creating a list of all the RegionInfo objects known to the .NET app. This code was working fine when running under ASP.NET on Windows so, why was it failing with this error?
Culture 4096 (0x1000) is an invalid culture identifier - CultureNotFoundException
The problem: there's a lot of cultures!
As far as I can see, this is actually not a problem specific to .NET Core, or even Linux. Rather, it's a consequence of the fact there are a lot of possible locales! This Stack Overflow post contains a great description (and solution) for the problem:
Almost all of the new locales in Windows are not assigned explicit LCIDs - because there is not enough "room" for the thousands of languages in hundreds of countries problem. They all get assigned 0x1000.
So the problem is that the new locales haven't all been given a new LCID. RegionInfo doesn't know what to do with the "placeholder" 0x1000, so it throws a CultureNotFoundException.
The solution is to use the name of the culture (e.g. en-GB or es-ES) instead of the dummy LCID:
public static IEnumerable<RegionInfo> AllRegionInfo { get; } =
CultureInfo.GetCultures(CultureTypes.SpecificCultures)
.Select(culture => new RegionInfo(culture.Name)) // using Name instead of LCID
.ToList();
To be honest, I'm not entirely sure why the original code was working previously. I would have expected Windows 10 to have various locales with the placeholder LCID, and to have seen this error before!
With this change the error went away! I focused on deploying the app to the alpine OS Docker images, and carried-on with my testing. Surprise, surprise, another error was waiting for me in the logs.
Problem two: The InvariantCulture appearing in SpecificCultures
In the log files, with the same stack trace as before, I found this error:
ArgumentException: There is no region associated with the Invariant Culture (Culture ID: 0x7F)
The Invariant Culture? But the docs say the Invariant Culture is s a neutral culture, not a specific culture. Why was it in the list? My initial thought was "pfft, who knows, I'll just filter it out":
public static IEnumerable<RegionInfo> AllRegionInfo { get; } =
CultureInfo.GetCultures(CultureTypes.SpecificCultures)
.Where(culture => culture.LCID != 0x7F) // filer invariant culture
.Select(culture => new RegionInfo(culture.Name)) // using Name instead of LCID
.ToList();
At this point, I should point out that I had added unit tests around this method, confirming that AllRegionInfo contained regions I expected (e.g. "en-US") and didn't contain nonsense regions ("xx-XX" or the invariant culture). These tests all passed both when running on Windows, and in the build phase of my Dockerfiles.
However, when testing the app, I again found a problem, this time getting an exception almost immediately:
FailFast: Couldn't find a valid ICU package installed on the system. Set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support.
The heart of the issue: Alpine is small
This finally pointed me in the direction of the real problem. There was no issue running on the build-time Docker images that contain the .NET Core SDK. But on the small, runtime Docker images an error was thrown, seemingly because there were no cultures installed. This was exactly the case, because I was using the Alpine Docker images.
Alpine has a much smaller footprint than a Linux distribution like Debian, so it's generally ideal for using in Docker containers, especially when coupled with multi-stage builds. I've been using it for all my .NET Core apps without any problems for some time. However, a quick bit of googling quickly revealed this issue: alpine images have no cultures installed.
.NET Core takes its list of cultures from the OS. On *nix systems, these come from the ICU library which is typically installed by default. However, in the interest of making the distro as small as possible, Alpine doesn't include the ICU libraries.
To account for this, .NET Core 2.0 introduced a Globalization Invariant Mode. You can read about everything it does in the linked document, but the important point for this discussion is:
When enabling the invariant mode, all cultures behave like the invariant culture.
also
All cultures LCID will have value 0x1000 (which means Custom Locale ID). The exception is the invariant cultures which will still have 0x7F.
Which explains the behaviour I was seeing! You can see this mode being enabled with the DOTNET_SYSTEM_GLOBALIZATION_INVARIANT environment variable in the Alpine runtime-deps Dockerfile that serves as the base image for all the other .NET Core Alpine Docker images.
The fix: install the ICU cultures and disable Globalization Invariant Mode
All of which brings us to a solution: install the ICU libraries in the Alpine runtime images, and disable the Globalization Invariant Mode. You can actually see how to do this in the Alpine SDK Docker images.
Yes, you read that correctly. The Alpine SDK image does have cultures installed. The runtime images don't have cultures installed. That meant my culture unit tests were completely failing to spot the issue, as they were essentially running with .NET Core in a different mode!
You can install the ICU libraries in Alpine using apk add icu-libs. Your runtime Dockerfile will need to start with something like this:
FROM mcr.microsoft.com/dotnet/core/aspnet:2.1.11-alpine3.9
# Install cultures (same approach as Alpine SDK image)
RUN apk add --no-cache icu-libs
# Disable the invariant mode (set in base image)
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false
# ... other setup
With this change, everything was back working correctly, but I retained creating the RegionInfo using Name instead of LCID as the better approach:
public static IEnumerable<RegionInfo> AllRegionInfo { get; } =
CultureInfo.GetCultures(CultureTypes.SpecificCultures)
.Select(culture => new RegionInfo(culture.Name))
.ToList();
As a side note, another way to "spot" you're running in Globalization Invariant Mode is that all your currency symbols have turned into ¤ symbols, as in this issue.
Summary
In this post I described an issue with cultures I found when porting an application from Windows to Linux. The problem was that the Alpine Docker images I was using didn't have any cultures installed, so was running in the Globalization Invariant Mode. To fix the issue, I installed the ICU libraries, and disabled invariant mode.
.NET Core works great cross platform, but when you're moving between Windows and Linux, it's not just the Windows-specific features you have to keep an eye-out for. There's the common issues like file-path-separators and line endings, but also more subtle differences like the one in this post. If you've been bitten by anything else moving to Linux I'd be interested to hear about it in the comments!