In my previous post, I described a common problem in which primitive arguments (e.g. System.Guid or string) are passed in the wrong order to a method, resulting in bugs. This problem is a symptom of primitive obsession; using primitive types to represent higher-level concepts.

This post directly follow on from my previous post, so I strongly recommend reading that one first.

In my previous post I described a strongly-typed ID that could be used to represent the ID of an object, for example an OrderId or a UserId. As a reminder, the implementation looked something like this:

public readonly struct OrderId : IComparable<OrderId>, IEquatable<OrderId>
{
    public Guid Value { get; }

    public OrderId(Guid value)
    {
        Value = value;
    }

    public static OrderId New() => new OrderId(Guid.NewGuid());

    // various overloads, overrides, and implementations
}

One of the common complaints when fighting primitive obsession like this, is that it makes things more complex at the "edges" of the system, when converting between Guid and OrderId for example. The best answer to this is to try to use the strongly-typed IDs everywhere.

With the implementation described so far, this is easier said than done, so in this post I'll describe some helper classes you can use with strongly-typed IDs to make working with ASP.NET Core APIs simpler.

tl;dr; You can skip to a complete example implementation including all the converters or a Visual Studio snippet if you wish.

Strongly-typed IDs make for ugly JSON APIs

Lets imagine that you have a standard eCommerce app, as in the previous post. You have an MVC API controller for your Orders, containing a single Post action for creating Orders.

[ApiController]
public class OrderController : ControllerBase
{
    [HttpPost]
    public IActionResult Post(Order order);
}

As we have a strongly-typed IDs Orders and Users, the Order object now looks something like the following (instead of using Guids for IDs):

public class Order
{
    public OrderId Id { get; set; }
    public UserId UserId { get; set; }
    public decimal Total { get; set; }
}

The problem is that our strongly-typed IDs mean that for MVC Model Binding to work as we expect, the posted JSON body would have to look something like this:

{
    "Id": {
        "Value": "da63f7a0-a4a6-4dbe-a9a4-4bb72dde30dd"
    },
    "UserId": {
        "Value": "4bb20f98-f6d4-43bc-9fdf-5b74ce4ef751"
    },
    "Total": 123.45
}

Reading over this again, I actually don't think model binding would work at all in this case, though I haven't tested it since.

Urgh, that's a bit of a mess. Luckily, we can simplify this using a custom JsonConverter.

Creating a custom JsonConverter

JsonConverter in Newtonsoft.Json can be used to customise how types are converted to and from JSON. in ASP.NET Core 2.x that also allows you to customize how types are serialised and deserialised during model binding.

Note that in ASP.NET Core 3.0 JSON serialization will be changing. See this GitHub issue for details.

The following example shows how to create a JsonConverter as a nested class of the strongly-typed ID. I've hidden the bulk of the OrderId class for brevity, but make sure to decorate the main strongly-typed ID class with the [JsonConverter] attribute:

[JsonConverter(typeof(OrderIdJsonConverter))]
public readonly struct OrderId : IComparable<OrderId>, IEquatable<OrderId>
{
    // Strongly-typed ID implementation elided 

    class OrderIdJsonConverter : JsonConverter
    {
        public override bool CanConvert(Type objectType)
        {
            return objectType == typeof(OrderId);
        }

        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            var id = (OrderId)value;
            serializer.Serialize(writer, id.Value);
        }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            var guid = serializer.Deserialize<Guid>(reader);
            return new OrderId(guid);
        }
    }
}

Implementing the custom JsonConverter is relatively simple, and relies on the fact Newtonsoft.Json already knows how to serialize and deserialize Guids:

• Override CanConvert: This converter can only convert OrderId types.
• Override WriteJson: To serialize an OrderId, extract the Guid Value, and serialize that.
• Override ReadJson: Start by deserializing a Guid, and create an OrderId from that.

By using a custom JsonConverter, the serialized Order looks much cleaner and easier to work with:

{
    "Id": "da63f7a0-a4a6-4dbe-a9a4-4bb72dde30dd",
    "UserId": "4bb20f98-f6d4-43bc-9fdf-5b74ce4ef751",
    "Total": 123.45
}

In fact, it's exactly the same as the original Order object was before we converted to strongly-typed IDs.

So that's the JSON support working, lets move on to looking at another API method, a GET method.

Using strongly-typed IDs in route constraints

A common pattern with REST APIs is to include the ID of a resource in the URL. For example:

[ApiController]
public class OrderController : ControllerBase
{
    [HttpGet("{id}")]
    public ActionResult<Order> Get(OrderId id);
}

In this example, you'd expect to be able to retrieve an Order object from the API by sending a GET request to /Order/7b-46-0c4, where 7b-46-0c4 is the ID of the order (shortened for brevity). Unfortunately, if you try this, you'll get a slightly confusing 415 Unsupported Media Type response:

{"type":"https://tools.ietf.org/html/rfc7231#section-6.5.13","title":"Unsupported Media Type","status":415,"traceId":"0HLLI5VFOFT3C:00000003"}

The problem is that the MVC framework doesn't know how to convert the string route segment "7b-46-0c4" into your OrderId type. We have a JSON converter that can convert strings to the OrderId type, but we're not converting from a JSON body in this case.

Creating a custom type converter

There's a couple of different ways you could solve this problem:

• Create a custom model binder
• Create a custom type converter

Creating a custom model binder is a relatively involved affair, but it gives you complete control over the binding process. In our case, we just need a simple string to OrderId conversion, and the documentation suggests you should use a type converter in this case.

Type converters provide "a unified way of converting types of values to other types". In our case, all we need to support is converting from a string to OrderId.

[JsonConverter(typeof(OrderIdJsonConverter))]
[TypeConverter(typeof(OrderIdTypeConverter))]
public readonly struct OrderId : IComparable<OrderId>, IEquatable<OrderId>
{
    // Strongly-typed ID implementation elided 

    class OrderIdTypeConverter : TypeConverter
    {
        public override bool CanConvertFrom(ITypeDescriptorContext context, Type sourceType)
        {
            return sourceType == typeof(string) || base.CanConvertFrom(context, sourceType);
        }

        public override object ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, object value)
        {
            var stringValue = value as string;
            if (!string.IsNullOrEmpty(stringValue)
                && Guid.TryParse(stringValue, out var guid))
            {
                return new OrderId(guid);
            }

            return base.ConvertFrom(context, culture, value);

        }
    }
}

Derive from the base TypeConverter class, and override the CanConvertFrom method to indicate that you can handle strings. I've created the implementation as a nested class of OrderId for tidiness.

In the ConvertFrom method override, cast the provided value to a string, and try to parse it into a Guid. If all goes well, you can return a new OrderId, otherwise, just delegate to the base implementation.

Finally, decorate your strongly-typed ID with the [TypeConverter] attribute, and reference your implementation.

That's all you need to fix your issues - no extra types to register with the MVC framework and no messing with custom model binding or providers. I was actually surprised how simple this approach was, having never used TypeConverters before!

Other type converters for interfacing with the world.

With the two converters described above, you should be able to work seamlessly with your ASP.NET Core APIs, so there's no excuse for passing on using strongly-typed IDs there!

At the other end of the application, at the database, you may want to create similar converters. Given the number of possible ORMs and micro-ORMs, I won't go into the details here, but most will provide this functionality. For example, you can create a custom TypeHandler<T> for Dapper which would look something like the following:

class OrderIdTypeHandler : SqlMapper.TypeHandler<OrderId>
{
    public override void SetValue(IDbDataParameter parameter, OrderId value)
    {
        parameter.Value = value.Value;
    }

    public override OrderId Parse(object value)
    {
        return new OrderId((Guid)value);
    }
}

You would just need to register the custom handler with Dapper using SqlMapper.AddTypeHandler(new OrderIdTypeHandler());

A full example implementation

I've been dribbling bits of implementation out in this post, so below is a full example implementation for an imaginary FooId type, including custom JsonConverter and a custom TypeConverter:

[JsonConverter(typeof(FooIdJsonConverter))]
[TypeConverter(typeof(FooIdTypeConverter))]
public readonly struct FooId : IComparable<FooId>, IEquatable<FooId>
{
    public Guid Value { get; }

    public FooId(Guid value)
    {
        Value = value;
    }

    public static FooId New() => new FooId(Guid.NewGuid());
    public static FooId Empty { get; } = new FooId(Guid.Empty);

    public bool Equals(FooId other) => this.Value.Equals(other.Value);
    public int CompareTo(FooId other) => Value.CompareTo(other.Value);

    public override bool Equals(object obj)
    {
        if (ReferenceEquals(null, obj)) return false;
        return obj is FooId other && Equals(other);
    }

    public override int GetHashCode() => Value.GetHashCode();

    public override string ToString() => Value.ToString();
    public static bool operator ==(FooId a, FooId b) => a.CompareTo(b) == 0;
    public static bool operator !=(FooId a, FooId b) => !(a == b);

    class FooIdJsonConverter : JsonConverter
    {
        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            var id = (FooId)value;
            serializer.Serialize(writer, id.Value);
        }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            var guid = serializer.Deserialize<Guid>(reader);
            return new FooId(guid);
        }

        public override bool CanConvert(Type objectType)
        {
            return objectType == typeof(FooId);
        }
    }

    class FooIdTypeConverter : TypeConverter
    {
        public override bool CanConvertFrom(ITypeDescriptorContext context, Type sourceType)
        {
            return sourceType == typeof(string) || base.CanConvertFrom(context, sourceType);
        }

        public override object ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, object value)
        {
            var stringValue = value as string;
            if (!string.IsNullOrEmpty(stringValue)
                && Guid.TryParse(stringValue, out var guid))
            {
                return new FooId(guid);
            }

            return base.ConvertFrom(context, culture, value);

        }
    }
}

That's a lot of boilerplate code!

Yes, I know. That's a lot of code for a simple ID type. I still think it's worth it, but there's no denying it's verbose…

All the F# devs shouting at the screen right now…

To try and counteract that somewhat, I've created a Visual Studio Snippet as described in the docs. Copy the XML below into a file (or download the snippet from here) and import it into your IDE.

Once it's imported, you can type typedid, hit Tab twice, type a new name for the ID and auto-generate all of that code! Note that you may need to add Newtonsoft.Json as a NuGet reference to your project if it's not already referenced.

The snippet - feel free to customize as you see fit!

<?xml version="1.0" encoding="utf-8"?>
<CodeSnippets xmlns="http://schemas.microsoft.com/VisualStudio/2005/CodeSnippet">
  <CodeSnippet Format="1.0.0">
    <Header>
      <Title>Strongly Typed ID</Title>
      <Description>Create a strongly typed ID struct</Description>
      <Shortcut>typedid</Shortcut>
      <HelpUrl>https://andrewlock.net/using-strongly-typed-entity-ids-to-avoid-primitive-obsession-part-2/</HelpUrl>
    </Header>
    <Snippet>
      <Declarations>
        <Literal>
          <ID>TypedId</ID>
          <ToolTip>Replace with the name of your type</ToolTip>
          <Default>TypedId</Default>
        </Literal>
      </Declarations>
      <Code Language="csharp"><![CDATA[[JsonConverter(typeof($TypedId$JsonConverter))]
    [TypeConverter(typeof($TypedId$TypeConverter))]
    public readonly struct $TypedId$ : IComparable<$TypedId$>, IEquatable<$TypedId$>
    {
        public Guid Value { get; }

        public $TypedId$(Guid value)
        {
            Value = value;
        }

        public static $TypedId$ New() => new $TypedId$(Guid.NewGuid());
        public static $TypedId$ Empty { get; } = new $TypedId$(Guid.Empty);

        public bool Equals($TypedId$ other) => this.Value.Equals(other.Value);
        public int CompareTo($TypedId$ other) => Value.CompareTo(other.Value);

        public override bool Equals(object obj)
        {
            if (ReferenceEquals(null, obj)) return false;
            return obj is $TypedId$ other && Equals(other);
        }

        public override int GetHashCode() => Value.GetHashCode();

        public override string ToString() => Value.ToString();
        public static bool operator ==($TypedId$ a, $TypedId$ b) => a.CompareTo(b) == 0;
        public static bool operator !=($TypedId$ a, $TypedId$ b) => !(a == b);

        class $TypedId$JsonConverter : JsonConverter
        {
            public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
            {
                var id = ($TypedId$)value;
                serializer.Serialize(writer, id.Value);
            }

            public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
            {
                var guid = serializer.Deserialize<Guid>(reader);
                return new $TypedId$(guid);
            }

            public override bool CanConvert(Type objectType)
            {
                return objectType == typeof($TypedId$);
            }
        }

        class $TypedId$TypeConverter : TypeConverter
        {
            public override bool CanConvertFrom(ITypeDescriptorContext context, Type sourceType)
            {
                return sourceType == typeof(string) || base.CanConvertFrom(context, sourceType);
            }

            public override object ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, object value)
            {
                var stringValue = value as string;
                if (!string.IsNullOrEmpty(stringValue)
                    && Guid.TryParse(stringValue, out var guid))
                {
                    return new $TypedId$(guid);
                }

                return base.ConvertFrom(context, culture, value);

            }
        }
    }]]>
      </Code>
      <Imports>
        <Import>
          <Namespace>System</Namespace>
        </Import>
        <Import>
          <Namespace>System.ComponentModel</Namespace>
        </Import>
        <Import>
          <Namespace>System.Globalization</Namespace>
        </Import>
        <Import>
          <Namespace>Newtonsoft.Json</Namespace>
        </Import>
      </Imports>
    </Snippet>
  </CodeSnippet>
</CodeSnippets>

Summary

Strongly-typed IDs can help avoid simple, but hard-to-spot, argumentation transposition errors. By using the types defined in this post, you can get all the benefits of the C# type system, without making your APIs clumsy to use, or adding translation code back-and-forth between Guids and your strongly-typed IDs. In this post I showed how to create a custom TypeConverter and a custom JsonConverter for your types. Finally, I provided a complete example implementation, and a Visual Studio snippet for generating strongly-typed IDs in your own project.

In a previous post, I described a common problem in which primitive arguments (e.g. System.Guid or string) are passed in the wrong order to a method, resulting in bugs. This problem is a symptom of primitive obsession: using primitive types to represent higher-level concepts. In my second post, I showed how to create a JsonConverter and TypeConverter to make using the strongly-typed IDs easier with ASP.NET Core.

Martin Liversage noted that JSON.NET will use a TypeConverter where one exists, so you generally don't need the custom JsonConverter I provided in the previous post!

In this post, I discuss using strongly-typed IDs with EF Core. I personally don't use EF Core a huge amount, but after a little playing I came up with something that I thought worked pretty well. Unfortunately, there's one huge flaw which puts a cloud over the whole approach, as I'll describe later 🙁.

Interfacing with external system using strongly typed IDs

As a very quick reminder, strongly-typed IDs are types that can be used to represent the ID of an object, for example an OrderId or a UserId. A basic implementation (ignoring the overloads and converters etc.) looks something like this:

public readonly struct OrderId : IComparable<OrderId>, IEquatable<OrderId>
{
    public Guid Value { get; }
    public OrderId(Guid value)
    {
        Value = value;
    }

    // various helpers, overloads, overrides, implementations, and converters
}

You only get the full benefit of strongly-typed IDs if you can use them throughout your application. That includes at the "edges" of the app, where you interact with external systems. In the previous post I described the interaction at the public-facing end of your app, in ASP.NET Core MVC controllers.

The other main external system you will likely need to interface with is the database. At the end of the last post, I very briefly described a converter for using strongly-typed IDs with Dapper by creating a custom TypeHandler:

class OrderIdTypeHandler : SqlMapper.TypeHandler<OrderId>
{
    public override void SetValue(IDbDataParameter parameter, OrderId value)
    {
        parameter.Value = value.Value;
    }

    public override OrderId Parse(object value)
    {
        return new OrderId((Guid)value);
    }
}

This needs to be registered globally using SqlMapper.AddTypeHandler(new OrderIdTypeHandler()); to be used directly in your Dapper database queries.

Dapper is the ORM that I use the most in my day job, but EF Core is possibly going to be the most common ORM in ASP.NET Core apps. Making EF Core play nicely with the strongly-typed IDs is possible, but requires a bit more work.

Building an EF Core data model using strongly typed IDs

We'll start by creating a very simple data model that uses strongly-typed IDs. The classic ecommerce Order/OrderLine example contains everything we need:

public class Order
{
    public OrderId OrderId { get; set; }
    public string Name { get; set; }

    public ICollection<OrderLine> OrderLines { get; set; }
}

public class OrderLine
{
    public OrderId OrderId { get; set; }
    public OrderLineId OrderLineId { get; set; }
    public string ProductName { get; set; }
}

We have two entities:

• Order which has an OrderId, and has a collection of OrderLines.
• OrderLine which has as OrderLineId and an OrderId. All of the IDs are strongly-typed.

After creating these entities, we need to add them to the EF Core DbContext. We create a DbSet<Order> for the collection of Orders, and let EF Core discover the OrderLine entity itself:

 public class ApplicationDbContext : DbContext
{
    public ApplicationDbContext(DbContextOptions<ApplicationDbContext> options)
        : base(options)
    {
    }

    public DbSet<Order> Orders { get; set; }
}

Unfortunately, if we try and generate a new migration for updated model using the dotnet ef tool, we'll get an error:

> dotnet ef migrations add OrderSchema

System.InvalidOperationException: The property 'Order.OrderId' could not be mapped, 
because it is of type 'OrderId' which is not a supported primitive type or a valid 
entity type. Either explicitly map this property, or ignore it using the 
'[NotMapped]' attribute or by using 'EntityTypeBuilder.Ignore' in 'OnModelCreating'.

EF Core complains that it doesn't know how to map our strongly-typed IDs (OrderId) to a database type. Luckily, there's a mechanism we can use to control this as of EF Core 2.1: value converters.

Creating a custom ValueConverter for EF Core

As described in the EF Core documentation:

Value converters allow property values to be converted when reading from or writing to the database. This conversion can be from one value to another of the same type (for example, encrypting strings) or from a value of one type to a value of another type (for example, converting enum values to and from strings in the database.)

The latter conversion, converting from one type to another, is what we need for the strongly-typed IDs. By using a value converter, we can convert our IDs into a Guid, just before they're written to the database. When reading a value, we convert the Guid value from the database into a strongly typed ID.

EF Core allows you to configure value converters manually for each property in your modelling code using lambdas. Alternatively, you can create reusable, standalone, custom value converters for each type. That's the approach I show here.

To implement a custom value converter you create an instance of a ValueConverter<TModel, TProvider>. TModel is the type being converted (the strongly-typed ID in our case), while TProvider is the database type. To create the converter you provide two lambda functions in the constructor arguments:

• convertToProviderExpression: an expression that is used to convert the strongly-typed ID to the database value (a Guid)
• convertFromProviderExpression: an expression that is used to convert the database value (a Guid) into an instance of the strongly-typed ID.

You can create an instance of the generic ValueConverter<> directly, but I chose to create a derived converter to simplify instantiating a new converter. Taking the OrderId example, we can create a custom ValueConverter<> using the following:

public class OrderIdValueConverter : ValueConverter<OrderId, Guid>
{
    public OrderIdValueConverter(ConverterMappingHints mappingHints = null)
        : base(
            id => id.Value,
            value => new OrderId(value),
            mappingHints
        ) { }
}

The lambda functions are simple - to obtain a Guid we use the Value property of the ID, and to create a new instance of the ID we pass the Guid to the constructor. The ConverterMappingHints parameter can allow setting things such as Scale and Precision for some database types. We don't need it here but I've included it for completeness in this example.

Registering the custom ValueConverter with EF Core's DB Context

The value converters describe how to store our strongly-typed IDs in the database, but EF Core need's to know when to use each converter. There's no way to do this using attributes, so you need to customise the model in DbContext.OnModelCreating. That makes for some pretty verbose code:

public class ApplicationDbContext : IdentityDbContext
{
    public ApplicationDbContext(DbContextOptions<ApplicationDbContext> options)
        : base(options)
    { }

    public DbSet<Order> Orders { get; set; }

    protected override void OnModelCreating(ModelBuilder builder)
    {
        base.OnModelCreating(builder);

        builder
            .Entity<Order>()
            .Property(o => o.OrderId)
            .HasConversion(new OrderIdValueConverter());

        builder
            .Entity<OrderLine>()
            .Property(o => o.OrderLineId)
            .HasConversion(new OrderLineIdValueConverter());

        builder
            .Entity<Order>()
            .Property(o => o.OrderId)
            .HasConversion(new OrderIdValueConverter());
    }
}

It's clearly not optimal having to add a manual mapping for each usage of a strongly-typed ID in your entities. Luckily we can simplify this code somewhat.

Automatically using a value converter for all properties of a given type.

Ideally our custom value converters would be used automatically by EF Core every time a given strongly-typed ID is used. There's an issue on GitHub for exactly this functionality, but in the meantime, we can emulate the behaviour by looping over all the model entities, as described in a comment on that issue.

In the code below, we loop over every entity in the model, and for each entity, find all those properties that are of the required type (OrderId for the OrderIdValueConverter). For each property we register the ValueConverter, in a process similar to the manual registrations above:

public static class ModelBuilderExtensions
{
    public static ModelBuilder UseValueConverter(
        this ModelBuilder modelBuilder, ValueConverter converter)
    {
        // The-strongly typed ID type
        var type = converter.ModelClrType;

        // For all entities in the data model
        foreach (var entityType in modelBuilder.Model.GetEntityTypes())
        {
            // Find the properties that are our strongly-typed ID
            var properties = entityType
                .ClrType
                .GetProperties()
                .Where(p => p.PropertyType == type);

            foreach (var property in properties)
            {
                // Use the value converter for the property
                modelBuilder
                    .Entity(entityType.Name)
                    .Property(property.Name)
                    .HasConversion(converter);
            }
        }

        return modelBuilder;
    }
}

All that remains is to register the value converter for each strongly-typed ID type in the DbContext:

public class ApplicationDbContext : IdentityDbContext
{
    public ApplicationDbContext(DbContextOptions<ApplicationDbContext> options)
        : base(options)
    { }

    public DbSet<Order> Orders { get; set; }

    protected override void OnModelCreating(ModelBuilder builder)
    {
        base.OnModelCreating(builder);

        builder.UseValueConverter(new OrderIdValueConverter())
        builder.UseValueConverter(new OrderLineIdValueConverter())
    }
}

It's a bit frustrating having to manually register each of these value converters - every time you create a new strongly typed ID you have to remember to register it in the DbContext.

Creating the ValueConverter implementation itself for every strongly-typed ID is not a big deal if you're using snippets to generate your IDs, like I described in the last post.

It would be nice if we were able to generate a new ID, use it in an entity, and not have to remember to update the OnModelCreating method.

Automatically registering value converters for strongly typed IDs

We can achieve this functionality with a little bit of reflection and some attributes. We'll start by creating an attribute that we can use to link each strongly-typed ID to a specific value converter, called EfCoreValueConverterAttribute:

public class EfCoreValueConverterAttribute : Attribute
{
    public EfCoreValueConverterAttribute(Type valueConverter)
    {
        ValueConverter = valueConverter;
    }

    public Type ValueConverter { get; }
}

We'll decorate each strongly typed ID with the attribute as part of the snippet generation, which will give something like the following:

// The attribute links the OrderId to OrderIdValueConverter
[EfCoreValueConverter(typeod(OrderIdValueConverter))]
public readonly struct OrderId : IComparable<OrderId>, IEquatable<OrderId>
{
    public Guid Value { get; }
    public OrderId(Guid value)
    {
        Value = value;
    }

    // The ValueConverter implementation
    public class OrderIdValueConverter : ValueConverter<OrderId, Guid>
    {
        public OrderIdValueConverter()
            : base(
                id => id.Value,
                value => new OrderId(value)
            ) { }
    }
}

Next, we'll add another method to the ModelBuilderExtensions this loops through all the types in an Assembly and finds all those that are decorated with the EfCoreValueConverterAttribute (i.e. the strongly typed IDs). The Type of the value converter is extracted from the custom attribute, and an instance of the value converter is created using the ValueConverter. We can then pass that to the UseValueConverter method we created previously.

public static class ModelBuilderExtensions
    {
        public static ModelBuilder AddStronglyTypedIdValueConverters<T>(
            this ModelBuilder modelBuilder)
        {
            var assembly = typeof(T).Assembly;
            foreach (var type in assembly.GetTypes())
            {
                // Try and get the attribute
                var attribute = type
                    .GetCustomAttributes<EfCoreValueConverterAttribute>()
                    .FirstOrDefault();

                if (attribute is null)
                {
                    continue;
                }

                // The ValueConverter must have a parameterless constructor
                var converter = (ValueConverter) Activator.CreateInstance(attribute.ValueConverter);

                // Register the value converter for all EF Core properties that use the ID
                modelBuilder.UseValueConverter(converter);
            }

            return modelBuilder;
        }

        // This method is the same as shown previously
        public static ModelBuilder UseValueConverter(
            this ModelBuilder modelBuilder, ValueConverter converter)
        {
            var type = converter.ModelClrType;

            foreach (var entityType in modelBuilder.Model.GetEntityTypes())
            {
                var properties = entityType
                    .ClrType
                    .GetProperties()
                    .Where(p => p.PropertyType == type);

                foreach (var property in properties)
                {
                    modelBuilder
                        .Entity(entityType.Name)
                        .Property(property.Name)
                        .HasConversion(converter);
                }
            }

            return modelBuilder;
        }
    }

With this code in place, we can register all our value converters in one fell swoop in the DbContext.OnModelCreating method:

public class ApplicationDbContext : IdentityDbContext
{
    public ApplicationDbContext(DbContextOptions<ApplicationDbContext> options)
        : base(options)
    {
    }

    public DbSet<Order> Orders { get; set; }

    protected override void OnModelCreating(ModelBuilder builder)
    {
        base.OnModelCreating(builder);

        // add all value converters
        builder.AddStronglyTypedIdValueConverters<OrderId>();
    }
}

The type parameter OrderId in the above example is used to identify the Assembly to scan to find the strongly-typed IDs. If required, it would be simple to add another overload to allowing scanning multiple assemblies.

With the code above, we don't have to touch the DbContext when we add a new strongly-typed ID, which is a much better experience for developers. If we run the migrations now, all is well:

> dotnet ef migrations add OrderSchema

Done. To undo this action, use 'ef migrations remove'

If you check the generated migration, you'll see that the OrderId column is created as a non-nullable Guid, and is the primary key, as you'd expect.

public partial class OrderSchema : Migration
{
    protected override void Up(MigrationBuilder migrationBuilder)
    {
        migrationBuilder.CreateTable(
            name: "Orders",
            columns: table => new
            {
                OrderId = table.Column<Guid>(nullable: false),
                Name = table.Column<string>(nullable: true)
            },
            constraints: table =>
            {
                table.PrimaryKey("PK_Orders", x => x.OrderId);
            });
    }
}

This solves most of the problems you'll encounter using strongly typed IDs with EF Core, but there's one place where this doesn't quite work, and unfortunately, it might be a deal breaker.

Custom value converters result in client-side evaluation

Saving entities that use your strongly-typed IDs to the database is no problem for EF Core. However, if you try and load an entity from the database, and filter based on the strongly-typed ID:

var order = _dbContext.Orders
    .Where(order => order.OrderId == orderId)
    .FirstOrDefault();

then you'll see a warning in the logs that the where clause must be evaluated client-side:

warn: Microsoft.EntityFrameworkCore.Query[20500]
      The LINQ expression 'where ([x].OrderId == __orderId_0)' 
      could not be translated and will be evaluated locally.

info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (12ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      SELECT [x].[OrderId], [x].[Name]
      FROM [Orders] AS [x]

That's terrible. This query has got to be a contender for the most common thing you'll ever do, and the above solution will not be good enough. Fetching an Order by ID with client-side execution involves loading all Orders into memory and filtering in memory!

In fairness the documentation does mention this limitation right at the bottom of the page (emphasis mine):

Use of value conversions may impact the ability of EF Core to translate expressions to SQL. A warning will be logged for such cases. Removal of these limitations is being considered for a future release.

But this value converter is pretty much the most basic you could imagine - if this converter results in client-side evaluation, they all will!

There is an issue to track this problem, but unfortunately there's no easy work around to this one. 🙁

All is not entirely lost. It's not pretty, but after some playing I eventually found something that will let you use strongly-typed IDs in your EF Core models that doesn't force client-side evaluation.

Avoiding client-side evaluation in EF Core with conversion operators

The key is adding implicit or explicit conversion operators to the strongly-typed IDs, such that EF Core doesn't bork on seeing the strongly-typed ID in a query. There's two possible options, an explicit conversion operator, or an implicit conversion operator.

Using an explicit conversion operator with strongly typed IDs

The first approach is to add an explicit conversion operator to your strongly-typed ID to go from the ID type to a Guid:

public readonly struct OrderId
{
    public static explicit operator Guid(OrderId orderId) => orderId.Value;

    // Remainder of OrderId implementation ...
}

Adding this sort of operator means you can cast an OrderId to a Guid, for example:

var orderId = new OrderId(Guid.NewGuid());
var result = (Guid) orderId; // Only compiles with explicit operator

So how does that help? Essentially we can trick EF Core into running the query server-side, by using a construction similar to the following:

Guid orderIdValue = orderId.Value; // extracted for clarity, can be inlined
var order = _dbContext.Orders
    .Where(order => (Guid) order.OrderId == orderIdValue) // explicit conversion to Guid
    .FirstOrDefault();

The key point is the explicit conversion of order.OrderId to a Guid. When EF Core evaluates the query, it no longer sees an OrderId type that it doesn't know what to do with, and instead generates the SQL we wanted in the first place:

info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (7ms) [Parameters=[@__orderId_Value_0='?' (DbType = Guid)], CommandType='Text', CommandTimeout='30']
      SELECT TOP(1) [x].[OrderId], [x].[Name]
      FROM [Orders] AS [x]
      WHERE [x].[OrderId] = @__orderId_Value_0

This shows the where clause being sent to the database, so all is well again. Well, apart from the fact it's an ugly hack. 😕 Implicit operators make the process very slightly less ugly.

Using an implicit conversion operator with strongly typed IDs

The implicit conversion operator implementation is almost identical to the explicit implementation, just with a different keyword:

public readonly struct OrderId
{
    public static implicit operator Guid(OrderId orderId) => orderId.Value;

    // Remainder of OrderId implementation ...
}

With this code, you no longer need an explicit (Guid) cast to convert an OrderId to a Guid, so we can write the query as:

Guid orderIdValue = orderId.Value; // extracted for clarity, can be inlined
var order = _dbContext.Orders
    .Where(order => order.OrderId == orderIdValue) // Implicit conversion to Guid
    .FirstOrDefault();

This query generates identical SQL, so technically you could use either approach. But which should you choose?

Implicit vs Explicit operators

For simple ugliness, the implicit operator seems slightly preferable, as you don't have to add the extra cast, but I'm not sure if that's a bad thing. The trouble is that the implicit conversions apply throughout your code base, so suddenly code like this will compile:

public Order GetOrderForUser(Guid orderId, Guid userId)
{
    // Get the User
}

OrderId orderId = OrderId.New();
UserId userId = UserId.New();

var order = GetOrderForUser(userId, orderId); // arguments reversed, the bug is back!

The GetOrderForUser() method should obviously be using the strongly-typed IDs, but the fact that this is possible without any indication of errors just makes me a little uneasy. For that reason, I think I prefer the explicit operators.

Either way, you should definitely hide away the cast from callers wherever possible:

// with explicit operator
public class OrderService
{
    // public API uses strongly-typed ID
    public Order GetOrder(OrderId orderId) => GetOrder(orderId.Value);

    // private implementation to handle casting
    private Order GetOrder(Guid orderId)
    {
        return _dbContext.Orders
            .Where(x => (Guid) x.OrderId == orderId)
            .FirstOrDefault();
    }
}

// with implicit operator 
public class OrderService
{
    // public API uses strongly-typed ID
    public Order GetOrder(OrderId orderId) => GetOrder(orderId.Value);

    // private implementation to handle casting
    private Order GetOrder(Guid orderId)
    {
        return _dbContext.Orders
            .Where(x => x.OrderId == orderId) // Only change is no cast required here
            .FirstOrDefault();
    }
}

It's probably also worth configuring your DbContext to throw an error when client-side evaluation occurs so you don't get client-side errors creeping in without you noticing. Override the DbContext.OnConfiguring method, and configure the options:

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    optionsBuilder.ConfigureWarnings(warning => 
            warning.Throw(RelationalEventId.QueryClientEvaluationWarning));
}

Even with all this effort, there's still gotchas. As well as standard IQueryable<T> LINQ syntax, the DbSet<> exposes a Find method which is effectively a shorthand for SingleOrDefault() for querying by an entities primary key. Unfortunately, nothing we do here will work:

var orderId = new OrderId(Guid.NewGuid());
_dbContext.Find(orderId); // using the strongly typed ID directly causes client-side evaluations
_dbContext.Find(order.Value); // passing in a Guid causes an exception : The key value at position 0 of the call to 'DbSet<Order>.Find' was of type 'Guid', which does not match the property type of 'OrderId'.

So close…

This post is plenty long enough, and I haven't quite worked out a final solution but I have a couple of ideas. Check back in a couple of days, and hopefully I'll have it figured out 🙂

Summary

In this post I explored possible solutions that would allow you to use strongly-typed IDs directly in your EF Core entities. The ValueConverter approach described in this post gets you 90% of the way there, but unfortunately the fact that queries will be executed client-side really makes the whole approach more difficult until this issue is resolved. You can get some success by using explicit or implicit conversions, but there are still edge cases. I'm playing with a different approach as we speak, and hope to have something working in a couple of days, so check back soon!

This is another post in my series on strongly-typed IDs. In the first and second posts, I looked at the reasons for using strongly-typed IDs, and how to add converters to interface nicely with ASP.NET Core. In the previous post I looked at ways of using strongly-typed IDs with EF Core. Unfortunately, there was a significant issue with the approach I outlined: querying by a strongly-typed ID could result in client-side evaluation. The workarounds I proposed only partially fixed the problem.

In this post, I show a workaround that seems to solve the issue. EF Core is not my speciality, so it's possible there's some hidden issues, but from my testing so far it works perfectly! 🤞 The secret-sauce is ValueConverterSelector.

Strongly-typed IDs in EF Core

As a quick recap, the solution I proposed in the previous post centred around value converters. As their name suggests, these can be used to convert instances of one type (for example a strongly-typed ID like OrderId) into a type that is supported by a database provider (for example a Guid or an int).

In the last post I showed an example implementation of a custom ValueConverter for an OrderId that is stored in the database as a Guid. The version below is slightly modified to be a nested class of the strongly-typed ID OrderId, which is how we would generate it if using the "snippet" approach from my second post. For this post, we don't need the [StronglyTypedIdEfValueConverter] attribute I previously described.

public readonly struct OrderId 
{
    // Not shown: the OrderId implementation and other converters

    public class StronglyTypedIdEfValueConverter : ValueConverter<OrderId, Guid>
    {
        public StronglyTypedIdEfValueConverter(ConverterMappingHints mappingHints = null)
            : base(id => id.Value, value => new OrderId(value), mappingHints) 
        {
        }
    }
}

You could manually map every use of OrderId in your EF Core model properties to use this converter as before. But as well as being verbose, this would leave you with the client-side evaluation problem from the last post.

Instead, we're going to look at one of the "internal" services of EF Core - the ValueConverterSelector. If you're not interested in why the solution works and just want to see the final code, skip ahead.

A semi-deep dive into ValueConverterSelector - handling built-in conversions

After reaching the conclusion of my last post, I felt like I had hit a brick wall trying to get strongly-typed IDs to work smoothly. There were all sorts of work arounds you could use, but ultimately you were going to get a sub-par experience no matter what.

This got me thinking: EF Core has all sorts of "built-in" value converters that convert between primitive types. These do conversions like Char to string, number to byte[], or string to Guid. Using these value converters doesn't trigger the client-side evaluation problem, and they doesn't require you to register them against each property - they're used automatically.

These converters aren't built into the BCL or anything, so they must be registered somewhere in EF Core. After a bit of searching, I tracked the answer down to the ValueConverterSelector class and the IValueConverterSelector interface.

I've reproduced the interface (from version 2.2.4) below, as the xmldocs describe exactly what this type does:

/// <summary>
/// A registry of ValueConverterInfo that can be used to find
/// the preferred converter to use to convert to and from a given model type
/// to a type that the database provider supports.
/// </summary>
public interface IValueConverterSelector
{
    /// <summary>
    /// Returns the list of ValueConverterInfo instances that can be
    /// used to convert the given model type. Converters nearer the front of
    /// the list should be used in preference to converters nearer the end.
    /// </summary>
    IEnumerable<ValueConverterInfo> Select(Type modelClrType, Type providerClrType = null);
}

EF Core uses an implementation of this interface to find the value converters for built-in types. It appears to use these types early in the query generation pipeline, so they don't cause the client-side evaluation issues you see with custom value converters.

The below code is a snippet taken from the Select() method of the default implementation, ValueConverterSelector. This method is essentially a giant if/else statement that finds all the applicable converters for a given modelClrType (the type used in your EF Core entities) and providerClrType (the type stored in the database).

Given the number of built-in converters, this method is big, so I've only shown a snippet of it below:

private readonly ConcurrentDictionary<(Type ModelClrType, Type ProviderClrType), ValueConverterInfo> _converters
    = new ConcurrentDictionary<(Type ModelClrType, Type ProviderClrType), ValueConverterInfo>();

public virtual IEnumerable<ValueConverterInfo> Select(Type modelClrType, Type providerClrType = null)
{
    // Extract the "real" type T from Nullable<T> if required
    var underlyingModelType = modelClrType.UnwrapNullableType();
    var underlyingProviderType = providerClrType?.UnwrapNullableType();

    // lots of code...

    if (underlyingModelType == typeof(Guid))
    {
        if (underlyingProviderType == null
            || underlyingProviderType == typeof(byte[]))
        {
            yield return _converters.GetOrAdd(
                (underlyingModelType, typeof(byte[])),
                k => GuidToBytesConverter.DefaultInfo);
        }

        if (underlyingProviderType == null
            || underlyingProviderType == typeof(string))
        {
            yield return _converters.GetOrAdd(
                (underlyingModelType, typeof(string)),
                k => GuidToStringConverter.DefaultInfo);
        }
    }

    // lots more code...
}

So what is this code doing? First, the method "unwraps" any nullable types - so if the type is a Guid?, it returns a Guid and so on. If the type is not nullable, this is a no-op. It's worth noting that the providerClrType can be null: null here means "give me all the value converters for the modelClrType".

After unwrapping the types, we enter the nested if/else statements - I've shown the if statement for Guid above. There are two built-in converters for Guid: the GuidToBytesConverter, and the GuidToStringConverter. If the underlyingProviderType is null or the correct type, the method uses yield return to return a default instance of ValueConverterInfo.

The implementation uses a ConcurrentDictionary to avoid creating multiple ValueConverterInfo objects, keyed on the underlyingModelType and underlyingProviderType.

The ValueConverterInfo object is a simple DTO that contains a factory method for creating a ValueConverter instance:

public readonly struct ValueConverterInfo
{
    private readonly Func<ValueConverterInfo, ValueConverter> _factory;

    public Type ModelClrType { get; }
    public Type ProviderClrType { get; }
    public ConverterMappingHints MappingHints { get; }
    public ValueConverter Create() => _factory(this);
}

If we look at one of the built-in value converters GuidToStringConverter for example, we see the DefaultInfo property that returns a ValueConverterInfo object:

public class GuidToStringConverter : StringGuidConverter<Guid, string>
{
    public GuidToStringConverter(ConverterMappingHints mappingHints = null)
        : base(ToString(), ToGuid(), _defaultHints.With(mappingHints))
    { }

    public static ValueConverterInfo DefaultInfo { get; }
        = new ValueConverterInfo(
            typeof(Guid), 
            typeof(string), 
            i => new GuidToStringConverter(i.MappingHints), 
            _defaultHints);
}

I haven't shown the base classes involved, so the code above isn't entirely complete, but the DefaultInfo property implementation is pretty simple. It creates a new ValueConverterInfo object, providing the ModelClrType (Guid) the ProviderClrType (string), a function for creating a new GuidToStringConverter given the current ValueConverterInfo instance (i.e. call the constructor), and the default mapping hints to use (for controlling the size of the string column in the database etc).

That's as far as I went digging into the ValueConverterSelector. I haven't worked out quite how it fits in to the overall EF Core query translation system (other than it's used in the ITypeMappingSource implementations), but I know enough now to be dangerous - lets get back to fixing the original problem, strongly-typed IDs.

Creating a custom ValueConverterSelector for strongly-typed IDs

To recap, we have a number of strongly-typed IDs that are used in our EF Core entities. For each strongly-typed ID we have a nested ValueConverter implementation. In this section, we're going to create a custom ValueConverterSelector to automatically register our value converters so they're used in the same way as the built-in value converters.

Luckily, the ValueConverterSelector implementation isn't sealed, and the Select() method is even virtual, so we can easily create our own implementation, while preserving the existing behaviour for built-in converters. The following code is the entire StronglyTypedIdValueConverterSelector - I'll walk through and explain it afterwards.

public class StronglyTypedIdValueConverterSelector : ValueConverterSelector
{
    // The dictionary in the base type is private, so we need our own one here.
    private readonly ConcurrentDictionary<(Type ModelClrType, Type ProviderClrType), ValueConverterInfo> _converters
        = new ConcurrentDictionary<(Type ModelClrType, Type ProviderClrType), ValueConverterInfo>();

    public StronglyTypedIdValueConverterSelector(ValueConverterSelectorDependencies dependencies) : base(dependencies)
    { }

    public override IEnumerable<ValueConverterInfo> Select(Type modelClrType, Type providerClrType = null)
    {
        var baseConverters = base.Select(modelClrType, providerClrType);
        foreach (var converter in baseConverters)
        {
            yield return converter;
        }

        // Extract the "real" type T from Nullable<T> if required
        var underlyingModelType = UnwrapNullableType(modelClrType);
        var underlyingProviderType = UnwrapNullableType(providerClrType);

        // 'null' means 'get any value converters for the modelClrType'
        if (underlyingProviderType is null || underlyingProviderType == typeof(Guid))
        {
            // Try and get a nested class with the expected name. 
            var converterType = underlyingModelType.GetNestedType("StronglyTypedIdEfValueConverter");

            if (converterType != null)
            {
                yield return _converters.GetOrAdd(
                    (underlyingModelType, typeof(Guid)),
                    k =>
                    {
                        // Create an instance of the converter whenever it's requested.
                        Func<ValueConverterInfo, ValueConverter> factory =
                            info => (ValueConverter) Activator.CreateInstance(converterType, info.MappingHints);

                        // Build the info for our strongly-typed ID => Guid converter
                        return new ValueConverterInfo(modelClrType, typeof(Guid), factory);
                    }
                );
            }
        }
    }

    private static Type UnwrapNullableType(Type type)
    {
        if (type is null) { return null; }

        return Nullable.GetUnderlyingType(type) ?? type;
    }
}

The StronglyTypedIdValueConverterSelector is written to follow the same patterns as the ValueConverterSelector it overrides, so I've created a ConcurrentDictionary<> for tracking the value converters in the same way the base class does. The dictionary in the base class is private so we have to create a new instance of it here, but that's not a big deal. The constructor passes through the required ValueConverterSelectorDependencies object to the base class.

The meat of the implementation is in the Select method. We start by fetching all of the applicable built-in value converters by calling base.Select(), and yield return on all of the returned implementations. That preserves existing behaviour.

Next, we have to "unwrap" nullable types, just as the base class did. We call the simple static UnwrapNullableType() method defined at the end of the class. If the provider type is either null or Guid, then we try and create a converter, otherwise we're done.

When testing the converter, I found that the method was only ever called with providerClrType=null. That's likely due to something specific about my models, I just thought I'd point it out.

Assuming the if() branch returns true, we now need to see if the modelClrType is a strongly-typed ID type with a value converter implementation. This is where the change to the value converter implementation at the start of this post makes sense:

public readonly struct OrderId 
{
    public class StronglyTypedIdEfValueConverter : ValueConverter<OrderId, Guid> { }
}

By creating the value converter as a nested class, and using the same name across all strongly-typed ID types (StronglyTypedIdEfValueConverter), we can both fetch the converter type and test for a strongly-typed ID at the same time with a small bit of reflection:

var converterType = underlyingModelType.GetNestedType("StronglyTypedIdEfValueConverter");
if (converterType != null)
{
    // we have a Type for the converter
}

At this point we know we have a value converter for the modelClrType, so we need to create the correct ValueConverterInfo and yield return it. The base class simplifies this code by using a static DefaultInfo property, but we'd have to invoke a similar method using reflection, and it all gets a bit more hassle than it's worth. Instead, I opted for creating a factory function that creates an instance of the converter by calling Activator.CreateInstance(), passing in the required ConverterMappingHints argument:

Func<ValueConverterInfo, ValueConverter> factory =
    info => (ValueConverter) Activator.CreateInstance(converterType, info.MappingHints);

Don't be fooled by the fact that the StronglyTypedIdEfValueConverter mappingHints parameter has a default value of null. Even though you don't need to provide this value when invoking the constructor normally, you must provide it when invoking the method using reflection (and Activator.CreateInstance())

Finally, we can create an instance of ValueConverterInfo, add it to the dictionary, and yield return it.

This implementation looks a bit complicated because of the reflection required, but I'm pretty confident there's nothing untoward going on there. All that remains is for us to replace the default instance of IValueConverterSelector with our custom class.

Replacing the default IValueConverterSelector with a custom implementation

Replacing "framework" EF Core services is relatively painless thanks to the ReplaceService method exposed by DbContextOptionsBuilder. You can call this method as part of your EF Core configuration in Startup.ConfigureServices, in the AddDbContext<> configuration method:

public void ConfigureServices(IServiceCollection services)
{
    services.AddDbContext<ApplicationDbContext>(options =>
        options
            .ReplaceService<IValueConverterSelector, StronglyTypedIdEfValueConverter>() // add this line
            .UseSqlServer(
                Configuration.GetConnectionString("DefaultConnection")));
}

That's it.

No more custom DbContext.OnModelCreating code.

No marker attributes.

No more implicit/explicit conversions to force use of the value converter.

And most importantly, no more client-side evaluation.

I'm actually kind of surprised by how well it works, but it all does seem to work. Even the following code (which was broken in the implementation from my last post), works:

public Order GetOrder(OrderId orderId)
{
    return _dbContext.Orders
                .Where(order => order.Id == orderId)
                .FirstOrDefault();
}

public Order GetOrderUsingFind(OrderId orderId)
{
    return _dbContext.Orders
                .Find(orderId);
}

Both of these usages generate the same SQL, which has a server-side where clause:

info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (6ms) [Parameters=[@__get_Item_0='?' (DbType = Guid)], CommandType='Text', CommandTimeout='30']

      SELECT TOP(1) [e].[OrderId], [e].[Name]
      FROM [Orders] AS [e]
      WHERE [e].[OrderId] = @__get_Item_0

Success! One more reason to use strongly-typed IDs in your next ASP.NET Core app 😃.

Summary

In this post I describe a solution to using strongly-typed IDs in your EF Core entities by using value converters and a custom IValueConverterSelector. The base ValueConverterSelector in the EF Core framework is used to register all built-in value conversions between primitive types. By deriving from this class, we can add our strongly-typed ID converters to this list, and get seamless conversion throughout our EF Core queries. As well as reducing the configuration required, this solves the client-side evaluation problem that plagued the previous implementation.

ASP.NET Core Identity is a membership system that adds login and user functionality to ASP.NET Core apps. It includes many features out of the box and has basic support for storing a phone number for a user. You can improve the robustness of ASP.NET Core’s phone number validation and provide a better user experience by integrating Twilio’s telephony features in your application.

By default the phone number in ASP.NET Core Identity is validated with a regular expression, but that's too basic to confirm whether the number is really valid, whether it includes the country dialling code, or whether it can receive SMS messages. You could implement improved validation using the library libphonenumber-csharp, as described in a previous Twilio blog post. Alternatively you could use various Twilio APIs to thoroughly validate the phone number, lookup details about the number (such as carrier or type), and prove ownership of the phone with a text or voice verification message.

Multiple stages of validation are required to determine if a phone number can be associated with an application user for a specific purpose, but I'll only be looking at the first step in this post: ensuring a number is valid for a specific country and determining if it is likely to be able to receive text messages. Twilio's Lookup API can provide this functionality in a Razor Pages app that uses ASP.NET Core Identity for user management.

Prerequisites

To follow along with this post you'll need

• A Twilio account (sign up for a free Twilio account here)
• .NET Core 2.2 SDK (version 2.2.102 or greater)
• VS Code, Visual Studio 2017, or other editor
• A familiarity with Razor Pages (see my previous post for an introduction to Razor Pages)

You can find the complete code for this post on GitHub.

Data validation in ASP.NET Core Razor Pages

Razor Pages is a new aspect of ASP.NET Core MVC that was introduced in ASP.NET Core 2.0. It's very similar to the traditional Model-View-controller (MVC) pattern commonly used in ASP.NET Core, but uses a "page-based" approach. See my previous post for an introduction on Razor Pages, and how they differ from ASP.NET Core MVC.

Razor Pages uses the same model binding and model validation processes as MVC, but applies them to a "Page Model" instead of action method parameters. For example, [Required], and [EmailAddress] are validation attributes that check the value of the InputModel.Email property is correct.

public partial class IndexModel : PageModel
{
    [BindProperty]
    public InputModel Input { get; set; }

    public class InputModel
    {
        [Required]
        [EmailAddress]
        public string Email { get; set; }
    }
}

These attributes are found in the System.ComponentModel.DataAnnotations namespace, which includes a [Phone] attribute for validating phone numbers. Unfortunately, this attribute is simplistic and doesn't take into account the highly complex set of rules and exceptions that apply to phone numbers.

For better validation you could use the open source library that Google created and uses for validating phone numbers: libphonenumber. There's a C# port of the library, libphonenumber-csharp, that you can install in your application using the NuGet package. This post on the Twilio blog shows how to use it in your applications.

Alternatively, you could use the Twilio Lookup API to validate phone numbers. This uses the same libphonenumber validation library behind the scenes, but it can also provide extra details such as carrier information (in the US), the type of the phone number (landline/mobile/VOIP), or details of suspected fraud associated with the number.

In this post I'll show how you can use the Twilio Lookup API in a new Razor Pages app to validate the optional phone number provided by users. We'll validate that the number the user entered is valid for their selected country, that it can likely receive SMS messages, and, if so, we'll save the number in the standard E.164 format.

Customizing ASP.NET Core Identity Razor Pages

In older versions of ASP.NET Core, creating a new MVC or Razor Pages app using the ASP.NET Core Identity templates would dump thousands of lines of code into your app. That changed in ASP.NET Core 2.1 with the introduction of Razor Class Libraries.

Razor Class Libraries allow you to bundle views, Razor Pages, and View Components into a NuGet package. To include the UI elements in an app, you simply reference the NuGet package. All of the Razor Pages from the library will then be available in your app with no additional work required on your side.

This is how ASP.NET Core Identity ships now. When you create a new ASP.NET Core app with Identity, your project directory will be virtually empty instead of having thousands of files. That's generally a good thing—you don't have to manage the infrastructure code or keep it up to date.

The obvious downside is that if you want to customize the default experience, for example to improve phone validation like we are, you can't just start updating the code. Luckily, Razor Class Libraries let you override any of the included views or Razor Pages by placing a Page at the same path in your application.

ASP.NET Core provides tools for scaffolding these "override" files for ASP.NET Core Identity. The Identity scaffolder is built into Visual Studio 2017, or there is a cross-platform global tool dotnet-aspnet-codegenerator you can run from the command line. I'll show how to use the global tool to scaffold the /Identity/Account/Manage page that we need to customize.

Creating the case study project

Using Visual Studio 2017+ or the .NET CLI, create a new solution and project with the following characteristics:

• Type: ASP.NET Core 2.2 Web Application (not MVC) with Visual C#
• Name: ValidatePhoneNumberDemo
• Solution directory
• Git repository
• https
• Authentication: Individual user accounts, Store user accounts in-app

ASP.NET Core Identity uses Entity Framework Core to store the users in the database, so be sure to run the database migrations after building your app:

dotnet ef database update

Using the .NET CLI to scaffold Identity pages

Install the aspnet-codegenerator global tool. Note that there is a bug in the latest version of the tool at the time of writing (2.2.0), so the following installs the last known good version (2.1.6).

dotnet tool install -g dotnet-aspnet-codegenerator --version 2.1.6

The Microsoft.VisualStudio.Web.CodeGeneration.Design NuGet is required by the aspnet-codegenerator global tool so, install it in to into your project using the NuGet Package Manager, Package Manager Console CLI, or by editing the the ValidatePhoneNumberDemo.csproj file. Be sure to set the PrivateAssets attribute to All, so that the package is used only during development.

<PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="2.2.1" PrivateAssets="All" />

Open a console window inside your app's project folder (not the solution folder). Run the scaffolding tool to generate the Account.Manage.Index Razor Page. As well as listing the files to generate, you need to specify the full name of the EF Core DB Context for your project.

dotnet aspnet-codegenerator identity --files Account.Manage.Index -dc ValidatePhoneNumberDemo.Data.ApplicationDbContext 

The scaffolder should complete in a few seconds, and afterwards you'll see new files in your project. The scaffolder generates the requested file, plus layout view files and partials.

You can view a list of all possible pages to generate using dotnet aspnet-codegenerator identity --listFiles.

Once your app is running, explore the existing phone number validation for users in ASP.NET Core Identity

• Register a new user by clicking Register from the menu bar
• Enter an email and password
• Click Hello <email>! from the menu bar to see the profile/manage account page, or navigate to /Account/Manage
• Try saving various phone numbers to see what's valid. Hint: 0 is apparently a valid phone number!

Initializing the Twilio API

We'll be using the Twilio helper library to simplify calling the Twilio HTTP API. Install the Twilio NuGet package (version 5.22.0 or later) using the NuGet Package Manager, Package Manager Console CLI, or by editing the ValidatePhoneNumberDemo.csproj file. After using any of these methods the <ItemGroup> section of the project file should look like this (version numbers may be higher):

<ItemGroup>
  <PackageReference Include="Microsoft.AspNetCore.App"/>
  <PackageReference Include="Microsoft.AspNetCore.Razor.Design" Version="2.2.0" PrivateAssets="All" />
  <PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="2.2.1" PrivateAssets="All" />
  <PackageReference Include="Twilio" Version="5.22.0" />
</ItemGroup>

To call the Twilio API you'll need your Twilio Account Sid and Auth Token (found in the Twilio Dashboard). When developing locally these should be stored using the Secrets Manager so they don't get accidentally committed to your source code repository. You can read about how and why to do that in this post on the Twilio Blog. Your resulting Secrets.json should look something like this:

"TwilioAccountDetails": {
  "AccountSID": "ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "AuthToken": "your_auth_token"
}

The Twilio helper libraries use a singleton instance of the Twilio client, which means you only need to set it up once in your app. The best place to configure things like this are in the Startup.cs file. Add using Twilio; at the top of Startup.cs, and add the following at the end of ConfigureServices:

public void ConfigureServices(IServiceCollection services)
{
    // existing configuration

    var accountSid = Configuration["TwilioAccountDetails:AccountSID"];
    var authToken = Configuration["TwilioAccountDetails:AuthToken"];
    TwilioClient.Init(accountSid, authToken);
}

This sets up the static Twilio client using your credentials using the ASP.NET Core configuration system. If you need to customize the requests made to Twilio (e.g. using a proxy server), or want to make use of HttpClientFactory features introduced in ASP.NET Core 2.1, see my previous post on the Twilio blog for an alternative approach.

An inconvenient truth: phone numbers are hard

As any developer who has had the (dis-)pleasure of working with time zones will know; time zones are hard. Unfortunately, phone numbers are hard too! There's a great number of misconceptions that developers have to tackle when they really start working with phone numbers. This document from the libphonenumber library on Falsehoods Programmers Believe About Phone Numbers is a great introduction.

One such problem is that different countries have different rules about what constitutes a valid phone number. So to unambiguously validate a phone number you need to know the country in which it was issued. See the falsehoods document linked above for some examples of why that's necessary.

In order to correctly validate the provided phone number, we need users to choose the issuing country at the same time. That means when we validate the phone number with Twilio we can include the 2-digit ISO 3166 country code, and confirm whether the phone number is valid specifically for that country.

Adding a country code dropdown

We'll add a dropdown of countries to the ASP.NET Core Identity management page to make it easy for users to choose the appropriate country code.

To build the dropdown, we'll load this list of countries from a JSON file countries.json stored in the project folder. A snippet of the file is shown below; you can find a complete file in the sample here.

Ellipsis (“...”) in code blocks represents a section redacted for brevity.

[
    {
        "Text": "United States of America",
        "Value": "US"
    },
    {
        "Text": "United Kingdom of Great Britain and Northern Ireland",
        "Value": "GB"
    },
    {
        "Text": "Canada",
        "Value": "CA"
    },
...
]

Next, we'll create a service to load the JSON file, and convert it into a List<SelectListItem>. Create the CountryService.cs file in the root of your project and add the following code:

using System;
using System.Collections.Generic;
using System.IO;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc.Rendering;
using Newtonsoft.Json;

namespace ValidatePhoneNumberDemo
{
    public class CountryService
    {
        private readonly IHostingEnvironment _environment;
        private readonly Lazy<List<SelectListItem>> _countries;

        public CountryService(IHostingEnvironment environment)
        {
            _environment = environment;
            _countries = new Lazy<List<SelectListItem>>(LoadCountries);
        }

        public List<SelectListItem> GetCountries()
        {
            return _countries.Value;
        }

        private List<SelectListItem> LoadCountries()
        {
            var fileInfo = _environment.ContentRootFileProvider.GetFileInfo("countries.json");

            using (var stream = fileInfo.CreateReadStream())
            using (var streamReader = new StreamReader(stream))
            using (var jsonTextReader = new JsonTextReader(streamReader))
            {
                var serializer = new JsonSerializer();
                return serializer.Deserialize<List<SelectListItem>>(jsonTextReader);
            }
        }
    }
}

This service uses Lazy<> to provide lazy initialization, an optimization that only loads the countries from the JSON file once, when they’re needed, in a thread-safe manner. It also uses the streaming support of Newtonsoft.Json to deserialize the contents directly to a List<SelectListItem>, instead of loading it as a string first. The deserialized list of countries is exposed via the GetCountries() method.

In order to use the CountryService you must register it with the app's dependency injection (DI) container. Open up Startup.cs and add the following line at the end of the ConfigureServices method:

services.AddSingleton<CountryService>();

This registers the service with the DI container as a singleton so we can inject it into our Razor Page. Open the code behind for the account management Razor Page at Areas\Identity\Pages\Account\Manage\Index.cshtml.cs, add using Microsoft.AspNetCore.Mvc.Rendering; at the top of the file, and inject an instance of CountryService into the constructor. Create a new property AvailableCountries on your page model and use the injected CountryService to assign the list of countries. Note that if you're using Visual Studio, you need to expand the Index.cshtml file node in Solution Explorer to see the code-behind file, Index.cshtml.cs.

namespace ValidatePhoneNumberDemo.Areas.Identity.Pages.Account.Manage
{
    public partial class IndexModel : PageModel
    {
        public IndexModel(
            UserManager<IdentityUser> userManager,
            SignInManager<IdentityUser> signInManager,
            IEmailSender emailSender,
            CountryService countryService)
        {
            _userManager = userManager;
            _signInManager = signInManager;
            _emailSender = emailSender;
            // Load the countries from the service
            AvailableCountries = countryService.GetCountries();
        }

        public List<SelectListItem> AvailableCountries { get; }
        ...
    }
}

Our page model now has the list of available countries so we can populate the dropdown, but we also need somewhere to store the user's selection. Update the nested InputModel and add the PhoneNumberCountryCode property:

namespace ValidatePhoneNumberDemo.Areas.Identity.Pages.Account.Manage
{
    public partial class IndexModel : PageModel
    {
        public class InputModel
        {
            [Required]
            [EmailAddress]
            public string Email { get; set; }

            [Phone]
            [Display(Name = "Phone number")]
            public string PhoneNumber { get; set; }

            // The country selected by the user.
            [Display(Name = "Phone number country")]
            public string PhoneNumberCountryCode { get; set; }
        }
    }
}

Finally, add the following Razor to Areas\Identity\Pages\Account\Manage\Index.cshtml, just above the Razor code for the phone number input.

<div class="form-group">
    <label asp-for="Input.PhoneNumberCountryCode"></label>
    <select asp-for="Input.PhoneNumberCountryCode" asp-items="Model.AvailableCountries" class="form-control"></select>
    <span asp-validation-for="Input.PhoneNumberCountryCode" class="text-danger"></span>
</div>

This Razor uses the Select Tag Helper to generate a <select> element populated with the values in the AvailableCountries property, which binds to the Input.PhoneNumberCountryCode property on form POST. If you run your app now and navigate to the account management page, you should see the country dropdown displayed above the phone number option.

In this example, we don't have a "none" or "unselected" option in the <select> element. Consequently, the first item is selected and sent back to the user. That's OK for our purposes, but a more comprehensive approach might be to select the default country based on the user's location.

Adding phone number validation using the Twilio Lookup API

At this point we have everything in place to add our extra validation; we have the Twilio client configured, the CountryService loading our list of country codes, and our dropdown for the user to select their country. Now we'll add the validation call to the Twilio API.

Open Index.chstml.cs and add the following namespace using statements:

using Twilio.Exceptions;
using Twilio.Rest.Lookups.V1;

Now replace the following code found inside OnPostAsync:

...
var phoneNumber = await _userManager.GetPhoneNumberAsync(user);
if (Input.PhoneNumber != phoneNumber)
{
    var setPhoneResult = await _userManager.SetPhoneNumberAsync(user, Input.PhoneNumber);
    if (!setPhoneResult.Succeeded)
    {
        var userId = await _userManager.GetUserIdAsync(user);
        throw new InvalidOperationException($"Unexpected error occurred setting phone number for user with ID '{userId}'.");
    }
}
...

with our updated validation code. Take care to preserve the code both above and below the snippet that you're replacing:

...
var phoneNumber = await _userManager.GetPhoneNumberAsync(user);
if (Input.PhoneNumber != phoneNumber)
{
    try
    {
        var numberDetails = await PhoneNumberResource.FetchAsync(
            pathPhoneNumber: new Twilio.Types.PhoneNumber(Input.PhoneNumber),
            countryCode: Input.PhoneNumberCountryCode,
            type: new List<string> { "carrier" });

        // only allow user to set phone number if capable of receiving SMS
        var phoneNumberType = numberDetails.GetPhoneNumberType();
        if (phoneNumberType != null 
            && phoneNumberType == PhoneNumberResource.TypeEnum.Landline)
        {
            ModelState.AddModelError($"{nameof(Input)}.{nameof(Input.PhoneNumber)}",
                $"The number you entered does not appear to be capable of receiving SMS ({phoneNumberType}). Please enter a different value and try again");
            return Page();
        }

        var numberToSave = numberDetails.PhoneNumber.ToString();
        var setPhoneResult = await _userManager.SetPhoneNumberAsync(user, numberToSave);
        if (!setPhoneResult.Succeeded)
        {
            var userId = await _userManager.GetUserIdAsync(user);
            throw new InvalidOperationException($"Unexpected error occurred setting phone number for user with ID '{userId}'.");
        }
    }
    catch (ApiException ex)
    {
        ModelState.AddModelError($"{nameof(Input)}.{nameof(Input.PhoneNumber)}",
            $"The number you entered was not valid (Twilio code {ex.Code}), please check it and try again");
        return Page();
    }
}
...

That's a lot of code to digest, so I'll walk through it below.

Before we do anything with the phone number, we load the existing number for the Identity user and check whether it's changed. If the number hasn't changed, there's no need to validate it, call the Twilio API, or update the value.

var phoneNumber = await _userManager.GetPhoneNumberAsync(user);
if (Input.PhoneNumber != phoneNumber)
{
    // Phone number has changed, so validate and save it
}

If the number has changed, we need to validate the value using the Twilio Lookup API. We use the PhoneNumberResource helper to make an asynchronous call to the Twilio API, passing in the PhoneNumber and PhoneNumberCountryCode from the InputModel. We wrap the call in a try-catch block, as it will throw an exception if it's invalid. In that case we display a generic error and redisplay the form. You could also log the exception, but be wary of storing personally identifiable information (PII) in log messages.

try
{
    var numberDetails = await PhoneNumberResource.FetchAsync(
        pathPhoneNumber: new Twilio.Types.PhoneNumber(Input.PhoneNumber),
        countryCode: Input.PhoneNumberCountryCode,
        type: new List<string> { "carrier" });

    // validation successful
}
catch (ApiException ex)
{
    ModelState.AddModelError($"{nameof(Input)}.{nameof(Input.PhoneNumber)}",
            $"The number you entered was not valid (Twilio code {ex.Code}), please check it and try again");
    return Page();
}

Note that we've added the error as a property error by including the field name in the call to AddModelError(). That means the error will be shown next to the phone number field, as well as in the Validation Summary Tag Helper by default. I've also included the Twilio code in the error message for demonstration purposes. In practice you probably won't want to expose that to users, but you may well want to use it for logging and metric purposes.

As well as validating the number, in this example we also requested the carrier information for the number by passing "carrier" to the type parameter of FetchAsync(). This is not necessary for the validation itself, but it can provide some useful additional information for some business use-cases. Including the "carrier" also returns the likely phone number type of the number.

We extract the phone number type from the response, taking care to handle nulls and missing values. The phone number type has one of three string values: landline, mobile, or voip. We use that value to try and determine if the number can receive SMS. If it can't, we redisplay the form and make the user enter a different value.

// only allow user to set phone number if capable of receiving SMS
var phoneNumberType = numberDetails.GetPhoneNumberType();
if (phoneNumberType != null 
    && phoneNumberType == PhoneNumberResource.TypeEnum.Landline)
{
    ModelState.AddModelError($"{nameof(Input)}.{nameof(Input.PhoneNumber)}",
        $"The number you entered does not appear to be capable of receiving SMS ({phoneNumberType}). Please enter a different value and try again");
    return Page();
}

This behavior may be undesirable in many cases; a better approach might be to return a warning, but still save the number. In any case, the only way to be sure whether the number can receive SMS or voice is to try to message or call it, using the Twilio Verify API for example.

Once we've validated the phone number with Twilio's API and confirmed it's not a landline number, we save the phone number to the Identity user. This code is essentially the same as before we added our customizations with one exception—instead of saving the number as it was entered by the user, we store the number formatted in E.164 format. This has the benefit of not requiring us to store the country code (as it's contained in the formatted number).

var numberToSave = numberDetails.PhoneNumber.ToString();
var setPhoneResult = await _userManager.SetPhoneNumberAsync(user, numberToSave);
if (!setPhoneResult.Succeeded)
{
    var userId = await _userManager.GetUserIdAsync(user);
    throw new InvalidOperationException($"Unexpected error occurred setting phone number for user with ID '{userId}'.");
}

With this code in place you can test out the validation in your own app. When a number fails Twilio's validation, you will see the error message shown below on the top left. When it passes validation but is a landline, you'll see the error message on the bottom left. Finally, when it passes all validation and is not a landline, you'll see the number is updated to the E.164 format (the +1 format) as shown on the right.

The extension method for extracting the phone number type from the PhoneResource object is shown below. It handles nulls, as well as cases where you may not have requested the type using the "carrier" argument.

using Twilio.Rest.Lookups.V1;

namespace ValidatePhoneNumberDemo
{
    public static class PhoneNumberResourceExtensions
    {
        public static PhoneNumberResource.TypeEnum GetPhoneNumberType(this PhoneNumberResource phoneNumber)
        {
            if (phoneNumber?.Carrier != null 
                && phoneNumber.Carrier.TryGetValue("type", out string rawType))
            {
                // implicitly convert from string to PhoneNumberResource.TypeEnum
                return rawType;
            }

            return null;
        }
    }
}

It's important to remember that validating a number using libphonenumber or the Twilio Lookup API (as we did in this post) is typically only the first step in a multi stage process to verify a user controls a phone number. Before using the phone number for business purposes you should confirm the user has access to the device either by sending a verification message/phone call, using the Twilio Verify API for example.

Summary

In this post I showed how you can improve the phone number validation in ASP.NET Core Identity by using the Twilio Lookup API. I showed how to scaffold ASP.NET Core Identity pages so they can be customized using the dotnet-aspnet-codegenerator global tool, and how to load countries for a dropdown list from a JSON file. It's important to include the country when validating phone numbers as different countries have different rules.

The Twilio Lookup API uses the libphonenumber library to check the phone number is valid, but it can also return additional information like the carrier, phone number type, or fraud details. For thorough verification that the user has control of the phone number, you should consider using the Twilio Verify API to send a message/call to the phone number.

Additional Resources

For an introduction to Razor Pages, see my previous post, or the excellent resource, https://www.learnrazorpages.com/. For more information on phone numbers and why they're difficult, see The Falsehoods Programmers Believe About Phone Numbers. For a post on using the libphonenumber directly in your ASP.NET Core application, see this post on the Twilio blog.

Some time ago I wrote a post on the default ASP.NET Core Identity PasswordHasher<> implementation, and how it enables backwards compatibility between password hashing algorithms. In a follow up post, I showed how to create a custom IPasswordHasher<> to slowly migrate existing BCrypt password hashes to the default ASP.NET Core Identity hashing format.

Unfortunately, the implementation in that post is no good for migrating weak password hashes to something more secure. If you are migrating from a weak hashing strategy, you'll end up with some of your passwords continually stored using the weak strategy. Passwords are only stored securely for users that had logged in recently. In this post I'll show a better implementation that solves that problem by taking a hash of a hash.

Disclaimer: You should always think carefully before replacing security-related components, as a lot of effort goes into making the default components secure by default. This article solves a specific problem, but you should only use it if you need it!

The code I'm going to show is based on the ASP.NET Core 2.2 release, but it should work with any 2.x version of ASP.NET Core Identity.

Background

As I discussed in the previous post, the IPasswordHasher<> interface has two responsibilities:

• Hash a password so it can be stored in a database
• Verify a provided plain-text password matches a previously stored hash

In this post I'm focusing on the scenario where you want to add ASP.NET Core Identity to an existing app, and you already have a database that contains usernames and password hashes.

The problem is that your password hashes are stored using a hash format that isn't compatible with ASP.NET Core Identity and is generally insecure (e.g. SHA1 or MD5). In this example, I'm going to assume your passwords are hashed using MD5, but you could easily apply it to other hashing algorithms. The Md5PasswordHasher<> we will create allows you to verify existing password MD5 hashes, while allowing you to create new hashes using the default ASP.NET Core Identity hashing algorithm.

In my previous implementation, I achieved this by creating a custom IPasswordHasher<> implementation that used existing password hashes (BCrypt in that case) to verify the password, and then re-hashed the password on successful login. The downside with that approach is that the IdentityUser.PasswordHash column ends up with a mixture of different hashes stored in it: ASP.NET Core Identity v3 (PBKDF2) hashes for users that have logged in recently, and BCrypt for those that haven't.

From a technical point of view, this isn't a big issue - the format marker byte allows the IPasswordHasher<> implementation to interpret and handle the different hash formats.

However this approach is problematic if your "old" hash format is weak or obsolete, e.g. MD5. If that's the case, you'll have a mix of secure (ASP.NET Core Identity PBKDF2) hashes and thoroughly insecure MD5 hashes stored in your database. If your app ends up having a data breach and appearing on HaveIBeenPwned, those MD5 hashes will be cracked very quickly. 😟

So how can you handle this if your current passwords are stored as MD5 hashes? You can't just "convert" them to a secure format, as that requires knowing the plaintext password for every user.

The correct approach is to apply the new hash algorithm to the MD5 hashes themselves, not the plaintext password. This gives you a "hash-inside-a-hash", so in the event of a data breach your users' passwords have much better protection. As users sign in, you can slowly re-hash their passwords to the un-nested form, but in the mean time you're not exposing insecure MD5 hashes.

When a user logs in and verifies their password, you can re-hash the password using the ASP.NET Core Identity default hash function. That way, hashes will slowly migrate from the legacy hash-inside-a-hash format to the default hash format.

Using a format byte to distinguish hash implementations

As discussed in a previous post, the default PasswordHasher<> implementation already handles multiple hashing formats, namely two different versions of PBKDF2. It does this by storing a single-byte "format-marker" along with the password hash. The whole combination is then Base64 encoded and stored in the database as a string.

When a password needs to be verified and compared to a stored hash, the hash is read from the database, decoded from Base64 to bytes, and the first byte is inspected. If the byte is 0x00, the password hash was created using v2 of the hashing algorithm. If the byte is 0x01, then v3 was used.

We can maintain compatibility with the base PasswordHasher algorithm by storing our own custom format marker in the first byte of the password hash in a similar way. 0x00 and 0x01 are already taken, so I chose 0xF0 for this case as it seems like it should be safe for a while!

A slightly abbreviated version of the Md5PasswordHasher implementation is shown below (you can find the complete source code on GitHub). When a password hash and plain-text password are provided for verification, we follow a similar approach to the default PasswordHasher<>. We convert the password from Base64 into bytes, and examine the first byte. If the hash starts with 0xF0 then we have a hash-inside-a-hash. If it starts with something else, then we pass the original stored hashed and provided plain-text password to the base PasswordHasher<> implementation.

If we find we are working with a hash-inside-a-hash, then we replace the 0xF0 format-marker with 0x01, and convert it back to a Base64 string for use with the base PasswordHasher<> implementation. We also take the provided password and create an MD5 hash of it. We then pass the MD5 hash as the "provided password" to the base VerifyHashedPassword method. This then hashes it using the Identity v3 PBKDF2 format (thanks to the 0x01 format marker we added) and compares the result with storedPassword.

public class Md5PasswordHasher<TUser> : PasswordHasher<TUser> where TUser : class
{
    public override PasswordVerificationResult VerifyHashedPassword(TUser user, string hashedPassword, string providedPassword)
    {
        byte[] decodedHashedPassword = Convert.FromBase64String(hashedPassword);

        // read the format marker from the hashed password
        if (decodedHashedPassword.Length == 0)
        {
            return PasswordVerificationResult.Failed;
        }

        // ASP.NET Core uses 0x00 and 0x01 for v2 and v3
        if (decodedHashedPassword[0] == 0xF0)
        {
            // replace the 0xF0 prefix in the stored password with 0x01 (ASP.NET Core Identity V3) and convert back to Base64
            decodedHashedPassword[0] = 0x01;
            var storedPassword = Convert.ToBase64String(decodedHashedPassword);

            // md5 hash the provided password
            var md5ProvidedPassword = GetM5Hash(providedPassword);

            // call the base implementation with the new values
            var result = base.VerifyHashedPassword(user, storedPassword, md5ProvidedPassword);

            return result == PasswordVerificationResult.Success
                ? PasswordVerificationResult.SuccessRehashNeeded
                : result;
        }

        return base.VerifyHashedPassword(user, hashedPassword, providedPassword);
    }

    public static string GetM5Hash(string input)
    {
        using (MD5 md5Hash = MD5.Create())
        {
            var bytes = md5Hash.ComputeHash(Encoding.UTF8.GetBytes(input));

            return Convert.ToBase64String(bytes);
        }
    }
}

If the provided password was correct (the base implementation returned PasswordVerificationResult.Success) then we force the ASP.NET Core Identity system to re-hash the password. This strips out the MD5 layer from the hash, leaving you with a "raw" ASP.NET Core Identity v3 PBKDF2 format hash stored in the database.

New passwords will always be created with the default v3 PBKDF2 format anyway, as we don't override the HashPassword method.

You can replace the default PasswordHasher<> implementation in your application by registering the Md5PasswordHasher in Startup.ConfigureServices(). There's a number of ways to do this, but I show how you can use the Replace() extension method below. Make sure to add this line after calling AddDefaultIdentity<>() or AddIdentity<>():

public void ConfigureServices(IServiceCollection services)
{
    // ...
    services.AddDefaultIdentity<IdentityUser>()
        .AddDefaultUI(UIFramework.Bootstrap4)
        .AddEntityFrameworkStores<ApplicationDbContext>();
    // ...

    // Replace the existing scoped IPasswordHasher<> implementation
    services.Replace(new ServiceDescriptor(
        serviceType: typeof(IPasswordHasher<IdentityUser>),
        implementationType: typeof(Md5PasswordHasher<IdentityUser>),
        ServiceLifetime.Scoped));

}

This is all you need in the normal operation of your application, but before you can run your app you need to create your hash-inside-a-hash values.

Converting stored MD5 passwords to support the Md5PasswordHasher

The approach of extending the default PasswordHasher<> implementation shown in this post requires you to have already stored your passwords against each IdentityUser using the hash-inside-a-hash mechanism and the 0xF0 format-marker. That means you'll need to re-hash your existing MD5 hashes with the ASP.NET Core Identity password hasher.

How you do this is highly dependent on how and where your passwords are stored. I've provided a basic extension method below that takes an IdentityUser and an existing MD5 hash string and produces a string in a format compatible with the Md5PasswordHasher.

public static class UserManagerExtensions
{
    public static async Task<IdentityResult> SetMd5PasswordForUser(
        this UserManager<IdentityUser> userManager, 
        IdentityUser user, 
        string md5Password)
    {
        // Performs v3 PBKDF2 hash of provided MD5 hash
        var reHashedPassword = userManager.PasswordHasher.HashPassword(user, md5Password);

        // Replace the format marker so we know to MD5 hash 
        // provided passwords during password verification
        var passwordToStore = ReplaceFormatMarker(reHashedPassword, 0xF0);

        // Replace the old hash with the "updated marker" hash
        user.PasswordHash = passwordToStore;

        // Roll the security stamp for the user (invalidates security-related tokens)
        await userManager.UpdateSecurityStampAsync(user);

        // Save the changes to the DB and return the result
        return await userManager.UpdateAsync(user);
    }

    // Replace the fomat marker in Base64 encoded string 
    // Not the most efficient but does the job
    private static string ReplaceFormatMarker(string passwordHash, byte formatMarker)
    {
        var bytes = Convert.FromBase64String(passwordHash);
        bytes[0] = formatMarker;
        return Convert.ToBase64String(bytes);
    }
}

During your migration to ASP.NET Core Identity you would create a new IdentityUser for each of your existing users, and then call SetMd5PasswordForUser, passing in the md5 formatted password.

_userManager.SetMd5PasswordForUser(user, md5Password);

I have a basic proof of concept for this in the sample app on GitHub. It's a little contrived, but you can register as a new user in the sample (which stores the password hash as v3 PBKDF2). The home page then lets you enter a new password which is MD5 hashed and saved to the current IdentityUser using the SetMd5PasswordForUser extension method.

If you log out, and then sign back in with the new password, the nested hash-within-a-hash will automatically be re-hashed to strip out the MD5 layer again, leaving the "raw" v3 PBKDF2 format hash (as per the Md5PasswordHasher implementation).

Summary

In this post I showed how you could extend the default ASP.NET Core Identity PasswordHasher<> implementation to allow migrating from insecure hash formats. This lets you verify hashes created using a legacy format (MD5 in this example), and update them to use the default Identity password hashing algorithm so the vulnerable hashes are protected in the event of a data breach.

ASP.NET Core Identity is a membership system that adds user sign in and user management functionality to ASP.NET Core apps. It includes many features out of the box and has basic support for storing a phone number for a user. By default, ASP.NET Core Identity doesn't try to verify ownership of phone numbers, but you can add that functionality yourself by integrating Twilio’s identity verification features into your application.

In this post you'll learn how you can prove ownership of a phone number provided by a user using Twilio Verify in an ASP.NET Core application using Razor Pages. This involves sending a code in an SMS message to the provided phone number. The user enters the code received and Twilio confirms whether it is correct. If so, you can be confident the user has control of the provided phone number.

You typically only confirm phone number ownership once for a user. This is in contrast to two-factor authentication (2FA) where you might send an SMS code to the user every time they login. Twilio has a separate Authy API for performing 2FA checks at login, but it won't be covered in this post.

Note that this post uses version 1 of the Twilio Verify API. Version 2.x of the API is currently in Beta.

Prerequisites

To follow along with this post you'll need:

• A Twilio account (sign up for a free Twilio account here)
• A Twilio Verify Application (create a new Verify application here)
• .NET Core 2.2 SDK (version 2.2.102 or greater)
• VS Code, Visual Studio 2017, or other editor
• A familiarity with Razor Pages (see a previous post for an introduction to Razor Pages)

You can find the complete code for this post on GitHub.

Creating the case study project

Using Visual Studio 2017+ or the .NET CLI, create a new solution and project with the following characteristics:

• Type: ASP.NET Core 2.2 Web Application (not MVC) with Visual C#
• Name: SendVerificationSmsDemo
• Solution directory
• Git repository
• https
• Authentication: Individual user accounts, Store user accounts in-app

ASP.NET Core Identity uses Entity Framework Core to store the users in the database, so be sure to run the database migrations in the project folder after building your app. Execute one of the following command line instructions to build the database:

.NET CLI

dotnet ef database update

Package Manager Console

update-database

The Twilio C# SDK and the Twilio Verify API

The Twilio API is a typical REST API, but to make it easier to work with Twilio provides helper SDK libraries in a variety of languages. Previous posts have shown how to use the C# SDK to validate phone numbers, and how to customize it to work with the ASP.NET Core dependency injection container.

Unfortunately, the C# SDK doesn't support the current version of the Twilio Verify API (v1.x), so you have to fall back to making "raw" HTTP requests with an HttpClient. The basic code required to do so is described in the documentation, but for conciseness it uses a bad practice: it creates an HttpClient manually in the code.

In ASP.NET Core 2.1 and above, you should use HttpClientFactory wherever possible. This class manages the lifetime of the underlying handlers and sockets for you, and so avoids performance issues you can hit at times of high load. You can learn about using HttpClientFactory with the Twilio SDK in a previous post on the Twilio blog.

Creating a Typed client for the Twilio Verify API

To make it easier to work with the Verify API, and to support HttpClientFactory, you will create a small Typed client. Create the TwilioVerifyClient.cs file in the root of your project, and replace the contents with the following code:

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.AspNetCore.WebUtilities;
using Newtonsoft.Json;

namespace SendVerificationSmsDemo
{
    public class TwilioVerifyClient
    {
        private readonly HttpClient _client;
        public TwilioVerifyClient(HttpClient client)
        {
            _client = client;
        }

        public async Task<TwilioSendVerificationCodeResponse> StartVerification(int countryCode, string phoneNumber)
        {
            var requestContent = new FormUrlEncodedContent(new[] {
                new KeyValuePair<string, string>("via", "sms"),
                new KeyValuePair<string, string>("country_code", countryCode.ToString()),
                new KeyValuePair<string, string>("phone_number", phoneNumber),
            });

            var response = await _client.PostAsync("protected/json/phones/verification/start", requestContent);

            var content = await response.Content.ReadAsStringAsync();

            // this will throw if the response is not valid
            return JsonConvert.DeserializeObject<TwilioSendVerificationCodeResponse>(content);
        }

        public async Task<TwilioCheckCodeResponse> CheckVerificationCode(int countryCode, string phoneNumber, string verificationCode)
        {
            var queryParams = new Dictionary<string, string>()
            {
                {"country_code", countryCode.ToString()},
                {"phone_number", phoneNumber},
                {"verification_code", verificationCode },
            };

            var url = QueryHelpers.AddQueryString("protected/json/phones/verification/check", queryParams);

            var response = await _client.GetAsync(url);

            var content = await response.Content.ReadAsStringAsync();

            // this will throw if the response is not valid
            return JsonConvert.DeserializeObject<TwilioCheckCodeResponse>(content);
        }

        public class TwilioCheckCodeResponse
        {
            public string Message { get; set; }
            public bool Success { get; set; }
        }

        public class TwilioSendVerificationCodeResponse
        {
            public string Carrier { get; set; }
            public bool IsCellphone { get; set; }
            public string Message { get; set; }
            public string SecondsToExpire { get; set; }
            public Guid Uuid { get; set; }
            public bool Success { get; set; }
        }
    }
}

The TwilioVerifyClient has two methods, StartVerification() and CheckVerificationCode(), which handle creating HTTP requests with the correct format and calling the Twilio Verify API. The typed client accepts an HttpClient in its constructor, which will be created by the HttpClientFactory automatically for you.

The responses for the method are implemented as simple POCO objects that match the response returned by the Twilio Verify API. For simplicity, these are implemented here as nested classes of the TwilioVerifyClient, but you can move these to another file if you prefer.

Configuring authentication for the typed client

HttpClientFactory is not part of the base ASP.NET Core libraries, so you need to install the Microsoft.Extensions.Http NuGet package (version 2.2.0 or later). You can use the NuGet Package Manager, Package Manager Console CLI, or edit the SendVerificationSmsDemo.csproj file. After using any of these methods the <ItemGroup> section of the project file should look like this (version numbers may be higher):

<ItemGroup>
  <PackageReference Include="Microsoft.AspNetCore.App" />
  <PackageReference Include="Microsoft.AspNetCore.Razor.Design" Version="2.2.0" PrivateAssets="All" />
  <PackageReference Include="Microsoft.Extensions.Http" Version="2.2.0" />
</ItemGroup>

To call the Twilio Verify API you'll need the Authy API Key for your Verify application (found in the Twilio Dashboard). When developing locally you should store this using the Secrets Manager so it doesn't get accidentally committed to your source code repository. You can read about how and why to do that in this post on the Twilio Blog. Your resulting secrets.json should look something like this:

{
  "Twilio": {
    "VerifyApiKey": "DBxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

Finally, configure your TwilioVerifyClient with the correct BaseAddress and your API key by adding the following at the end of ConfigureServices in Startup.cs:

public void ConfigureServices(IServiceCollection services)
{
    // existing configuration

    var apiKey = Configuration["Twilio:VerifyApiKey"];

    services.AddHttpClient<TwilioVerifyClient>(client =>
    {
        client.BaseAddress = new Uri("https://api.authy.com/");
        client.DefaultRequestHeaders.Add("X-Authy-API-Key", apiKey);
    });
}

Don't let the authy base URL confuse you: version 1 of the Verify API followed an interface structure that preceded the creation of Verify as a separate product. The version 2 Verify API normalizes the URI.

With the typed client configuration complete, you can start adding the phone verification functionality to your ASP.NET Core Identity application.

Adding the required scaffolding files

In this post we're going to be adding some additional pages to the Identity area. Typically when you're adding or editing Identity pages in ASP.NET Core you should use the built-in scaffolding tools to generate the pages, as shown in this post. If you've already done that, you can skip this section.

Rather than adding all the Identity scaffolding, all you need for this post is a single file. Create the file __ViewImports.cshtml_ in the Areas/Identity/Pages folder and add the following code:

@using Microsoft.AspNetCore.Identity
@using SendVerificationSmsDemo.Areas.Identity
@namespace SendVerificationSmsDemo.Areas.Identity.Pages
@addTagHelper *, Microsoft.AspNetCore.Mvc.TagHelpers

This adds all the required namespaces and tag helpers required by your Razor Pages. It will also light up the IntelliSense in Visual Studio. If you've already scaffolded Identity pages you'll already have this file!

Sending a verification code to a phone number

The default ASP.NET Core Identity templates provide the functionality for storing a phone number for a user, but don't provide the capability to verify ownership of the number. In the post Validating Phone Numbers in ASP.NET Core Identity Razor Pages with Twilio Lookup you can learn how to validate a phone number by using the Twilio Lookup API. As noted in the post it’s a good idea to store the result formatted as an E.164 number.

In version 1 of the Verify API you must provide the country dialing code and phone number as separate parameters. Consequently, if you're using version 1 of the Verify API it may be best to store these values separately for the IdentityUser user, instead of (or as well as) storing an E.164 number.

For simplicity, this post will gloss over storing the values separately. You’ll create a form that lets the user enter the values separately. In practice, you would automatically use the values attached to the user, rather than allowing them to enter in a new number.

Create a new Razor Page in the Areas/Identity/Pages/Account folder called VerifyPhone.cshtml. In the code-behind file VerifyPhone.cshtml.cs add the following using statements to the top of the file:

using System.ComponentModel.DataAnnotations;
using Microsoft.AspNetCore.Authorization;

Next, replace the VerifyPhoneModel class with the following:

[Authorize]
public class VerifyPhoneModel : PageModel
{
    private readonly TwilioVerifyClient _client;

    public VerifyPhoneModel(TwilioVerifyClient client)
    {
        _client = client;
    }

    [BindProperty]
    public InputModel Input { get; set; }

    public class InputModel
    {
        [Required]
        [Display(Name = "Country dialing code")]
        public int DialingCode { get; set; }

        [Required]
        [Phone]
        [Display(Name = "Phone number")]
        public string PhoneNumber { get; set; }
    }

    public async Task<IActionResult> OnPostAsync()
    {
        if (!ModelState.IsValid)
        {
            return Page();
        }

        try
        {
            var result = await _client.StartVerification(Input.DialingCode, Input.PhoneNumber);
            if (result.Success)
            {
                return RedirectToPage("ConfirmPhone", new {Input.DialingCode, Input.PhoneNumber});
            }

            ModelState.AddModelError("", $"There was an error sending the verification code: {result.Message}");
        }
        catch (Exception)
        {
            ModelState.AddModelError("", 
                "There was an error sending the verification code, please check the phone number is correct and try again");
        }

        return Page();
    }
}

The InputModel for the page is used to bind a simple form (screenshot below) for collecting the country dialing code and phone number to verify. The code doesn’t include an OnGet handler as the framework provides one implicitly. The OnPost handler is where the verification process begins.

The code provides some basic validation using DataAnnotation attributes (see the post Validating Phone Numbers in ASP.NET Core Identity Razor Pages with Twilio Lookup to learn about performing robust validation) and, if successful, it uses the injected TwilioVerifyClient to start verification. If the Verify API call is successful, the user is redirected to the ConfirmPhone page, which you'll create shortly. If the Verify API indicates the request failed, or if an exception is thrown, an error is added to the ModelState, and the page is re-displayed to the user.

The form itself consists of two text boxes and a submit button. Replace the contents of VerifyPhone.cshtml with the following Razor markup:

@page
@model VerifyPhoneModel
@{
    ViewData["Title"] = "Verify phone number";
}

<h4>@ViewData["Title"]</h4>
<div class="row">
    <div class="col-md-8">
        <form method="post">
            <div asp-validation-summary="ModelOnly" class="text-danger"></div>
            <div class="form-row">
                <div class="form-group col-md-4">
                    <label asp-for="Input.DialingCode"></label>
                    <input asp-for="Input.DialingCode" class="form-control" />
                    <span asp-validation-for="Input.DialingCode" class="text-danger"></span>
                </div>
                <div class="form-group col-md-8">
                    <label asp-for="Input.PhoneNumber"></label>
                    <input asp-for="Input.PhoneNumber" class="form-control" />
                    <span asp-validation-for="Input.PhoneNumber" class="text-danger"></span>
                </div>
            </div>
            <button type="submit" class="btn btn-primary">Send verification code</button>
        </form>
    </div>
</div>

@section Scripts {
    <partial name="_ValidationScriptsPartial" />
}

When rendered, the form looks like the following:

To test the form, run your app, and navigate to /Identity/Account/VerifyPhone. Enter your country code and phone number and click Send verification code. If the phone number is valid, you’ll receive an SMS similar to the message shown below. Note that you can customize this message, including the language, terminology, and code length: see the Verify API documentation for details.

Your Twilio Verify API demo verification code is: 2933

Now you need to create the page where the user enters the code they receive.

Checking the verification code

The check verification code page contains a single text box where the user enters the code they receive. Create a new Razor Page in the Areas/Identity/Pages/Account folder called ConfirmPhone.cshtml. In the code-behind file ConfirmPhone.cshtml.cs add the following using statements to the top of the file:

using System.ComponentModel.DataAnnotations;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Identity;

Next replace the ConfirmPhoneModel class with the following:

[Authorize]
public class ConfirmPhoneModel : PageModel
{
    private readonly TwilioVerifyClient _client;
    private readonly UserManager<IdentityUser> _userManager;

    public ConfirmPhoneModel(TwilioVerifyClient client, UserManager<IdentityUser> userManager)
    {
        _client = client;
        _userManager = userManager;
    }

    [BindProperty(SupportsGet = true)]
    public InputModel Input { get; set; }

    public class InputModel
    {
        [Required]
        [Display(Name = "Country dialing code")]
        public int DialingCode { get; set; }

        [Required]
        [Phone]
        [Display(Name = "Phone number")]
        public string PhoneNumber { get; set; }

        [Required]
        [Display(Name = "Code")]
        public string VerificationCode { get; set; }
    }

    public async Task<IActionResult> OnPostAsync()
    {
        if (!ModelState.IsValid)
        {
            return Page();
        }

        try
        {
            var result = await _client.CheckVerificationCode(Input.DialingCode, Input.PhoneNumber, Input.VerificationCode);
            if (result.Success)
            {
                var identityUser = await _userManager.GetUserAsync(User);
                identityUser.PhoneNumberConfirmed = true;
                var updateResult =  await _userManager.UpdateAsync(identityUser);

                if (updateResult.Succeeded)
                {
                    return RedirectToPage("ConfirmPhoneSuccess");
                }
                else
                {
                    ModelState.AddModelError("", "There was an error confirming the verification code, please try again");
                }
            }
            else
            {
                ModelState.AddModelError("", $"There was an error confirming the verification code: {result.Message}");
            }
        }
        catch (Exception)
        {
            ModelState.AddModelError("",
                "There was an error confirming the code, please check the verification code is correct and try again");
        }

        return Page();
    }
}

As before, we can skip the OnGet handler, as that's provided implicitly by the framework. The InputModel has three parameters 1) the dialing code, 2) the phone number provided in the previous step, and 3) the verification code entered by the user.

For simplicity, the code passed the phone number and country code from the previous step to this page via the querystring. In practice, you would load these from the IdentityUser itself, as described earlier.

Calling the Twilio Verify API is simple thanks to the typed client. The dialing code and phone number from the previous step and the authentication code entered by the user are passed to the API. If the check is successful, the Verify API will returnresult.Success=true. You can store the confirmation result on the IdentityUser object directly by setting the PhoneNumberConfirmed property and saving the changes.

If everything completes successfully, you redirect the user to a simple ConfirmPhoneSuccess page (that you'll create shortly). If there are any errors or exceptions, an error is added to the ModelState and the page is redisplayed.

Replace the contents of ConfirmPhone.cshtml with the Razor markup below. For usability, the provided phone number is redisplayed in the page.

@page
@model ConfirmPhoneModel
@{
    ViewData["Title"] = "Confirm phone number";
}

<h4>@ViewData["Title"]</h4>
<div class="row">
    <div class="col-md-6">
        <form method="post">
            <p>
                We have sent a confirmation code to (@Model.Input.DialingCode) @Model.Input.PhoneNumber
                Enter the code you receive to confirm your phone number.
            </p>
            <div asp-validation-summary="All" class="text-danger"></div>
            <input asp-for="Input.DialingCode" type="hidden" />
            <input asp-for="Input.PhoneNumber" type="hidden" />

            <div class="form-group">
                <label asp-for="Input.VerificationCode"></label>
                <input asp-for="Input.VerificationCode" class="form-control" type="number" />
                <span asp-validation-for="Input.VerificationCode" class="text-danger"></span>
            </div>
            <button type="submit" class="btn btn-primary">Confirm</button>
        </form>
    </div>
</div>

@section Scripts {
    <partial name="_ValidationScriptsPartial" />
}

When rendered, this looks like the following:

Once the user successfully confirms their phone number you can be confident they have access to it and you can use it in other parts of your application with confidence.

Showing a confirmation success page

To create a simple "congratulations" page for the user, create a new Razor Page in the Areas/Identity/Pages/Account folder called ConfirmPhoneSuccess.cshtml. You don't need to change the code-behind for this page, just add the following markup to ConfirmPhoneSuccess.cshtml:

@page
@model ConfirmPhoneSuccessModel
@{
    ViewData["Title"] = "Phone number confirmed";
}

<h1>@ViewData["Title"]</h1>
<div>
    <p>
        Thank you for confirming your phone number.
    </p>
    <a asp-page="/Index">Back to home</a>
</div>

After entering a correct verification code, users will be redirected to this page. From here, they can return to the home page.

Trying out the Twilio Verify functionality

Try out what you’ve just built by running the app. Follow these steps to validate a user’s ownership of a phone number with Verify:

Navigate to https://localhost:44348/Identity/Account/VerifyPhone in your browser. Because this page is protected by ASP.NET Core Identity authorization, you’ll be redirected to the account log in page.

Register as a new user. You will be redirected to the /Identity/Account/VerifyPhone route and will see the rendered VerifyPhone.cshtml Razor page. At this point you can see the record for the user you created in the dbo.AspNetUsers table of the database aspnet-SendVerificationSmsDemo-<GUID>. Note that the phone number is null.

Enter a valid country code and phone number capable of receiving SMS text messages. Click Send verification code. You should be routed to a URI similar to /Identity/Account/ConfirmPhone?DialingCode=44&PhoneNumber=07123456789, where the DialingCode and PhoneNumber reflect the values you entered.

In a matter of moments you should receive an SMS message with a verification code. Note that the message reflects the name of the application you created in Twilio Verify.

At this point you can go to the Verify console at https://www.twilio.com/console/verify/applications. You should see a value of 1 in the SMS VERIFICATION STARTED column. Select the application matching the SMS message you received.

Enter the numeric code from the SMS message in the Code box on the Confirm phone number page and click Confirm. (Validation codes expire, so you need to do this within 10 minutes of receiving the code.)

If everything worked correctly you should be redirected to the /Identity/Account/ConfirmPhoneSuccess page. If you refresh the Verify Insights for your application in the Twilio Console you should see the successful validation reflected in statistics for the application.

Good work! You've successfully integrated Twilio Verify with ASP.NET Core 2.2 Identity.

Possible improvements

This post showed the basic approach for using version 1 of the Verify API with ASP.NET Core Identity, but there are many improvements you could make:

• Store the country dialing code and phone number on the IdentityUser. This would be required for practical implementations, as you want to be sure that the phone number stored against the user is the one you are verifying! This would also simplify the code somewhat, as described previously.
• Include a link the VerifyPhone page. Currently you have to navigate manually to Identity/Account/VerifyPhone, but in practice you would want to add a link to it somewhere in your app.
• Show the verification status of the phone number in the app. By default, ASP.NET Core Identity doesn't display the IdentityUser.PhoneNumberConfirmed property anywhere in the app.
• Only verify unconfirmed numbers. Related to the previous improvement, you probably only want to verify phone numbers once, so you should check for PhoneNumberConfirmed=true in the VerifyPhone page, as well as hide any verification links.
• Allow re-sending the code. In some cases, users might find the verification code doesn't arrive. For a smoother user experience, you could add functionality to allow re-sending a confirmation code to the ConfirmPhone page.

Summary

In this post you saw how to use version 1 of the Twilio Verify API to confirm phone number ownership in an ASP.NET Core Identity application. You learned how to create a typed client for calling the API that uses best practices like HttpClientFactory, and how to use it from Razor Pages. This example took the simplistic route of asking users to re-enter their country dialing code and phone number, but in real applications you should store these in the IdentityUser directly.

You can find the complete sample code for this post on GitHub.

In this post I describe how to run Quartz.NET jobs using an ASP.NET Core hosted service. I show how to create a simple IJob, a custom IJobFactory, and a QuartzHostedService that runs jobs while your application is running. I'll also touch on some of the issues to aware of, namely of using scoped services inside singleton classes.

Introduction - what is Quartz.NET?

As per their website:

Quartz.NET is a full-featured, open source job scheduling system that can be used from smallest apps to large scale enterprise systems.

It's an old staple of many ASP.NET developers, used as a way of running background tasks on a timer, in a reliable, clustered, way. Using Quartz.NET with ASP.NET Core is pretty similar - Quartz.NET supports .NET Standard 2.0, so you can easily use it in your applications.

Quartz.NET has two main concepts:

• A job. This is the background tasks that you want to run on some sort of schedule.
• A scheduler. This is responsible for running jobs based on triggers, on a time-based schedule.

ASP.NET Core has good support for running "background tasks" via way of hosted services. Hosted services are started when your ASP.NET Core app starts, and run in the background for the lifetime of the application. By creating a Quartz.NET hosted service, you can use a standard ASP.NET Core application for running your tasks in the background.

This sort of non-HTTP scenario is also possible with the "generic host", but for various reasons I generally don't use those at the moment. This should hopefully improve in ASP.NET Core 3.0 with the extra investment going into these non-HTTP scenarios.

While it's possible to create a "timed" background service, (that runs a tasks every 10 minutes, for example), Quartz.NET provides a far more robust solution. You can ensure tasks only run at specific times of the day (e.g. 2:30am), or only on specific days, or any combination by using a Cron trigger. It also allows you to run multiple instances of your application in a clustered fashion, so that only a single instance can run a given task at any one time.

In this post I'll show the basics of creating a Quartz.NET job and scheduling it to run on a timer in a hosted service.

Installing Quartz.NET

Quartz.NET is a .NET Standard 2.0 NuGet package, so it should be easy to install in your application. For this test I created an ASP.NET Core project and chose the Empty template. You can install the Quartz.NET package using dotnet add package Quartz. If you view the .csproj for the project, it should look something like this:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.2</TargetFramework>
    <AspNetCoreHostingModel>InProcess</AspNetCoreHostingModel>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="Quartz" Version="3.0.7" />
  </ItemGroup>

</Project>

Creating an IJob

For the actual background work we are scheduling, we're just going to use a "hello world" implementation that writes to an ILogger<> (and hence to the console). You should implement the Quartz interface IJob which contains a single asynchronous Execute() method. Note that we're using dependency injection here to inject the logger into the constructor.

using Microsoft.Extensions.Logging;
using Quartz;
using System.Threading.Tasks;

[DisallowConcurrentExecution]
public class HelloWorldJob : IJob
{
    private readonly ILogger<HelloWorldJob> _logger;
    public HelloWorldJob(ILogger<HelloWorldJob> logger)
    {
        _logger = logger;
    }

    public Task Execute(IJobExecutionContext context)
    {
        _logger.LogInformation("Hello world!");
        return Task.CompletedTask;
    }
}

I also decorated the job with the [DisallowConcurrentExecution] attribute. This attribute prevents Quartz.NET from trying to run the same job concurrently.

Creating an IJobFactory

Next, we need to tell Quartz how it should create instances of IJob. By default, Quartz will try and "new-up" instances of the job using Activator.CreateInstance, effectively calling new HelloWorldJob(). Unfortunately, as we're using constructor injection, that won't work. Instead, we can provide a custom IJobFactory that hooks into the ASP.NET Core dependency injection container (IServiceProvider):

using Microsoft.Extensions.DependencyInjection;
using Quartz;
using Quartz.Spi;
using System;

public class SingletonJobFactory : IJobFactory
{
    private readonly IServiceProvider _serviceProvider;
    public SingletonJobFactory(IServiceProvider serviceProvider)
    {
        _serviceProvider = serviceProvider;
    }

    public IJob NewJob(TriggerFiredBundle bundle, IScheduler scheduler)
    {
        return _serviceProvider.GetRequiredService(bundle.JobDetail.JobType) as IJob;
    }

    public void ReturnJob(IJob job) { }
}

This factory takes an IServiceProvider in the constructor, and implements the IJobFactory interface. The important method is the NewJob() method, in which the factory has to return the IJob requested by the Quartz scheduler. In this implementation we delegate directly to the IServiceProvider, and let the DI container find the required instance. The cast to IJob at the end is required because the non-generic version of GetRequiredService returns an object.

The ReturnJob method is where the scheduler tries to return (i.e. destroy) a job that was created by the factory. Unfortunately, there's no mechanism for doing so with the built-in IServiceProvider. We can't create a new IScopeService that fits into the required Quartz API, so we're stuck only being able to create singleton jobs.

This is important. With the above implementation, it is only safe to create IJob implementations that are Singletons.

Configuring the Job

I'm only showing a single IJob implementation here, but we want the Quartz hosted service to be a generic implementation that works for any number of jobs. To help with that, we create a simple DTO called JobSchedule that we'll use to define the timer schedule for a given job type:

using System;

public class JobSchedule
{
    public JobSchedule(Type jobType, string cronExpression)
    {
        JobType = jobType;
        CronExpression = cronExpression;
    }

    public Type JobType { get; }
    public string CronExpression { get; }
}

The JobType is the .NET type of the job (HelloWorldJob for our example), and CronExpression is a Quartz.NET Cron expression. Cron expressions allow complex timer scheduling so you can set rules like "fire every half hour between the hours of 8 am and 10 am, on the 5th and 20th of every month". Just be sure to check the documentation for examples as not all Cron expressions used by different systems are interchangeable.

We'll add the job to DI and configure its schedule in Startup.ConfigureServices():

using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.DependencyInjection;
using Quartz;
using Quartz.Impl;
using Quartz.Spi;

public void ConfigureServices(IServiceCollection services)
{
    // Add Quartz services
    services.AddSingleton<IJobFactory, SingletonJobFactory>();
    services.AddSingleton<ISchedulerFactory, StdSchedulerFactory>();

    // Add our job
    services.AddSingleton<HelloWorldJob>();
    services.AddSingleton(new JobSchedule(
        jobType: typeof(HelloWorldJob),
        cronExpression: "0/5 * * * * ?")); // run every 5 seconds
}

This code adds four things as singletons to the DI container:

• The SingletonJobFactory shown earlier, used for creating the job instances.
• An implementation of ISchedulerFactory, the built-in StdSchedulerFactory, which handles scheduling and managing jobs
• The HelloWorldJob job itself
• An instance of JobSchedule for the HelloWorldJob with a Cron expression to run every 5 seconds.

There's only one piece missing now that brings them all together, the QuartzHostedService.

Creating the QuartzHostedService

The QuartzHostedService is an implementation of IHostedService that sets up the Quartz scheduler, and starts it running in the background. Due to the design of Quartz, we can implement IHostedService directly, instead of the more common approach of deriving from the base BackgroundService class. The full code for the service is listed below, and I'll discuss it afterwards.

using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;
using Quartz;
using Quartz.Spi;

public class QuartzHostedService : IHostedService
{
    private readonly ISchedulerFactory _schedulerFactory;
    private readonly IJobFactory _jobFactory;
    private readonly IEnumerable<JobSchedule> _jobSchedules;

    public QuartzHostedService(
        ISchedulerFactory schedulerFactory,
        IJobFactory jobFactory,
        IEnumerable<JobSchedule> jobSchedules)
    {
        _schedulerFactory = schedulerFactory;
        _jobSchedules = jobSchedules;
        _jobFactory = jobFactory;
    }
    public IScheduler Scheduler { get; set; }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        Scheduler = await _schedulerFactory.GetScheduler(cancellationToken);
        Scheduler.JobFactory = _jobFactory;

        foreach (var jobSchedule in _jobSchedules)
        {
            var job = CreateJob(jobSchedule);
            var trigger = CreateTrigger(jobSchedule);

            await Scheduler.ScheduleJob(job, trigger, cancellationToken);
        }

        await Scheduler.Start(cancellationToken);
    }

    public async Task StopAsync(CancellationToken cancellationToken)
    {
        await Scheduler?.Shutdown(cancellationToken);
    }

    private static IJobDetail CreateJob(JobSchedule schedule)
    {
        var jobType = schedule.JobType;
        return JobBuilder
            .Create(jobType)
            .WithIdentity(jobType.FullName)
            .WithDescription(jobType.Name)
            .Build();
    }

    private static ITrigger CreateTrigger(JobSchedule schedule)
    {
        return TriggerBuilder
            .Create()
            .WithIdentity($"{schedule.JobType.FullName}.trigger")
            .WithCronSchedule(schedule.CronExpression)
            .WithDescription(schedule.CronExpression)
            .Build();
    }
}

The QuartzHostedService has three dependencies: the ISchedulerFactory and IJobFactory we configured in Startup, and an IEnumerable<JobSchedule>. We only added a single JobSchedule to the DI container (for the HelloWorldJob), but if you register more job schedules with the DI container they'll all be injected here.

StartAsync is called when the application starts up and is where we configure Quartz. We start by creating an instance of IScheduler, assigning it to a property for use later, and setting the JobFactory for the scheduler to the injected instance:

public async Task StartAsync(CancellationToken cancellationToken)
{
    Scheduler = await _schedulerFactory.GetScheduler(cancellationToken);
    Scheduler.JobFactory = _jobFactory;

    // ...
}

Next, we loop through the injected job schedules, and create a Quartz IJobDetail and ITrigger for each one using the CreateJob and CreateTrigger helper methods at the end of the class. If you don't like how this part works, or need more control over the configuration, you can easily customise it by extending the JobSchedule DTO as you see fit.

public async Task StartAsync(CancellationToken cancellationToken)
{
    // ...
    foreach (var jobSchedule in _jobSchedules)
    {
        var job = CreateJob(jobSchedule);
        var trigger = CreateTrigger(jobSchedule);

        await Scheduler.ScheduleJob(job, trigger, cancellationToken);
    }
    // ...
}

private static IJobDetail CreateJob(JobSchedule schedule)
{
    var jobType = schedule.JobType;
    return JobBuilder
        .Create(jobType)
        .WithIdentity(jobType.FullName)
        .WithDescription(jobType.Name)
        .Build();
}

private static ITrigger CreateTrigger(JobSchedule schedule)
{
    return TriggerBuilder
        .Create()
        .WithIdentity($"{schedule.JobType.FullName}.trigger")
        .WithCronSchedule(schedule.CronExpression)
        .WithDescription(schedule.CronExpression)
        .Build();
}

Finally, once all the jobs are scheduled, you call Scheduler.Start() to actually start the Quartz.NET scheduler processing in the background. When the app shuts down, the framework will call StopAsync(), at which point you can call Scheduler.Stop() to safely shut down the scheduler process.

public async Task StopAsync(CancellationToken cancellationToken)
{
    await Scheduler?.Shutdown(cancellationToken);
}

You can register the hosted service using the AddHostedService() extension method in Startup.ConfigureServices:

public void ConfigureServices(IServiceCollection services)
{
    // ...
    services.AddHostedService<QuartzHostedService>();
}

If you run the application, you should see the background task running every 5 seconds and writing to the Console (or wherever you have logging configured)

Using scoped services in jobs

There's one big problem with the implementation as described in this post: you can only create Singleton jobs. That means you can't use any dependencies that are registered as Scoped services. For example, you can't inject an EF Core DatabaseContext into your IJob implementation, as you'll have a captive dependency problem.

Working around this isn't a big issue: you can inject an IServiceProvider and create your own scope, similar to the solution for a similar problem in a previous post. For example, if you need to use a scoped service in your HelloWorldJob, you could use something like the following:

public class HelloWorldJob : IJob
{
    // Inject the DI provider
    private readonly IServiceProvider _provider;
    public HelloWorldJob( IServiceProvider provider)
    {
        _provider = provider;
    }

    public Task Execute(IJobExecutionContext context)
    {
        // Create a new scope
        using(var scope = _provider.CreateScope())
        {
            // Resolve the Scoped service
            var service = scope.ServiceProvider.GetService<IScopedService>();
            _logger.LogInformation("Hello world!");
        }

        return Task.CompletedTask;
    }
}

This ensures a new scope is created every time the job runs, so you can retrieve (and dispose) scoped services inside the IJob. Unfortunately things do get a little messy. In the next post I'll show a variation on this approach that is a little cleaner.

Summary

In this post I introduced Quartz.NET and showed how you could use it to schedule background jobs to run in ASP.NET Core using IHostedService. The example shown in this post is best for singleton jobs, which isn't ideal, as consuming scoped services is clumsy. In the next post, I'll show a variation on this approach that makes using scoped services easier.

In my previous post I showed how you can create a Quartz.NET hosted service with ASP.NET Core and use it to run background tasks on a schedule. Unfortunately, due to the way way the Quartz.NET API works, using Scoped dependency injection services in your Quartz jobs is somewhat cumbersome.

In this post I show one way to make it easier to use scoped services in your jobs. You can use the same approach for managing "unit-of-work" patterns with EF Core, and other cross-cutting concerns.

This post follows on directly from the previous post, so I suggest reading that post first, if you haven't already.

Recap - the custom JobFactory and singleton IJob

In the last post, we had a HelloWorldJob implementing IJob that simply wrote to the console.

public class HelloWorldJob : IJob
{
    private readonly ILogger<HelloWorldJob> _logger;
    public HelloWorldJob(ILogger<HelloWorldJob> logger)
    {
        _logger = logger;
    }

    public Task Execute(IJobExecutionContext context)
    {
        _logger.LogInformation("Hello world!");
        return Task.CompletedTask;
    }
}

We also had an IJobFactory implementation that retrieved an instance of the job from the DI container when required:

public class SingletonJobFactory : IJobFactory
{
    private readonly IServiceProvider _serviceProvider;
    public SingletonJobFactory(IServiceProvider serviceProvider)
    {
        _serviceProvider = serviceProvider;
    }

    public IJob NewJob(TriggerFiredBundle bundle, IScheduler scheduler)
    {
        return _serviceProvider.GetRequiredService(bundle.JobDetail.JobType) as IJob;
    }

    public void ReturnJob(IJob job) { }
}

These services were registered in Startup.ConfigureServices() as singletons:

services.AddSingleton<IJobFactory, SingletonJobFactory>();
services.AddSingleton<HelloWorldJob>();