Parameter count mismatch between X-Forwarded-For and X-Forwarded-Proto

These are my fieldnotes from experimenting with the ForwardedHeaders middleware on ASP.NET Core 1.1 and 2.0 - sparked by having a production log full of Parameter count mismatch between X-Forwarded-For and X-Forwarded-Proto warnings.

I've set up a github repo and a simple Postman example here.

Background

The X-Forwarded-* headers are set by proxies or load-balancers in between the client browser and your web server. Each proxy usually adds their incoming IP, protocol and host to the existing set of header values - resulting in a list of header values reaching the web host. And most commonly - allowing the web host to know the IP of the end-user.

X-Forwarded-For - IP address (x.x.x.x)
X-Forwarded-Proto - Protocol (http, https)
X-Forwarded-Host - Host (something.com:54321)

All headers use comma-separated value sets and each hop adds to the end of the list: client, proxy1, proxy2, proxy3 etc.

ForwardedHeaders middleware

The ForwardedHeaders middleware that comes with ASP.NET Core reads the incoming X-Forwarded-* headers and assigns the values to properties on the http context making them available throughout the application:

Mapping
X-Forwarded-For - HttpContext.Connection.RemoteIpAddress & HttpContext.Connection.RemotePort
X-Forwarded-Proto - HttpContext.Request.Scheme
X-Forwarded-Host - HttpContext.Request.Host

The minimal ASP.NET Core MVC application does not have ForwardedHeaders enabled by default, but if you're using the IISIntegration middleware it is enabled and set to forward the IP and the protocol. For reference, the default ForwardedHeadersOptions can be found here.

The most notable options for the ForwardedHeaders middleware are:

ForwardedHeaders - Specify which headers should be forwarded (assigned according to the mapping table above)
RequireHeaderSymmetry - Set to true if the mapping should be aborted and a warning logged when the headers are asymmetric (more about symmetry in the examples below).
ForwardLimit - Max number of entries per header that should be read - going from right to left.
KnownProxies - Addresses of known proxies to accept headers from. These will be skipped when performing the mapping.

In ASP.NET Core 1.1, the default value of RequireHeaderSymmetry is true. In 2.0 this has been changed to false.

These properties and their effects are not very well documented. The rest of this post will dive into a couple of examples to clearify their use.

Examples

I'm using a stripped-down ASP.NET core 2.0 project and sending requests through Postman. The code is posted here along with a Postman collection.

Basic scenario
The simplest scenario is just forwarding the protocol and IP, and having a single proxy. We can set up the middleware like this:

app.UseForwardedHeaders(new ForwardedHeadersOptions
{
  ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto,
  RequireHeaderSymmetry = true
});

To test the headers I'll use Postman to act as the proxy, first sending off a request without any headers. The result is:

{
    "remoteIpAddress": "::1",
    "remotePort": 5153,
    "requestHost": "localhost:2395",
    "requestScheme": "http"
}

And when setting X-Forwarded-For header to 80.80.80.80:1337 and X-Forwarded-Proto to https, I get:

{
    "remoteIpAddress": "80.80.80.80",
    "remotePort": 1337,
    "requestHost": "localhost:2395",
    "requestScheme": "https"
}

So far so good, but what happens if the proxy did not set the X-Forwarded-Proto header? Then the other header is also ignored:

{
    "remoteIpAddress": "::1",
    "remotePort": 5153,
    "requestHost": "localhost:2395",
    "requestScheme": "http"
}

And a warning has been logged:

warn: Microsoft.AspNetCore.HttpOverrides.ForwardedHeadersMiddleware[1]. Parameter count mismatch between X-Forwarded-For and X-Forwarded-Proto.

In other words: when RequireHeaderSymmetry is true, the proxy needs to apply all the expected headers, otherwise all headers will be ignored.

Advanced scenario
Now we assume three proxies, setting up our request in Postman like this (with Postman acting as the third proxy):

And changing the middleware configuration to:

app.UseForwardedHeaders(new ForwardedHeadersOptions
{
  ForwardedHeaders = 
    ForwardedHeaders.XForwardedFor 
    | ForwardedHeaders.XForwardedProto
    | ForwardedHeaders.XForwardedHost,
  ForwardLimit = 2,
  KnownProxies = { 
    IPAddress.Parse("70.70.70.70"), 
    IPAddress.Parse("80.80.80.80") 
  },
  RequireHeaderSymmetry = true
});

What is the expected result? This is what we get:

{
    "remoteIpAddress": "70.70.70.70",
    "remotePort": 1337,
    "requestHost": "firstproxy.com",
    "requestScheme": "https"
}

The first proxy is returned even though I have defined this as a known proxy indicating it should be skipped. The reason is that we have only allowed two forwards, and as the middleware starts from right to left when looking for values it stops at the first proxy.

Setting ForwardLimit to 3 resolves the issue:

{
    "remoteIpAddress": "60.60.60.60",
    "remotePort": 1336,
    "requestHost": "enduser.com",
    "requestScheme": "http"
}

Awesome! So the last thing worth mentioning is that if all the expected headers are not set by all proxies you'll get assymetry and the headers are ignored. Example request:

Outputs:

{
    "remoteIpAddress": "::1",
    "remotePort": 5437,
    "requestHost": "localhost:2395",
    "requestScheme": "http"
}

And logs Parameter count mismatch between X-Forwarded-For and X-Forwarded-Proto.

Conclusion

You need to know your infrastructure very well to be able to set up the ForwardedHeaders middleware correctly - but when it works you should be able to retrieve your end-user's IP.

Seeing header mismatch warnings in the log means that at least one proxy is not setting the expected header correctly. If you are unable to correct your proxy you can turn this off by setting RequireHeaderSymmetry to false. In ASP.NET Core 2 the default has been changed from true to false.