Solve Nginx bad gateway with Identity Server and Asp.Net Core on Kubernetes

When deploying components on Kubernetes it is best practice to use Kubernetes Ingress as a way to control the traffic to your actual applications. One of the most popular components to use on Kubernetes for ingress is Nginx. My colleague Pascal Naber has written an excellent Post on how to configure Ingress using Nginx. However when you deploy Identity Server and a client web application that uses identity server it fails to do a proper login round trip. You are then presented with the following error
502 Bad gateway.

After some searching around it seems that the request is failing because the response from IdentityServer is to large for the default Nginx buffer size. These buffer sizes can be changes in the nginx.conf file. However because we were using the default nginx-ingress-controller docker image that wasn’t an easy fix.

There were two solutions for this problem:

  • Create your own Nginx Ingress controller docker image with a modified nginx.conf
  • Pass in parameters to our Ingress controller using a Kubernetes configmap

In this case we used an Kubernetes configmap to configure Nginx properly for the Identity Server responses. For Nginx as a reverse proxy to function properly with Identity Server add the following configmap.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app: ingress-nginx
data:
  proxy-buffer-size: "128k"
  proxy-buffers: "4 256k"
  proxy-busy-buffers-size: "256k"
  large-client-header-buffers: "4 16k"

You can find all the options for configuration on Nginx configuration.

Make sure to replace any values with your own if you want to use this configmap. Deploy this configmap.yaml with

kubectl apply -f configmap.yaml

The only quirky thing is that you will have to restart your Nginx ingress pods to have them load in the new configuration. However, currently there is no restart option for pods. So you will have to a kubectl delete pod POD_NAME. If you have used the default options when deploying Ingress then Kubernetes will automagically restart your pods when you delete them. Now Nginx ingress runs with the new configuration.

Addendum

If you are deploying .NET Core applications in Kubernetes on Linux behind a reverse proxy such as Nginx then also make sure to configure your middleware correct. Instructions for that can be found at MS Docs.