How to troubleshoot HTTP 400 response codes from ingress-nginx ingresses

Task

This article details how to troubleshoot failing requests, with a HTTP 400 response code, when using an ingress to access a service in a Kubernetes cluster.

This error message is typically returned due to a bad request, or as a result of an issue with the request headers or cookies.

This article is not intended to be exhaustive as there are a wide variety of causes, however some possible issues are covered.

Pre-requisites

• A Rancher Kubernetes Engine (RKE) CLI or Rancher v2.x provisioned Kubernetes cluster

Resolution

Large request headers exceeding header buffer

Nginx has a client header buffer configuration of 4x 8KB by default. Per the Nginx documentation:

• "A request line cannot exceed the size of one buffer, or the 414 (Request-URI Too Large) error is returned to the client."
• "A request header field cannot exceed the size of one buffer as well, or the 400 (Bad Request) error is returned to the client."

The buffer sizes can be extended by defining the large-client-header-buffers option in the nginx-configuration ConfigMap (see below).

Large URI is duplicated into a new header passed to the backend

This issue is commonly seen where the app itself responds when queried directly with a large path, but returns a 400 error when queried through an ingress.

By default, when ingress-nginx receives a request, it adds the original request's URI to the X-Original-Uri header that it passes on to the backend. This can result in the app being unable to handle the large sized headers, in addition to the long path.

This behaviour can be disabled by setting the proxy-add-original-uri-header option to false in your nginx-configuration ConfigMap (see below).

Adding options to the ingress-nginx nginx-configuration ConfigMap

This configuration map is populated by RKE from configuration defined in the cluster config:

If the cluster is provisioned by Rancher v2.x:

1. Edit the cluster (navigate to the cluster within Rancher, select the triple-dot button and then "Edit")
2. Select "Edit as YAML" to open the cluster configuration as YAML, instead of a form.

3. Add the desired configuration within the ingress block in the following format:

     ingress:
       provider: nginx
       options:
         name: value

   

4. Save the cluster configuration changes. RKE will go through and apply the config defined during its update process.

If the cluster is provisioned by the RKE CLI:

The process is largely the same as the Rancher process above, but the configuration is defined in the cluster.yml for this cluster:

1. Open the cluster configuration yaml with your editor and add the ingress.options block:

   yaml ingress: provider: nginx options: name: value

   

2. Apply this config with rke up --config <cluster configuration yaml>

Ensure you have an up-to-date cluster.rkestate within the same directory before running rke up

Further reading

• ingress-nginx ConfigMap documentation
• RKE documentation on ingress-nginx configuration