upstream sent too big header while reading response header from upstream

[wramadan]

Hello everyone,

I have installed API Umbrella on docker and everything is working fine, but for one of the api requests I get this 502- Bad Gateway and I checked the current log file and found this error.

The same API is accessible if the parameters sent in the header are shorter, but I need to have a longer header for this API so I wonder if some one knows where's the problem exactly

Here's a sample for the bad request from the log file

2018-01-18T11:54:57.19806 2018/01/18 11:54:57 [error] 158#0: *13345 upstream sent too big header while reading response header from upstream, client: 172.24.0.1, server: _, request: "GET /connect/authorize?client_id=js&redirect_uri=http%3A%2F%2Flocalhost%3A5010%2Fcallback
.html&response_type=id_token%20token&scope=openid%20profile%20api1%20PortalLikeServiceAPI%20ImageProviderApi%20FileServiceApi%20ProfileNetworkServiceApi%20WallPostServiceApi%20commentservice%20PortalLoggingServiceAPI%20ProfileService%20IdentityService%20DataAggregationSer
viceApi&state=7062da78fbb04b6098e9ffd180232269&nonce=a7ee216f4db44fa3be03196f25733834&ui_locales=de-DE HTTP/1.1", upstream: "http://172.24.0.11:80/connect/authorize?client_id=js&redirect_uri=http%3A%2F%2Flocalhost%3A5010%2Fcallback.html&response_type=id_token%20token&scop
e=openid%20profile%20api1%20PortalLikeServiceAPI%20ImageProviderApi%20FileServiceApi%20ProfileNetworkServiceApi%20WallPostServiceApi%20commentservice%20PortalLoggingServiceAPI%20ProfileService%20IdentityService%20DataAggregationServiceApi&state=7062da78fbb04b6098e9ffd1802
32269&nonce=a7ee216f4db44fa3be03196f25733834&ui_locales=de-DE", host: "43e6c84f-180e-4b81-9426-3c688c3e85ec", referrer: "http://localhost:5010/"

[kcharter]

I've seen a similar problem and it looks like to fix it API Umbrella needs to include in its internal nginx configuration settings for proxy_buffer_size and some related parameters.

Although the nginx configuration includes proxy_buffering: off, according to the nginx reference documentation at http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering when proxy buffering is off the proxy_buffer_size parameter still determines the size of the largest chunk that nginx will read from the upstream and pass to the client. By default this buffer size is a single memory page (in my particular case, 4 KiB). It looks like in practice it needs to be large enough to hold all the headers in the response.

Experimenting, for version 0.14.4 I found that I had to override proxy_buffer_size in two of the nginx configuration files, the ones generated from

• in templates/etc/nginx/frontend_defaults.conf.mustache; and
• in templates/etc/nginx/router.conf.mustache

I used a larger value, N for proxy_buffer_size, and found that with the explicit override I had to also include settings for proxy_buffers and proxy_busy_buffers_size to keep nginx from rejecting the configuration. I used

proxy_buffer_size N;
proxy_buffers 2 N;
proxy_busy_buffer_size N;

This kept nginx happy and works fine for our purposes, but I don't know if it's optimal; 2 is the smallest allowed first argument for proxy_buffers.

It looks like in the current master head it would be sufficient to make a small addition in templates/etc/nginx/frontend_defaults.conf.moustache, to allow a value of proxy_buffer_size in the main YAML file, and to use that configuration value when expanding the template. I'm planning to make the modification for just our fork of 0.14.4 to try it out.

If somebody knows for sure that I'm barking up the wrong tree, I'd be happy to hear it.

[GUI]

Sorry for the belated response on this, but this should now be more easily configurable in the v0.15.1 release. You can now configure this in your /etc/api-umbrella/api-umrella.yml file via the following settings:

nginx:
  proxy_buffer_size: 8k
  proxy_buffers: 8 8k

The default value in v0.15.1 is 8k, which may be higher than the previous default value of either 4k or 8k (it depended on the platform). But if you need to raise this default it should hopefully be more easily configurable. And thanks to @kcharter for providing more details and workarounds before v0.15.1's release!