It is not a secret that we used nginx as backend Web server for nghttp2.org. We use nghttpx as frontend “Edge” reverse proxy, and all requests except for the ones to /httpbin were routed to nginx via HTTP/2 protocol.

Today, we have replaced backend nginx with h2o. h2o is a relatively new brazing fast Web server. Actually, we don’t require performance in our low traffic Web site. The reason we chose h2o is its better HTTP/2 implementation. Since we use HTTP/2 as backend protocol between proxy and backend server, we’d like to use better backend HTTP/2 server. h2o supports both HTTP/1.1 as well as HTTP/2 without TLS by default without extra configuration. Currently, HTTP/2 priority in the frontend proxy connection is not propagated to the backend connection, so we couldn’t utilize the great scheduling mechanism built in h2o. When we implement it, we definitely want clever HTTP/2 scheduler like h2o.

We still use nghttpx as frontend reverse proxy.

Previously, with nginx, we added link header field in nginx configuration so that nghttpx could push resource based on it. Now we initiate push using nghttpx’s mruby scripting feature. The following script initiates the push for the resource “/stylesheet/screen.css” when request path is either “/” or “/index.html”:

1
2
3
4
5
6
7
8
9
10
11

class App
  def on_req(env)
    req = env.req
    path = req.path
    if path == '/' || path == '/index.html'
      req.push '/stylesheets/screen.css'
    end
  end
end

App.new

And pass the file path containing the above script to nghttpx using --mruby-file option.

We released nghttp2 v1.9.2.

This release fixes several stability issues of nghttpx.

We released nghttp2 v1.9.1.

This release fixes several bugs in nghttpx found in the previous release.

We released nghttp2 v1.9.0.

This release adds new callback functions to libnghttp2 for better debugging, and potential performance enhancements. We refactored nghttpx basic interface, and it gets many powerful features in this release. We fixed several bugs in h2load when it is used against HTTP/1.1 server. We also now have cmake build support.

New callback functions for better debugging and performance

We have added 2 new callback functions. nghttp2_error_callback is a callback that tells application about the detailed error message for human consumption. This is intended for debugging purpose.

The 2nd new callback function is nghttp2_on_header_callback2. This function is similar to existing nghttp2_on_header_callback. The crucial difference between these two is that new callback uses reference counted buffers for header field name/value. Application can increase their reference count by nghttp2_rcbuf_incref, and store its reference without copying the content. When its usage is done, don’t forget to call nghttp2_rcbuf_decref. Previously, the buffer storing header field name/value is owned solely by libnghttp2 library, and application has to copy them out if it wants to retain them for future use.

We also added new API function nghttp2_http2_strerror. This function returns text version of HTTP/2 error code (e.g., PROTOCOL_ERROR). This is useful to output debugging information about error code contained in RST_STREAM or GOAWAY frame.

We added new option nghttp2_option_set_no_auto_ping_ack, which disables automatic ping reply. Application can submit ping reply using nghttp2_submit_ping with NGHTTP2_FLAG_ACK in flags parameter.

cmake build, and more

Peter Wu has done a stellar job to add cmake build support for nghttp2. According to the PR documents, cmake build is faster than autotool build. It also supports Windows build at least for libnghttp2.

Jan-E fixed several rough edges in Makefile.msvc.

h2load bug fixes

We fixed 2 bugs in h2load when HTTP/1.1 is used. The first bug is that it did not try to connect to server again. This happens if server shutdowns the connection if it serves certain number of requests. This kind of behaviour is enabled by default for some server software.

The 2nd bug is that initial max concurrent streams was too large, and it causes undefined behaviour if -m option is not used.

nghttpx: better configuration for frontend/backend protocol and encryption

In this release, we reworked nghttpx command-line (and thus its configuration) interface. Previously, it had --http2-bridge, --client, and --client-proxy options to change its major mode. But they were quite inflexible, and became obstacles when we are extending nghttpx features. To ensure the further feature enhancements, they have been removed. Now nghttpx gets much simpler, and only has 2 modes: default mode, and HTTP/2 proxy mode (-s option). The removed modes can be achieved using other options. Read Migration from nghttpx v1.8.0 or earlier to know how to migrate from earlier release.

Now backend connections are not encrypted by default regardless of the used protocol. The exciting new feature is that backend protocol can be specified per routing pattern basis. Also the TLS can be enabled per routing pattern as well:

1
2

backend=127.0.0.1,8080;;proto=h2;tls
backend=unix:/var/unix/httpbinsv.sock;/httpbin/;proto=http/1.1

With above configuration, requests to /httpbin/ are routed to unix:/var/unix/httpbinsv via HTTP/1.1 protocol over cleartext TCP. The other requests are routed to 127.0.0.1:8080 via HTTP/2 protocol over TLS. tls keyword in --backend option enables encryption.

We now allow wildcard in routing pattern in --backend option. When we write:

1

backend=127.0.0.1,8080;*.nghttp2.org

All requests which have host (or :authority) header field whose suffix is .nghttp2.org are routed to 127.0.0.1:8080.

Since the previous release, nghttpx has got multiple frontend addresses support. Now its feature has been extended, and TLS can be enabled or disabled per frontend address. This means that single nghttpx instance finally can serve both TLS and non-TLS contents:

1
2

frontend=*,443
frontend=*,80;no-tls

With the above configuration, nghttpx listens to port 443 for incoming TLS connection. It also listens to port 80, but this time for incoming cleartext connection. no-tls keyword in --frontend option disables encryption. --frontend-no-tls options has been removed in favor of no-tls keyword.

The encryption for memcached connections has been available since the previous release. In this release, we changed how to enable TLS. Now we use similar syntax for --frontend option. To enable TLS over memcached connection to get TLS ticket keys, use the following configuration:

1

tls-ticket-key-memcached=127.0.0.1,11211;tls

In the above configuration, the tls keyword enables encryption.

nghttpx supports server push with Link header field with rel=preload. Now it recognizes nopush target attribute (see preload).

There are several deprecated options. If they are used, nghttpx will output warning level logging message. Please be careful for them, and they may contain the idea how to migrate to the new or existing other options.

nghttp2.org has been powered by nghttpx reverse proxy. It forwards most of the requests to backend nginx. And resources under /httpbin is forwarded to httpbin server. Previously, all backend forward link is cleartext HTTP/1.1. Now, with the new feature added to nghttpx, we forwards requests to nginx in HTTP/2 protocol.

nghttpx in the current nghttp2 master branch has a feature to specify backend protocol per host/request path routing pattern. So we just specify the following 2 options to make the above routing work:

1
2

backend=127.0.0.1,8080;;proto=h2
backend=unix:/var/unix/httpbinsv.sock;/httpbin/;proto=http/1.1

The above configuration means that all requests whose path starts with /httpbin/ are forwarded to UNIX domain socket unix:/var/unix/httpbinsv.sock via HTTP/1.1 protocol. The rest of the requets are forwarded to server listening 127.0.0.1:8080 via HTTP/2 protocol.

The microbench mark shows that using HTTP/2 backend connection reduces backend TCP connection to about 1/100 (because of the concurrency in HTTP/2 protocol, and we coalesce frontend connections into fewer backend HTTP/2 connections), and roughly 2 times faster than HTTP/1.1 backend. HTTP/2 backend is a bit slower with default 64KiB stream window when transferring large objects, but it could be remedied by using larger window.

We released nghttp2 v1.8.0.

This release adds new library APIs to send and receive non-critical HTTP/2 extension frames. It also adds new features to nghttpx and nghttpd, and polishes many rough edges.

We added the APIs to send and receive non-critical HTTP/2 extension frames. “Non-critical” means that it won’t change HTTP/2 standard protocol rules, and those frames may be ignored by a receiver. We have added a set of functions and callbacks for this.

To send HTTP/2 extension frames, implement nghttp2_pack_extension_callback to encode data into wire format, and set it using nghttp2_session_callbacks_set_pack_extension_callback(). If the application wants to send multiple different types of extension frames, it should handle them in this single callback. Then use nghttp2_submit_extension() to submit a frame.

To receive HTTP/2 extension frames, implement 2 callbacks: nghttp2_unpack_extension_callback and nghttp2_on_extension_chunk_recv_callback. nghttp2_unpack_extension_callback implements the way how to decode wire format. nghttp2_on_extension_chunk_recv_callback implements how to buffer the incoming extension payload. These callbacks must be set using nghttp2_session_callbacks_set_unpack_extension_callback and nghttp2_session_callbacks_set_on_extension_chunk_recv_callback. The application also must tell the library which extension frame type it is willing to receive using nghttp2_option_set_user_recv_extension_type(). Note that the application has to create nghttp2_option object for that purpose, and initialize session with it.

Peter Wu sends many patches to fix various bugs, and rough edges, including out-of-tree documentation build and integration tests.

David Beitey documented how to build h2load, and detect the configuration error.

Bernard Spil fixed configure error related to dlopen/libdl detection on *BSD platform.

Reza Tavakoli fixed nghttp2 package version detection in Windows Makefile.

Jay Satiro refactored the portion of the code to check against HTTP/2 cipher black list. It is pretty neat.

We have updated default cipher list used in bundled applications using Security/Server Side TLS compiled by Mozilla.

We fixed compile error with gcc-6 which enables C++14 by default.

In libnghttp2_asio, we fixed the bug that connect timeout did not work. In the part of the fix, we removed nghttp2::asio_http2::client::session::connect_timeout() functon, and instead added connect timeout parameter to constructor. This will break backward compatibility.

We fixed bug in nghttpd that Trailer header field was not added when non-200 status response was returned.

We added -w and -W options to nghttpd to change stream-level and connection-level window size respectively.

nghttpx now supports multiple frontend addresses. --frontend (or -f) can be used multiple times to specify more than one frontend addresses.

nghttpx now interleaves text/html pushed resources with associated resource as well as text/css and application/javascript for Polymer use case.

We fixed the bug that nghttpx did not add headers given in add-response-headers to the response generated by mruby.

In nghttpx, we deprecated --backend-ipv4 and --backend-ipv6 in favor of new --backend-address-family option.

We have added TLS encryption support for memcached connections. We use memcached for sharing and storing TLS session cache and TLS ticket keys. We also added options to set address family, and client certificates for memcached connections.

We have added TLS encryption support for backend HTTP/1. Unlike HTTP/2 backend, backend HTTP/1 encryption is disabled by default for backward compatibility. To enable TLS, use --backend-http1-tls option. The existing option to specify client certificate is working with this as well.

We have added --no-http2-cipher-black-list to allow black listed cipher suite.

In nghttpx, we added --request-header-field-buffer and --max-request-header-fields options, and they deprecated --header-field-buffer and --max-header-fields options. To limit response headers as well, we added its response side counterparts, --response-header-field-buffer and --max-response-header-fields options.

We released nghttp2 v1.7.1.

This release addresses following security issue.

Security Advisory

CVE-2016-1544: Out of memory in nghttpd, nghttp, and libnghttp2_asio applications due to unlimited incoming HTTP header fields.

Vulnerability

nghttpd, nghttp, and libnghttp2_asio applications do not limit the memory usage for the incoming HTTP header field. If peer sends specially crafted HTTP/2 HEADERS frames and CONTINUATION frames, they will crash with out of memory error.

HTTP/2 uses HPACK to compress header fields. The basic idea is that HTTP header field is stored in the receiver with the numeric index number. The memory used by this storage is tightly constrained, and it is 4KiB by default. When sender sends the same header field, it just sends the corresponding numeric index number, which is usually 1 or 2 bytes. This means that after sender makes the receiver store the relatively large header field (e.g., 4KiB), and it can send specially crafted HEADERS/CONTINUATION frames which contain a lot of references to the stored header field, sender easily effectively send lots of big header fields to the receiver quite easily. nghttpd, nghttp, and libnghttp2_asio applications do not limit the memory usage for received header fields, so if the peer performs the procedure described above, they will crash due to out of memory.

Note that libnghttp2 itself is not affected by this vulnerability.

Affected Versions

• Affected versions: nghttp2 <= 1.7.0
• Not affected versions: nghttp2 >= 1.7.1

The Solution

Install nghttp2 v1.7.1

Time Line

It was first reported to the nghttp2 team February 3 2016.

nghttp2 v1.7.1 was released on February 11 2016.

Credits

Reported by Noam Mazor. Fixed by the nghttp2 team.

Thank you for all who involved.

This security advisory format is inspired from curl/libcurl project.

We released nghttp2 v1.7.0.

This release contains the more strict error handling in libnghttp2 code to provide more diagnostics to applications. We made many improvements to h2load, and nghttpx in this release.

For libnghttp2, we now validate important header fields, :authority, host, and :scheme, strictly by checking the characters used in them. These header fields are usually used by server as routing purpose, and they may appear in different header fields when forwarded. It is better to basic error handling here so that we can reduce possible attack vectors.

Previously, nghttp2_submit_headers and nghttp2_submit_request functions did not return error when self dependency was made. Now it returns error code when such argument is passed.

Previously, when linking OpenSSL library statically, build was failed because -ldl was not passed to linker, so we had to add it using LIBS. Now it is automatically added to linker options.

In libnghttp2_asio library, nghttp2::asio_http2::server::http2::get_io_services() has been renamed as nghttp2::asio_http2::server::http2::io_services() to follow nameing convention. Previously, calling nghttp2::asio_http2::server::stop() did not stop acceptor. Now it stops acceptor too.

h2load now support UNIX domain socket to connect to server. Previously, -m option of h2load was auto, which defaulted to the number of URIs passed in command-line. Now it is removed, and the default value is 1. This is because -m option specifies the number of pipelined requests in HTTP/1, and disabling HTTP pipelining more reflects the practical usage of HTTP/1. The statistics calculation of h2load is now sampling based, instead of taking into account of all requets/clients to reduce memory consumption.

nghttpd now gets --no-content-length option to omit content-length in response.

nghttpx now interleaves pushed streams with the associated stream if pushed streams are JavaScript or CSS resources, which is determined by content-type response header field. The initial value of request/response buffer size (--backend-request-buffer and --backend-response-buffer options) is now increased to 128K. We have fixed the bug that --listener-disable-timeout option is not used. Now nghttpx does not emit :authority if request does not contain authority information. This happens when :authority header field is missing in HTTP/2 request, and origin or asterisk form is used in HTTP/1.1 request. nghttpx now supports RFC 7239 Forwarded header field. By default, Forwarded header field is not added. When it is instructed to be added, nghttpx emits obfuscated strings for by and for parameters. Vernon Tang fixed the bug which causes crash at start up when running on IPv6 only. Now nghttpx does not apply --max-header-fields and --header-field-buffer to response header fields, since they are meant to be applied to request header fields only. ayanamist fixed the bug that nghttpx wrongly lower-cased header field value supplied via --add-request-header-field and add-response-header-field options.

We released nghttp2 v1.6.0.

This release fixes heap-use-after-free bug in idle stream handling code. We strongly recommend to upgrade the older installation to this latest version as soon as possible. Other than that we have minor polish up in libnghttp2 code base, and some new features to asio library, and h2load.

We tightened up the error condition when we received frame which is not allowed in that state. We now make it connection error if we are sure that peer really violates the protocol.

Previously, the default remote SETTINGS_MAX_CONCURRENT_STREAMS value was unlimited as per RFC 7540. But this was problematic, and this could lead to many REFUSED_STREAM, or ENHANCE_YOUR_CALM error. To avoid this situation, and make it more friendly to peer, we assume that the default remote SETTINGS_MAX_CONCURRENT_STREAMS is 100. When we get this SETTINGS value from peer, we will update it. The application can change this initial value using nghttp2_option_set_peer_max_concurrent_streams() API.

Previously, stream object for pushed stream was not created when nghttp2_submit_push_promise() returned. It was created after nghttp2_before_frame_send_callback was called. So application had to wait for its invocation when it wanted to use stream object. Now stream object is created right after successful nghttp2_submit_push_promise() call.

We added new API, nghttp2_session_create_idle_stream(). This function creates idle stream without telling it the remote endpoint.

Previously, if we wanted to build libnghttp2 only, we had to give --disable-app, --disable-examples, --disable-hpack-tools, and --disable-python-bindings options to configure script. We added --lib-only configure option as a short hand for these options.

The previous package build failed without OpenSSL 1.0.2. This was fixed in this release by the patch contributed by Sunpoet Po-Chuan Hsieh.

Andreas Pohl added several new server APIs. nghttp2::asio_http2::server::http2::get_io_services() returns all underlying io_service objects. nghttp2::asio_http2::server::request::remote_endpoint() returns the remote endpoint address.

We added configurable connect, and read timeout for asio client API. See nghttp2::asio_http2::client::session::connect_timeout(), and nghttp2::asio_http2::client::session::read_timeout() API. This involved backend incompatible change, and the application must be recompiled to use new asio library (this applies to asio library only, and libnghttp2 is fully backward compatible to 1.0.0).

Similarly, we added configurable TLS handshake, and read timeout for asio server API. See nghttp2::asio_http2::server::http2::tls_handshake_timeout(), and nghttp2::asio_http2::server::http2::read_timeout() API.

h2load now shows min, max, mean and sd (standard deviation) of request per second value calculated per connection. This is useful to see how performance differs between connections.

We released nghttp2 v1.5.0.

This release introduces new APIs, and fixes several bugs, and adds new features to bundled applications.

First, let’s talk about new APIs.

The new API nghttp2_session_change_stream_priority() changes the priority of given stream without sending PRIORITY frame. This is very useful for server to change priority of streams, especially for pushed streams. By default, pushed streams depend on the associated stream, but if we are pushing resources included in critical rendering path, it would be better to at least interleave them with the associated stream. In this case, using this new API is very handy.

The new API nghttp2_session_check_server_session() is very simple function, and returns nonzero if given HTTP/2 session is configured as server side.

The new API nghttp2_session_check_request_allowed() is for client to check that it can send request at this time. There are several reasons that request can no longer be issued (e.g., all available stream IDs have been spent; or GOAWAY has been received).

The new API nghttp2_session_upgrade2() deprecates existing nghttp2_session_upgrade(). nghttp2_session_upgrade() is called in both client and server when connection was upgraded to HTTP/2 from HTTP/1.1 via HTTP Upgrade mechanism. Originally reported in curl issue #521, we discovered that it has a design flaw. That is it does not tell the library the request method upgraded request used. By default, nghttp2 library checks HTTP messaging rules. One of them is that the length of response body matches the length included in content-length. In particular, when method is HEAD, in client side library checks that response body is really 0, no matter what the content-length header field contains. Suppose the first request with HTTP Upgrade has HEAD method, and it is upgraded. Since nghttp2_session_upgrade() cannot tell the library what method is, library attempts to check the response body length matches content-length header field. But this will fail since server returns 0 length body, and non-zero length header field most likely exists. This results in sending RST_STREAM from client side. The new nghttp2_session_upgrade2() has additional parameter to tell library the method is HEAD or not, and it solves this issue. All programs using nghttp2_session_upgrade() are advised to migrate to nghttp2_session_upgrade2() as soon as possible. To workaround the above problem in nghttp2_session_upgrade(), it now does not check response body length against content-length for the upgraded request, but this is just a gas stop measure.

The new flags NGHTTP2_NV_FLAG_NO_COPY_NAME and NGHTTP2_NV_FLAG_NO_COPY_VALUE have been introduced in this release. They are used to specify flags field of nghttp2_nv. Previously, when application passed header fields to functions like nghttp2_submit_request() or nghttp2_submit_response(), they are all copied. This is very convenient and quite safe, since caller does not have to worry about the memory leak or dangling pointers. But obviously this memory copy could add overhead especially if header fileds are long or many. The new flags can tell the library that it does not have to copy header name or value, thereby eliminates the overhead of copying. It is a responsibility of applications to retain the header fields until the library finished using them. For server application, usually response headers are retained until streams are closed, due to logging variables, so there is a good chance for them to optimize a bit and squeeze the more speed.

OK, that is all for new stuff for library API. Let’s see the bug fix in library. The documentation of nghttp2_session_find_stream() said that it returns root stream (stream ID is 0) if 0 is passed as stream ID, but previously it returned always NULL. Now it is corrected, and it returns root stream.

Kamil Dudka fixed borken linage with --disable-static in unit tests.

We added ALPN support for asio library.

For h2load, we made some bug fixes, and added new features. We fixed the bug which caused crash when dealing with “connection: close” from HTTP/1.1 server. Kit Chan fixed the bug that h2load goes into infinite loop when timing script file starts with 0.0 in first line. Previously, user cannot override user-agent header field with -H option, but now they can do that. We now show “space savings” to measure header compression efficiency. This is only useful for HTTP/2 or SPDY. We fixed bug that application protocol is not shown with OpenSSL < 1.0.2. We introduced --h1 option which is a short hand of --npn-list=http/1.1 and --no-tls-proto=http/1.1, and effectively forces http/1.1 usage for both http and https URI.

For nghttpx, we finally support server push from HTTP/2 backend. PUSH_PROMISE frame from backend is forwarded to frontend session, and delivered to client. We fixed the bug that causes connection failure with backend proxy URI. We now use the host name given in --backend-tls-sni-field to verify certificate host name. We now log :authority header field value as $http_host if available. This is useful since HTTP/2 request might not contain host header field.

For nghttpd, we fixed crash with CONNECT request. Previously, nghttpd does not set content-type header fields. In this release, nghttpd reads /etc/mime.types and set content-type header field depending on the extension of the requested resource.

For nghttp, request method is now record in har correctly. The HTTP method in HTTP Upgrade is now configurable using -H with :method.

Finally, we upgraded clang-format-3.6. We also upgraded mruby to 1.2.0.