We released nghttp2 v1.5.0.

This release introduces new APIs, and fixes several bugs, and adds new features to bundled applications.

First, let’s talk about new APIs.

The new API nghttp2_session_change_stream_priority() changes the priority of given stream without sending PRIORITY frame. This is very useful for server to change priority of streams, especially for pushed streams. By default, pushed streams depend on the associated stream, but if we are pushing resources included in critical rendering path, it would be better to at least interleave them with the associated stream. In this case, using this new API is very handy.

The new API nghttp2_session_check_server_session() is very simple function, and returns nonzero if given HTTP/2 session is configured as server side.

The new API nghttp2_session_check_request_allowed() is for client to check that it can send request at this time. There are several reasons that request can no longer be issued (e.g., all available stream IDs have been spent; or GOAWAY has been received).

The new API nghttp2_session_upgrade2() deprecates existing nghttp2_session_upgrade(). nghttp2_session_upgrade() is called in both client and server when connection was upgraded to HTTP/2 from HTTP/1.1 via HTTP Upgrade mechanism. Originally reported in curl issue #521, we discovered that it has a design flaw. That is it does not tell the library the request method upgraded request used. By default, nghttp2 library checks HTTP messaging rules. One of them is that the length of response body matches the length included in content-length. In particular, when method is HEAD, in client side library checks that response body is really 0, no matter what the content-length header field contains. Suppose the first request with HTTP Upgrade has HEAD method, and it is upgraded. Since nghttp2_session_upgrade() cannot tell the library what method is, library attempts to check the response body length matches content-length header field. But this will fail since server returns 0 length body, and non-zero length header field most likely exists. This results in sending RST_STREAM from client side. The new nghttp2_session_upgrade2() has additional parameter to tell library the method is HEAD or not, and it solves this issue. All programs using nghttp2_session_upgrade() are advised to migrate to nghttp2_session_upgrade2() as soon as possible. To workaround the above problem in nghttp2_session_upgrade(), it now does not check response body length against content-length for the upgraded request, but this is just a gas stop measure.

The new flags NGHTTP2_NV_FLAG_NO_COPY_NAME and NGHTTP2_NV_FLAG_NO_COPY_VALUE have been introduced in this release. They are used to specify flags field of nghttp2_nv. Previously, when application passed header fields to functions like nghttp2_submit_request() or nghttp2_submit_response(), they are all copied. This is very convenient and quite safe, since caller does not have to worry about the memory leak or dangling pointers. But obviously this memory copy could add overhead especially if header fileds are long or many. The new flags can tell the library that it does not have to copy header name or value, thereby eliminates the overhead of copying. It is a responsibility of applications to retain the header fields until the library finished using them. For server application, usually response headers are retained until streams are closed, due to logging variables, so there is a good chance for them to optimize a bit and squeeze the more speed.

OK, that is all for new stuff for library API. Let’s see the bug fix in library. The documentation of nghttp2_session_find_stream() said that it returns root stream (stream ID is 0) if 0 is passed as stream ID, but previously it returned always NULL. Now it is corrected, and it returns root stream.

Kamil Dudka fixed borken linage with --disable-static in unit tests.

We added ALPN support for asio library.

For h2load, we made some bug fixes, and added new features. We fixed the bug which caused crash when dealing with “connection: close” from HTTP/1.1 server. Kit Chan fixed the bug that h2load goes into infinite loop when timing script file starts with 0.0 in first line. Previously, user cannot override user-agent header field with -H option, but now they can do that. We now show “space savings” to measure header compression efficiency. This is only useful for HTTP/2 or SPDY. We fixed bug that application protocol is not shown with OpenSSL < 1.0.2. We introduced --h1 option which is a short hand of --npn-list=http/1.1 and --no-tls-proto=http/1.1, and effectively forces http/1.1 usage for both http and https URI.

For nghttpx, we finally support server push from HTTP/2 backend. PUSH_PROMISE frame from backend is forwarded to frontend session, and delivered to client. We fixed the bug that causes connection failure with backend proxy URI. We now use the host name given in --backend-tls-sni-field to verify certificate host name. We now log :authority header field value as $http_host if available. This is useful since HTTP/2 request might not contain host header field.

For nghttpd, we fixed crash with CONNECT request. Previously, nghttpd does not set content-type header fields. In this release, nghttpd reads /etc/mime.types and set content-type header field depending on the extension of the requested resource.

For nghttp, request method is now record in har correctly. The HTTP method in HTTP Upgrade is now configurable using -H with :method.

Finally, we upgraded clang-format-3.6. We also upgraded mruby to 1.2.0.

I gave a talk at HTTP/2 + ATS meetup in Yahoo! Japan Tokyo office on November, 6, 2015. My talk was titled “Stream scheduling utilizing HTTP/2 priority”, which described detailed mechanism of stream scheduling algorithm implemented in nghttp2. As written in last slide, the original idea came from h2o project. I made some modifications to suit my taste.

We released nghttp2 v1.4.0.

This release includes number of fixes for libnghttp2. We briefly explain notable bug fixes here. Previously, libnghttp2 ignored CONTINUATION frames if preceding HEADERS frame contained padding. The appearance of CONTINUATION is rare these days, but padding is used in some services already, and we may see CONTINUATION somewhere too. The second and third bugs are SETTINGS and HPACK dynamic table size related bugs. The second bug is that previously libnghttp2 did not shrink to minimum size of requested dynamic table size contained in SETTINGS frame sent from local endpoint if it contains several SETTINGS_HEADER_TABLE_SIZE. Now it is corrected, and libnghttp2 shrinks to the minimum size. The third bug is that due to the ambiguous text in RFC 7540 and 7541, we interpreted that if receiver received SETTINGS containing SETTINGS_HEADER_TABLE_SIZE, it always has to send dynamic table size update in the next compressed header block. But it turns out that it is not the intention of the specification author. The intended behaviour is the receiver is required to send dynamic table size update only when it really changed maximum dynamic table size. Depending on the SETTINGS_HEADER_TABLE_SIZE and the current maximum dynamic table size, the table size may not change. The original bug report is GH-396.

To protect from suspicious peer, if libnghttp2 detected peer is flooding local endpoint with excessive SETTINGS and PING, nghttp2_session_mem_recv() and nghttp2_session_recv() now return fatal error code NGHTTP2_ERR_FLOODED.

Application now can return NGHTTP2_ERR_PAUSE from nghttp2_send_data_callback to indicate that application wants to make nghttp2_session_mem_send() or nghttp2_session_send() return immediately without processing next outgoing frames.

libnghttp2 now hides all internal private symbols using -fvisibility=hidden, if compiler supports it. This should not affect current applications since they should use public APIs only. It will increase performance, but the downside is that now static library is required to run unit tests. To offer the same functionality of python HPACK binding, deflatehd and inflatehd applications which previously used internal private functions, we added the following functions to public API: nghttp2_hd_deflate_get_num_table_entries(), nghttp2_hd_deflate_get_table_entry(), nghttp2_hd_deflate_get_dynamic_table_size(), nghttp2_hd_deflate_get_max_dynamic_table_size() and their nghttp2_hd_inflater counterparts. These are used to get header table entry using 1-based index, and current and maximum dynamic table size.

configuration summary now prints out more detailed information, including CFLAGS and LIBS for each detected third party library.

Applications under src directory can be now compiled with BoringSSL. Building programs under examples, including libevent-client and libevent-server, with BoringSSL is a bit tricky, since you have to build libevent with BoringSSL first. We haven’t verified that works.

nghttpx has new options, and some bug fixes. The bug that PUT method is replaced with POST in HTTP/1 frontend was fixed. Lucas Pardue added TLS dynamic record size behaviour command line options. --tls-dyn-rec-warmup-threshold option sets the number of bytes before switching to full TLS record size. --tls-dyn-rec-idle-timeout option specifies the idle time duration. When it is passed, TLS record size is back to 1300 bytes. Peeyush Aggarwal added --fastopen option to enable RFC 7413 (TCP Fast Open) for listening connections. We also added neverbleed support to nghttpx. It is disabled by default, and is enabled by --with-neverbleed configure option. mruby scripting API has changed along with the option name (now unified --mruby-file option), see the manual page for details. We changed default timeouts for read sockets to 1 minutes. Previous timeout is a bit too long.

h2load also gets new features. Lucas Pardue added option --rate-period for user-definable rate period. Previously, it was fixed to 1 second, but now user can specify any value, from subseconds period to more than 1 second. Previously, if HTTP/1 is used, h2load did not reconnect to the server if server closed connection gracefully with “Connection: close”. Apache web server does this actively after processing several number of requests. If that happened, h2load marked all remaining requests assigned to that connection failure. Now it is corrected, and h2load connects to the server again, and continues to load testing. We also enabled SSL/TLS session resumption in h2load client.

We simplified h2load rate mode in this release. In this change, we removed -C option. Instead, -c option is used to specify the number of connections to be made, and it is now required argument if more than 1 client are required (this is usually the case). The number of requests made per connection is calculated simply by -n / -c.

In h2load, we changed the handling of -n option when --timing-script-file is used. If -n is used with --timing-script, it specifies the number of requests each client will make rather than the total number of requests h2load will perform across clients. This handling applies to rate mode as well.

nghttp now allows multiple -c option occurrences, and takes min and last value, and sends both of them in SETTINGS to simulate multiple dynamic table size changes.

We released nghttp2 v1.3.4.

This release fixes improper signal handling in nghttpx.

Janusz Dziemidowicz improved nghttpx init script so that it now checks whether new process actually runs on upgrade.

We released nghttp2 v1.3.3.

In this release, we fixed the bug that handled padding in DATA frame wrongly. We added additional compiler warning flags in the hope that it will make library code more rigid. We have made an improvement in HPACK dynamic table search, and it is now faster than the previous release using the real data.

Lucas Pardue added HTTP/1.1 support to h2load. Now it can be used for HTTP/1.1 tests as well as HTTP/2. This makes comparison between HTTP/1.1 and HTTP/2 a lot easier.

nghttpx now consists of 2 processes. One process is called master process, and reads configuration, and setup listening ports. It spawns a process called worker process, which handles all networking operations from remote clients. The latter drops privileges if –user is specified. The master process does not drops privileges so that when new process using new binary is spawned from master process as hotswap, it can still read private keys which requires privileged user permission.

nghttpx now changes permission of UNIX domain socket file to the user given in --user.

We fixed the bug in nghttpx that PROXY protocol did not work if frontend is configured to use TLS.

We are excited about the announcement of neverbleed, privilege separation engine for OpenSSL / LibreSSL, from Kazuho Oku. We have local repository to integrate this work into nghttpx. Hopefully, we can include it in the next release.

We released nghttp2 v1.3.2.

In this release, we fixed assertion failure in nghttpx on premature connection termination during last phase of TLS handshake. In library code, when request HEADERS frame cannot be sent due to the too large header fields, we previously did not open stream, so on nghttp2_on_frame_not_send_callback, we could not retrieve per-stream user data using nghttp2_session_get_stream_user_data(). Now it first opens stream and then checks the size of headers, so nghttp2_session_get_stream_user_data() can work.

For nghttpx, we added x-http2-push header field for pushed resource. This practice is also done in h2o web server.

We released nghttp2 v1.3.1.

In this release, we fixed several bugs in library and added improvements to h2load and nghttpx.

We fixed rough edges of stream priority handling in nghttp2 library code. It was minor, but worth updating.

We shipped init script for nghttpx, but it missed --daemon option. So unless it is enabled in its configuration file, start/stop script hangs. This bug was fixed by Tomasz Buchert.

Lucas Pardue improved validation of timing script input in h2load.

Previously, in h2load, we measured TTFB (time to first byte) in first socket read, rather than first byte of response body. Now it is corrected.

We added PROXY protocol version 1 support to nghttpx. If --accept-proxy-protocol option is used, nghttpx expects PROXY protocol line on frontend connection. The obtained client address is used in X-Forwarded-For header field.

We extended push functionality in nghttpx, and it can now push resources specified in Link header fields which come from HTTP/2 backend. Also we now allow absolute URI in Link header fields.

We added mruby support in nghttpx. It is disabled by default. Use --with-mruby configure option to enable it. The nghttp2 archive includes mruby source code, so user doesn’t have to download it themselves. Please note that mruby package in Debian/Ubuntu does not work with nghttpx, since it does not enable C++ ABI mode. nghttpx runs given mruby scripts in 2 phases: request phase, and response phase. The request phase is when set of request header fields are received from client. The response phase is when set of response header fields are received from backend server. We have documented all mruby APIs. User can override request/response header fields, change request path, and even return custom response without forwarding request to backend or discarding response from backend. Checkout examples.

We released nghttp2 v1.3.0.

In this release, we added upper limit for the number of the streams pushed from server and not yet opened for client side session. Specifically, these streams are in “reserved (remote)” state as described in RFC 7540, Section 5.1.. The thing is RFC 7540 does not say anything about the upper limit of these streams. It says the number of maximum concurrent streams, but it only applies to the opened streams, not applied to streams in “reserved (remote)” state. From this release, nghttp2 library limits the number of these streams less than or equal to 200 by default. Applications can change this limit by using nghttp2_option_set_max_reserved_remote_streams() function. Those streams which exceed this limit will be automatically closed with RST_STREAM frame.

We added public APIs to expose dependency tree information. The nghttp2 library handles HTTP/2 dependency priority just fine, but there is a need to access to the dependency tree from applications. One of the use case is to schedule stream handling in threads. In the most of the cases, the number of available threads are smaller than incoming streams. Applications can get the dependency information from nghttp2 library, and schedule them in the order by the priority. nghttp2_session_find_stream() function will return stream object with the given stream ID. Applications can traverse dependency tree from this object. nghttp2_session_get_root_stream() will return the stream object which is the root of whole dependency tree. See nghttp2_stream_* functions to access the properties of stream object.

We rewrite priority tree handling. Now it is simpler and robust. We now use priority queue per stream, which contains the stream which has ready to send a frame, or one of its descendants have a frame to send. We maintain invariant that if a stream is queued, then its ancestors are also queued (except for root). When we re-schedule stream after transmission, we re-schedule all ancestors, so that streams on the other path can get a chance to send. The basic idea is the same with h2o’s scheduler, but there are differences in the details.

We found that openssl ocsp command exits withs status code 0 even if ocsp query or validation was failed. We added extra code to fetch-ocsp-response script to handle these situations.

Now nghttpx and nghttpd terminate HTTP/2 connection with GOAWAY of error code INADEQUATE_SECURITY if one of black listed cipher suites (see RFC 7540, Appendix A) was negotiated.

Tomasz Buchert improved handling of /dev/stdout and /dev/stderr for logging in nghttpx.

We added HEAD method support to nghttpd.

Lucas Pardue added Timing-script and base URI support to h2load. See #330 for more details.

Nora Shoemaker added timeout options to h2load. See #331 for more details.

We released nghttp2 v1.2.1.

We released v1.2.0 in Aug 9, but immediately after the release, we found some stability issues in nghttpx. We have done some hard work to fix them, and here is the v1.2.1 release.

Since most of the interesting stuff was done in v1.2.0, we also describe the changes in v1.2.0 in this blog post.

Previously, nghttp2 library only allow one outgoing in-flight SETTINGS frame. Now its limitation was gone, and application can issue multiple SETTINGS frame as it wants.

Previously, nghttp2 library allows incoming dynamic table size update in the middle of compressed header block. But RFC 7541 clearly states that it is restricted to the beginning of the header block. Now nghttp2 checks this restriction strictly. nghttp2 library also strictly checks whether peer sends dynamic table size update in response to header table change in SETTINGS.

Tom Harwood offered the patch to improve English text in tutorial documentation. Now they should be more readable.

nghttpx gets many advanced features in this release (and ironically (or inevitably?), it was the cause of the instability issue). Now it supports sharing session cache and TLS ticket keys among multiple nghttpx instance using memcached. We use OpenSSL, and basically it does not support asynchronous session cache lookup, but we use the same trick invented by Kazuho Oku in his awesome h2o project. If memcached based TLS ticket key sharing is not used, nghttpx generates TLS ticket keys internally as usual. From this release, it now generates new key every 1 hour, and its life time is 12 hours.

We added --tls-ticket-key-cipher option to change the cipher to encrypt session ticket. Currently, AES-128-CBC and AES-256-CBC are supported. Previously, we did not have this option, and cipher is always AES-128-CBC. Because of limitation in current OpenSSL implementation, we cannot use AES-GCM for ticket encryption at the moment.

Previously nghttpx HTTP/2 backend connection did not enable any TLS resumption. Now it is enabled.

Previously, by default, nghttpx rewrote Host (or :authority) header field for backend request using backend server’s address. This was done because apache and nginx do this by default. But we have heard that in most of the use case for nghttpx, users disable this feature using --no-host-rewrite. So we decided to not to rewrite Host header field by default. Instead, we added --host-rewrite option to enable rewrite.

We fixed bug in nghttpd that it sent response body even if status code was 304.

Nora Shoemaker sent us a patch to add connection rate based execution. See #299 for the intended use case.

We released nghttp2 v1.1.2.

This release fixes the linker error when libnghttp2_asio is linked. It contains a fix to allow custom installation location for Python bindings.