I gave a talk at HTTP/2 + ATS meetup in Yahoo! Japan Tokyo office on November, 6, 2015. My talk was titled “Stream scheduling utilizing HTTP/2 priority”, which described detailed mechanism of stream scheduling algorithm implemented in nghttp2. As written in last slide, the original idea came from h2o project. I made some modifications to suit my taste.

We released nghttp2 v1.4.0.

This release includes number of fixes for libnghttp2. We briefly explain notable bug fixes here. Previously, libnghttp2 ignored CONTINUATION frames if preceding HEADERS frame contained padding. The appearance of CONTINUATION is rare these days, but padding is used in some services already, and we may see CONTINUATION somewhere too. The second and third bugs are SETTINGS and HPACK dynamic table size related bugs. The second bug is that previously libnghttp2 did not shrink to minimum size of requested dynamic table size contained in SETTINGS frame sent from local endpoint if it contains several SETTINGS_HEADER_TABLE_SIZE. Now it is corrected, and libnghttp2 shrinks to the minimum size. The third bug is that due to the ambiguous text in RFC 7540 and 7541, we interpreted that if receiver received SETTINGS containing SETTINGS_HEADER_TABLE_SIZE, it always has to send dynamic table size update in the next compressed header block. But it turns out that it is not the intention of the specification author. The intended behaviour is the receiver is required to send dynamic table size update only when it really changed maximum dynamic table size. Depending on the SETTINGS_HEADER_TABLE_SIZE and the current maximum dynamic table size, the table size may not change. The original bug report is GH-396.

To protect from suspicious peer, if libnghttp2 detected peer is flooding local endpoint with excessive SETTINGS and PING, nghttp2_session_mem_recv() and nghttp2_session_recv() now return fatal error code NGHTTP2_ERR_FLOODED.

Application now can return NGHTTP2_ERR_PAUSE from nghttp2_send_data_callback to indicate that application wants to make nghttp2_session_mem_send() or nghttp2_session_send() return immediately without processing next outgoing frames.

libnghttp2 now hides all internal private symbols using -fvisibility=hidden, if compiler supports it. This should not affect current applications since they should use public APIs only. It will increase performance, but the downside is that now static library is required to run unit tests. To offer the same functionality of python HPACK binding, deflatehd and inflatehd applications which previously used internal private functions, we added the following functions to public API: nghttp2_hd_deflate_get_num_table_entries(), nghttp2_hd_deflate_get_table_entry(), nghttp2_hd_deflate_get_dynamic_table_size(), nghttp2_hd_deflate_get_max_dynamic_table_size() and their nghttp2_hd_inflater counterparts. These are used to get header table entry using 1-based index, and current and maximum dynamic table size.

configuration summary now prints out more detailed information, including CFLAGS and LIBS for each detected third party library.

Applications under src directory can be now compiled with BoringSSL. Building programs under examples, including libevent-client and libevent-server, with BoringSSL is a bit tricky, since you have to build libevent with BoringSSL first. We haven’t verified that works.

nghttpx has new options, and some bug fixes. The bug that PUT method is replaced with POST in HTTP/1 frontend was fixed. Lucas Pardue added TLS dynamic record size behaviour command line options. --tls-dyn-rec-warmup-threshold option sets the number of bytes before switching to full TLS record size. --tls-dyn-rec-idle-timeout option specifies the idle time duration. When it is passed, TLS record size is back to 1300 bytes. Peeyush Aggarwal added --fastopen option to enable RFC 7413 (TCP Fast Open) for listening connections. We also added neverbleed support to nghttpx. It is disabled by default, and is enabled by --with-neverbleed configure option. mruby scripting API has changed along with the option name (now unified --mruby-file option), see the manual page for details. We changed default timeouts for read sockets to 1 minutes. Previous timeout is a bit too long.

h2load also gets new features. Lucas Pardue added option --rate-period for user-definable rate period. Previously, it was fixed to 1 second, but now user can specify any value, from subseconds period to more than 1 second. Previously, if HTTP/1 is used, h2load did not reconnect to the server if server closed connection gracefully with “Connection: close”. Apache web server does this actively after processing several number of requests. If that happened, h2load marked all remaining requests assigned to that connection failure. Now it is corrected, and h2load connects to the server again, and continues to load testing. We also enabled SSL/TLS session resumption in h2load client.

We simplified h2load rate mode in this release. In this change, we removed -C option. Instead, -c option is used to specify the number of connections to be made, and it is now required argument if more than 1 client are required (this is usually the case). The number of requests made per connection is calculated simply by -n / -c.

In h2load, we changed the handling of -n option when --timing-script-file is used. If -n is used with --timing-script, it specifies the number of requests each client will make rather than the total number of requests h2load will perform across clients. This handling applies to rate mode as well.

nghttp now allows multiple -c option occurrences, and takes min and last value, and sends both of them in SETTINGS to simulate multiple dynamic table size changes.

We released nghttp2 v1.3.4.

This release fixes improper signal handling in nghttpx.

Janusz Dziemidowicz improved nghttpx init script so that it now checks whether new process actually runs on upgrade.

We released nghttp2 v1.3.3.

In this release, we fixed the bug that handled padding in DATA frame wrongly. We added additional compiler warning flags in the hope that it will make library code more rigid. We have made an improvement in HPACK dynamic table search, and it is now faster than the previous release using the real data.

Lucas Pardue added HTTP/1.1 support to h2load. Now it can be used for HTTP/1.1 tests as well as HTTP/2. This makes comparison between HTTP/1.1 and HTTP/2 a lot easier.

nghttpx now consists of 2 processes. One process is called master process, and reads configuration, and setup listening ports. It spawns a process called worker process, which handles all networking operations from remote clients. The latter drops privileges if –user is specified. The master process does not drops privileges so that when new process using new binary is spawned from master process as hotswap, it can still read private keys which requires privileged user permission.

nghttpx now changes permission of UNIX domain socket file to the user given in --user.

We fixed the bug in nghttpx that PROXY protocol did not work if frontend is configured to use TLS.

We are excited about the announcement of neverbleed, privilege separation engine for OpenSSL / LibreSSL, from Kazuho Oku. We have local repository to integrate this work into nghttpx. Hopefully, we can include it in the next release.

We released nghttp2 v1.3.2.

In this release, we fixed assertion failure in nghttpx on premature connection termination during last phase of TLS handshake. In library code, when request HEADERS frame cannot be sent due to the too large header fields, we previously did not open stream, so on nghttp2_on_frame_not_send_callback, we could not retrieve per-stream user data using nghttp2_session_get_stream_user_data(). Now it first opens stream and then checks the size of headers, so nghttp2_session_get_stream_user_data() can work.

For nghttpx, we added x-http2-push header field for pushed resource. This practice is also done in h2o web server.

We released nghttp2 v1.3.1.

In this release, we fixed several bugs in library and added improvements to h2load and nghttpx.

We fixed rough edges of stream priority handling in nghttp2 library code. It was minor, but worth updating.

We shipped init script for nghttpx, but it missed --daemon option. So unless it is enabled in its configuration file, start/stop script hangs. This bug was fixed by Tomasz Buchert.

Lucas Pardue improved validation of timing script input in h2load.

Previously, in h2load, we measured TTFB (time to first byte) in first socket read, rather than first byte of response body. Now it is corrected.

We added PROXY protocol version 1 support to nghttpx. If --accept-proxy-protocol option is used, nghttpx expects PROXY protocol line on frontend connection. The obtained client address is used in X-Forwarded-For header field.

We extended push functionality in nghttpx, and it can now push resources specified in Link header fields which come from HTTP/2 backend. Also we now allow absolute URI in Link header fields.

We added mruby support in nghttpx. It is disabled by default. Use --with-mruby configure option to enable it. The nghttp2 archive includes mruby source code, so user doesn’t have to download it themselves. Please note that mruby package in Debian/Ubuntu does not work with nghttpx, since it does not enable C++ ABI mode. nghttpx runs given mruby scripts in 2 phases: request phase, and response phase. The request phase is when set of request header fields are received from client. The response phase is when set of response header fields are received from backend server. We have documented all mruby APIs. User can override request/response header fields, change request path, and even return custom response without forwarding request to backend or discarding response from backend. Checkout examples.

We released nghttp2 v1.3.0.

In this release, we added upper limit for the number of the streams pushed from server and not yet opened for client side session. Specifically, these streams are in “reserved (remote)” state as described in RFC 7540, Section 5.1.. The thing is RFC 7540 does not say anything about the upper limit of these streams. It says the number of maximum concurrent streams, but it only applies to the opened streams, not applied to streams in “reserved (remote)” state. From this release, nghttp2 library limits the number of these streams less than or equal to 200 by default. Applications can change this limit by using nghttp2_option_set_max_reserved_remote_streams() function. Those streams which exceed this limit will be automatically closed with RST_STREAM frame.

We added public APIs to expose dependency tree information. The nghttp2 library handles HTTP/2 dependency priority just fine, but there is a need to access to the dependency tree from applications. One of the use case is to schedule stream handling in threads. In the most of the cases, the number of available threads are smaller than incoming streams. Applications can get the dependency information from nghttp2 library, and schedule them in the order by the priority. nghttp2_session_find_stream() function will return stream object with the given stream ID. Applications can traverse dependency tree from this object. nghttp2_session_get_root_stream() will return the stream object which is the root of whole dependency tree. See nghttp2_stream_* functions to access the properties of stream object.

We rewrite priority tree handling. Now it is simpler and robust. We now use priority queue per stream, which contains the stream which has ready to send a frame, or one of its descendants have a frame to send. We maintain invariant that if a stream is queued, then its ancestors are also queued (except for root). When we re-schedule stream after transmission, we re-schedule all ancestors, so that streams on the other path can get a chance to send. The basic idea is the same with h2o’s scheduler, but there are differences in the details.

We found that openssl ocsp command exits withs status code 0 even if ocsp query or validation was failed. We added extra code to fetch-ocsp-response script to handle these situations.

Now nghttpx and nghttpd terminate HTTP/2 connection with GOAWAY of error code INADEQUATE_SECURITY if one of black listed cipher suites (see RFC 7540, Appendix A) was negotiated.

Tomasz Buchert improved handling of /dev/stdout and /dev/stderr for logging in nghttpx.

We added HEAD method support to nghttpd.

Lucas Pardue added Timing-script and base URI support to h2load. See #330 for more details.

Nora Shoemaker added timeout options to h2load. See #331 for more details.

We released nghttp2 v1.2.1.

We released v1.2.0 in Aug 9, but immediately after the release, we found some stability issues in nghttpx. We have done some hard work to fix them, and here is the v1.2.1 release.

Since most of the interesting stuff was done in v1.2.0, we also describe the changes in v1.2.0 in this blog post.

Previously, nghttp2 library only allow one outgoing in-flight SETTINGS frame. Now its limitation was gone, and application can issue multiple SETTINGS frame as it wants.

Previously, nghttp2 library allows incoming dynamic table size update in the middle of compressed header block. But RFC 7541 clearly states that it is restricted to the beginning of the header block. Now nghttp2 checks this restriction strictly. nghttp2 library also strictly checks whether peer sends dynamic table size update in response to header table change in SETTINGS.

Tom Harwood offered the patch to improve English text in tutorial documentation. Now they should be more readable.

nghttpx gets many advanced features in this release (and ironically (or inevitably?), it was the cause of the instability issue). Now it supports sharing session cache and TLS ticket keys among multiple nghttpx instance using memcached. We use OpenSSL, and basically it does not support asynchronous session cache lookup, but we use the same trick invented by Kazuho Oku in his awesome h2o project. If memcached based TLS ticket key sharing is not used, nghttpx generates TLS ticket keys internally as usual. From this release, it now generates new key every 1 hour, and its life time is 12 hours.

We added --tls-ticket-key-cipher option to change the cipher to encrypt session ticket. Currently, AES-128-CBC and AES-256-CBC are supported. Previously, we did not have this option, and cipher is always AES-128-CBC. Because of limitation in current OpenSSL implementation, we cannot use AES-GCM for ticket encryption at the moment.

Previously nghttpx HTTP/2 backend connection did not enable any TLS resumption. Now it is enabled.

Previously, by default, nghttpx rewrote Host (or :authority) header field for backend request using backend server’s address. This was done because apache and nginx do this by default. But we have heard that in most of the use case for nghttpx, users disable this feature using --no-host-rewrite. So we decided to not to rewrite Host header field by default. Instead, we added --host-rewrite option to enable rewrite.

We fixed bug in nghttpd that it sent response body even if status code was 304.

Nora Shoemaker sent us a patch to add connection rate based execution. See #299 for the intended use case.

We released nghttp2 v1.1.2.

This release fixes the linker error when libnghttp2_asio is linked. It contains a fix to allow custom installation location for Python bindings.

We released nghttp2 v1.1.1.

We should have written this blog after v1.1.0 was released, but we found some memory leaks in nghttpx, and while tracing this, we have found another issue. Although these issues have existed well before v1.1.0, we decided to fix these issues, and make an another release.

As a result, v1.1.0 became very short term release, but it contains many changes, so we describe changes since v1.0.5 here.

First thing first, the bug fixes in libnghttp2 library. We found bug which causes receive window exhaustion if automatic flow control feature is disabled, and peer sends illegal HTTP message body.

Gabi Davar kindly offered a patch to enhance msvc build and python setup script.

Klaus Ziegler sent us a patch to compile nghttp2 with IRIX gcc-4.7.

nghttp now gets --max-concurrent-streams option to control the concurrency. Regarding nghttp, acesso sent us a patch to add comment on HAR on pushed objects.

We fixed memory leaks and stability issues in nghttpx.

nghttpx now supports backend routing based on request host and path by extending -b option. The detailed syntax is explained in nghttpx man page. The routing pattern is very similar to ServeMux in net/http package from Go programming language. For example, if we want to route request to “/httpbin/” to 127.0.0.1 port 8000, and rest of the requests to 127.0.0.1 port 8081, we can write configuration file like so:

1
2

backend=127.0.0.1,8000;/httpbin/
backend=127.0.0.1,8081

The second rule above is equivalent to 127.0.0.1,8001;/, which matches all request paths. This is called catch-all pattern, and nghttpx requires this catch-all pattern.

You can use also request host to route request:

1
2

backend=127.0.0.1,8000;example.org/
backend=127.0.0.1,8080;blog.example.org/

All requests whose host header field (:authority header field for HTTP/2) are routed to 127.0.0.1 port 8000, and all requests for blog.example.org are routed to 127.0.0.1 port 8080.

This host-path based routing is available for both HTTP/1 and HTTP/2 backend (in other words, it is enabled in default mode, --http2-bridge and --client). It is disabled if -s or -p is given.

nghttpx also gets --include option to include configurations from another file. This is useful to share configurations between different nghttpx instances. The accesslog format variable now can be enclosed with curly braces for disambiguation (e.g., ${remote_addr}). SSL/TLS related log variables, such as TLS protocol version, ciphers, etc, have been also added.

h2load gets --cipher option to specify specific cipher suites to use.