Tutorial: HPACK API

In this tutorial, we describe basic use of nghttp2's HPACK API. We briefly describe the APIs for deflating and inflating header fields. The full example of using these APIs, deflate.c, is attached at the end of this page. It also resides in the examples directory in the archive or repository.

Deflating (encoding) headers

First we need to initialize a nghttp2_hd_deflater object using the nghttp2_hd_deflate_new() function:

int nghttp2_hd_deflate_new(nghttp2_hd_deflater **deflater_ptr,
                           size_t deflate_hd_table_bufsize_max);

This function allocates a nghttp2_hd_deflater object, initializes it, and assigns its pointer to *deflater_ptr. The deflate_hd_table_bufsize_max is the upper bound of header table size the deflater will use. This will limit the memory usage by the deflater object for the dynamic header table. If in doubt, just specify 4096 here, which is the default upper bound of dynamic header table buffer size.

To encode header fields, use the nghttp2_hd_deflate_hd() function:

ssize_t nghttp2_hd_deflate_hd(nghttp2_hd_deflater *deflater,
                              uint8_t *buf, size_t buflen,
                              const nghttp2_nv *nva, size_t nvlen);

The deflater is the deflater object initialized by nghttp2_hd_deflate_new() described above. The encoded byte string is written to the buffer buf, which has length buflen. The nva is a pointer to an array of headers fields, each of type nghttp2_nv. nvlen is the number of header fields which nva contains.

It is important to initialize and assign all members of nghttp2_nv. For security sensitive header fields (such as cookies), set the NGHTTP2_NV_FLAG_NO_INDEX flag in nghttp2_nv.flags. Setting this flag prevents recovery of sensitive header fields by compression based attacks: This is achieved by not inserting the header field into the dynamic header table.

nghttp2_hd_deflate_hd() processes all headers given in nva. The nva must include all request or response header fields to be sent in one HEADERS (or optionally following (multiple) CONTINUATION frame(s)). The buf must have enough space to store the encoded result, otherwise the function will fail. To estimate the upper bound of the encoded result length, use nghttp2_hd_deflate_bound():

size_t nghttp2_hd_deflate_bound(nghttp2_hd_deflater *deflater,
                                const nghttp2_nv *nva, size_t nvlen);

Pass this function the same parameters (deflater, nva, and nvlen) which will be passed to nghttp2_hd_deflate_hd().

Subsequent calls to nghttp2_hd_deflate_hd() will use the current encoder state and perform differential encoding, which yields HPAC's fundamental compression gain.

If nghttp2_hd_deflate_hd() fails, the failure is fatal and any further calls with the same deflater object will fail. Thus it's very important to use nghttp2_hd_deflate_bound() to determine the required size of the output buffer.

To delete a nghttp2_hd_deflater object, use the nghttp2_hd_deflate_del() function.

Inflating (decoding) headers

A nghttp2_hd_inflater object is used to inflate compressed header data. To initialize the object, use nghttp2_hd_inflate_new():

int nghttp2_hd_inflate_new(nghttp2_hd_inflater **inflater_ptr);

To inflate header data, use nghttp2_hd_inflate_hd2():

ssize_t nghttp2_hd_inflate_hd2(nghttp2_hd_inflater *inflater,
                               nghttp2_nv *nv_out, int *inflate_flags,
                               const uint8_t *in, size_t inlen,
                               int in_final);

nghttp2_hd_inflate_hd2() reads a stream of bytes and outputs a single header field at a time. Multiple calls are normally required to read a full stream of bytes and output all of the header fields.

The inflater is the inflater object initialized above. The nv_out is a pointer to a nghttp2_nv into which one header field may be stored. The in is a pointer to input data, and inlen is its length. The caller is not required to specify the whole deflated header data via in at once: Instead it can call this function multiple times as additional data bytes become available. If in_final is nonzero, it tells the function that the passed data is the final sequence of deflated header data.

The inflate_flags is an output parameter; on success the function sets it to a bitset of flags. It will be described later.

This function returns when each header field is inflated. When this happens, the function sets the NGHTTP2_HD_INFLATE_EMIT flag in inflate_flags, and a header field is stored in nv_out. The return value indicates the number of bytes read from in processed so far, which may be less than inlen. The caller should call the function repeatedly until all bytes are processed. Processed bytes should be removed from in, and inlen should be adjusted appropriately.

If in_final is nonzero and all given data was processed, the function sets the NGHTTP2_HD_INFLATE_FINAL flag in inflate_flags. When you see this flag set, call the nghttp2_hd_inflate_end_headers() function.

If in_final is zero and the NGHTTP2_HD_INFLATE_EMIT flag is not set, it indicates that all given data was processed. The caller is required to pass additional data.

Example usage of nghttp2_hd_inflate_hd2() is shown in the inflate_header_block() function in deflate.c.

Finally, to delete a nghttp2_hd_inflater object, use nghttp2_hd_inflate_del().

deflate.c

/*
 * nghttp2 - HTTP/2 C Library
 *
 * Copyright (c) 2014 Tatsuhiro Tsujikawa
 *
 * Permission is hereby granted, free of charge, to any person obtaining
 * a copy of this software and associated documentation files (the
 * "Software"), to deal in the Software without restriction, including
 * without limitation the rights to use, copy, modify, merge, publish,
 * distribute, sublicense, and/or sell copies of the Software, and to
 * permit persons to whom the Software is furnished to do so, subject to
 * the following conditions:
 *
 * The above copyright notice and this permission notice shall be
 * included in all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
 * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
 * LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
 * OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
 * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 */
#ifdef HAVE_CONFIG_H
#include <config.h>
#endif /* !HAVE_CONFIG_H */

#include <stdio.h>
#include <string.h>

#include <nghttp2/nghttp2.h>

#define MAKE_NV(K, V)                                                          \
  {                                                                            \
    (uint8_t *)K, (uint8_t *)V, sizeof(K) - 1, sizeof(V) - 1,                  \
        NGHTTP2_NV_FLAG_NONE                                                   \
  }

static void deflate(nghttp2_hd_deflater *deflater,
                    nghttp2_hd_inflater *inflater, const nghttp2_nv *const nva,
                    size_t nvlen);

static int inflate_header_block(nghttp2_hd_inflater *inflater, uint8_t *in,
                                size_t inlen, int final);

int main(int argc _U_, char **argv _U_) {
  int rv;
  nghttp2_hd_deflater *deflater;
  nghttp2_hd_inflater *inflater;
  /* Define 1st header set.  This is looks like a HTTP request. */
  nghttp2_nv nva1[] = {
      MAKE_NV(":scheme", "https"), MAKE_NV(":authority", "example.org"),
      MAKE_NV(":path", "/"), MAKE_NV("user-agent", "libnghttp2"),
      MAKE_NV("accept-encoding", "gzip, deflate")};
  /* Define 2nd header set */
  nghttp2_nv nva2[] = {MAKE_NV(":scheme", "https"),
                       MAKE_NV(":authority", "example.org"),
                       MAKE_NV(":path", "/stylesheet/style.css"),
                       MAKE_NV("user-agent", "libnghttp2"),
                       MAKE_NV("accept-encoding", "gzip, deflate"),
                       MAKE_NV("referer", "https://example.org")};

  rv = nghttp2_hd_deflate_new(&deflater, 4096);

  if (rv != 0) {
    fprintf(stderr, "nghttp2_hd_deflate_init failed with error: %s\n",
            nghttp2_strerror(rv));
    exit(EXIT_FAILURE);
  }

  rv = nghttp2_hd_inflate_new(&inflater);

  if (rv != 0) {
    fprintf(stderr, "nghttp2_hd_inflate_init failed with error: %s\n",
            nghttp2_strerror(rv));
    exit(EXIT_FAILURE);
  }

  /* Encode and decode 1st header set */
  deflate(deflater, inflater, nva1, sizeof(nva1) / sizeof(nva1[0]));

  /* Encode and decode 2nd header set, using differential encoding
     using state after encoding 1st header set. */
  deflate(deflater, inflater, nva2, sizeof(nva2) / sizeof(nva2[0]));

  nghttp2_hd_inflate_del(inflater);
  nghttp2_hd_deflate_del(deflater);

  return 0;
}

static void deflate(nghttp2_hd_deflater *deflater,
                    nghttp2_hd_inflater *inflater, const nghttp2_nv *const nva,
                    size_t nvlen) {
  ssize_t rv;
  uint8_t *buf;
  size_t buflen;
  size_t outlen;
  size_t i;
  size_t sum;

  sum = 0;

  for (i = 0; i < nvlen; ++i) {
    sum += nva[i].namelen + nva[i].valuelen;
  }

  printf("Input (%zu byte(s)):\n\n", sum);

  for (i = 0; i < nvlen; ++i) {
    fwrite(nva[i].name, 1, nva[i].namelen, stdout);
    printf(": ");
    fwrite(nva[i].value, 1, nva[i].valuelen, stdout);
    printf("\n");
  }

  buflen = nghttp2_hd_deflate_bound(deflater, nva, nvlen);
  buf = malloc(buflen);

  rv = nghttp2_hd_deflate_hd(deflater, buf, buflen, nva, nvlen);

  if (rv < 0) {
    fprintf(stderr, "nghttp2_hd_deflate_hd() failed with error: %s\n",
            nghttp2_strerror((int)rv));

    free(buf);

    exit(EXIT_FAILURE);
  }

  outlen = (size_t)rv;

  printf("\nDeflate (%zu byte(s), ratio %.02f):\n\n", outlen,
         sum == 0 ? 0 : (double)outlen / (double)sum);

  for (i = 0; i < outlen; ++i) {
    if ((i & 0x0fu) == 0) {
      printf("%08zX: ", i);
    }

    printf("%02X ", buf[i]);

    if (((i + 1) & 0x0fu) == 0) {
      printf("\n");
    }
  }

  printf("\n\nInflate:\n\n");

  /* We pass 1 to final parameter, because buf contains whole deflated
     header data. */
  rv = inflate_header_block(inflater, buf, outlen, 1);

  if (rv != 0) {
    free(buf);

    exit(EXIT_FAILURE);
  }

  printf("\n-----------------------------------------------------------"
         "--------------------\n");

  free(buf);
}

int inflate_header_block(nghttp2_hd_inflater *inflater, uint8_t *in,
                         size_t inlen, int final) {
  ssize_t rv;

  for (;;) {
    nghttp2_nv nv;
    int inflate_flags = 0;
    size_t proclen;

    rv = nghttp2_hd_inflate_hd(inflater, &nv, &inflate_flags, in, inlen, final);

    if (rv < 0) {
      fprintf(stderr, "inflate failed with error code %zd", rv);
      return -1;
    }

    proclen = (size_t)rv;

    in += proclen;
    inlen -= proclen;

    if (inflate_flags & NGHTTP2_HD_INFLATE_EMIT) {
      fwrite(nv.name, 1, nv.namelen, stderr);
      fprintf(stderr, ": ");
      fwrite(nv.value, 1, nv.valuelen, stderr);
      fprintf(stderr, "\n");
    }

    if (inflate_flags & NGHTTP2_HD_INFLATE_FINAL) {
      nghttp2_hd_inflate_end_headers(inflater);
      break;
    }

    if ((inflate_flags & NGHTTP2_HD_INFLATE_EMIT) == 0 && inlen == 0) {
      break;
    }
  }

  return 0;
}