Notes from the Road: QUIC Interim NYC
Last week, I attended the QUIC Working Group Interim meeting in New York City. Our group is one of many at the IETF. We are working on standardizing the IETF QUIC protocol. The QUIC protocol that is currently used on the Internet is called Google QUIC. It started as a Google experiment between the Chrome browser and some of the Google services a few years ago. We added support for Google QUIC to the LiteSpeed Web Server and ADC products in 2017. At around the same time, Google handed off the protocol to IETF for standardization. This way, everyone can implement QUIC and reap the benefits of a more performant web. I’ve blogged about previous meetings.
This meeting lasted four days: Monday and Tuesday were dedicated to the Interop Hackathon, while Wednesday and Thursday we worked on the QUIC drafts (agenda; minutes). What follows is a recap of the proceedings as well as some observations.
I have been looking forward to this Interim. It was to be the second QUIC Interim Meeting that I attended in person. Unlike the first meeting in Seattle, this one would be a cinch to attend: jump on a train and I am there!
On the morning of Monday, September 17th, I joined thousands of other commuters making their daily trip to the city. Young and old, white and black, skinny and well-rounded: all were on their way to earn their living. In this strange zone, where one is surrounded by people, but yet all alone, my mind wandered as I looked out the train window at houses, roads, stacks of railroad ties, puddles, junkyards, flags, trucks, bridges, baseball fields, and an occasional tent of a homeless person. Two hours door-to-door.
Lars Eggert’s employer, NetApp, was the host of the meeting. Each day at 9:30 am, we assembled in a comfortable conference room on the 19th floor of 285 Madison Ave.
View from the 19th floor (Madison and 41st Street)
The Interop Hackathon
The Interoperability Hackathon — or the Interop — is an opportunity for the different implementers of QUIC to meet face-to-face and to find and fix bugs quickly. This time, there were two interop events at once: the QUIC transport interop and the QPACK interop. I participated in the second one.
QPACK is the HTTP header compression protocol. Like most of our software, the LiteSpeed QPACK library is open-source. There are a few other open-source as well as closed-source implementations. The way to test for compatibility is to encode a list of headers using one QPACK implementation, decode the result using another QPACK implementation, and check whether the result matches the original input.
A snapshot of the QPACK Interop Matrix
By the end of the second day, the participants — Alan Frindell (Facebook), Martin Thomson (Mozilla), Mike Bishop (Akamai), Roma Kane (F5), Masakazu Kitajo (Apache Traffic Server), and yours truly — found and fixed dozens of bugs.
Spin Bit Experiments
On Wednesday, the official part of the QUIC Interim commenced with three spin bit experiments presentations. If you remember, the latency spin bit is meant to aid network operators — in other words, so oft-maligned “middleboxes” — to debug issues with QUIC connections. Because QUIC uses UDP and encrypts almost everything in its packets, the observer lacks most of the information that is present in a TCP connection.
Alexandre Ferrieux of Orange was first with his Spin bit and beyond @ Orange. He hit rough terrain early, as several people, including the chairs, interrupted him. The Working Group consensus, they said, is only to expose the spin bit. Alexandre’s experiments included additional square sequence and end-to-end bits to provide information on upstream loss and downstream loss, respectively. Thus, he did not have a chance to present all the reasons why one would want those three bits of information in each packet. Nevertheless, the reasons are listed in his presentation and well worth a read.
Next was Marcus Ilhar of Ericsson. His presentation was subtitled Testing the Robustness of the 1-bit Signal. He modified quic-go to add the spin bit and wrote a middlebox simulator that both introduced reordering and loss and measured the spin bit. The conclusion is that the spin bit is indeed useful for measuring the RTT of a connection.
Brian Trammell’s (ETH Zürich) presentation was Intraflow Performance Diagnostics.
A portion of a slide from Brian Trammell’s presentation
As he was waiting to board his plane, so went Brian’s story, he wondered why uploading images from his laptop to his home computer was so slow. He broke out tcptrace and analyzed the information it presented. The main insight is that one does not have to have information about congestion window to debug network problems: the RTT information is enough. The RTT information would be provided by the spin bit.
Gutenberg Bible
I craved soul food more than the earthly sustenance and so I visited the Morgan Library & Museum during lunch. Among other rare items, its Treasures from the Vault exhibition featured a real Gutenberg Bible.
Gutenberg Bible at the Morgan Library & Museum
Only 180 copies (120 on paper and 60 on vellum) were printed in 1455. Of those, just 49 are extant, 3 of them at the Morgan Library. The paper copy on display is a large, beautifully typeset and bound, tome.
Speed Dating
Martin Thomson’s selection of GitHub issues was subjected by what was called “Speed Dating.” Each of the issues was given three minutes for discussion by the group. Simplification of PNE (packet number encryption) was picked up by EKR (Eric Rescorla) to research, while most of the other issues were closed with no action.
Interop Results
Christian Huitema (Microsoft) and Alan Frindell updated the participants with the results of the interop. Among notable developments, Christian highlighted that there were two successful connection migrations and that Martin Thomson and Kazuho Oku (Fastly) are close to having interoperable HTTP stacks. Alan summarized the QPACK Interop and informed the group that Google, too, has a QPACK implementation close to shipping.
Wrapping QPACK Absolute Index
Alan led the discussion around my proposal to wrap the QPACK absolute index in order to improve compression. His presentation implied that the compression gains are not that significant, which I had to contradict.
This is because, I said, HTTP is not just for the web. Long lived connections or connections with high volume of requests (or both) may show significant degradation in compression performance. I ran a simulation of a scenario wherein server reply contains three headers — never-changing :status and content-type, and cookie that changes every ten requests. 10,000 responses have 0.13 compression ratio when absolute index wraps versus 0.15 when it does not (14% worse compression ratio). 100,000 responses is 0.13 compression ratio versus 0.16 for 19% deterioration. So the difference is not insignificant at all.
The room had to concur. Alan presented us with three options:
- Do nothing;
- Wrap the encoded value of the absolute index; or
- Change the definition of the absolute index.
(3) is the largest change, one that I originally proposed. There was no clear consensus in the room, with a bunch of people, including Martin Thomson, saying that we should do nothing. On the other hand, Brian Trammell and others argued in favor of wrapping. This change, said Brian, reflects the spirit of QPACK. We had to go to hums. The first hum would be whether to do nothing (1) or do something — (2) or (3). If the first hum resulted in doing something, then we’d hum to select (2) or (3).
The first hum was close, with one of the chairs, Mark Nottingham (Fastly), having to interpret it. His interpretation was that we should do something. It was decided that instead of humming, Alan, who is the editor of the QPACK draft, would select between (2) and (3) himself. The humble man was not particularly happy with the task assigned to him, but resigned himself to carrying the burden.
Designer Quadrupeds
Camel: Where a man belongs.
They say that the camel is a horse designed by a committee. The Working Group has a pattern of delegating a group of people to work on a component of the protocol and then subjecting their findings to on-the-spot (read: shallow) analysis. This is like the inverse of bikeshedding, where the body is eager to argue about, and change, the design of a nuclear power plant. This happened in Melbourne to the Compression Design Team, where weeks of work were discarded in twenty minutes and we ended up with the suboptimal solution that is QPACK. The index-wrapping proposal was close to suffering the same fate at the hands of the people who hummed for option (1). I understand the desire to complete the work more quickly, but we should not settle for shabby work!
Connection ID Management
Mike Bishop presented the output of the Connection ID Design Team. Because I participated in this design team, many of the issues raised by the working group sounded familiar to me; the design team had been over them several times.
Header Bits
Martin Thomson talked about the bits in the first byte of the QUIC packet. It would be nice to be able to multiplex QUIC and DTLS, for example.
QUIC Trace Tool
Victor Vasiliev (Google) described the quic-trace tool developed by Google. A QUIC peer can make a recording of a QUIC connection in a special format. quic-trace creates a visualization of the connection using this recording. The visualization can be used to analyze connection performance.
Wayne Thiebaud
Wayne Thiebaud is an American painter. During Thursday’s thankfully long — 90 minutes! — lunch, I again rushed off to the Morgan Library & Museum to view an exhibition of some of his works. While mostly known for his paintings of everyday objects such as cakes and gumball machines, I liked his charcoal drawings the most.
Three Roads, 1983
Flow Control Deadlocks
As part of my work on the Compression Design Team, I discovered a deadlock condition in QCRAM in January of this year. QPACK solves the problem by cautioning the implementers to write to encoder and data streams in specific order. This was part of Mike Bishop’s Flow Control Gotchas presentation. It was interesting to observe that one could also deadlock with just one stream, for example, if a non-streamable frame is preceded by its length and the application does not read the frame until all of it is available.
I asked for an example of a non-streamable frame. Mike replied that the header block is one such example, to which I nodded. Later, however, I changed my mind: it is not possible to write the header block as a stream, because it must be followed by length, but it is possible to read the header block in streaming mode. This is exactly what our QPACK library does. I will bring this up on the mailing list next week.
Upcoming Meetings
The last bit of business was to plan the next meetings. After a bit of haggling, we agreed that the pre-Bangkok (IETF 103) Interop target will be based on Internet Draft 15 (this Interop was targeting ID-14). The January Interim meeting is likely to be held in Tokyo.
The Road Ahead
QPACK is almost done. My suggestion that the encoder stream framing be removed was accepted and made it into the current QPACK draft. The wrapping of the absolute index just squeaked through this week, and this is the last non-editorial change that I foresee.
On the other side of the spectrum, the QUIC transport draft is still rather shaky. Version negotiation, first header bytes, and packet number encryption are all liable to change. I think there is virtually no chance that we make the November deadline.
The good news is that there is definitely progress. If only we can stop ourselves from changing the transport every few months, we will be in good shape.
Comments