Notes from the Road: QUIC Interop Zurich

The QUIC Interop event precedes the QUIC Interim. The Interop is meant to find bugs in different implementations — and in the specifications themselves! This year, these two events take place in Zurich, Switzerland, where Google is graciously hosting us.

Enter the Dragon

Kasernenstrasse 97

On Monday, February 3rd, I arrived at Kasernenstrasse 97 at 9 o’clock in the morning. The Zurich Google office is located in Sihlpost — a massive building on the left bank of the river Sihl. Being so massive, the building possesses two entrances, about 200 feet apart, and two addresses: Kasernenstrasse 97 and Kasernenstrasse 95. The former is the regular Google employee entrance; the latter is where the Google reception is located.

Visitors like me are to use the reception entrance where they are given a temporary guest badge. This I learned later. On that morning, having ignored the sign imploring the visitors to use the other entrance (“95? That must be on the next block!”), I tailgated my way to the fourth floor. There, having run into yet another badged entrance, I waited. Thankfully, a few seconds later, a Google employee emerged. I told her in my best German that I am here for the QUIC conference. “Oh! You should have used the other entrance. That’s OK, I will lead you to the reception.” Like the magical Sesame, gates fell open before us and, in as few as 30 seconds, I was at the reception. Thank you, my kind nameless guide!

Boys are Back in Town

I was one of the first people in the room. There, Brian Trammell, our host, greeted me and I met Stanislav Slusny (Akamai). Later, many familiar faces filled the room: Eric Kinnear (Apple) and his crew; Lars Eggert (NetApp); Lucas Pardue, the new QUIC Working Group co-chair, and Alessandro Ghedini (Cloudflare); Martin Duke (F5); David Schinazi and Ted Hardie (Google); Jana Iyengar (Fastly); Martin Thomson and Eric Rescorla (aka EKR) of Mozilla; and others. It was like the good old times again! We picked up where we left off: each running his QUIC client against the others’ servers.

The “Handshake Done” Bug

Christian Huitema, who attended the Interop remotely, and I had been debugging performance issues in lsquic/picoquic (Christian’s implementation) interactions for a few days. On Monday, I fetched the latest version of picoquic and saw the performance much improved. On the other hand, I also observed that some connections would fail during the handshake. After some digging and arguments, I had to admit that I was in the wrong and that lsquic server sent the “handshake done” confirmation too soon when TLS session was being resumed. I have a tentative bug fix that will be included in the next release of lsquic.

Later it turned out that F5 and Cloudflare QUIC implementations have also been afflicted with the same bug, which made me feel slightly less silly. This is the great value of interops: we are bound to find bugs in implementations of newly added protocol features (of which the HANDSHAKE_DONE frame is one).

Bug 2: Garbage Padding

Another new feature of the QUIC protocol is allowing UDP frames carrying Initial packets to be padded with garbage. An implementation is supposed to discard this padding. This was not happening in lsquic, which is what Andy Grover (Mozilla) discovered using the neqo client. I was able to fix this lsquic issue quickly and Andy successfully tested that it works. Thanks for a nice corner case, Andy!

GSO vs XDP

What’s better: GSO or XDP? This was one of the questions that Igor Lubashev (Akamai), Matt Joras (Facebook), Stanislav Slusny, and I discussed. (That is to say, I mostly listened.) This was one of those spontaneous moments that one can experience only when attending a QUIC meeting in person, not remotely. I can’t wait to apply these shared bits of wisdom in lsquic.

Delayed ACKs

Jana Iyengar and Ian Swett (Google), erstwhile coworkers and two of the principal movers of QUIC, teamed up to propose a protocol extension to control acknowledgement delay and other ACK generation parameters dynamically. This has the potential to reduce CPU overhead significantly by sending and processing fewer ACK packets. Getting this right is tricky, however. lsquic, picoquic, and quiche have experimental support for this feature and we played with it, looking for performance gains (or losses). lsquic’s mechanism for ACK control using this extension is quite bare-bones and improves CPU utilization in some cases, while in other cases it may cause significant throughput degradation. More work is required here.

Loss Bits

The loss bits extension (by Igor Lubashev, Alexandre Ferrieux (Orange), and your humble author) is meant to help operators troubleshoot network problems. In particular, inclusion of the loss bits into QUIC packets allows one to determine whether packet loss occurs upstream or downstream of a probe. To date, this extension has proved to be very controversial. The spin bit debates were epic — and they were over just a single bit. Now, we want to add two more bits — Q and L bits — to the unencrypted part of the QUIC packet header!

At LiteSpeed, we believe in being good citizens. This includes allowing network operators to do their job. From the practical side, availability of network tools should improve the chances that QUIC gets adopted. For these reasons, we had supported the spin bit when it was proposed. We now also support the loss bits extension.

Besides lsquic, one other implementation supports the loss bits extension — Christian Huitema’s picoquic. We had already achieved interoperability prior to this week’s event. On Tuesday, Igor and I ran several tests against picoquic and lsquic servers while introducing packet loss. These tests are ongoing.

And the Winner Is…

Just kidding — there is no winner! The Interop is a collaboration between many organizations, large and small, and individual developers. We proved once again that the protocol works. We found bugs. lsquic benefitted from this exercise and, in turn, helped others uncover bugs in their implementations.

The Interim meeting is tomorrow. This is the year that we ship the final spec. Let’s get it done!

P.S. lsquic server has most purple squares!