Thanks to Yan Zhu of Brave; also, our sincere thanks to Ilya Grigorik, Eric Lawrence, Mike West, and Andrew Howden of Google for their assistance.
Back-story: Brave users reported ads getting past our ad-blocking shields in previous Chromium versions, beginning with reports of ads displaying on YouTube.com on October 11, 2016. uBlock Origin users had reported similar bugs. We discovered during testing that disabling QUIC seemed to stop these ads. As a result, we pushed an update to disable QUIC in Brave on January 25, 2017. This update appears to have temporarily abated the incoming bug reports about ads getting past our shields.
We’ve concluded our investigation of YouTube ads getting past ad blockers, and can confirm that ad blocking with QUIC re-enabled operates as expected in Chrome 58 with uBlock Origin.
We would like to end on a positive note about QUIC. It realizes significant speedups, due to round trip reductions and other advantages, over the HTTPS/TCP/IP alternative. As the post details, we suggest others in ad tech become familiar with it to benefit from the same advantages Google is seeing. 05/10/17 – 3:45pm PDT
When we inspected web page traffic via chrome://net-internals, we discovered that QUIC requests were and still are being used for a majority of Google’s ad domains
What is QUIC?
Originally announced in 2013, QUIC (Quick UDP Internet Connections) is an experimental network protocol, which runs on top of the UDP protocol and is usually requested through port 443 with an Alternative Service HTTP request header flag (example:
alt-svc:quic="googleads.g.doubleclick.net:443"). A QUIC working group has been established to standardize the protocol; QUIC is currently still considered experimental.
From a high level, QUIC requests pack several round trips into a single, one-way request, including the security (TLS) handshake. Google’s diagram from their 2015 blog post on the topic helps illustrate the round trip savings:
- QUIC is enabled by default in Google’s Chrome browser and underlying Chromium open source browser code. As of March 2017, Chrome accounts for 58.9% of users browsing the web.
- Brave blocks QUIC requests. QUIC is an opt-in feature in Opera, and is currently not available in other Firefox, Edge, and Safari. HTTPS requests containing the
alt-svc: quic=":443"response header fall back to traditional TCP connections in other browsers, or when QUIC fails in Chrome.
- QUIC use has not flown completely under the radar. Google’s April 2015 blog post on QUIC mentioned that roughly 50% of Google traffic was being requested from Chrome/Chromium with QUIC. Missing from the announcement and other QUIC documentation is any mention of QUIC usage in Google’s Doubleclick ad requests, or in Google Analytics tracking requests.
How can I review QUIC sessions and requests?
QUIC runs on top of the UDP protocol. QUIC requests are often made through the same port (443) that is used for TCP requests. Aside from some corporate firewalls that block UDP requests by their protocol number, making QUIC requests to port 443 helps requests get through firewalls configured to allow TCP requests to that port, independent of the protocol number in the IP header. This is a clever hack that dramatically reduces adoption friction, by avoiding the need for firewall reconfiguration in most cases.
- Chrome shows QUIC requests in Chrome Developer Tools. If you want to distinguish between QUIC and HTTP in the Network interface, right click to select and reveal the Protocol column.
- QUIC requests are visible in the
chrome://internal browser URLs, and of course in WireShark and other lower-level packet sniffing tools.
Tips for observing QUIC traffic
The easiest way to capture and observe QUIC traffic in detail is within the
chrome://net-internals interface. Some
chrome://net-internalsshortcuts are included below for reference.
- To view the QUIC settings and session connections within the dedicated QUIC panel, enter the following address in the URL bar in Chrome: chrome://net-internals#quic
- To view QUIC session requests in the Events panel, enter the following address in the URL bar in Chrome: chrome://net-internals/#events&q=type:QUIC_SESSION
- To access the alt-svc panel to view domains that contain the
quic :443, Alternative Service HTTP response header, enter the following address in the URL bar in Chrome: chrome://net-internals/#alt-svc
- To export a JSON log that includes QUIC request traffic, enter the following address in the URL bar in Chrome: chrome://net-internals/#export
- The HTTP/2 and SPDY indicator Chrome extension provides a shortcut to the
- Within the Network section of the Chrome Developer Tools, users can right click to reveal the Protocol column, which will show
http/2+quic/36as the request protocol. This column is not displayed with default settings in Developer Tools.
To disable QUIC
- Enter chrome://flags/ into the URL bar in Chrome.
- Locate the Experimental QUIC protocol flag.
- Click to expand the drop down menu and select Disabled
QUIC and Google ad requests
When we inspected web page traffic via
chrome://net-internals, we discovered that QUIC requests were and still are being used for a majority of Google’s ad domains, including domains involved with bidding such as the one below:
bid.g.doubleclick.net Start Time: 2017-04-20 00:28:09.220 t= 95972 [st= 0] +QUIC_SESSION [dt=340129+] --> cert_verify_flags = 6 --> host = "bid.g.doubleclick.net" --> port = 443 --> privacy_mode = false --> require_confirmation = false t= 96028 [st= 56] QUIC_CHROMIUM_CLIENT_STREAM_SEND_REQUEST_HEADERS --> :authority: bid.g.doubleclick.net :method: GET :scheme: https accept: */* accept-encoding: gzip, deflate, sdch, br accept-language: en-US,en;q=0.8 cookie:id=224683e7072a008a||t=1492673205|et=730|cs=002213fd48a93fd86aff9b0172; IDE=AHWqTUnVMhtPcKMXwHTn_nS5yFVKx1XEjcVJo18Kg9O3XAc2HbLk0EBHtQ; DSID=NO_DATA referer: https://tpc.googlesyndication.com/safeframe/1-0-7/html/container.html user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36 --> quic_priority = 1 --> quic_stream_id = 5
Primary Google domains observed opening QUIC sessions
Google domains that made requests within QUIC sessions
Alternate Service Mappings https://pagead2.googlesyndication.com quic googleads.g.doubleclick.net:443, expires 2017-05-20 00:33:40,quic :443, expires 2017-05-20 00:33:40 https://tpc.googlesyndication.com quic :443, expires 2017-05-20 00:33:22 https://s0.2mdn.net quic :443, expires 2017-05-20 00:33:22 https://cm.g.doubleclick.net quic googleads.g.doubleclick.net:443, expires 2017-05-20 00:33:22,quic :443, expires 2017-05-20 00:33:22 https://www.google.com quic :443, expires 2017-05-20 00:33:22 https://adx.g.doubleclick.net quic googleads.g.doubleclick.net:443, expires 2017-05-20 00:33:22,quic :443, expires 2017-05-20 00:33:22 https://securepubads.g.doubleclick.net quic :443, expires 2017-05-20 00:33:22 https://ad.doubleclick.net quic :443, expires 2017-05-20 00:33:19 https://googleads4.g.doubleclick.net quic googleads.g.doubleclick.net:443, expires 2017-05-20 00:33:03,quic :443, expires 2017-05-20 00:33:03 https://content.googleapis.com quic :443, expires 2017-05-20 00:33:02 https://apis.google.com quic :443, expires 2017-05-20 00:33:01 https://www.googletagservices.com quic googleads.g.doubleclick.net:443, expires 2017-05-20 00:29:26,quic :443, expires 2017-05-20 00:29:26 https://accounts.google.com quic :443, expires 2017-05-20 00:33:01 https://www.google-analytics.com quic :443, expires 2017-05-20 00:33:00 https://ade.googlesyndication.com quic googleads.g.doubleclick.net:443, expires 2017-05-20 00:32:58,quic :443, expires 2017-05-20 00:32:58 https://googleads.g.doubleclick.net quic googleads.g.doubleclick.net:443, expires 2017-05-20 00:32:54,quic :443, expires 2017-05-20 00:32:54 https://www.gstatic.com quic :443, expires 2017-05-20 00:29:55 https://s1.2mdn.net quic :443, expires 2017-05-20 00:32:43 https://stats.g.doubleclick.net quic :443, expires 2017-05-20 00:32:40 https://pubads.g.doubleclick.net quic :443, expires 2017-05-20 00:32:06 https://static.doubleclick.net quic :443, expires 2017-05-20 00:32:07 https://www.youtube.com quic :443, expires 2017-05-20 00:32:07 https://video-ad-stats.googlesyndication.com quic :443, expires 2017-05-20 00:32:07 https://s.ytimg.com quic :443, expires 2017-05-20 00:32:06 https://google-analytics.com quic :443, expires 2017-05-20 00:32:03 https://imasdk.googleapis.com quic :443, expires 2017-05-20 00:32:00 https://fonts.googleapis.com quic :443, expires 2017-05-20 00:31:55 https://ssl.google-analytics.com quic :443, expires 2017-05-20 00:31:58 https://www.googletagmanager.com quic :443, expires 2017-05-20 00:31:55 https://fonts.gstatic.com quic :443, expires 2017-05-20 00:31:55 https://p4-fbm4tfy4du3vk-rsg77dtzm53vwr6k-if-v6exp3-v4.metric.gstatic.com quic :443, expires 2017-05-20 00:30:57 https://ssl.gstatic.com quic :443, expires 2017-05-20 00:27:03 https://bid.g.doubleclick.net quic :443, expires 2017-05-20 00:31:19 https://p4-fbm4tfy4du3vk-rsg77dtzm53vwr6k-854535-i2-v6exp3.ds.metric.gstatic.com quic :443, expires 2017-05-20 00:31:08 https://p4-fbm4tfy4du3vk-rsg77dtzm53vwr6k-854535-i1-v6exp3.v4.metric.gstatic.com quic :443, expires 2017-05-20 00:31:08 https://ajax.googleapis.com quic :443, expires 2017-05-20 00:31:02 https://www.googleadservices.com quic googleads.g.doubleclick.net:443, expires 2017-05-20 00:27:45,quic :443, expires 2017-05-20 00:27:45 https://plus.google.com quic :443, expires 2017-05-20 00:27:25 https://clients2.google.com quic :443, expires 2017-05-20 00:26:24
Examples of Doubleclick QUIC requests
Example of TCP a HTTP GET Request URL with the
alt-svc: quic=”443” HTTP response header
What does this mean for the ad ecosystem?
It would be helpful for Doubleclick to clarify how they’re using QUIC for ads and tracking. With DoubleClick’s dominance in the ad market, publishers and advertisers need to have a clear understanding and documentation regarding which protocols are being used for ad and tracking requests.
We want to emphasize that we’re fans of QUIC as a protocol, and we hope it is widely adopted. But from our contacts in ad tech, it appears that QUIC is not understood or used yet. While Doubleclick is not obligated to share its QUIC usage details, we hope that it will do so to increase QUIC adoption across the ad-tech ecosystem.
Additional QUIC info and resources:
- Google: QUIC FAQ for geeks
- Google: Flow control in QUIC
- Google: QUIC Wire Layout Specification
- Google: QUIC Crypto
- Google: Life of a URL Request
- Chromium Blog: A QUIC update on Google’s experimental transport
- Chromium Blog: Experimenting with QUIC
- ma.ttias.be: Google’s QUIC protocol: moving the web from TCP to UDP
- Digiday: WTF is advertising arbitrage?
- bugs.chromium.org: Google’s DFP ads not firing when QUIC protocol used
- Kate Pearce: HTTP/2 and QUIC: Teaching good protocols to do bad things (Black Hat 2016)