1150334 - Fragmentation in Safe Browsing chunks leads to 413 (Request Entity Too Large) during list updates

Status: RESOLVED FIXED

(Toolkit :: Safe Browsing, defect)

Description

[François Marier [:francois]]

Monica started looking into bug 1147212 and noticed that she would sometimes get HTTP response 413 (Request Entity Too Large) in response to adding goog-unwanted-shavar. Sample request body is attached.

That's a very fragmented chunk space. It could be a regression on our end.

Comment 1

[François Marier [:francois]]

Attachment: Example request body

Comment 2

[Gian-Carlo Pascutto [:gcp]]

Attachment: Logfile from first update

This is a log taken from within ProtocolParser showing what the server sends us the very first update. 

You can see for example that sub chunk number 47176 is missing, although 47175 and 47177 are there. (see first 3 lines)

Adds chunks are continuous, sub chunks are all over the place.

This makes me suspect this may not be at our end.

Comment 3

[Gian-Carlo Pascutto [:gcp]]

I've done a build of Firefox ESR 10, which more or less uses the original SafeBrowsing implementation dating back to Firefox 3 (and likely dates back to Google's original implementation in their toolbar).

It has the same problem, i.e. it ends up with a totally fragemented subchunk list. 

I conclude this is not a regression in our code.

Comment 4

[Gian-Carlo Pascutto [:gcp]]

Our alerts did pick up that something happened to the distribution of SafeBrowsing data at Firefox users:
http://alerts.telemetry.mozilla.org/index.html#/detectors/1/metrics/244/alerts/?from=2015-02-26&to=2015-02-26

Comment 5

[Gian-Carlo Pascutto [:gcp]]

Google's team pointed out in email conversation that the missing data is in the first redirect URL of the server responses. Looking in that direction now.

Comment 6

[Gian-Carlo Pascutto [:gcp]]

Further investigation didn't turn up much interesting. WireShark confirms that the list of redirects we process matches what the server sends back.

I can demonstrate that the list is fragmented via curl + grep, so I think that is more confirmation this is likely not our regression.

Comment 7

[François Marier [:francois]]

The request size limit on the server will be increased from 20KB to 64KB sometime this week or early
next week.

Comment 8

[François Marier [:francois]]

The fix is apparently in production, so I'll assume this is fixed.