1593250 - Investigate JSON Parsing failures during decode

Status: RESOLVED WONTFIX

(Data Platform and Tools :: General, task, P3)

Description

[Mark Reid [:mreid]]

We see quite a few JSON Parsing errors like:
com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 31)): only regular white space (\r, \n, \t) is allowed between tokens

The same error appears across many docTypes.

The contents of these unparseable documents looks suspiciously similar, this may indicate a bug in handling compression or character encodings on either the client or server.

The occurrence rate is fairly low, but for example there were 3144 of these parsing failures for event pings yesterday.

Comment 1

[Jeff Klukas [:klukas] (UTC-4)]

Also captured in https://github.com/mozilla/gcp-ingestion/issues/940

Comment 2

[Jeff Klukas [:klukas] (UTC-4)]

We've done some investigations as these have cropped up, and in most cases it appears likely that the payloads could be fuzzing. These illegal payloads are likely to be associated with a particular Geo or ISP. We have not been able to identify any recent parsing/encoding bugs within the pipeline.

Comment 3

[Rob Miller [:rmiller]]

Hello,

The Mozilla Data Engineering organization is currently going through our extensive backlog, consisting of hundreds of issues stretching back for nearly 10 years. We've done a pass through all of the open bugzilla bugs and have identified and tagged the ones that we think are relevant enough to still need attention. The rest, including the bug with which this comment is associated, we are closing as "WONTFIX" in a single bulk operation.

If you feel we have closed this (or any) issue in error, please feel free to take the following actions:

• Reopen the bug.
• Edit the bug to add the string [dataplatform] (including the brackets) to the Whiteboard field. (Note that you must edit the Whiteboard, not the similarly named QA Whiteboard.)

Doing this will ensure that we see the bug in our weekly triage process, where we will decide how to proceed.

Thank you.