1182249 - Test fx client retry behavior

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Description

[Katie Parlante]

Test client retry behavior if the server returns 503, etc.

Comment 1

[Katie Parlante]

When Firefox tries to send telemetry pings and gets errors (e.g. 503 errors), the client should try sending those same pings later. This bug is for testing that behavior. When filing the bug originally I did look for the original bug/spec that defines this but didn't find it. mreid, dexter and/or gfritzsche should be able to help with more information.

Comment 2

[Stuart Philp :sphilp]

Alessio added some details for me here, describing the backoff and retry logic, as well as the appropriate actions for each response code. 

Here's where they are defined in code:

Response codes:
http://mxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/TelemetrySend.jsm#867

Backoff logic:
http://mxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/TelemetrySend.jsm#434

Checked these on nightly OSX and 8.1, backoff appears to be working correctly in response to 500s (doubling after each failed request) and it does in fact send the data when normal service resumes.

Comment 3

[Vladan Djeric (:vladan)]

The 5xx rate is not insignificant:

https://app.datadoghq.com/dash/54494/data-pipeline?live=false&page=0&is_auto=false&from_ts=1436500800000&to_ts=1437191999999&tile_size=m&fullscreen=false

Stuart: did you confirm that the ping which failed to send becasue of 5xx error is saved at Firefox shutdown?

Comment 4

[Georg Fritzsche [:gfritzsche]]

(In reply to Katie Parlante from comment #1)
> When Firefox tries to send telemetry pings and gets errors (e.g. 503
> errors), the client should try sending those same pings later. This bug is
> for testing that behavior. When filing the bug originally I did look for the
> original bug/spec that defines this but didn't find it.

The behavior is to a part inherited, a partial redesign and cleanup happened in bug 1156712 et al.
I tried to document the current submission behavior here:
https://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/pings.html#submission

Comment 5

[Stuart Philp :sphilp]

Yeah, the failed pings are saved and sent the next time Firefox starts (assuming the server is working again). Also confirmed pings are saved and sent in batches of 10 if there are a lot saved.