Telemetry aggregation job is failing

RESOLVED FIXED

Status

Cloud Services
Metrics: Pipeline
P1
blocker
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: rvitillo, Assigned: mdoglio)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Assignee)

Comment 1

2 years ago
The reason of the failure is the following file:
telemetry-2/20160626/telemetry/4/saved_session/Firefox/beta/47.0/20160531183335/20160626051725.477_ip-172-31-5-69/20160626051725.477_ip-172-31-5-69:
The filename ends with a colon, which gets urlencoded by the aws python client. Moztelemetry tries then to retrieve the file content using the ulrencoded version and that raises an error.
:mreid do you know why a filename like that could be there? As a stop gap I'll add some logic to python_moztelemetry to skip (+ logging) this kind of files
Flags: needinfo?(mreid)

Comment 2

2 years ago
I'm not sure how this kind of file name would get in there. Should never happen.

@whd, any ideas?

The file in question is:
s3://net-mozaws-prod-us-west-2-pipeline-data/telemetry-2/20160626/telemetry/4/saved_session/Firefox/beta/47.0/20160531183335/20160626051725.477_ip-172-31-5-69:
Flags: needinfo?(mreid) → needinfo?(whd)
(Assignee)

Updated

2 years ago
Assignee: nobody → mdoglio
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Comment 3

2 years ago
I landed a patch to fix the problem in python_moztelemetry --> https://github.com/mozilla/python_moztelemetry/pull/67
I branched this out in bug 1282789 to cover the followup investigation work.

Comment 4

2 years ago
This was caused by the auto-recovery script doing greedy regexp matching on :, and occasionally we see errors that include colons. https://github.com/mozilla-services/puppet-config/pull/2103 has the fix.
Flags: needinfo?(whd)
You need to log in before you can comment on or make changes to this bug.