1636697 - Frequent Linux x64 asan reporter retries that end as exception

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Cosmin Sabou [:CosminS]]

Th link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&searchStr=linux%2Cx64%2Casan%2Creporter%2Copt%2Cpartial%2Cgeneration%2Cpartials-linux64-asan-reporter-nightly%2Fopt%2Cp%28n%29&fromchange=5d4199425f57cf5657ace9e868924164286e0f43&tochange=efb86c809f5c1bba866e1e8585dab7341bb9828a&selectedTaskRun=eB8vMwrGTWCwSylpUWy_Bg-0

Task with no logs: https://firefox-ci-tc.services.mozilla.com/tasks/eB8vMwrGTWCwSylpUWy_Bg/runs/6

After the first rerun the task is green.

This is the merge that it started from: https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=5d4199425f57cf5657ace9e868924164286e0f43

Justin, is this something you could look into? Thanks.

Comment 1

[Intermittent Failures Robot]

2 failures in 5276 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• linux64-asan-reporter: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1636697&startday=2020-05-04&endday=2020-05-10&tree=all

Comment 2

[Tom Prince [:tomprince]]

redirecting to :sfraser, who has been working on partial recently.

Comment 3

[Simon Fraser [:sfraser] ⌚️GMT]

That the claims expired points at an error in the workers, although I'm struggling to find the right place those errors might have been logged. Where do we go to find those?

From the stats in BigQuery it looks like we also had build failures for gecko-3/b-linux around these times for tasks on autoland, central and one on beta. Examples:

https://firefox-ci-tc.services.mozilla.com/tasks/bXHw8j6eToSN6ovIYCIULQ
https://firefox-ci-tc.services.mozilla.com/tasks/BhCaXdfDT5Gyfugh7_YUFA
https://firefox-ci-tc.services.mozilla.com/tasks/HrzOOe3TSPudj-LoWnARpg
https://firefox-ci-tc.services.mozilla.com/tasks/UVzRAu93R5-ivZ_dvWQPdg
https://firefox-ci-tc.services.mozilla.com/tasks/Awit6mLVTkG7xMcu3srQgg
https://firefox-ci-tc.services.mozilla.com/tasks/J60xJ_zXT6eposWZBjq1rw
https://firefox-ci-tc.services.mozilla.com/tasks/RT_2udJtS1WtE_pJhsnUQw
https://firefox-ci-tc.services.mozilla.com/tasks/S6NxkTzsRe-_dFW08o7o2Q
https://firefox-ci-tc.services.mozilla.com/tasks/A8imbS5zQoedlK05Sc3wXQ
https://firefox-ci-tc.services.mozilla.com/tasks/Wzqk-B-KS0yvTExQ-X-0zw
https://firefox-ci-tc.services.mozilla.com/tasks/PPOMBjuvSt6RnVFIeIcPAA
https://firefox-ci-tc.services.mozilla.com/tasks/H3XYH-78TQy6wiA12GVePA
https://firefox-ci-tc.services.mozilla.com/tasks/bUp8F1rvQNWI3kEM_U4lJQ
https://firefox-ci-tc.services.mozilla.com/tasks/NxNDOYZtTimkWCUPel1j2A

For posterity, the timings are:

0 exception claim-expired 2020-05-08T23:01:38.352Z 2020-05-08T23:21:38.938Z
1 exception claim-expired 2020-05-08T23:21:38.977Z 2020-05-08T23:41:40.277Z
2 exception claim-expired 2020-05-08T23:45:55.669Z 2020-05-09T00:05:57.996Z
3 exception claim-expired 2020-05-09T00:05:58.209Z 2020-05-09T00:25:58.322Z
4 exception claim-expired 2020-05-09T00:25:58.502Z 2020-05-09T00:45:59.470Z
5 exception claim-expired 2020-05-09T00:45:59.704Z 2020-05-09T01:06:01.288Z
6 exception claim-expired 2020-05-09T01:12:59.128Z 2020-05-09T01:33:00.077Z

0 exception claim-expired 2020-05-09T10:58:01.625Z 2020-05-09T11:18:03.489Z
1 exception claim-expired 2020-05-09T11:19:51.793Z 2020-05-09T11:39:52.763Z
2 exception claim-expired 2020-05-09T11:41:02.227Z 2020-05-09T12:01:02.293Z
3 exception claim-expired 2020-05-09T12:01:02.610Z 2020-05-09T12:21:03.283Z
4 exception claim-expired 2020-05-09T12:21:03.342Z 2020-05-09T12:41:04.756Z
5 exception claim-expired 2020-05-09T12:41:04.986Z 2020-05-09T13:01:05.080Z
6 completed completed 2020-05-09T13:27:15.399Z 2020-05-09T14:03:57.394Z

Comment 4

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: Bug 1636697 Stop ASAN partial generation

Due to the resource usage (over 8Gb memory per libxul), cost and low impact, stop generating partials for asan builds

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2c2cf2e0c319
Stop ASAN partial generation r=catlee

Comment 6

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/2c2cf2e0c319