1613938 - Raptor tests hang when downloading conditioned profiles

Status: RESOLVED FIXED

(Testing :: Raptor, defect)

Description

[Florin Strugariu [:Bebe]]

Raptor tests hang when downloading conditioned profile

ex:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=287878138&repo=mozilla-central&lineNumber=1603

[task 2020-02-07T05:20:47.987Z] 05:20:47     INFO -  raptor-perftest Info: Making temp_download_dir from inside get_conditioned_profile c:\users\task_1581044938\appdata\local\temp\tmpc_obng
[task 2020-02-07T05:20:47.987Z] 05:20:47     INFO -  raptor-perftest Info: Platform used: win64
[taskcluster:error] Aborting task...
[taskcluster 2020-02-07T05:45:39.225Z] SUCCESS: The process with PID 3432 (child process of PID 3120) has been terminated.
[taskcluster 2020-02-07T05:45:39.225Z] SUCCESS: The process with PID 3120 (child process of PID 4108) has been terminated.
[taskcluster 2020-02-07T05:45:39.225Z] SUCCESS: The process with PID 7992 (child process of PID 4868) has been terminated.

Comment 1

[Florin Strugariu [:Bebe]]

Tarek it looks like the tests are hanging around:
https://searchfox.org/mozilla-central/source/testing/raptor/raptor/perftest.py#202

we get the previous log message but not the one from cond profile
https://searchfox.org/mozilla-central/source/testing/condprofile/condprof/client.py#61

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Please note that the bug summary doesn't reflect the actual failure as visible on Treeherder. As such sheriffs will file a general task aborted intermittent bug soon. We might want to keep this as blocking bug for that.

Comment 3

[Tarek Ziadé (:tarek)]

so, the fact that you are not getting the first comment in the condprof function is weird.

what you can do is wrap this whole block here : https://searchfox.org/mozilla-central/source/testing/raptor/raptor/perftest.py#201-207
in a try... except Exception block, to see if there's another error, because it bubbles up all the way to creating the PerfTest() class, so maybe that's why it hangs.

Comment 4

[Florin Strugariu [:Bebe]]

Attachment: Bug 1613938 - Raptor tests hang when downloading conditioned profiles

Comment 5

[Florin Strugariu [:Bebe]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e41d19feda47b228cdbff2719ecae98a0a457a35https://treeherder.mozilla.org/#/jobs?repo=try&revision=e41d19feda47b228cdbff2719ecae98a0a457a35

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7707c38dda05
Raptor tests hang when downloading conditioned profiles r=tarek

Comment 7

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/7707c38dda05

Comment 8

[Geoff Brown [:gbrown]]

Seems like this is still happening:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=288516233&repo=mozilla-central&lineNumber=1068
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=288467706&repo=autoland&lineNumber=3040
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=288424467&repo=mozilla-central&lineNumber=883

Comment 9

[Florin Strugariu [:Bebe]]

Tarek any ideas or suggestion on how to fix this?

Comment 10

[Tarek Ziadé (:tarek)]

Maybe this is a resource issue (like a full disk maybe?) because from the Python side, the only thing we are doing is creating a temp dir then getting ready to fill it. Since we're not seeing any error and stuff just hangs, I am wondering if we could dump some system info just before it happens to try to understand. (memory/disk/)

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It's sad that we don't get the resource-usage.json file as artifact uploaded when jobs abort. :/