1323061 - assertion while running mochitest tests on win7 taskcluster causes tests to not run

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we fail to get tests to run due to an assertion, here is the reason we have an assertion:
09:23:46     INFO -  ### XPCOM_MEM_BLOAT_LOG defined -- unable to log bloat/leaks to z:\task\build\tmp8ggfif.mozrunner\runtests_leaks_tab_pid1648.log

I assume this is related to us not being able to access the log.  This is from a try push here (specifically mochitest-1):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c1ada0b9159e2b37f7a62c9c659a392ece5295aa

Once this is figured out we can turn leak logging back on and start working on any leaks which are related to the migration from buildbot to taskcluster.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:pmoore, can you focus on this or figure out how to bump this up in your priority queue a bit?

Comment 2

[Rob Thijssen [:grenade (EET/UTC+0300)]]

on buildbot windows tests, the log file is shown as: c:\users\cltbld\appdata\local\temp\tmph3otck.mozrunner\runtests_leaks_tab_pid3340.log

on taskcluster windows tests it's shown as z:\task\build\tmpihqmif.mozrunner\runtests_leaks_tab_pid196.log

i think what's happening is that the code (on tc) is failing to find the TEMP dir and thus assuming the path (tmpihqmif.mozrunner\runtests_leaks_tab_pid196.log) falls under the current work dir (z:\task\build) but it doesn't. i reckon this is caused because the temp directory is not set correctly for the task user when the task user is created (see bug 1319487) and may be fixed by running SHSetKnownFolderPath for the temp directory of the task user account.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

good find :grenade- that seems like an actionable step forward.

Comment 4

[Pete Moore [:pmoore][:pete]]

I'll be back at work tomorrow, and will take a look.

In the meantime, trying a quick test run here, on the off chance it might solve it:
https://tools.taskcluster.net/task-inspector/#JqSYzqJdQgOxtRnJqTcvTQ/0

Comment 5

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pmoore][:pete] from comment #4)
> I'll be back at work tomorrow, and will take a look.
> 
> In the meantime, trying a quick test run here, on the off chance it might
> solve it:
> https://tools.taskcluster.net/task-inspector/#JqSYzqJdQgOxtRnJqTcvTQ/0

That fixed it. Just don't set the env section. See bug 1319487 for a discussion of this topic. It seemed to run quite quickly.

Comment 6

[Rob Thijssen [:grenade (EET/UTC+0300)]]

it only fixes it in the case where runTasksAsCurrentUser is set, because the generic worker user folders are created by the autologon scheduled task. when the task is run as a task user created by the generic worker and the folders are not created, it is still broken.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

I am not clear on the difference between scheduled task and task user.  Which method are we targeting for the initial test roll out?

If this fixes things in our current model of running tests, I would like to get it deployed- possibly this would fix other issues with failure to remove files that we are seeing on 6 different xpcshell tests.

Comment 8

[Pete Moore [:pmoore][:pete]]

(In reply to Rob Thijssen (:grenade - GMT) from comment #6)
> it only fixes it in the case where runTasksAsCurrentUser is set, because the
> generic worker user folders are created by the autologon scheduled task.
> when the task is run as a task user created by the generic worker and the
> folders are not created, it is still broken.

I'm not usre - I suspect this is due to the inconsistency between the system setting, and the environment variable. Somewhere inside the test, I suspect it is querying the system to find the temp dir, and getting the value on the C: drive, which differs from the environment variable setting on the Z: drive.

(In reply to Joel Maher ( :jmaher) from comment #7)
> I am not clear on the difference between scheduled task and task user. 
> Which method are we targeting for the initial test roll out?

Running as current user, i.e. where removing these env vars solves the problem.

> If this fixes things in our current model of running tests, I would like to
> get it deployed- possibly this would fix other issues with failure to remove
> files that we are seeing on 6 different xpcshell tests.

Agreed.

If we are concerned with performance penalty, it would be valuable to measure with/without setting these env vars, to see what the difference is. However, we can get everything moved to the Z: drive by modifying the attend.xml file used during system preparation, as discussed in bug 1319487.

Comment 9

[Pete Moore [:pmoore][:pete]]

(In reply to Rob Thijssen (:grenade - GMT) from comment #6)
> it only fixes it in the case where runTasksAsCurrentUser is set, because the
> generic worker user folders are created by the autologon scheduled task.
> when the task is run as a task user created by the generic worker and the
> folders are not created, it is still broken.

To test this theory, I've created a task which explicitly creates the temp directory:

https://tools.taskcluster.net/task-inspector/#I9VjgQ6iTYSKZiHPmqzT4w/

Maybe this also fixes it.

Comment 10

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pmoore][:pete] from comment #9)

> To test this theory, I've created a task which explicitly creates the temp
> directory:
> 
> https://tools.taskcluster.net/task-inspector/#I9VjgQ6iTYSKZiHPmqzT4w/
> 
> Maybe this also fixes it.

OK that also turned it green - so all I did there was add a command `mkdir "%TEMP%"`.

So this is also a solution to the problem, if we don't want to remove the env vars, and keep everything on Z:.

Comment 11

[Pete Moore [:pmoore][:pete]]

Probably the cleanest solution is to set both the system TEMP and TMP env variables to:

Z:\%USERNAME%\AppData\Local\Temp

I'll test this, and then make an OCC pull request for it...

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

is there more work to do here, we are not blocked anymore and it would be nice to close this bug.

Comment 13

[Pete Moore [:pmoore][:pete]]

Agreed - thanks Joel! In short, this is fixed in other bugs, since we no longer set these env vars, so the default values are used, and those dirs exist.