424571 - Trunk windows talos boxes are failing

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Mike Connor [:mconnor]]

Seems to be an error with file permissions or the talos python script.  Odd that they all went at the same time, doesn't appear to be anything changed on that front (config or script) since yesterday.  Tree is closed until this is resolved.

Traceback (most recent call last):
  File "run_tests.py", line 345, in ?
    test_file(arg)
  File "run_tests.py", line 315, in test_file
    res, browser_dump, counter_dump = ttest.runTest(browser_config, tests[test])
  File "c:\talos-slave\vista-trunk-nochrome\talos\ttest.py", line 230, in runTest
    shutil.rmtree(temp_dir)
  File "C:\Python24\lib\shutil.py", line 163, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "C:\Python24\lib\shutil.py", line 163, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "C:\Python24\lib\shutil.py", line 168, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\Python24\lib\shutil.py", line 166, in rmtree
    os.remove(fullname)
OSError: [Errno 13] Permission denied: 'c:\\users\\mozill~1\\appdata\\local\\temp\\tmpcqt51m\\profile\\Cache\\_CACHE_001_'

Comment 1

[Dave Miller [:justdave]]

There is nothing in the IT support document at http://wiki.mozilla.org/Buildbot/IT_Talos_Support_Document that covers this situation.  Handing off to QA.

Comment 2

[Dave Miller [:justdave]]

joduinn has been paged.

Comment 3

[Dave Miller [:justdave]]

alice has been paged as well.

Comment 4

[Dave Miller [:justdave]]

joduinn's on the phone with me right now, and alice is on IRC and sounds like she's got it tracked down already.

Comment 5

[Dave Miller [:justdave]]

OK, branch is fine it's only the 5 trunk talos boxes affected by this.

Comment 6

[alice nodelman [:alice] [:anode]]

The errors that talos is throwing indicate that it is attempting to clean up a profile directory that is still locked and in use.  In this case, since all the machines went down at once on the same build, I'd assume that something was checked in that caused the browser to either a) close slower or b) close in an unclean fashion. 

I've checked in a talos patch that I was working on in bug 416911 that should handle/mask this problem by having talos do a better job of killing browser processes at the end of testing - but I still believe that there was a browser regression.

Comment 7

[alice nodelman [:alice] [:anode]]

I got to see some of the winxp machines run tests and found the problem.  A dialog is being raised during Ts:

Internet Security
A script from "file://" is requesting enhanced abilities that are UNSAFE and could be used to compromise your machine or data:"

From talk on irc it seems that we tightened up security around file:// links.  That change is interfering with the operation of talos tests that use file load.

Comment 8

[alice nodelman [:alice] [:anode]]

I've made local changes to all the production talos config files to set

security.fileuri.strict_origin_policy : false

That should override the new security setting and allow talos to complete tests.

Comment 9

[alice nodelman [:alice] [:anode]]

Filed bug 424594 to have talos comply with new security restrictions.

Comment 10

[alice nodelman [:alice] [:anode]]

These machines are all green now due to the temporary fix in comment #7.  Follow up with bug 424594 for talos complying with new file:// security.