Intermittent "How do you want to open this file?" dialog on Windows 10 test machines
Categories
(Infrastructure & Operations :: RelOps: Windows OS, task)
Tracking
(Not tracked)
People
(Reporter: johannh, Assigned: grenade)
References
Details
We recently enabled screenshot comparison testing on Windows 10, but unfortunately there's this dialog intermittently showing up in the middle of our screenshots: https://public-artifacts.taskcluster.net/e64PfNZtTK-Btd5LkkAGhg/0/public/test_info/20170614013944-permissionPrompts_04_noLWT_shareScreen.png This leads to a lot of falsely reported differences such as: https://screenshots.mattn.ca/comparisons/mozilla-central/b266a8d8fd595b84a7d6218d7b8c6b7af0b5027c/mozilla-central/ad3f1138ce6f199408ad58d65c7476636e924909/windows10-64-vm/permissionPrompts_04_noLWT_shareScreen.png You can find the whole gallery if you uncheck "known inconsistencies" on https://screenshots.mattn.ca/compare/?oldProject=mozilla-central&oldRev=b266a8d8fd595b84a7d6218d7b8c6b7af0b5027c&newProject=mozilla-central&newRev=ad3f1138ce6f199408ad58d65c7476636e924909&filter=windows10 This effectively prevents us from enabling automatic reporting on Windows 10. Since it's intermittent (and mozscreenshots is not opening a .txt file to my knowledge), I suspect it's not the test's fault. It would be interesting to know which file it's trying to open. The screenshots are from this job: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=b266a8d8fd595b84a7d6218d7b8c6b7af0b5027c&filter-tier=1&filter-tier=2&filter-tier=3&filter-searchStr=ss&selectedJob=106884521
Updated•7 years ago
|
Comment 1•7 years ago
|
||
:ahal, could you help figure out what might be causing this? Possibly this is something outside of the tests and has nothing to do, but I suspect this is related to tests running or something in the land of the firefox browser session
Comment 2•7 years ago
|
||
Hm, this isn't ringing any bells and I can't think of anytime Firefox would randomly try to open a text file unsolicited. To be clear, the tests aren't trying to open anything? You could check mimeTypes.rdf in the profile for any suspicious handling of .txt.. Maybe it could be set to always save to disk or something. Or maybe setting browser.download.loglevel=info might provide some useful info in the Gecko logs?
Reporter | ||
Comment 3•7 years ago
|
||
I haven't spent a lot of time looking into this yet, my idea that this is not coming from our tests is because: - There's no txt in mozscreenshots: https://searchfox.org/mozilla-central/search?q=txt&path=mozscreenshots - The download icon isn't blue, which it would be if we downloaded anything in this session. - Shouldn't it show the Firefox "Open With" dialog instead if it was opened with Firefox? - As I mentioned, it's intermittent.
Comment 4•7 years ago
|
||
It's intermittent, and cross-suite (see also bug 1373567, I'd be astonished if a jsreftest tries to open a text file). Much more likely to be one of "something in the pre-run setup intermittently fails in a way that tries to open a text file" or "something that only runs on the first pre-run setup of a new instance always tries to open a text file."
Reporter | ||
Comment 5•7 years ago
|
||
FWIW I think this is also failing my browser_contextmenu_touch.js test https://treeherder.mozilla.org/#/jobs?repo=try&revision=5f78950e60fe0c0b6570a9be6a3f2178f863371d&selectedJob=113820104 Rob, can you let us know how this is progressing? Having a consistent Windows 10 test environment is getting more and more important for Photon development as we progress.
Assignee | ||
Comment 6•7 years ago
|
||
apologies for the delay on sorting this. i'm hoping that explicitly setting notepad as the default text editor will prevent this dialogue from popping up. https://github.com/mozilla-releng/OpenCloudConfig/commit/efad455766c55d8166b984591adbf901b39f30ec
Reporter | ||
Comment 7•7 years ago
|
||
Ah, cool, thanks! Can we expect this to trickle into the normal build architecture now (i.e. are you looking for feedback on whether this works in the mentioned test cases) or is that an additional step? :)
Assignee | ||
Comment 8•7 years ago
|
||
yes, i'm hoping to see that we don't get further occurrences of the popup mentioned in comment 0. the change in OpenCloudConfic is propagated to the win 10 testers
Comment 9•7 years ago
|
||
I want to make sure we're on the same page here... :johannh are you testing on buildbot hardware testers or are you using on taskcluster testers? The changes rob is making will only show up on taskcluster. Rob, can you please make sure this also gets ported to the w10 buildbot testers since that's what people are currently using for production talos tests?
Reporter | ||
Comment 11•7 years ago
|
||
Hm, I thought this was fixed but I just saw another occurrence in this job (see uploaded artifacts): https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=b4e656e5a996dd385e9af43a4d9e207553377c51&filter-tier=1&filter-tier=2&filter-tier=3&exclusion_profile=false&filter-searchStr=ss&selectedJob=114664826
Assignee | ||
Comment 12•7 years ago
|
||
tried and failed to reproduce: - created a loaner (https://tools.taskcluster.net/groups/Gwx3p81ET7SCGswa0fjD8g/tasks/Gwx3p81ET7SCGswa0fjD8g/details) - rdp'd to loaner and created a new user account without admin privs, closed connection - rdp'd to loaner as unprivileged user and ran following commands (derived from failed task in comment 11): z: mkdir task_grenade && cd task_grenade wget https://queue.taskcluster.net/v1/task/SNsrvZ91SA21R-FN1-Td9A/runs/0/artifacts/public/build/mozharness.zip 7z x mozharness.zip c:\mozilla-build\python\python.exe -u mozharness\scripts\desktop_unittest.py --cfg mozharness\configs\unittests\win_taskcluster_unittest.py --mochitest-suite=browser-chrome-screenshots --no-read-buildbot-config --installer-url https://queue.taskcluster.net/v1/task/SNsrvZ91SA21R-FN1-Td9A/artifacts/public/build/target.zip --test-packages-url https://queue.taskcluster.net/v1/task/SNsrvZ91SA21R-FN1-Td9A/artifacts/public/build/target.test_packages.json --download-symbols ondemand --mochitest-suite=browser-chrome-screenshots - tests ran and failed but for other reasons. there were no screenshots containing the popup for txt file association and checking file associations by double clicking on an .txt file in explorer resulted in the file being opened by notepad.exe without a prompt. not sure where to go at this point as the problem appears to be intermittent. if we knew what process was attempting to open a text file and what text file, that would be useful too. generic-worker on this worker type (gecko-t-win10-64) is an older version (8.3.0) which runs under the GenericWorker user account. so i'll also attempt to set file associations for that user (https://github.com/mozilla-releng/OpenCloudConfig/commit/fe1f4bb718d00c10e57574af02460b55943738f8). i guess we'll see in the coming days if the problem persists.
Reporter | ||
Comment 13•7 years ago
|
||
Gah, I was about to declare this defeated because I haven't seen it occur in the last dozen or so mozscreenshots runs, but we had another two cases today: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=efc0b1525edbd357818dc7195537364e76f709e7&filter-tier=1&filter-tier=2&filter-tier=3&exclusion_profile=false&filter-searchStr=ss&selectedJob=115065825 https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=5e73b9798464c3f7106f0161dc9a49b234f42f9c&filter-tier=1&filter-tier=2&filter-tier=3&exclusion_profile=false&filter-searchStr=ss&selectedJob=115065353 This all seems to be using gecko-t-win10-64, but that's probably not a helpful observation.
Comment hidden (Intermittent Failures Robot) |
Comment 15•7 years ago
|
||
:grenade, can you look into this again, it seems as though we are still getting many instances of this (screenshots and marionette historically, now in jsreftests)- what looks to be when we start up the browser.
Reporter | ||
Comment 16•7 years ago
|
||
Another occurrence in screenshots today: https://screenshots.mattn.ca/compare/?oldProject=mozilla-central&oldRev=a9182f92641b24d0a972886aed27e784004a343f&newProject=mozilla-central&newRev=933a04a91ce3bd44b230937083a835cb60637084&filter=windows10
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 18•7 years ago
|
||
apologies for the delay, i will look into this again on monday
Comment hidden (Intermittent Failures Robot) |
Comment 22•7 years ago
|
||
:grenade, how is this investigation coming along? we continue to find more tests that are failing and suspect this is the root cause based on the screenshots.
Assignee | ||
Comment 23•7 years ago
|
||
still scratching my head on this one. i created a task [1] on windows 10 which just runs the commands `assoc` and `ftype` which show us (assoc) what filetype an extension is associated to and (ftype) what program is configured to open each filetype. the task log shows that: - .txt extensions are correctly associated with file type txtfile (.txt=txtfile) - txtfile file types are configured to be opened with notepad (txtfile="C:\Windows\System32\Notepad.exe" "%1"). the screenshot suggests that windows doesn't know what to open .txt files with, but the only mechanism i know of for setting or checking the file associations confirms that the file association is correctly set. [1]: https://tools.taskcluster.net/groups/N9RKmbmqROmVWKSbU9GZ0Q/tasks/N9RKmbmqROmVWKSbU9GZ0Q/runs/0
Assignee | ||
Comment 24•7 years ago
|
||
found a comment on the web suggesting that if windows detects a new program installed that has capability to open the filetype, it causes the popup. the registry setting below is supposed to prevent this [HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\Explorer] "NoNewAppAlert"=dword:00000001 testing now: https://github.com/mozilla-releng/OpenCloudConfig/commit/9df98cb867b1e47da36c390c7dcb32b75d7603ee
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Updated•7 years ago
|
Comment 26•7 years ago
|
||
(In reply to Rob Thijssen (:grenade - UTC+3) from comment #24) > found a comment on the web suggesting that if windows detects a new program > installed that has capability to open the filetype, it causes the popup. the > registry setting below is supposed to prevent this Interesting... but why is there a new program installed during testing?
Assignee | ||
Comment 27•7 years ago
|
||
(In reply to Xidorn Quan [:xidorn] UTC+10 from comment #26) > Interesting... but why is there a new program installed during testing? there isn't. it happens during the instance setup. it's just recognising a change from default (notepad only) setup at any point. not specifically during the test run.
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 31•7 years ago
|
||
pmoore: some of the failures being associated with this bug actually look like this: https://public-artifacts.taskcluster.net/RajWsgBMRuab7kL4wxTGXw/0/public/test_info/mozilla-test-fail-screenshot_dc4jdj.png as you can see, it's a different dialog to the "how do you want to open this file" popup. the screen obscuring the browser test is actually the cmd screen that launches generic worker. the output shown on the screen is from this command: https://github.com/mozilla-releng/OpenCloudConfig/blob/aaee6a1b/userdata/Configuration/GenericWorker/run-generic-worker-format-and-reboot.bat#L3 i'm confused as to why the script which launches generic worker is visible while generic worker is running. it seems like this is an intermittent issue rather than a persistent one. i'm guessing that its not the same instance of the script as the one actually running gw (eg: it's a subsequent run). perhaps we need some logic in this script that allows only a single running instance? any ideas?
Comment 32•7 years ago
|
||
No ideas - although this should go away with bug 1399401.
Comment 33•7 years ago
|
||
The reason I say this, is that generic-worker 10 runs as a windows service, rather than a login script triggered by a scheduled task. I'm guessing this worker type runs as current user too, I think not running as current user would also solve this, but I'm guessing there are reasons we can't do this. There may be other ways to fix this rather than the solution in bug 1399401 but I think getting that bug out is the best way forward as it simplifies (and standardises) our setup in so many ways. However, if that bug takes too long to roll out, there may be other approaches. I'm at a rust training this week in Paris, so won't get a chance to look at this properly until at least next week.
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 36•5 years ago
|
||
the "How do you want to open this file?" dialog has resurfaced on windows 10 arm64 instances. with a little debugging (selecting notepad as the file association), i see that the file in question contains the following text:
#!/bin/sh
# Copyright (C) 2002, Earnie Boyd
# mailto:earnie@users.sf.net
# This file is part of Minimal SYStem.
# http://www.mingw.org/msys.shtml
# File: cmd "$COMSPEC" "$@"
i'm guessing that something in the mozilla-build install (the installer contains msys) is triggering this.
i will do some more debugging and try to find a way to squash this.
Assignee | ||
Comment 37•5 years ago
|
||
after a little more debugging, i determined that:
-
the file-path being opened during the mozilla-build/msys install is
C:\mozilla-build\msys\bin\cmd
which contains:#!/bin/sh # Copyright (C) 2002, Earnie Boyd # mailto:earnie@users.sf.net # This file is part of Minimal SYStem. # http://www.mingw.org/msys.shtml # File: cmd "$COMSPEC" "$@"
-
crucially, the system
PATH
environment variable incorrectly contained an entry forC:\mozilla-build\msys\bin
before the entry forC:\Windows\System32
-
the default value setting of the system
COMSPEC
environment variable isCMD
these settings create an undesired or unexpected condition when the command cmd
is run without a .exe
suffix since the incorrect PATH
setting causes cmd
to be invoked at C:\mozilla-build\msys\bin\cmd
instead of C:\Windows\System32\cmd.exe
.
the invocation of cmd
by some/any process is then not seen as an executable by the operating system since windows does not treat a file as an executable if it does not have an extension that matches one contained in the PATHEXT
environment variable. instead, windows treats invocations of non-executables as a request to open a file with a name matching the invocation.
hence on our infra, whenever something invokes cmd
on a system where the PATH
contains a C:\mozilla-build\msys\bin
entry before its C:\Windows\System32
entry, windows assumes we want to open the text file at C:\mozilla-build\msys\bin\cmd
instead of invoking C:\Windows\System32\cmd.exe
which is normally what the caller expects to happen.
the fix is to simply correct the PATH
environment variable so that the C:\Windows\System32
entry precedes the C:\mozilla-build\msys\bin
entry.
in occ, we use a dsc component to set the PATH
environment variable. until now, this was achieved by prepending any entries in the manifest component to whatever was already in the system PATH
. eg:
{
"ComponentName": "env_PATH",
"ComponentType": "EnvironmentVariableUniquePrepend",
"Name": "PATH",
"Values": [
...,
"C:\\mozilla-build\\msys\\bin",
...
],
"Target": "Machine"
},
would result in a PATH
var containing: C:\mozilla-build\msys\bin;whatever-was-already-in-PATH
swapping EnvironmentVariableUniquePrepend
for EnvironmentVariableUniqueAppend
would have the desired effect on any new system and result in a PATH
var containing: whatever-was-already-in-PATH;C:\mozilla-build\msys\bin
.
however, i wanted a fix that would also correct any running systems that already have C:\mozilla-build\msys\bin
in PATH
in the wrong order (preceding C:\Windows\System32
). so i modified the implementations of EnvironmentVariableUniquePrepend
and EnvironmentVariableUniqueAppend
so that they first remove any entries from PATH
that also exist in the manifest component, before either appending or prepending all the manifest components to the PATH
, resulting in the desired order of PATH
components.
the fix patch is very simple and is implemented in the aarch64 branch here: https://github.com/mozilla-releng/OpenCloudConfig/commit/27e555c. it will be merged to master in due course
Description
•