Open Bug 1554821 Opened 7 years ago Updated 24 days ago

OSX Mojave (10.14) - browser/tools/mozscreenshots/ - Can't capture screenshots without macOS screenshot permission

Categories

(Testing :: mozscreenshots, defect, P3)

Product:
Testing
Component:
mozscreenshots
Type:
defect

Tracking

(Not tracked)

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

Filed by: egao [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=248622685&repo=try
Full log: https://queue.taskcluster.net/v1/task/MeAocohbQCyPUUdzNJ0WLg/runs/0/artifacts/public/logs/live_backing.log

Test: mochitest-browser-screenshots
Chunk: n/a
Platform: macosx1014-mojave

Context:
22:42:48 INFO - TEST-UNEXPECTED-FAIL | browser/tools/mozscreenshots/controlCenter/browser_controlCenter.js | Test timed out -
22:42:48 INFO - GECKO(1681) | MEMORY STAT vsizeMaxContiguous not supported in this build configuration.
22:42:48 INFO - GECKO(1681) | MEMORY STAT | vsize 7621MB | residentFast 357MB | heapAllocated 93MB
22:42:48 INFO - TEST-OK | browser/tools/mozscreenshots/controlCenter/browser_controlCenter.js | took 4507706ms
22:42:48 INFO - checking window state
22:42:48 INFO - GECKO(1681) | must wait for focus
22:43:40 INFO - GECKO(1681) | 73:75: execution error: Firefox Nightly got an error: AppleEvent timed out. (-1712)
22:43:42 INFO - Not taking screenshot here: see the one that was previously logged
22:43:42 INFO - TEST-UNEXPECTED-FAIL | browser/tools/mozscreenshots/controlCenter/browser_controlCenter.js | Uncaught exception received from previously timed out test - at resource://mozscreenshots/TestRunner.jsm:401 - TypeError: this.mochitestScope.info is not a function
22:43:42 INFO - Stack trace:
22:43:42 INFO - _onConfigurationReady@resource://mozscreenshots/TestRunner.jsm:401:25
22:43:42 INFO - async*_performCombo@resource://mozscreenshots/TestRunner.jsm:386:16
22:43:42 INFO - asyncstart@resource://mozscreenshots/TestRunner.jsm:130:18
22:43:42 INFO - asynccapture@chrome://mochitests/content/browser/browser/tools/mozscreenshots/controlCenter/browser_controlCenter.js:15:20
22:43:42 INFO - Tester_execTest/<@chrome://mochikit/content/browser-test.js:1116:34
22:43:42 INFO - asyncTester_execTest@chrome://mochikit/content/browser-test.js:1144:12
22:43:42 INFO - nextTest/<@chrome://mochikit/content/browser-test.js:1005:14
22:43:42 INFO - SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:803:59
22:49:52 INFO - Buffered messages finished
22:49:52 ERROR - TEST-UNEXPECTED-TIMEOUT | Last test finished | application timed out after 370 seconds with no output
22:49:52 ERROR - Force-terminating active process(es).
22:49:52 INFO - Determining child pids from psutil...
22:49:52 INFO - [1682, 1683, 1685, 1686, 1687, 1707, 1789, 1838]
22:49:52 INFO - ==> process 1681 launched child process 1682
22:49:52 INFO - ==> process 1681 launched child process 1683
22:49:52 INFO - ==> process 1681 launched child process 1685
22:49:52 INFO - ==> process 1681 launched child process 1686
22:49:52 INFO - ==> process 1681 launched child process 1687
22:49:52 INFO - ==> process 1681 launched child process 1699
22:49:52 INFO - ==> process 1681 launched child process 1707
22:49:52 INFO - ==> process 1681 launched child process 1782
22:49:52 INFO - ==> process 1681 launched child process 1789
22:49:52 INFO - ==> process 1681 launched child process 1838
22:49:52 INFO - Found child pids: set([1699, 1782, 1707, 1838, 1682, 1683, 1685, 1686, 1687, 1789])
22:49:52 INFO - Failed to get child procs
22:49:52 INFO - Killing process: 1699
22:49:52 INFO - Not taking screenshot here: see the one that was previously logged
22:49:52 INFO - Can't trigger Breakpad, process no longer exists
22:49:52 INFO - Killing process: 1782
22:49:52 INFO - Not taking screenshot here: see the one that was previously logged
22:49:52 INFO - Can't trigger Breakpad, process no longer exists
22:49:52 INFO - Killing process: 1707
22:49:52 INFO - Not taking screenshot here: see the one that was previously logged
22:49:52 INFO - Killing process: 1838
22:49:52 INFO - Not taking screenshot here: see the one that was previously logged
22:49:52 INFO - Killing process: 1682
22:49:52 INFO - Not taking screenshot here: see the one that was previously logged
22:49:52 INFO - Killing process: 1683
22:49:52 INFO - Not taking screenshot here: see the one that was previously logged
22:49:52 INFO - Killing process: 1685
22:49:52 INFO - Not taking screenshot here: see the one that was previously logged
22:49:52 INFO - Killing process: 1686
22:49:52 INFO - Not taking screenshot here: see the one that was previously logged
22:49:52 INFO - Killing process: 1687
22:49:52 INFO - Not taking screenshot here: see the one that was previously logged
22:49:52 INFO - Killing process: 1789
22:49:52 INFO - Not taking screenshot here: see the one that was previously logged
22:49:52 INFO - Killing process: 1681
22:49:52 INFO - Not taking screenshot here: see the one that was previously logged
22:49:52 INFO - psutil found pid 1681 dead
22:49:53 WARNING - runtests.py | Failed to get app exit code - running/crashed?
22:49:53 INFO - TEST-INFO | Main app process: exit 0
22:49:53 INFO - runtests.py | Application ran for: 1:22:15.209103
22:49:53 INFO - zombiecheck | Reading PID log: /var/folders/pw/g0947b2n6v1_lkbjc6np_vyr000017/T/tmpQv4GDwpidlog
22:49:53 INFO - ==> process 1681 launched child process 1682
22:49:53 INFO - ==> process 1681 launched child process 1683
22:49:53 INFO - ==> process 1681 launched child process 1685
22:49:53 INFO - ==> process 1681 launched child process 1686
22:49:53 INFO - ==> process 1681 launched child process 1687
22:49:53 INFO - ==> process 1681 launched child process 1699
22:49:53 INFO - ==> process 1681 launched child process 1707
22:49:53 INFO - ==> process 1681 launched child process 1782
22:49:53 INFO - ==> process 1681 launched child process 1789
22:49:53 INFO - ==> process 1681 launched child process 1838
22:49:53 INFO - zombiecheck | Checking for orphan process with PID: 1699
22:49:53 INFO - zombiecheck | Checking for orphan process with PID: 1782
22:49:53 INFO - zombiecheck | Checking for orphan process with PID: 1707
22:49:53 INFO - zombiecheck | Checking for orphan process with PID: 1838
22:49:53 INFO - zombiecheck | Checking for orphan process with PID: 1682
22:49:53 INFO - zombiecheck | Checking for orphan process with PID: 1683
22:49:53 INFO - zombiecheck | Checking for orphan process with PID: 1685
22:49:53 INFO - zombiecheck | Checking for orphan process with PID: 1686
22:49:53 INFO - zombiecheck | Checking for orphan process with PID: 1687
22:49:53 INFO - zombiecheck | Checking for orphan process with PID: 1789
22:49:53 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/cYy_ZTIqS1W25mBgk4x8Vw/artifacts/public/build/target.crashreporter-symbols.zip
22:49:55 INFO - mozcrash Copy/paste: /Users/cltbld/tasks/task_1558991507/build/macosx64-minidump_stackwalk /var/folders/pw/g0947b2n6v1_lkbjc6np_vyr000017/T/tmpFYjCr7.mozrunner/minidumps/BD22881D-3E59-4C51-8796-A22621DB4CF1.dmp /var/folders/pw/g0947b2n6v1_lkbjc6np_vyr000017/T/tmp9eoUZm
22:49:59 INFO - mozcrash Saved minidump as /Users/cltbld/tasks/task_1558991507/build/blobber_upload_dir/BD22881D-3E59-4C51-8796-A22621DB4CF1.dmp
22:50:00 INFO - PROCESS-CRASH | Main app process exited normally | application crashed [@ 0x7fff71616c2a]
22:50:00 INFO - Crash dump filename: /var/folders/pw/g0947b2n6v1_lkbjc6np_vyr000017/T/tmpFYjCr7.mozrunner/minidumps/BD22881D-3E59-4C51-8796-A22621DB4CF1.dmp
22:50:00 INFO - Operating system: Mac OS X
22:50:00 INFO - 10.14.0 18A391
22:50:00 INFO - CPU: amd64
22:50:00 INFO - family 6 model 69 stepping 1
22:50:00 INFO - 4 CPUs
22:50:00 INFO -
22:50:00 INFO - GPU: UNKNOWN
22:50:00 INFO -
22:50:00 INFO - Crash reason: EXC_SOFTWARE / SIGABRT
22:50:00 INFO - Crash address: 0x7fff71616c2a
22:50:00 INFO - Process uptime: 4935 seconds
22:50:00 INFO -
22:50:00 INFO - Thread 0 (crashed)
22:50:00 INFO - 0 0x7fff71616c2a
22:50:00 INFO - rax = 0x000000000100001f rdx = 0x0000000000000000
22:50:00 INFO - rcx = 0x00007ffeef33ce38 rbx = 0x0000000000000102
22:50:00 INFO - rsi = 0x0000000000000102 rdi = 0x00007ffeef33ced0
22:50:00 INFO - rbp = 0x00007ffeef33ce90 rsp = 0x00007ffeef33ce38
22:50:00 INFO - r8 = 0x0000000000008113 r9 = 0x0000000000001388
22:50:00 INFO - r10 = 0x000000000000041c r11 = 0x0000000000000202
22:50:00 INFO - r12 = 0x0000000000000102 r13 = 0x000000000000041c
22:50:00 INFO - r14 = 0x00007ffeef33ced0 r15 = 0x0000000000000000
22:50:00 INFO - rip = 0x00007fff71616c2a
22:50:00 INFO - Found by: given as instruction pointer in context
22:50:00 INFO - 1 XUL!google_breakpad::CrashGenerationClient::RequestDumpForException(int, int, int, unsigned int) [crash_generation_client.cc:944c410b7e9185a0cb90a4fbc0970299f1ff3e2b : 77 + 0x5c]
22:50:00 INFO - rbp = 0x00007ffeef33d740 rsp = 0x00007ffeef33cea0
22:50:00 INFO - rip = 0x000000010bf5e630
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 2 XUL!google_breakpad::ExceptionHandler::WriteMinidumpWithException(int, int, int, __darwin_ucontext, unsigned int, bool, bool) [exception_handler.cc:944c410b7e9185a0cb90a4fbc0970299f1ff3e2b : 382 + 0x1b]
22:50:00 INFO - rbp = 0x00007ffeef33d830 rsp = 0x00007ffeef33d750
22:50:00 INFO - rip = 0x000000010bf60385
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 3 XUL!google_breakpad::ExceptionHandler::SignalHandler(int, __siginfo*, void*) [exception_handler.cc:944c410b7e9185a0cb90a4fbc0970299f1ff3e2b : 628 + 0x21]
22:50:00 INFO - rbp = 0x00007ffeef33d860 rsp = 0x00007ffeef33d840
22:50:00 INFO - rip = 0x000000010bf609d7
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 4 0x7fff716c7b3d
22:50:00 INFO - rbp = 0x00007ffeef33d870 rsp = 0x00007ffeef33d870
22:50:00 INFO - rip = 0x00007fff716c7b3d
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 5 0x105f340b0
22:50:00 INFO - rbp = 0x00007ffeef33de40 rsp = 0x00007ffeef33d880
22:50:00 INFO - rip = 0x0000000105f340b0
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 6 libmozglue.dylib!<name omitted> [ConditionVariable_posix.cpp:944c410b7e9185a0cb90a4fbc0970299f1ff3e2b : 109 + 0x5]
22:50:00 INFO - rbp = 0x00007ffeef33de50 rsp = 0x00007ffeef33de50
22:50:00 INFO - rip = 0x0000000100c69c09
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 7 XUL!mozilla::ThreadEventQueue<mozilla::PrioritizedEventQueue>::GetEvent(bool, mozilla::EventQueuePriority*) [ThreadEventQueue.cpp:944c410b7e9185a0cb90a4fbc0970299f1ff3e2b : 153 + 0x51]
22:50:00 INFO - rbp = 0x00007ffeef33deb0 rsp = 0x00007ffeef33de60
22:50:00 INFO - rip = 0x0000000107665ce6
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 8 XUL!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:944c410b7e9185a0cb90a4fbc0970299f1ff3e2b : 1109 + 0x15]
22:50:00 INFO - rbp = 0x00007ffeef33e3a0 rsp = 0x00007ffeef33dec0
22:50:00 INFO - rip = 0x00000001076705d4
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 9 XUL!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:944c410b7e9185a0cb90a4fbc0970299f1ff3e2b : 486 + 0xd]
22:50:00 INFO - rbp = 0x00007ffeef33e3d0 rsp = 0x00007ffeef33e3b0
22:50:00 INFO - rip = 0x0000000107673649
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 10 XUL!mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) [MessagePump.cpp:944c410b7e9185a0cb90a4fbc0970299f1ff3e2b : 110 + 0xa]
22:50:00 INFO - rbp = 0x00007ffeef33e420 rsp = 0x00007ffeef33e3e0
22:50:00 INFO - rip = 0x0000000107d762ad
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 11 XUL!nsBaseAppShell::Run() [nsBaseAppShell.cpp:944c410b7e9185a0cb90a4fbc0970299f1ff3e2b : 137 + 0x45]
22:50:00 INFO - rbp = 0x00007ffeef33e460 rsp = 0x00007ffeef33e430
22:50:00 INFO - rip = 0x000000010a90ef6c
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 12 XUL!nsAppShell::Run() [nsAppShell.mm:944c410b7e9185a0cb90a4fbc0970299f1ff3e2b : 705 + 0x8]
22:50:00 INFO - rbp = 0x00007ffeef33e4a0 rsp = 0x00007ffeef33e470
22:50:00 INFO - rip = 0x000000010a993417
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 13 XUL!XRE_RunAppShell() [nsEmbedFunctions.cpp:944c410b7e9185a0cb90a4fbc0970299f1ff3e2b : 911 + 0x9]
22:50:00 INFO - rbp = 0x00007ffeef33e500 rsp = 0x00007ffeef33e4b0
22:50:00 INFO - rip = 0x000000010bfd15b2
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 14 XUL!XRE_InitChildProcess(int, char**, XREChildData const*) [message_loop.cc:944c410b7e9185a0cb90a4fbc0970299f1ff3e2b : 315 + 0xc]
22:50:00 INFO - rbp = 0x00007ffeef33e7f0 rsp = 0x00007ffeef33e510
22:50:00 INFO - rip = 0x000000010bfd119e
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 15 plugin-container!main [MozillaRuntimeMain.cpp:944c410b7e9185a0cb90a4fbc0970299f1ff3e2b : 23 + 0x2c]
22:50:00 INFO - rbp = 0x00007ffeef33e830 rsp = 0x00007ffeef33e800
22:50:00 INFO - rip = 0x00000001008c0f07
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO - 16 0x7fff714de085
22:50:00 INFO - rbp = 0x00007ffeef33e840 rsp = 0x00007ffeef33e840
22:50:00 INFO - rip = 0x00007fff714de085
22:50:00 INFO - Found by: previous frame's frame pointer
22:50:00 INFO -

:johannh - this appears to be a new failure unique to macosx1014 (mojave), and causes the mochitest-browser-screenshots chunk to time out after exhausting maximum runtime permitted. Would you be able to redirect the ni to someone that may know about this test?

Flags: needinfo?(jhofmann)

MattN might be interested in this but it's such low frequency that I don't think it warrants a needinfo...

Component: Site Identity and Permission Panels → mozscreenshots
Flags: needinfo?(jhofmann)
Product: Firefox → Testing

The frequency is low because it's a new platform not yet generally available, hence all tests run on that platform are triggered by myself. Once the tests are moved over to running on macosx1014 by default the failure for this test will increase in count.

:MattN - johannh mentioned that you might be interested in this issue. Browser screenshot tests are not able to complete running on macosx1014, instead timing out. The expected transition timeframe for rest of the macosx1010 machines to macosx1014 is end of this month. Could you take a look or pass on the ni to someone that may be able to comment, thanks.

Flags: needinfo?(MattN+bmo)

This is related to the accessibility API changes of 10.14 as I assume you didn't grant the mozscreenshots processes access. This is a widespread issue affecting automations on 10.14.

See https://bitsplitting.org/2018/07/11/reauthorizing-automation-in-mojave/ and https://www.felix-schwarz.org/blog/2018/08/new-apple-event-apis-in-macos-mojave etc.

I'm not sure if you can disable these restrictions on the machines. I imagine this affected other test harnesses too so you can probably use the same fix you used elsewhere.

Flags: needinfo?(MattN+bmo) → needinfo?(egao)

:MattN - thanks for input. You are correct, it seems that disabling the test isn't the solution, so in the meantime this test will remain running on macosx1010 until a fix for the test harness itself becomes available.

Flags: needinfo?(egao)

:dividehex - jmaher asked me to ping you to set up permission for mozscreenshots to run this test on 10.14.5.

Flags: needinfo?(jwatkins)

(In reply to Edwin Gao (:egao) from comment #8)

:dividehex - jmaher asked me to ping you to set up permission for mozscreenshots to run this test on 10.14.5.

I don't have a magic bullet to enable permissions for screenshots (AppleEvents) at the moment. It looks like it might be possible to allow this though the use of configuration profiles but the caveat is that it needs to be pushed via "a user approved MDM server in a device profile." We don't utilize mdm on the existing macos hardware. With that said, we are looking into mdm for possible future hardware purchases.

https://developer.apple.com/business/documentation/Configuration-Profile-Reference.pdf

Flags: needinfo?(jwatkins)

Given what has been said in this thread, perhaps browser-screenshots suite will have to continue running with macosx1010 machines until a solution can be figured out later down the road, either via infrastructure or by fixing the test.

I have been able to arrive at a solution via the following:

  • disable browser_controlCenter.js
  • disable browser_devtools.js
  • disable browser_permissionPrompts.js
  • disable browser_primaryUI.js

This results in a green mochitest-browser-screenshots run for macosx1014.

Enabling any one of these tests individually would cause the suite to time out at 60 minutes. Longer timeouts do not help, as the suite keeps expanding in time to consume any additional timeouts that are given.

The current proposal is to disable the tests mentioned above, which includes test detailed in this bug, so that browser-screenshots job can be moved to macosx1014 leading to decommisioning of macosx1010 machines.

:MattN - do you have thoughts for/against having this test disabled as outlined in the comment above?

Flags: needinfo?(MattN+bmo)

Can we combine bug 1570102 and this bug since the combination of the two bugs would mean disabling all of these tests that run automatically on pushes?

Flags: needinfo?(MattN+bmo)

:MattN - sure, I can combine the the other bug into this bug.

I've submitted a try push that re-enables these four tests on macosx1014. I'm not expecting them to pass, but hopefully it will provide a good starting point for the debugging of these tests to be re-enabled.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0866b8d25781551b84eb15b493f6b8bdfeb1ed30

:dividehex - needinfo'ing you to see if anything has changed with regards to 'MDM hardware' you mentioned back in comment 9. I am not well versed on this matter; is the hardware server an absolute requirement to address the series of timeouts in the screenshot related tests?

Flags: needinfo?(jwatkins)

(In reply to Edwin Takahashi (:egao, :etakahashi) from comment #14)

:dividehex - needinfo'ing you to see if anything has changed with regards to 'MDM hardware' you mentioned back in comment 9. I am not well versed on this matter; is the hardware server an absolute requirement to address the series of timeouts in the screenshot related tests?

Nothing has changed since comment 9 and there is no immediate plans to setup an MDM service. If this is a priority, let :fubar know so we can discuss it and plan it out.

Flags: needinfo?(jwatkins)
Priority: P5 → P3
Summary: OSX Mojave (10.14) - browser/tools/mozscreenshots/controlCenter/browser_controlCenter.js | Test timed out - → OSX Mojave (10.14) - browser/tools/mozscreenshots/ - Can't capture screenshots without macOS screenshot permission
Severity: normal → S3
See Also: → 2051353
You need to log in before you can comment on or make changes to this bug.