Open Bug 1030419 Opened 6 years ago Updated 5 years ago

Intermittent Android 4.3 (aws) /tests/layout/style/test/test_media_queries.html | Test timed out.

Categories

(Core :: CSS Parsing and Computation, defect, P5)

x86_64
Linux
defect

Tracking

()

People

(Reporter: gbrown, Unassigned)

References

Details

(Whiteboard: [test disabled on Android 4.3])

test_media_queries.html has been running reliably on Android 2.3, in an emulator run on ix slaves. We are now moving Android 2.3 mochitests to aws slaves. On aws, test_media_queries.html intermittently fails:

https://tbpl.mozilla.org/php/getParsedLog.php?id=42473078&full=1&branch=ash#error0

14:14:43     INFO -  177 INFO TEST-START | /tests/layout/style/test/test_media_queries.html
14:14:43     INFO -  178 INFO TEST-INFO | dumping last 100 message(s)
14:14:43     INFO -  179 INFO TEST-INFO | if you need more context, please use SimpleTest.requestCompleteLog() in your test
14:14:43     INFO -  180 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | Media query 'not all and (min-resolution: 39dpcm)' should apply after cloning
14:14:43     INFO -  181 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (scan)' should not change serialization
14:14:43     INFO -  182 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (scan: progressive)' should not change serialization
14:14:43     INFO -  183 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (scan:interlace)' should not change serialization
14:14:43     INFO -  184 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query '(scan)' should not change serialization
14:14:43     INFO -  185 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | Media query '(scan)' should NOT apply after cloning
14:14:43     INFO -  186 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query '(scan: progressive)' should not change serialization
14:14:43     INFO -  187 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | Media query '(scan: progressive)' should NOT apply after cloning
14:14:43     INFO -  188 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query '(scan: interlace)' should not change serialization
14:14:43     INFO -  189 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | Media query '(scan: interlace)' should NOT apply after cloning
14:14:43     INFO -  190 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'not all and (scan)' should not change serialization
14:14:43     INFO -  191 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | Media query 'not all and (scan)' should apply after cloning
14:14:43     INFO -  192 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'not all and (scan: progressive)' should not change serialization
14:14:43     INFO -  193 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | Media query 'not all and (scan: progressive)' should apply after cloning
14:14:43     INFO -  194 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'not all and (scan: interlace)' should not change serialization
14:14:43     INFO -  195 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | Media query 'not all and (scan: interlace)' should apply after cloning
14:14:43     INFO -  196 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (grid)' should not change serialization
14:14:43     INFO -  197 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (grid: 0)' should not change serialization
14:14:43     INFO -  198 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (grid: 1)' should not change serialization
14:14:43     INFO -  199 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (grid: 1)' should not change serialization
14:14:43     INFO -  200 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query '(grid)' should not change serialization
14:14:43     INFO -  201 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | Media query '(grid)' should NOT apply after cloning
14:14:43     INFO -  202 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query '(grid: 0)' should not change serialization
14:14:43     INFO -  203 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | Media query '(grid: 0)' should apply after cloning
14:14:43     INFO -  204 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query '(grid: 1)' should not change serialization
14:14:43     INFO -  205 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | Media query '(grid: 1)' should NOT apply after cloning
14:14:43     INFO -  206 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query '(grid: 2)' should not change serialization
14:14:43     INFO -  207 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | Media query '(grid: 2)' should NOT apply after cloning
14:14:43     INFO -  208 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query '(grid: -1)' should not change serialization
14:14:43     INFO -  209 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | Media query '(grid: -1)' should NOT apply after cloning
14:14:43     INFO -  210 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-start-backward)' should not change serialization
14:14:43     INFO -  211 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-start-forward)' should not change serialization
14:14:43     INFO -  212 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-end-backward)' should not change serialization
14:14:43     INFO -  213 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-end-forward)' should not change serialization
14:14:43     INFO -  214 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-thumb-proportional)' should not change serialization
14:14:43     INFO -  215 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-images-in-menus)' should not change serialization
14:14:43     INFO -  216 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-images-in-buttons)' should not change serialization
14:14:43     INFO -  217 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-overlay-scrollbars)' should not change serialization
14:14:43     INFO -  218 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-default-theme)' should not change serialization
14:14:43     INFO -  219 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-mac-graphite-theme)' should not change serialization
14:14:43     INFO -  220 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-mac-lion-theme)' should not change serialization
14:14:43     INFO -  221 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-mac-yosemite-theme)' should not change serialization
14:14:43     INFO -  222 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-compositor)' should not change serialization
14:14:43     INFO -  223 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-classic)' should not change serialization
14:14:43     INFO -  224 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-glass)' should not change serialization
14:14:43     INFO -  225 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-touch-enabled)' should not change serialization
14:14:43     INFO -  226 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-swipe-animation-enabled)' should not change serialization
14:14:43     INFO -  227 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-start-backward: 0)' should not change serialization
14:14:43     INFO -  228 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-start-forward: 0)' should not change serialization
14:14:43     INFO -  229 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-end-backward: 0)' should not change serialization
14:14:43     INFO -  230 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-end-forward: 0)' should not change serialization
14:14:43     INFO -  231 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-thumb-proportional: 0)' should not change serialization
14:14:43     INFO -  232 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-images-in-menus: 0)' should not change serialization
14:14:43     INFO -  233 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-images-in-buttons: 0)' should not change serialization
14:14:43     INFO -  234 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-overlay-scrollbars: 0)' should not change serialization
14:14:43     INFO -  235 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-default-theme: 0)' should not change serialization
14:14:43     INFO -  236 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-mac-graphite-theme: 0)' should not change serialization
14:14:43     INFO -  237 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-mac-lion-theme: 0)' should not change serialization
14:14:43     INFO -  238 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-mac-yosemite-theme: 0)' should not change serialization
14:14:43     INFO -  239 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-compositor: 0)' should not change serialization
14:14:43     INFO -  240 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-classic: 0)' should not change serialization
14:14:43     INFO -  241 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-glass: 0)' should not change serialization
14:14:43     INFO -  242 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-touch-enabled: 0)' should not change serialization
14:14:43     INFO -  243 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-swipe-animation-enabled: 0)' should not change serialization
14:14:43     INFO -  244 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-start-backward: 1)' should not change serialization
14:14:43     INFO -  245 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-start-forward: 1)' should not change serialization
14:14:43     INFO -  246 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-end-backward: 1)' should not change serialization
14:14:43     INFO -  247 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-end-forward: 1)' should not change serialization
14:14:43     INFO -  248 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-scrollbar-thumb-proportional: 1)' should not change serialization
14:14:43     INFO -  249 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-images-in-menus: 1)' should not change serialization
14:14:43     INFO -  250 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-images-in-buttons: 1)' should not change serialization
14:14:43     INFO -  251 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-overlay-scrollbars: 1)' should not change serialization
14:14:43     INFO -  252 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-default-theme: 1)' should not change serialization
14:14:43     INFO -  253 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-mac-graphite-theme: 1)' should not change serialization
14:14:43     INFO -  254 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-mac-lion-theme: 1)' should not change serialization
14:14:43     INFO -  255 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-mac-yosemite-theme: 1)' should not change serialization
14:14:43     INFO -  256 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-compositor: 1)' should not change serialization
14:14:43     INFO -  257 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-classic: 1)' should not change serialization
14:14:43     INFO -  258 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-glass: 1)' should not change serialization
14:14:43     INFO -  259 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-touch-enabled: 1)' should not change serialization
14:14:43     INFO -  260 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-swipe-animation-enabled: 1)' should not change serialization
14:14:43     INFO -  261 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-theme: aero)' should not change serialization
14:14:43     INFO -  262 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-theme: aero-lite)' should not change serialization
14:14:43     INFO -  263 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-theme: luna-blue)' should not change serialization
14:14:43     INFO -  264 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-theme: luna-olive)' should not change serialization
14:14:43     INFO -  265 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-theme: luna-silver)' should not change serialization
14:14:43     INFO -  266 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-theme: royale)' should not change serialization
14:14:43     INFO -  267 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-theme: generic)' should not change serialization
14:14:43     INFO -  268 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-theme: zune)' should not change serialization
14:14:43     INFO -  269 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-windows-theme: garbage)' should not change serialization
14:14:43     INFO -  270 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-os-version: windows-xp)' should not change serialization
14:14:43     INFO -  271 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-os-version: windows-vista)' should not change serialization
14:14:43     INFO -  272 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-os-version: windows-win7)' should not change serialization
14:14:43     INFO -  273 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-os-version: windows-win8)' should not change serialization
14:14:43     INFO -  274 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query '(-moz-is-glyph)' should not change serialization
14:14:43     INFO -  275 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-is-glyph)' should not change serialization
14:14:43     INFO -  276 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'not all and (-moz-is-glyph)' should not change serialization
14:14:43     INFO -  277 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'only all and (-moz-is-glyph)' should not change serialization
14:14:43     INFO -  278 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query '(-moz-is-glyph:0)' should not change serialization
14:14:43     INFO -  279 INFO TEST-PASS | /tests/layout/style/test/test_media_queries.html | cloning query 'all and (-moz-is-glyph:0)' should not change serialization
14:14:43     INFO -  280 INFO TEST-UNEXPECTED-FAIL | /tests/layout/style/test/test_media_queries.html | Test timed out.
14:14:43     INFO -  281 INFO TEST-INFO | MEMORY STAT vsize after test: 425168896
14:14:43     INFO -  282 INFO TEST-INFO | MEMORY STAT residentFast after test: 174690304
14:14:43     INFO -  283 INFO TEST-INFO | MEMORY STAT heapAllocated after test: 33629824
14:14:43     INFO -  284 INFO TEST-END | /tests/layout/style/test/test_media_queries.html | finished in 320264ms
Do we know how many CPU cycles per second of system clock advance we're guaranteed in this testing setup?  (In particular, is it a number greater than 0?)
Flags: needinfo?(gbrown)
See Also: → 1030432
I do not know of any such guarantee.

But, except for a handful of exceptions, all of the other mochitests seem to run reliably on Android 2.3 on aws, usually just a little slower than on ix.
Flags: needinfo?(gbrown)
Was this test almost at the timeout threshold before?  Is the problem here that it's intermittently crossing just over the boundary, or intermittently hanging or taking massively longer?

(One of the reasons we explicitly stopped running tests on VMs about 5 years ago was that our test timeouts were in system clock time, and the VMs we were using didn't seem to offer any guarantee about how much work we could do per any advance in wall clock time, so we had low frequency intermittent timeouts of just about everything.)
Disabled on 2.3, so we can switch to aws: https://hg.mozilla.org/integration/mozilla-inbound/rev/2780e1149082
Whiteboard: [leave open] [test disabled on Android 2.3]
I don't think this test should be disabled without a better understanding of why it's timing out and whether the platform on which it's running is actually suitable for running our test suites.  Could you at least answer the questions in comment 3?
Flags: needinfo?(gbrown)
Sorry David. I have not communicated well or even sufficiently here.

Let me try to clarify what I'm doing and how we got here.

Historically, we have run our Android tests on Tegra boards (Android 2.2) and Panda boards (Android 4.0). The older Tegra boards are now in short supply, prone to failure, and cannot be replaced.

Independent of test considerations, the Firefox for Android team has decided to end support for Android 2.2, making Android 2.3 the earliest supported version.

For some months now, we have worked to set up a new test environment based on the Android emulator: Android 2.3 is run in the Android arm emulator, and the emulator is run on an in-house ix slave running Linux. This removes the need and cost of purchasing, housing, cabling, etc device boards. As on any new test platform, there have been numerous new cases of test failures -- some of those have been investigated and resolved, but others remain. Bug 979921 tracks tests failures on the new Android 2.3 platform. Many of the remaining failures are consequences of bug 1025968 (SwapBuffers crash seen in many robocop tests) and bug 1020227 (deadlock in opensl_stream_destroy).

A few weeks ago, we thought that most of the test failures and all other issues were sorted out and we started running a full suite of Android 2.3 tests on trunk trees on the emulator, on ix slaves. Test failure rates have been quite low (typically 10x lower than Android 2.2 tegra rates). Test run times for reftests have been about 3x slower than tegras, while most other tests, like mochitests, have run times similar to tegras.

However, when we tried to introduce Android 2.3 *armv6* tests on trunk trees, we ran into a capacity problem: We don't have enough ix slaves to service the test load. To avoid the ix capacity issue, we looked at running on aws (preferred also for scalability and cost). Running reftests on emulator-on-aws seems impractical -- they take at least 4x as long as on ix. But other test types run in about the same time (looking at the average time for test jobs), or up to 50% longer (we had to increase mochitest chunks from 8 to 12 for aws). Running {mochitest, robocop, xpcshell} on aws seems practical: With a few exceptions, like this test, tests run reliably and just a little slower on average than on ix.


If we disable or fix this test and a few others on Android 2.3 on aws, we can run Android 2.3 and Android 2.3 armv6 tests on trunk trees, retire the Android 2.2 tests on Tegras, and the Firefox for Android team can end support for Android 2.2 in Firefox 33, or perhaps even 32. An alternative is to order more ix slaves, but there is both cost and delay involved in that decision and that seems unwarranted if only a handful of tests fail on aws.

I wish we could have identified this failure earlier in this project, but the ix capacity issue and switch to aws was a last minute surprise. Now there is a lot of pressure to get these tests running on trunk ASAP.


Now, as for test_media_queries...

> Was this test almost at the timeout threshold before?  Is the problem here that it's intermittently 
> crossing just over the boundary, or intermittently hanging or taking massively longer?

It is taking much longer on 2.3-aws compared to 2.3-ix or 2.2-tegra:

2.3-aws:   test_media_queries.html | finished in 320264ms (timeout)
2.3-ix:    test_media_queries.html | finished in 62252ms
2.2-tegra: test_media_queries.html | finished in 25131ms

On 2.3-aws, it fails frequently (40%?). When it passes, it takes a long time, but typically 60 seconds less than the timeout threshold:

2.3-aws:   test_media_queries.html | finished in 266679ms (pass)
2.3-aws:   test_media_queries.html | finished in 243644ms (pass)
2.3-aws:   test_media_queries.html | finished in 260503ms (pass)

There are several retries of Android 2.3 Armv6 Opt mochitest-12 at https://tbpl.mozilla.org/?tree=Ash&rev=c135c74b985e, showing both failures and passes.

Let me know if I can provide any other information -- I want to see this running again too.
Flags: needinfo?(gbrown)
Priority: -- → P5
See Also: → 1118001
Whiteboard: [leave open] [test disabled on Android 2.3] → [leave open] [test disabled on Android 2.3] [test disabled on Android 4.3]
Android 2.3 is no longer supported in Firefox 48+.

Test manifests were updated in bug 1251013.

This test remains disabled on Android 4.3.
Whiteboard: [leave open] [test disabled on Android 2.3] [test disabled on Android 4.3] → [test disabled on Android 4.3]
Summary: Intermittent Android 2.3 (aws) /tests/layout/style/test/test_media_queries.html | Test timed out. → Intermittent Android 4.3 (aws) /tests/layout/style/test/test_media_queries.html | Test timed out.
You need to log in before you can comment on or make changes to this bug.