Closed Bug 1538725 Opened 9 months ago Closed 8 months ago

aarch64 is too slow to run many wpt in less than 10 seconds

Categories

(Testing :: web-platform-tests, enhancement)

Version 3
enhancement
Not set

Tracking

(firefox68 fixed)

RESOLVED FIXED
mozilla68
Tracking Status
firefox68 --- fixed

People

(Reporter: bwc, Assigned: egao)

References

Details

Attachments

(1 file)

A lot of web-platform-tests have been disabled on aarch64, and it looks like this is simply because the test machine is too slow. Maybe we need to increase the default timeout on this platform?

Duplicate of this bug: 1536917
Duplicate of this bug: 1536916
Duplicate of this bug: 1536919
Duplicate of this bug: 1538283
Duplicate of this bug: 1536922
Duplicate of this bug: 1536925
Duplicate of this bug: 1536931

Once we try this out, we should try backing out bug 1536906 and see if there is still stuff that needs to be disabled.

I think from bug 1536919 it is clear there is a simple way to extend the default timeout for win/aarch64, I think we should double it.

Currently we only run on opt, and that might be all we do for a while. Given that, we should make this run like we do for debug (multiplier of 3 or 4):
https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py#46

:egao, is this something you could look at and try removing some of the disable/timeout conditions already set.

Flags: needinfo?(egao)

Pushed a try run with the following changes:

  • multiplier set to 6x if windows10-aarch64
  • beacon tests re-enabled

https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=5ddaed69222a33634109babe1c2a8c6bf1a0427a

If the beacon tests then pass, we may be able to investigate other web-platform-tests that were disabled.

Flags: needinfo?(egao)

Do we really expect it to be 6x slower than x86?

Note that we always have some tests timing out, so setting this value too high is also problematic.

I figured 6x would be a good starting point, because past experience with mochitest shows that aarch64 could be quite a bit slower than x86_64. The value will be tweaked, and will probably end up closer to 2-4x but for the time being, my goal is to test the hypothesis of longer timeout preventing the beacon tests from failing.

As :jgraham suspected, try push results from comment 10 shows some tests that now expected a TIMEOUT instead experiences a CRASH with 6x timeout multiplier.

I'm trying to pin down a good number for windows10-aarch64 with multiple try pushes containing 2x, 3x and 4x multipliers.

On the other hand, beacon tests as well as a test previously that consistently timed out RTCPeerConnection-getStats.https.html and both scenarios now produced the expected behavior (instead of TIMEOUT).


My concern with this approach (to extend timeout multiplier for certain platforms) is this delicate balancing act, which we won't achieve 100% satisfaction either way. Some tests rely on it timing out, others require longer timeouts as we've seen. It doesn't seem feasible to set one global multiplier, whether it may be 1 or 100. This also means in the future as windows10-aarch64 is optimized and faster hardware becomes available this multiplier is bound to break at that point.

Setting a single global multiplier is indeed a tradeoff, but I'm not sure a better option exists. Indeed if hardware gets faster the timeout will be too long, but that's relatively easy to adjust compared to many per-test multipliers. Picking a single number is hard, but I hope we have a feeling for what X is in "this hardware is about X times slower than the AWS x86 instances we are using". If we know X then the multiplier should be set to that.

Results are in for 2x, 3x, 4x and 6x runs for web-platform-test-10 on windows10-aarch64:

2x:

17:59:23     INFO - TEST-UNEXPECTED-OK | /html/infrastructure/urls/resolving-urls/query-encoding/navigation.sub.html?encoding=windows-1252 | expected TIMEOUT
17:59:38     INFO - TEST-UNEXPECTED-OK | /html/infrastructure/urls/resolving-urls/query-encoding/navigation.sub.html?encoding=utf8 | expected TIMEOUT
17:59:54     INFO - TEST-UNEXPECTED-OK | /html/infrastructure/urls/resolving-urls/query-encoding/navigation.sub.html?encoding=x-cp1251 | expected TIMEOUT
18:19:27     INFO - TEST-UNEXPECTED-TIMEOUT | /webrtc/RTCPeerConnection-getStats.https.html | getStats() on track associated with RtpReceiver should return stats report containing inbound-rtp stats - Test timed out
18:19:27     INFO - TEST-UNEXPECTED-TIMEOUT | /webrtc/RTCPeerConnection-getStats.https.html | expected OK

3x:

17:49:10     INFO - TEST-UNEXPECTED-OK | /html/infrastructure/urls/resolving-urls/query-encoding/navigation.sub.html?encoding=windows-1252 | expected TIMEOUT
17:49:26     INFO - TEST-UNEXPECTED-OK | /html/infrastructure/urls/resolving-urls/query-encoding/navigation.sub.html?encoding=utf8 | expected TIMEOUT
17:49:41     INFO - TEST-UNEXPECTED-OK | /html/infrastructure/urls/resolving-urls/query-encoding/navigation.sub.html?encoding=x-cp1251 | expected TIMEOUT

4x:

18:06:35     INFO - TEST-UNEXPECTED-OK | /html/infrastructure/urls/resolving-urls/query-encoding/navigation.sub.html?encoding=windows-1252 | expected TIMEOUT
18:06:51     INFO - TEST-UNEXPECTED-OK | /html/infrastructure/urls/resolving-urls/query-encoding/navigation.sub.html?encoding=utf8 | expected TIMEOUT
18:07:07     INFO - TEST-UNEXPECTED-OK | /html/infrastructure/urls/resolving-urls/query-encoding/navigation.sub.html?encoding=x-cp1251 | expected TIMEOUT

6x:

01:28:46     INFO - TEST-UNEXPECTED-CRASH | /WebCryptoAPI/wrapKey_unwrapKey/wrapKey_unwrapKey.https.worker.html | expected TIMEOUT
01:54:53     INFO - TEST-UNEXPECTED-OK | /html/infrastructure/urls/resolving-urls/query-encoding/navigation.sub.html?encoding=windows-1252 | expected TIMEOUT
01:55:10     INFO - TEST-UNEXPECTED-OK | /html/infrastructure/urls/resolving-urls/query-encoding/navigation.sub.html?encoding=utf8 | expected TIMEOUT
01:55:27     INFO - TEST-UNEXPECTED-OK | /html/infrastructure/urls/resolving-urls/query-encoding/navigation.sub.html?encoding=x-cp1251 | expected TIMEOUT

I will go ahead and run other web-platform-test chunks for the 3x and 4x as they hold the most promise.

All of web-platform-test suites were run:

3x

https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=b9dd3345b33fd9b62290984ea67fc2cfca0289c1

4x

https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=fd24a73c89e3b6b81688a76988a9af85215adaef

In its current state, with various tests disabled for windows10-aarch64, both timeout multipliers are a wash. With that noted, I think it is a good idea to go with 4x multiplier.

Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/173bc2937a7b
extend web-platform-tests multiplier to 4x for windows10-aarch64 r=bwc,jmaher
Blocks: 1525434
Status: NEW → RESOLVED
Closed: 8 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/16190 for changes under testing/web-platform/tests
You need to log in before you can comment on or make changes to this bug.