1697345 - Perma Android 8.0 jsreftest failures, e.g. js/src/tests/non262/<test> | load failed: timed out waiting for reftest-wait to be removed | test must provide a function getTestCases().

Status: RESOLVED WONTFIX

(Core :: JavaScript Engine, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: malexandru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=332610251&repo=mozilla-beta
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JBqLC-2yQ9CHGwwrO7Ys2w/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JBqLC-2yQ9CHGwwrO7Ys2w/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

[task 2021-03-09T23:12:45.654Z] 23:12:45     INFO -  REFTEST TEST-PASS | js/src/tests/non262/Array/regress-451906.js | Index array by numeric string  item 1
[task 2021-03-09T23:12:45.654Z] 23:12:45     INFO -  REFTEST TEST-END | js/src/tests/non262/Array/regress-451906.js
[task 2021-03-09T23:12:45.654Z] 23:12:45     INFO -  REFTEST TEST-START | js/src/tests/non262/Array/regress-456845.js
[taskcluster 2021-03-09T23:17:38.360Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/JBqLC-2yQ9CHGwwrO7Ys2w/0/on/bitbar/pixel2-25/until/1615333058.336
[task 2021-03-09T23:17:52.659Z] 23:12:45     INFO -  REFTEST TEST-LOAD | http://10.7.205.225:8854/jsreftest/tests/js/src/tests/jsreftest.html?test=non262/Array/regress-456845.js | 94 / 7216 (1%)
[task 2021-03-09T23:17:52.659Z] 23:17:52  WARNING -  REFTEST TEST-UNEXPECTED-FAIL | js/src/tests/non262/Array/regress-456845.js | load failed: timed out waiting for reftest-wait to be removed
[task 2021-03-09T23:17:52.659Z] 23:17:52     INFO -  REFTEST INFO | Saved log: START http://10.7.205.225:8854/jsreftest/tests/js/src/tests/jsreftest.html?test=non262/Array/regress-456845.js
[task 2021-03-09T23:17:52.659Z] 23:17:52     INFO -  REFTEST INFO | Saved log: [CONTENT] OnDocumentLoad triggering WaitForTestEnd
[task 2021-03-09T23:17:52.659Z] 23:17:52     INFO -  REFTEST INFO | Saved log: [CONTENT] WaitForTestEnd: Adding listeners
[task 2021-03-09T23:17:52.659Z] 23:17:52     INFO -  REFTEST INFO | Saved log: Initializing canvas snapshot
[task 2021-03-09T23:17:52.659Z] 23:17:52     INFO -  REFTEST INFO | Saved log: [CONTENT] MakeProgress
[task 2021-03-09T23:17:52.659Z] 23:17:52     INFO -  REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_TO_FIRE_INVALIDATE_EVENT
[task 2021-03-09T23:17:52.659Z] 23:17:52     INFO -  REFTEST INFO | Saved log: [CONTENT] MakeProgress: dispatching MozReftestInvalidate
[task 2021-03-09T23:17:52.659Z] 23:17:52     INFO -  REFTEST INFO | Saved log: [CONTENT] MakeProgress
[task 2021-03-09T23:17:52.659Z] 23:17:52     INFO -  REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_FOR_REFTEST_WAIT_REMOVAL
[task 2021-03-09T23:17:52.659Z] 23:17:52     INFO -  REFTEST INFO | Saved log: [CONTENT] MakeProgress: waiting for reftest-wait to be removed
[task 2021-03-09T23:17:52.659Z] 23:17:52     INFO -  REFTEST TEST-END | js/src/tests/non262/Array/regress-456845.js
[task 2021-03-09T23:17:52.659Z] 23:17:52     INFO -  REFTEST TEST-START | js/src/tests/non262/Array/regress-465980-01.js```

Comment 2

[Intermittent Failures Robot]

25 failures in 838 pushes (0.03 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 17
• mozilla-beta: 8

Platform and build breakdown:

• android-hw-p2-8-0-arm7-api-16: 19
  • debug: 16
  • opt: 3
• android-hw-p2-8-0-arm7-api-16-shippable: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697345&startday=2021-03-11&endday=2021-03-11&tree=all

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

• Try push with recent central revision: hits the issue
• Try push based on https://hg.mozilla.org/mozilla-central/rev/5f0f6477c734369a72fec1211b608eb14d33bd4a - hits the issue
• central push for above revision - didn't hit issue.

Comment 4

[Intermittent Failures Robot]

18 failures in 846 pushes (0.021 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 18

Platform and build breakdown:

• android-hw-p2-8-0-arm7-api-16: 10
  • debug: 10
• android-hw-p2-8-0-arm7-api-16-shippable: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697345&startday=2021-03-12&endday=2021-03-12&tree=all

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The frequent failures also affect previous pushes whose initially executed jsreftests were successful. The tasks which succeeded and failed share the same dependencies and artifacts. Some of them got updated the day before the failures started by the changes in bug 1695773.

Comment 6

[Intermittent Failures Robot]

96 failures in 4622 pushes (0.021 failures/push) were associated with this bug in the last 7 days.

This is the #9 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 21
• mozilla-central: 60
• autoland: 15

Platform and build breakdown:

• android-hw-p2-8-0-arm7-api-16-shippable: 33
  • opt: 33
• android-hw-p2-8-0-arm7-api-16: 63
  • debug: 60
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697345&startday=2021-03-08&endday=2021-03-14&tree=all

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

These Android 8.0 arm7 tests continue to fail very frequently. This can be also observed for mozilla-release and hence shouldn't be related to a check-in. Can you check with Bitbar again regarding differences to the gecko-t-bitbar-gw-unit-p2 pool?

Comment 8

[Andrew Erickson [:aerickson]]

Yes, I'll check with Bitbar again.

It's confusing that the test executor continues to successfully execute tests after seeing a few failures... could the test framework just be firing jobs too quickly?

example: https://treeherder.mozilla.org/logviewer?job_id=333423674&repo=mozilla-central&lineNumber=14529

• "TEST-PASS" instances happen after the error "test must provide a function getTestCases()." occurs

Comment 9

[Andrew Erickson [:aerickson]]

The Bitbar issues were due to networking. Sakari is relatively confident all issues are resolved.

These jobs are running in the unit-p2 pool (which is the same as the other pools). Bitbar doesn't actually know/manage what pools our devices are in, it's strictly something we control in our provisioner (devicepool) layer. Bitbar's infrastructure connects 6-8 phones to a Docker host. If we saw a concentrated subset of devices (e.g. pixel2-490 could likely be on the same docker host as pixe2-492) that were having issues, it could be due to a bad Docker host. I didn't see any particular hot spots for this bug (some hosts have fewer failures, but not sure if we have a statistically significant amount of runs).

How do the jsreftests work? It seems from the logs that the tests push everything to the device and then use the local filesystem to load the test cases. That would seem to rule out any networking-related issues.

Comment 10

[Andrew Erickson [:aerickson]]

This can be also observed for mozilla-release and hence shouldn't be related to a check-in.

The linked mozilla-release failures are due to the task exceeding 1 hour (Bug 1697835. I'm not sure why that's happening. I didn't see any artifact fetching failures.), not "test must provide a function getTestCases()."

Have we seen this particular error in jobs run on other trees?

Comment 11

[Andrew Erickson [:aerickson]]

(In reply to Andrew Erickson [:aerickson] from comment #10)

The linked mozilla-release failures are due to the task exceeding 1 hour (Bug 1697835. I'm not sure why that's happening. I didn't see any artifact fetching failures.), not "test must provide a function getTestCases()."

Have we seen this particular error in jobs run on other trees?

Aah, I do see that "test must provide a function getTestCases()." is mentioned in two of the failures. I was looking at the ultimate reason for failure (the mozilla-release jobs fail due to a timeout, while the central jobs are reaching the end of the run and failing due to this bug). It seems the networking issues related to fetching mentioned in Bug 1697835 are better, but this issue persists.

Comment 12

[Intermittent Failures Robot]

17 failures in 754 pushes (0.023 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 6
• try: 11

Platform and build breakdown:

• android-hw-p2-8-0-arm7-api-16: 10
  • debug: 5
  • opt: 5
• android-hw-p2-8-0-arm7-api-16-shippable: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697345&startday=2021-03-17&endday=2021-03-17&tree=all

Comment 13

[Intermittent Failures Robot]

29 failures in 4257 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

This is the #41 most frequent failure this week.

Repository breakdown:

• mozilla-central: 17
• try: 11
• mozilla-release: 1

Platform and build breakdown:

• android-hw-p2-8-0-arm7-api-16-shippable: 12
  • opt: 12
• android-hw-p2-8-0-arm7-api-16: 17
  • debug: 11
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697345&startday=2021-03-15&endday=2021-03-21&tree=all

Comment 14

[Intermittent Failures Robot]

6 failures in 3967 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2
• mozilla-central: 4

Platform and build breakdown:

• android-hw-p2-8-0-arm7-api-16-shippable: 2
  • opt: 2
• android-hw-p2-8-0-arm7-api-16: 4
  • opt: 1
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697345&startday=2021-03-22&endday=2021-03-28&tree=all

Comment 16

[Intermittent Failures Robot]

3 failures in 3125 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• mozilla-beta: 1

Platform and build breakdown:

• android-hw-p2-8-0-arm7-api-16-shippable: 2
  • opt: 2
• android-hw-p2-8-0-arm7-api-16: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697345&startday=2021-03-29&endday=2021-04-04&tree=all

Comment 17

[Intermittent Failures Robot]

2 failures in 3854 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• android-hw-p2-8-0-arm7-api-16: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697345&startday=2021-04-05&endday=2021-04-11&tree=all

Comment 18

[Intermittent Failures Robot]

2 failures in 4255 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• android-hw-p2-8-0-arm7-api-16: 1
  • debug: 1
• android-hw-p2-8-0-arm7-api-16-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697345&startday=2021-04-12&endday=2021-04-18&tree=all

Comment 19

[Intermittent Failures Robot]

5 failures in 3224 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5

Platform and build breakdown:

• android-hw-p2-8-0-arm7-api-16: 4
  • debug: 4
• android-hw-p2-8-0-arm7-api-16-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697345&startday=2021-04-19&endday=2021-04-25&tree=all

Comment 20

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

jsreftests on AArch64 got moved to macOS in bug 1692091.