1389500 - |mach mochitest --verify| fails: KeyError: 'pop from an empty set'

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Geoff Brown [:gbrown]]

Error running mach:

    ['mochitest', 'testing/mochitest/tests/Harness_sanity/test_sanity.html', '--verify']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

KeyError: 'pop from an empty set'

  File "/home/gbrown/src/testing/mochitest/mach_commands.py", line 409, in run_mochitest_general
    **harness_args)
  File "/home/gbrown/src/testing/mochitest/mach_commands.py", line 159, in run_desktop_test
    result = mochitest.run_test_harness(parser, options)
  File "/home/gbrown/objdirs/firefox/_tests/testing/mochitest/runtests.py", line 2917, in run_test_harness
    result = runner.verifyTests(options)
  File "/home/gbrown/objdirs/firefox/_tests/testing/mochitest/runtests.py", line 2407, in verifyTests
    result = step()
  File "/home/gbrown/objdirs/firefox/_tests/testing/mochitest/runtests.py", line 2351, in step2
    result = self.runTests(stepOptions)
  File "/home/gbrown/objdirs/firefox/_tests/testing/mochitest/runtests.py", line 2468, in runTests
    prefs = self.prefs_by_manifest[m].pop()

Comment 1

[Geoff Brown [:gbrown]]

Attachment: save and restore prefs_by_manifest

In runtests.py, self.prefs_by_manifest is populated in getActiveTests() and
its elements are removed in runTests(). runTests() calls getActiveTests()
so I'd expect that to be safe, but getActiveTests() has an early return when
called more than once:

https://dxr.mozilla.org/mozilla-central/rev/5322c03f4c8587fe526172d3f87160031faa6d75/testing/mochitest/runtests.py#1394

In --verify mode, runTests() is called multiple times. The first call to runTests() succeeds, but the second one fails.

Here's a simple, brute-force fix that works for me; I'm certainly open to alternatives.

Comment 2

[Andrew Halberstadt [:ahal]]

Comment on attachment 8896288 [details] [diff] [review]
save and restore prefs_by_manifest

Review of attachment 8896288 [details] [diff] [review]:
-----------------------------------------------------------------

Instead of pop() let's just do:
prefs = list(self.prefs_by_manifest[m])[0]

Comment 3

[Geoff Brown [:gbrown]]

Attachment: avoid pop from prefs_by_manifest

Thanks!

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/28d22d9b12a5
Fix |mochitest --verify| by not popping prefs; r=ahal

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/28d22d9b12a5