1182817 - In some cases, running tests on Try with the --tag option blows up with python error

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

See [1]. My goal was to run just the tests in gfx/layers/apz/test/mochitest.ini via "--tag apz" with extra logging enabled so that I could investigate an intermittent failure that I can't reproduce locally. Some of the tests ran ok, in chunks 2 and 5. But the one test I cared about (test_basic_pan.html) didn't get run, and instead chunk 8 has some python error:

21:47:20     INFO -  Traceback (most recent call last):
21:47:20     INFO -    File "runtestsb2g.py", line 481, in run_remote_mochitests
21:47:20     INFO -      retVal = mochitest.run_tests(options)
21:47:20     INFO -    File "runtestsb2g.py", line 127, in run_tests
21:47:20     INFO -      self.logPreamble(self.getActiveTests(options))
21:47:20     INFO -    File "/builds/slave/test/build/tests/mochitest/runtests.py", line 1961, in getActiveTests
21:47:20     INFO -      exists=False, disabled=disabled, filters=filters, **info)
21:47:20     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/manifestparser/manifestparser.py", line 748, in active_tests
21:47:20     INFO -      tests = fn(tests, values)
21:47:20     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/manifestparser/filters.py", line 192, in __call__
21:47:20     INFO -      end = tests.index(chunk_tests[end])
21:47:20     INFO -  IndexError: list index out of range
21:47:20     INFO -  Automation Error: Exception caught while running tests

Same thing applies to the debug suite, except chunks 17, 18, and 19 all had the error.

I'm trying again with the tests rearranged [2] in mochitest.ini on the assumption that this is a bug in the python code that barfs on the last test when spread too thinly over many chunks.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=141a9039e029
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=ba7429d94f13

Comment 1

[Ed Morley [:emorley]]

(Treeherder doesn't schedule the jobs :-) )

Comment 2

[Andrew Halberstadt [:ahal]]

Thanks, I'll take a look. Looks like there are a total of 3 unskipped tests with the 'apz' tag being chunked, so all the "No tests found" messages are expected. The bug here is that the last one fails.

Comment 3

[Andrew Halberstadt [:ahal]]

This happens anytime there are at least twice as many chunks as there are tests. It makes the 'end' index == len(tests):
https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/manifestparser/manifestparser/filters.py#178

Patch coming shortly.

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: MozReview Request: Bug 1182817 - [manifestparser] Fix exception in chunk_by_slice when there are two times more chunks than tests, r=chmanchester

Bug 1182817 - [manifestparser] Fix exception in chunk_by_slice when there are two times more chunks than tests, r=chmanchester

Comment 5

[Andrew Halberstadt [:ahal]]

Here's a similar try run to the one in comment 0:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d19f997f96c3

Comment 6

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8632947 [details]
MozReview Request: Bug 1182817 - [manifestparser] Fix exception in chunk_by_slice when there are two times more chunks than tests, r=chmanchester

https://reviewboard.mozilla.org/r/13155/#review11755

Ship It!

Comment 7

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/43eea082d662

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/43eea082d662