1142553 - Increase chunks for Android 4.0 robocop

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Geoff Brown [:gbrown]]

Android 4.0 robocop, especially rc2, is frequently retrying -- the same old problem of panda jobs running too long.

The mobile team keeps adding robocop tests; maybe split into 10 chunks?

Comment 1

[Geoff Brown [:gbrown]]

Attachment: increase chunks in buildbot-configs

Comment 2

[Geoff Brown [:gbrown]]

Attachment: increase chunks in mozharness

How can these changes be landed in a mozharness-pinned world?

Comment 3

[Kim Moir [:kmoir] ET]

Regarding your question re mozharness + pinning

I think you just need to land them and then update the mh revision in mozharness.json that is used for the specific branch

http://armenzg.blogspot.ca/2014/11/pinning-mozharness-from-in-tree-aka.html

Comment 4

[Geoff Brown [:gbrown]]

We discussed this on irc and decided to check these in, wait for a reconfig, then bump mozharness (and likely uplift the bump).

After the reconfig, we expect the new rc8, rc9 and rc10 jobs to fail until the mozharness bump catches up (but all tests will still be running in chunks 1-7).

Comment 5

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/build/buildbot-configs/rev/3c76bc12cb25
https://hg.mozilla.org/build/mozharness/rev/36d9bbd7bf41

Comment 6

[[e-mail] (:bhearsum)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/3c76bc12cb25

Comment 7

[[e-mail] (:bhearsum)]

In production: https://hg.mozilla.org/build/mozharness/rev/36d9bbd7bf41

Comment 8

[Geoff Brown [:gbrown]]

Thanks to a mozharness bump (m-i, m-c, aurora - bug 1094364) closely synchronized with a reconfig, we switched to 10 chunks without any errors. There was at least one job that ran just 7 of 10 chunks because the mozharness bump was in effect before the reconfig, but that's fine!

All's well and frequent retries have stopped on m-i and m-c and should be fine on aurora.

Comment 9

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

rc10 on Android has been failing with https://treeherder.mozilla.org/logviewer.html#?job_id=674949&repo=mozilla-aurora ever since this landed, despite a mozharness bump to pick up the changes.

It's saying:
15:28:57 INFO - Running command: ['python', '-u', '/builds/panda-0529/test/build/tests/mochitest/runtestsremote.py', '--deviceIP=10.26.133.72', '--xre-path=../hostutils/xre', '--utility-path=../hostutils/bin', '--certificate-path=certs', '--app=org.mozilla.fennec_aurora', '--console-level=INFO', '--http-port=30529', '--ssl-port=31529', '--symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-android-api-11/1426280929/fennec-38.0a2.en-US.android-arm.crashreporter-symbols.zip', '--robocop=mochitest/robocop.ini', '--total-chunks=10', '--this-chunk=10'] in /builds/panda-0529/test/build/tests
15:28:57 INFO - Copy/paste: python -u /builds/panda-0529/test/build/tests/mochitest/runtestsremote.py --deviceIP=10.26.133.72 --xre-path=../hostutils/xre --utility-path=../hostutils/bin --certificate-path=certs --app=org.mozilla.fennec_aurora --console-level=INFO --http-port=30529 --ssl-port=31529 --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-android-api-11/1426280929/fennec-38.0a2.en-US.android-arm.crashreporter-symbols.zip --robocop=mochitest/robocop.ini --total-chunks=10 --this-chunk=10
15:28:57 INFO - Using env: {'DM_TRANS': 'sut',
15:28:57 INFO - 'HOME': '/home/cltbld',
15:28:57 INFO - 'LANG': 'en_US.UTF-8',
15:28:57 INFO - 'LOGNAME': 'cltbld',
15:28:57 INFO - 'MINIDUMP_SAVE_PATH': '/builds/panda-0529/test/build/blobber_upload_dir',
15:28:57 INFO - 'MINIDUMP_STACKWALK': '/builds/minidump_stackwalk',
15:28:57 INFO - 'MOZ_UPLOAD_DIR': '/builds/panda-0529/test/build/blobber_upload_dir',
15:28:57 INFO - 'OLDPWD': '/home/cltbld',
15:28:57 INFO - 'PATH': '/builds/panda-0529/test/build/venv/bin:/usr/local/bin:/usr/local/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin',

Comment 10

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

It's not clear to me if this chunk is truly empty (so no tests are missing) or if there are tests that are supposed to be in chunk 10 that aren't being run (so some tests are missing). I'm closing Aurora until someone can determine or fix this.

Comment 11

[Jordan Lund (:jlund)]

(In reply to Wes Kocher (:KWierso) from comment #10)
> It's not clear to me if this chunk is truly empty (so no tests are missing)
> or if there are tests that are supposed to be in chunk 10 that aren't being
> run (so some tests are missing). I'm closing Aurora until someone can
> determine or fix this.

18:01:19 <jlund> KWierso|sheriffduty: so I dug around in unfamiliar code and I suspect the reason m-c robo-10 is working is because: https://hg.mozilla.org/mozilla-central/diff/187d5f6e0b03/testing/mochitest/runtestsremote.py#l1.72
18:01:48 <jlund> notice aurora doesn't have that yet: https://hg.mozilla.org/releases/mozilla-aurora/annotate/17f9ac0d4f40/testing/mochitest/runtestsremote.py#l804

ahal, jmaher: does ^ that sound accurate?

Comment 12

[Geoff Brown [:gbrown]]

(In reply to Jordan Lund (:jlund) from comment #11)

That sounds right to me.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

if this is on aurora, we need to uplift the work that :ahal has been doing to chunking.  We have a series of changes that could be uplifted to aurora, they are all tracked in this bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1124182

Comment 14

[Andrew Halberstadt [:ahal]]

I'm not sure I follow what is happening here. Are we saying that robocop chunking was broken prior to my patch? It seemed like it was working to me... but yes, if my change in comment 11 is making it work, there are a whole slew of dependent bugs that will also need to be uplifted.

Comment 15

[Andrew Halberstadt [:ahal]]

I'll also add that if we do go that route, I'd recommend only uplifting the manifestparser changes and the specific part wrt robocop. Uplifting the entire patch wholesale will likely break one of the other mochitest suites.

Comment 16

[Andrew Halberstadt [:ahal]]

Oh, so my algorithm doesn't count disabled tests when doing the chunking. It looks like the algorithm on aurora is naive and could be including disabled tests in its chunking. It's possible the new chunk only contains disabled tests, which would explain the error.

Comment 17

[Geoff Brown [:gbrown]]

(In reply to Wes Kocher (:KWierso) from comment #10)
> It's not clear to me if this chunk is truly empty (so no tests are missing)
> or if there are tests that are supposed to be in chunk 10 that aren't being
> run (so some tests are missing). 

All tests are being run in rc1 through rc9. 

There have been occasional problems with empty robocop chunks before; there's something wrong with the chunking algorithm...likely its treatment of disabled tests.

I note that the excessive Android 4.0 retries problem was not affecting aurora, so if we could decrease chunks on aurora again, we'd be happy. But pinning aurora to an old mozharness would be restrictive. And worse, buildbot would still try to run rc8, rc9, rc10, and they would all report an error -- no cleaner than the current situation.

I think the reasonable solutions here are:
1. uplift ahal's changes (sounds like there are some non-trivial dependencies though)
2. write a custom chunking fix for aurora
3. do nothing; ignore the failure on rc10

I can look into (2) this morning.

Comment 18

[Ryan VanderMeulen [:RyanVM]]

Sounds like we can at least reopen Aurora and ignore the rc10 failures for now.

Comment 19

[Andrew Halberstadt [:ahal]]

(In reply to Geoff Brown [:gbrown] from comment #17)
> (In reply to Wes Kocher (:KWierso) from comment #10)
> > It's not clear to me if this chunk is truly empty (so no tests are missing)
> > or if there are tests that are supposed to be in chunk 10 that aren't being
> > run (so some tests are missing). 
> 
> All tests are being run in rc1 through rc9. 
> 
> There have been occasional problems with empty robocop chunks before;
> there's something wrong with the chunking algorithm...likely its treatment
> of disabled tests.
> 
> I note that the excessive Android 4.0 retries problem was not affecting
> aurora, so if we could decrease chunks on aurora again, we'd be happy. But
> pinning aurora to an old mozharness would be restrictive. And worse,
> buildbot would still try to run rc8, rc9, rc10, and they would all report an
> error -- no cleaner than the current situation.
> 
> I think the reasonable solutions here are:
> 1. uplift ahal's changes (sounds like there are some non-trivial
> dependencies though)
> 2. write a custom chunking fix for aurora
> 3. do nothing; ignore the failure on rc10
> 
> I can look into (2) this morning.

Please note the change that Jordan linked fixes the chunking algorithm to disregard disabled tests. If you want to do a one-off fix to the robocop algorithm on aurora, you can copy the algorithm from here (use the disabled=False code path):
https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/manifestparser/manifestparser/filters.py#153

Comment 20

[Geoff Brown [:gbrown]]

It looks like the relevant difference here is in rounding when calculating the number of tests per chunk.

On aurora, there are 81 enabled tests to run and the robocop algorithm chooses to run 9 tests in each chunk, leaving 0 tests for the 10th chunk.

On central, there are 82 enabled tests to run and the new algorithm chooses to run 8 tests in 8 chunks and 9 tests in 2 chunks (rc3 and rc8).

old:
            tests_per_chunk = math.ceil(
                len(tests) / (options.totalChunks * 1.0))
            start = int(round((options.thisChunk - 1) * tests_per_chunk))
            end = int(round(options.thisChunk * tests_per_chunk))
            if end > len(tests):
                end = len(tests)

new:

        tests_per_chunk = float(len(chunk_tests)) / self.total_chunks
        start = int(round((self.this_chunk - 1) * tests_per_chunk))
        end = int(round(self.this_chunk * tests_per_chunk))

Comment 21

[Geoff Brown [:gbrown]]

Attachment: fix rounding in robocop chunking algorithm

Fix for aurora. Seems to work fine:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c43bcce2edec

Comment 22

[Andrew Halberstadt [:ahal]]

Comment on attachment 8578196 [details] [diff] [review]
fix rounding in robocop chunking algorithm

Review of attachment 8578196 [details] [diff] [review]:
-----------------------------------------------------------------

Lgtm, though it's still possible for a chunk to only contain disabled tests, but that will be fixed with bug 1142112

Comment 23

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/7f90fdddb5ef

Comment 24

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/0e0204877015