Closed Bug 1517363 Opened 3 years ago Closed 3 years ago

Rebalance mozmill test chunks

Categories

(Thunderbird :: Testing Infrastructure, enhancement)

enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
Thunderbird 66.0

People

(Reporter: darktrojan, Assigned: darktrojan)

Details

Attachments

(1 file, 1 obsolete file)

On most platforms* we split mozmill tests into four chunks (Z1-Z4), and we do this by taking the list of files and directories in the tests dir and splitting it into four. Simple enough, except that the directories contain multiple tests, and by quirk of naming, most of the big directories are near the start of the alphabet.

Currently this leads to:

Z1 runs 80 tests
Z2 runs 51 tests
Z3 runs 29 tests
Z4 runs 17 tests

I propose we split the tests up in a more intelligent way, which accounts for the size of the directories. This will mean some tests move into a different set from where we're used to them being, but that's okay.


* Also I think we should change OS X to also run four chunks.
Attached patch 1517363-mozmill-chunks-1.diff (obsolete) — Splinter Review
Well this didn't give me the timing improvement I was hoping for, but that can be a bit all-over-the-place anyway.

Z1: 61 (-19), content-policy through eventDialog moved to Z2
Z2: 48 ( -3), folder-pane through message-header moved to Z3
Z3: 29 (  0), multiple-identities through session-store moved to Z4
Z4: 39 (+22)
Attachment #9034099 - Flags: review?(jorgk)
Comment on attachment 9034099 [details] [diff] [review]
1517363-mozmill-chunks-1.diff

(In reply to Geoff Lankow (:darktrojan) from comment #1)
> Well this didn't give me the timing improvement I was hoping for, but that
> can be a bit all-over-the-place anyway.

That's because it didn't work. The tests ran in the same chunks as before. :(
Attachment #9034099 - Flags: review?(jorgk)
Okay, now that I've made this change actually work, the time taken by the slowest chunks on each platform has reduced by 3 to 7 minutes. That's before splitting OSX into four, which I still want to do.
Attachment #9034099 - Attachment is obsolete: true
Attachment #9034280 - Flags: review?(jorgk)
Also, the number of tests does not say much about the time it will take as not all tests take equally long. Some are trivial, some (like the folder-display) do many operations in one test file.
Best would be to time each test folder manually and then split them in equally long groups.
Comment on attachment 9034280 [details] [diff] [review]
1517363-mozmill-chunks-2.diff

Review of attachment 9034280 [details] [diff] [review]:
-----------------------------------------------------------------

So this is chunking up the tests, not the directories any more. Looks like an improvement. Aceman's idea is good, too, but we'd have to assign a "weight" to each test. Next rev ;-)
Attachment #9034280 - Flags: review?(jorgk) → review+
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/4530e74cd168
Rebalance mozmill test chunks. r=jorgk DONTBUILD
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
The patch didn't quite apply, there was no 'shutil' in the patch but there was in the file which hadn't changed in ages. I removed it since it wasn't referenced in the file, so I hope it doesn't blow up now.
Target Milestone: --- → Thunderbird 66.0
Umm, upon closer inspection, you increased the Xpcshell chunks on Mac to 4, but not Mozmill:
https://hg.mozilla.org/comm-central/rev/4530e74cd16819dabcb07061938af5dec07e86cc#l2.12

Mozmill is a couple of lines down in the file. Reading comment #0, that was not the intention.

Xpshell now runs in 7, 5, 11 and 6 minutes. I don't know whether to revert that. Mozmill runs in 23 and 19, so it's balanced but not faster.
Flags: needinfo?(geoff)
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/a5791d3c3cc6
Rebalance mozmill test chunks, take 2. r=me
(In reply to Jorg K (GMT+1) from comment #8)
> Umm, upon closer inspection, you increased the Xpcshell chunks on Mac to 4,
> but not Mozmill:
> https://hg.mozilla.org/comm-central/rev/
> 4530e74cd16819dabcb07061938af5dec07e86cc#l2.12

Oh, whoops. That change got added and removed several times as I went back and forward to Try. Must've screwed it up!
Flags: needinfo?(geoff)
As per IRC conversation: We'll leave Xpcshell tests on Mac split into four.
You need to log in before you can comment on or make changes to this bug.