Closed Bug 1517363 Opened 3 years ago Closed 3 years ago
Rebalance mozmill test chunks
On most platforms* we split mozmill tests into four chunks (Z1-Z4), and we do this by taking the list of files and directories in the tests dir and splitting it into four. Simple enough, except that the directories contain multiple tests, and by quirk of naming, most of the big directories are near the start of the alphabet. Currently this leads to: Z1 runs 80 tests Z2 runs 51 tests Z3 runs 29 tests Z4 runs 17 tests I propose we split the tests up in a more intelligent way, which accounts for the size of the directories. This will mean some tests move into a different set from where we're used to them being, but that's okay. * Also I think we should change OS X to also run four chunks.
Well this didn't give me the timing improvement I was hoping for, but that can be a bit all-over-the-place anyway. Z1: 61 (-19), content-policy through eventDialog moved to Z2 Z2: 48 ( -3), folder-pane through message-header moved to Z3 Z3: 29 ( 0), multiple-identities through session-store moved to Z4 Z4: 39 (+22)
Comment on attachment 9034099 [details] [diff] [review] 1517363-mozmill-chunks-1.diff (In reply to Geoff Lankow (:darktrojan) from comment #1) > Well this didn't give me the timing improvement I was hoping for, but that > can be a bit all-over-the-place anyway. That's because it didn't work. The tests ran in the same chunks as before. :(
Okay, now that I've made this change actually work, the time taken by the slowest chunks on each platform has reduced by 3 to 7 minutes. That's before splitting OSX into four, which I still want to do.
Also, the number of tests does not say much about the time it will take as not all tests take equally long. Some are trivial, some (like the folder-display) do many operations in one test file. Best would be to time each test folder manually and then split them in equally long groups.
Comment on attachment 9034280 [details] [diff] [review] 1517363-mozmill-chunks-2.diff Review of attachment 9034280 [details] [diff] [review]: ----------------------------------------------------------------- So this is chunking up the tests, not the directories any more. Looks like an improvement. Aceman's idea is good, too, but we'd have to assign a "weight" to each test. Next rev ;-)
Attachment #9034280 - Flags: review?(jorgk) → review+
Pushed by email@example.com: https://hg.mozilla.org/comm-central/rev/4530e74cd168 Rebalance mozmill test chunks. r=jorgk DONTBUILD
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
The patch didn't quite apply, there was no 'shutil' in the patch but there was in the file which hadn't changed in ages. I removed it since it wasn't referenced in the file, so I hope it doesn't blow up now.
Target Milestone: --- → Thunderbird 66.0
Umm, upon closer inspection, you increased the Xpcshell chunks on Mac to 4, but not Mozmill: https://hg.mozilla.org/comm-central/rev/4530e74cd16819dabcb07061938af5dec07e86cc#l2.12 Mozmill is a couple of lines down in the file. Reading comment #0, that was not the intention. Xpshell now runs in 7, 5, 11 and 6 minutes. I don't know whether to revert that. Mozmill runs in 23 and 19, so it's balanced but not faster.
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/comm-central/rev/a5791d3c3cc6 Rebalance mozmill test chunks, take 2. r=me
(In reply to Jorg K (GMT+1) from comment #8) > Umm, upon closer inspection, you increased the Xpcshell chunks on Mac to 4, > but not Mozmill: > https://hg.mozilla.org/comm-central/rev/ > 4530e74cd16819dabcb07061938af5dec07e86cc#l2.12 Oh, whoops. That change got added and removed several times as I went back and forward to Try. Must've screwed it up!
As per IRC conversation: We'll leave Xpcshell tests on Mac split into four.
You need to log in before you can comment on or make changes to this bug.