Rebalance mozmill test chunks

RESOLVED FIXED in Thunderbird 66.0

Status

enhancement
RESOLVED FIXED
5 months ago
5 months ago

People

(Reporter: darktrojan, Assigned: darktrojan)

Tracking

unspecified
Thunderbird 66.0

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

Assignee

Description

5 months ago
On most platforms* we split mozmill tests into four chunks (Z1-Z4), and we do this by taking the list of files and directories in the tests dir and splitting it into four. Simple enough, except that the directories contain multiple tests, and by quirk of naming, most of the big directories are near the start of the alphabet.

Currently this leads to:

Z1 runs 80 tests
Z2 runs 51 tests
Z3 runs 29 tests
Z4 runs 17 tests

I propose we split the tests up in a more intelligent way, which accounts for the size of the directories. This will mean some tests move into a different set from where we're used to them being, but that's okay.


* Also I think we should change OS X to also run four chunks.
Assignee

Comment 1

5 months ago
Posted patch 1517363-mozmill-chunks-1.diff (obsolete) — Splinter Review
Well this didn't give me the timing improvement I was hoping for, but that can be a bit all-over-the-place anyway.

Z1: 61 (-19), content-policy through eventDialog moved to Z2
Z2: 48 ( -3), folder-pane through message-header moved to Z3
Z3: 29 (  0), multiple-identities through session-store moved to Z4
Z4: 39 (+22)
Attachment #9034099 - Flags: review?(jorgk)
Assignee

Comment 2

5 months ago
Comment on attachment 9034099 [details] [diff] [review]
1517363-mozmill-chunks-1.diff

(In reply to Geoff Lankow (:darktrojan) from comment #1)
> Well this didn't give me the timing improvement I was hoping for, but that
> can be a bit all-over-the-place anyway.

That's because it didn't work. The tests ran in the same chunks as before. :(
Attachment #9034099 - Flags: review?(jorgk)
Assignee

Comment 3

5 months ago
Okay, now that I've made this change actually work, the time taken by the slowest chunks on each platform has reduced by 3 to 7 minutes. That's before splitting OSX into four, which I still want to do.
Attachment #9034099 - Attachment is obsolete: true
Attachment #9034280 - Flags: review?(jorgk)

Comment 4

5 months ago
Also, the number of tests does not say much about the time it will take as not all tests take equally long. Some are trivial, some (like the folder-display) do many operations in one test file.
Best would be to time each test folder manually and then split them in equally long groups.

Comment 5

5 months ago
Comment on attachment 9034280 [details] [diff] [review]
1517363-mozmill-chunks-2.diff

Review of attachment 9034280 [details] [diff] [review]:
-----------------------------------------------------------------

So this is chunking up the tests, not the directories any more. Looks like an improvement. Aceman's idea is good, too, but we'd have to assign a "weight" to each test. Next rev ;-)
Attachment #9034280 - Flags: review?(jorgk) → review+

Comment 6

5 months ago
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/4530e74cd168
Rebalance mozmill test chunks. r=jorgk DONTBUILD
Status: ASSIGNED → RESOLVED
Last Resolved: 5 months ago
Resolution: --- → FIXED

Comment 7

5 months ago
The patch didn't quite apply, there was no 'shutil' in the patch but there was in the file which hadn't changed in ages. I removed it since it wasn't referenced in the file, so I hope it doesn't blow up now.
Target Milestone: --- → Thunderbird 66.0

Comment 8

5 months ago
Umm, upon closer inspection, you increased the Xpcshell chunks on Mac to 4, but not Mozmill:
https://hg.mozilla.org/comm-central/rev/4530e74cd16819dabcb07061938af5dec07e86cc#l2.12

Mozmill is a couple of lines down in the file. Reading comment #0, that was not the intention.

Xpshell now runs in 7, 5, 11 and 6 minutes. I don't know whether to revert that. Mozmill runs in 23 and 19, so it's balanced but not faster.
Flags: needinfo?(geoff)

Comment 9

5 months ago
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/a5791d3c3cc6
Rebalance mozmill test chunks, take 2. r=me
Assignee

Comment 10

5 months ago
(In reply to Jorg K (GMT+1) from comment #8)
> Umm, upon closer inspection, you increased the Xpcshell chunks on Mac to 4,
> but not Mozmill:
> https://hg.mozilla.org/comm-central/rev/
> 4530e74cd16819dabcb07061938af5dec07e86cc#l2.12

Oh, whoops. That change got added and removed several times as I went back and forward to Try. Must've screwed it up!
Flags: needinfo?(geoff)

Comment 11

5 months ago
As per IRC conversation: We'll leave Xpcshell tests on Mac split into four.
You need to log in before you can comment on or make changes to this bug.