Closed Bug 1054308 Opened 11 years ago Closed 10 years ago

Investigate switching Thunderbird comm-central MozMill tests to mozharness

Categories

(Thunderbird :: Testing Infrastructure, defect)

Product:
Thunderbird
Component:
Testing Infrastructure
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
Thunderbird 39.0

People

(Reporter: standard8, Assigned: jcranmer)

References

Details

Attachments

(4 files, 3 obsolete files)

[mozharness] Add mozmill to the list of suites [checkin: see comment 15]
10.95 KB, patch
jlund
: review+
Details | Diff | Splinter Review
[mozilla-central] Add the in-tree config for mozmill tests [checkin: see comment 18]
3.85 KB, patch
ted
: review+
Details | Diff | Splinter Review
[buildbot] Port Thunderbird mozmill tests to mozharness,
3.37 KB, patch
Callek
: review+
Details | Diff | Splinter Review
[buildbot] Mozharness config fixes for mozmill
4.07 KB, patch
Callek
: review+
Details | Diff | Splinter Review
Things to do here: - Switch in-tree mozmill to use mozharness - or all the moztools(?) stuff surrounding it. - Work out how to do that in packaged format & switch tbpl
Will the tests need to be changed in any way (the testing code itself) ?
See Also: → 1055926
This is as far as I've gotten: * Add a mozmill suite to mozharness * Unpack extra directories * Install mozmill virtualenv +--> This is where things go wrong. Creating a custom virtualenv is problematic in its own right, since it screws up our ability to run the correct virtualenv later. I can try hacking around that, but I'm guessing that the eventual reviewer won't like that. The other option is to try ditching the install mozmill virtualenv script altogether and manually run it in the already-existing virtualenv on test machines. Here, though, we run into the problem that mozmill wants a pinned mozrunner==2.5.13 and the virtualenv already has mozrunner 6.2.
Conclusion: mozmill and jsbridge do not work on the newer version of mozrunner already in the virtualenv.
Depends on: 1121107
Not sure who needs to/can review this mozilla-central patch. This is needed because the desktop_unittests script fails if the testsuite name isn't in the in-tree config.
Assignee: nobody → Pidgeot18
Status: NEW → ASSIGNED
Attachment #8560820 - Flags: review?(ted)
Comment on attachment 8560820 [details] [diff] [review] [mozilla-central] Add the in-tree config for mozmill tests Er, oops, guess I was too premature on this patch. :-(
Attachment #8560820 - Attachment is obsolete: true
Attachment #8560820 - Flags: review?(ted)
tracking-thunderbird38: --- → ?
What is the motivation for tracking-thunderbird38? Why is this important?
(In reply to Kent James (:rkent) from comment #6) > What is the motivation for tracking-thunderbird38? Why is this important? This reduces the maintenance burden of the buildbot infrastructure for testing. The most important piece for TB38 tracking is the mozilla-central change (to add mozmill to the list of acceptable testsuites); the buildbot changes could be done later and backported as necessary.
OK let's try to push this. You probably have a reasonable chance of getting it aurora I suppose if we miss the cutoff.
tracking-thunderbird38: ? → +
Don't know if there's a better reviewer, but here goes. I've mostly copy-pasted the mozharness stuff from xpcshell.
Attachment #8564611 - Flags: review?(jlund)
This should be the correct change to make to mozilla-central to add the in-tree config stuff.
Attachment #8564612 - Flags: review?(ted)
I don't have the full buildbot accumen to build/test that patch, but once the above two land and go into production, it should be a matter of copying <http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/thunderbird_config.py#129> with an s/xpcshell/mozmill/g, and then adding in the code to make it ride the 38 train.
Flags: needinfo?(bugspam.Callek)
Comment on attachment 8564611 [details] [diff] [review] [mozharness] Add mozmill to the list of suites [checkin: see comment 15] Review of attachment 8564611 [details] [diff] [review]: ----------------------------------------------------------------- looks good so far. Have you tried this out against anything? I'm fine landing and then testing if we roll it out slowly. i.e. on a project branch. ::: scripts/desktop_unittest.py @@ +509,5 @@ > + abs_app_extensions_dir, > + overwrite='overwrite_if_exists') > + modules = ['jsbridge', 'mozmill'] > + for module in modules: > + self.install_module(module=os.path.join(dirs['abs_mozmill_dir'], where is this getting installed? It appears off glance that we are adding them to a current virtualenv and it sounded like you suggested that has conflicting modules: https://bugzilla.mozilla.org/show_bug.cgi?id=1054308#c3 ?
(In reply to Jordan Lund (:jlund) from comment #12) > looks good so far. Have you tried this out against anything? I'm fine > landing and then testing if we roll it out slowly. i.e. on a project branch. I've tested the mozharness in a local semi-reproduction of Linux buildbots in a docker container and on an OS X builder. > where is this getting installed? It appears off glance that we are adding > them to a current virtualenv and it sounded like you suggested that has > conflicting modules: https://bugzilla.mozilla.org/show_bug.cgi?id=1054308#c3 > ? I fixed the conflicting module issue in bug 1121107.
Comment on attachment 8564611 [details] [diff] [review] [mozharness] Add mozmill to the list of suites [checkin: see comment 15] Review of attachment 8564611 [details] [diff] [review]: ----------------------------------------------------------------- ship it
Attachment #8564611 - Flags: review?(jlund) → review+
Comment on attachment 8564611 [details] [diff] [review] [mozharness] Add mozmill to the list of suites [checkin: see comment 15] Checked into the mozharness repository: http://hg.mozilla.org/build/mozharness/
Attachment #8564611 - Attachment description: [mozharness] Add mozmill to the list of suites → [mozharness] Add mozmill to the list of suites [checkin: see comment 15]
Sorry, that last comment should say http://hg.mozilla.org/build/mozharness/rev/08cd66cbfacd
Comment on attachment 8564612 [details] [diff] [review] [mozilla-central] Add the in-tree config for mozmill tests [checkin: see comment 18] Review of attachment 8564612 [details] [diff] [review]: ----------------------------------------------------------------- Sorry for the delay!
Attachment #8564612 - Flags: review?(ted) → review+
Attachment #8564612 - Attachment description: [mozilla-central] Add the in-tree config for mozmill tests → [mozilla-central] Add the in-tree config for mozmill tests [checkin: see comment 18]
I've not tested this patch, and I have no idea how.
Flags: needinfo?(bugspam.Callek)
Attachment #8570984 - Flags: review?(bugspam.Callek)
Attachment #8570984 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 8570984 [details] [diff] [review] [buildbot] Switch TB 39+ to mozharness Review of attachment 8570984 [details] [diff] [review]: ----------------------------------------------------------------- Per request over IRC, landed: https://hg.mozilla.org/build/buildbot-configs/rev/8342d69fe6b7
(In reply to Chris Cooper [:coop] from comment #22) > In production: https://hg.mozilla.org/build/buildbot-configs/rev/8342d69fe6b7 Err I had to backout due to an issue with buildbot-configs testing (duplicate buildernames), it seems I forgot to comment here... I'll have to try and help joshua on this one I suspect.
Attachment #8570984 - Attachment is obsolete: true
Attachment #8580167 - Attachment description: Port Thunderbird mozmill tests to mozharness, → [buildbot] Port Thunderbird mozmill tests to mozharness,
Attachment #8580167 - Flags: review?(bugspam.Callek)
Comment on attachment 8580167 [details] [diff] [review] [buildbot] Port Thunderbird mozmill tests to mozharness, r+ pending a passing result at: https://travis-ci.org/Callek/build-buildbot-configs/builds/55140520
Attachment #8580167 - Flags: review?(bugspam.Callek) → review+
(In reply to Joshua Cranmer [:jcranmer] from comment #24) > Created attachment 8580167 [details] [diff] [review] > [buildbot] Port Thunderbird mozmill tests to mozharness, Pushed as https://hg.mozilla.org/build/buildbot-configs/rev/5c63a6b51c55
Since March 23, Mozmill runs in comm-central are failing with: self.buildbot_config isn't set after running read_buildbot_config! Running post_fatal callback... Exiting -1 Is that because we need the second half of this bug landed?
16:15:26 INFO - Running main action method: read_buildbot_config 16:15:26 INFO - buildbot_json_path is not set. Skipping... 16:15:26 FATAL - self.buildbot_config isn't set after running read_buildbot_config! 16:15:26 FATAL - Running post_fatal callback... so it looks like we need to add: 'buildbot_json_path': os.environ.get('PROPERTIES_FILE'), to the mozharness config for mozmill
jlund, can you look at Callek's comment 29 and see what we can do to get mozmill up again on comm-central? It has been completely failing for days.
Flags: needinfo?(jlund)
apologies for not getting to this yet. I am on point for merge & uplift release work today so I won't be able to look into this until my EOD.
so essentially what is happening is: we look for some json that is saved on our slave in a step before mozharness is called. This json holds all the properties that buildbot knows about with regards to the current job. e.g. installer binary url, test zip url, branch, builder name, etc However, Mozharness has to be told where to look for this json file: http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/buildbot.py#63 usually this is in our test configs like here: http://mxr.mozilla.org/build/source/mozharness/configs/unittests/mac_unittest.py#11 I notice for thunderbird xpcshell tests, you use the unittests/{windows,linux,mac}_config.py files (which include buildbot_json_path) and so we make it past the buildbot_json_path error: - example log without buildbot_json_path error: http://ftp.mozilla.org/pub/mozilla.org/thunderbird/tinderbox-builds/comm-central-macosx64-debug/1427479936/comm-central_snowleopard-debug_test-xpcshell-bm107-tests1-macosx-build83.txt.gz - snippet: 12:02:24 INFO - Run as scripts/scripts/desktop_unittest.py --cfg unittests/mac_unittest.py --xpcshell-suite xpcshell --cfg unittests/thunderbird_extra.py --blob-upload-branch comm-central --download-symbols true where as in the mozmill tests there is only the --cfg unittests/thunderbird_extra.py but not the --cfg unittests/{windows,linux}_config.py configs: - example log with buildbot_json_path error: http://ftp.mozilla.org/pub/mozilla.org/thunderbird/tinderbox-builds/comm-central-linux64-debug/1427479936/comm-central_ubuntu64_vm-debug_test-mozmill-bm67-tests1-linux64-build2.txt.gz - snippet: 11:58:16 INFO - Run as scripts/scripts/desktop_unittest.py --mozmill-suite mozmill --cfg unittests/thunderbird_extra.py --blob-upload-branch comm-central --download-symbols true so, buildbot-configs is where we set up the mozharness call and associated args. Is there a reason we are not including the platform specific configs or was it just a mistake? - where we set unittests/thunderbird_extra.py for all mozmill tests: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/thunderbird_config.py#144 - notice we do the same for xpcshell: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/thunderbird_config.py#133 - where we set the platform specific configs for xpcshell (we don't do this for mozmill): - http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/thunderbird_config.py#171 - http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/thunderbird_config.py#209 - ... and so on hope this helps
Flags: needinfo?(jlund)
Is this a reason why mozmill on treeherder is reported "busted" and not run at all? I have noticed new errors in my LOCAL |make mozmill| runs, and I am afraid that these errors will creep in while mozmill test runs are not run at all. I wonder how we can help. TIA
Given these insights and that its been busted a while, do you want the "mozharness based mozmill" change backed out, or is someone on your end working on changing/approving the mozharness config and we'll move forward that way?
Flags: needinfo?(rkent)
Flags: needinfo?(Pidgeot18)
(In reply to Jordan Lund (:jlund) from comment #285) > so essentially what is happening is: ... > > where as in the mozmill tests there is only the --cfg > unittests/thunderbird_extra.py but not the --cfg > unittests/{windows,linux}_config.py configs: I surmise that these -cfg path can be given multiple times then. > - example log with buildbot_json_path error: > http://ftp.mozilla.org/pub/mozilla.org/thunderbird/tinderbox-builds/comm- > central-linux64-debug/1427479936/comm-central_ubuntu64_vm-debug_test-mozmill- > bm67-tests1-linux64-build2.txt.gz > - snippet: 11:58:16 INFO - Run as scripts/scripts/desktop_unittest.py > --mozmill-suite mozmill --cfg unittests/thunderbird_extra.py > --blob-upload-branch comm-central --download-symbols true > > > so, buildbot-configs is where we set up the mozharness call and associated > args. Is there a reason we are not including the platform specific configs > or was it just a mistake? From what I observe as a third party [I mean I know nothing about these buildbot thingy], I think it is a pure oversight, not intentional. (I think Joshua wondered how this could be tested, and the real-world usage is the only test method available.) > - where we set unittests/thunderbird_extra.py for all mozmill tests: > http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/ > thunderbird_config.py#144 > - notice we do the same for xpcshell: > http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/ > thunderbird_config.py#133 > > - where we set the platform specific configs for xpcshell (we don't do > this for mozmill): > - > http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/ > thunderbird_config.py#171 > - > http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/ > thunderbird_config.py#209 > - ... and so on > This "and so on" has to be repeated for the architecture/debug combinations, correct? In today's tbpl, er, treeherder page, I see combinations as follows. Linux opt Linux debug Linux x64 opt Linux x64 debug OS X 10.6 opt OS X 10.6 debug OS X 10.8 opt OS X 10.8 debug OS X 10.10 opt OS X 10.10 debug Windows XP opt Windows XP debug Windows 7 opt Windows 7 debug http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/thunderbird_config.py#157 157 PLATFORM_UNITTEST_VARS = { 158 'linux': { 159 'product_name': 'thunderbird', 160 'app_name': 'mail', 161 'brand_name': 'Daily', 162 'builds_before_reboot': 1, 163 'unittest-env': {'DISPLAY': ':0'}, 164 'enable_opt_unittests': True, 165 'enable_debug_unittests': True, 166 'ubuntu32_vm': { 167 'opt_unittest_suites': UNITTEST_SUITES['opt_unittest_suites'][:], 168 'debug_unittest_suites': UNITTEST_SUITES['debug_unittest_suites'][:], 169 'suite_config': { * 170 'xpcshell': { * 171 'config_files': ["unittests/linux_unittest.py"], * 172 }, 173 }, 174 }, 175 }, I am not familiar with the code, but basically we need to add something like as follows immediately after line 172 above? And repeat similar such additions to the listed OS entries? Caveat: I don't know if |mozmill| is referred as 'mozmill' as the label in JSON file, and I don't know the pathname for linux-specific mozmill config file which is needed above.. E.g.: fir 'linux' part, add the following 'mozmill': { <--- Is this the correct name? 'config_files': ["path-to-the-mozmil-linux-specific-config.py"], }, Also, in the above PLATFOR_UNITTEST_VARS, we have entry for macos64 http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/thunderbird_config.py#223 But there I see only two OS versions, namely 'snowleopard' and 'yosemite' whereas treeherder lists three: OS X 10.6, OS X 10.8, and OS X 10.10. I wonder why there numbers do not match. Anyway I believe Joshua has better handle on this matter. Also, I believe that, if a simple fix above gets us past the hurdle, many of us would like to see the infrastructure of TB to be the same as that of FF and other software products of mozilla. But this I need to hear other people's opinions. (This will go a long way to remove the dual hg repository nature of C-C: we have a copy of M-C as subdirectory. We can probably shuffle the directory so that C-C does not have to have M-C directory as its subdirectory anymore once the same testing infastructure is shared: this is my understanding.) TIA
(In reply to Jordan Lund (:jlund) from comment #285) > so, buildbot-configs is where we set up the mozharness call and associated > args. Is there a reason we are not including the platform specific configs > or was it just a mistake? Probably a mistake. My grasp of anything in the buildbot configs is extremely tenuous at best and I was rather hoping to get a lot more help from release engineering than I have been getting in this regard. > - where we set the platform specific configs for xpcshell (we don't do > this for mozmill): > - > http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/ > thunderbird_config.py#171 > - > http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/ > thunderbird_config.py#209 > - ... and so on > > hope this helps So it sounds like we just need to copy those blocks for mozmill?
> > - where we set the platform specific configs for xpcshell (we don't do > > this for mozmill): > > - > > http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/ > > thunderbird_config.py#171 > > - > > http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/ > > thunderbird_config.py#209 > > - ... and so on > > > > hope this helps > > So it sounds like we just need to copy those blocks for mozmill? yup you got it. you will want to point to the same platform config (linux, windows, or mac) and use the 'mozmill' key everywhere you are running mozmill. I'd imagine this will be beside every place you are running xpcshell. as for testing changes, you could attempt to try setting up your own environment: https://wiki.mozilla.org/ReleaseEngineering/How_To/Setup_Personal_Development_Master#Setup.2FRunning_local_master_scheduler_on_laptop_-_not_dev-master and then diff'n your changes via: https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques#builder_list.py_.2F_dump_master.py
So, let's try this patch?
Flags: needinfo?(Pidgeot18)
Attachment #8588679 - Flags: review?(bugspam.Callek)
>_> Discovered typo.
Attachment #8588679 - Attachment is obsolete: true
Attachment #8588679 - Flags: review?(bugspam.Callek)
Attachment #8588680 - Flags: review?(bugspam.Callek)
Attachment #8588680 - Flags: review?(bugspam.Callek) → review+
Second buildbot changeset pushed: https://hg.mozilla.org/build/buildbot-configs/rev/7eca3284bf95
Flags: needinfo?(rkent)
And mozmill tests are working again, so this bug is now fixed!
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Keywords: leave-open
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 39.0
Target Milestone: Thunderbird 39.0 → Thunderbird 40.0
TB 39 is the correct target--the buildbot configs apply mozharness-based mozmill starting at Gecko 39, Gecko 40.
Target Milestone: Thunderbird 40.0 → Thunderbird 39.0
Is there any reason this should be uplifted to Thunderbird 38? You originally requested tracking to TB 38.
The rationale of supporting TB 38 is that releng has to support TB 38 builds a year from now. Given the current changes in buildbot infrastructure, non-mozharness-based tests may disappear before then. It would literally be a one-line change in a buildbot config file to get this working on TB 38 (since the mozharness/mozmill side of things landed in 38), so there is little immediate pressure to do any more backporting.
OK, then let's do this. Leave things as they are for the moment, as things seem to be working for TB38 and I don't want some new breakage when we are trying to release. But after the initial release, perhaps we can try to finish this then. I'll leave the tracking.
Fallen, is this what is needed to get mozmill tests working in c-esr38 again?
Yes, I've filed bug 1191922 to take care, after 300+ comments it might be good to separate it. Thanks for mentioning this bug!
Blocks: 1191922
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: