Closed Bug 1054308 Opened 8 years ago Closed 8 years ago

Investigate switching Thunderbird comm-central MozMill tests to mozharness

Categories

(Thunderbird :: Testing Infrastructure, defect)

Product:
Thunderbird
Component:
Testing Infrastructure
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
Thunderbird 39.0

People

(Reporter: standard8, Assigned: jcranmer)

References

Details

Attachments

(4 files, 3 obsolete files)

[mozharness] Add mozmill to the list of suites [checkin: see comment 15]
10.95 KB, patch
jlund
: review+
Details | Diff | Splinter Review
[mozilla-central] Add the in-tree config for mozmill tests [checkin: see comment 18]
3.85 KB, patch
ted
: review+
Details | Diff | Splinter Review
[buildbot] Port Thunderbird mozmill tests to mozharness,
3.37 KB, patch
Callek
: review+
Details | Diff | Splinter Review
[buildbot] Mozharness config fixes for mozmill
4.07 KB, patch
Callek
: review+
Details | Diff | Splinter Review 
Things to do here:

- Switch in-tree mozmill to use mozharness - or all the moztools(?) stuff surrounding it.
- Work out how to do that in packaged format & switch tbpl
Will the tests need to be changed in any way (the testing code itself) ?
See Also: → 1055926
This is as far as I've gotten:

* Add a mozmill suite to mozharness
* Unpack extra directories
* Install mozmill virtualenv
+--> This is where things go wrong.

Creating a custom virtualenv is problematic in its own right, since it screws up our ability to run the correct virtualenv later. I can try hacking around that, but I'm guessing that the eventual reviewer won't like that.

The other option is to try ditching the install mozmill virtualenv script altogether and manually run it in the already-existing virtualenv on test machines. Here, though, we run into the problem that mozmill wants a pinned mozrunner==2.5.13 and the virtualenv already has mozrunner 6.2.
Conclusion: mozmill and jsbridge do not work on the newer version of mozrunner already in the virtualenv.
Depends on: 1121107
Not sure who needs to/can review this mozilla-central patch.

This is needed because the desktop_unittests script fails if the testsuite name isn't in the in-tree config.
Assignee: nobody → Pidgeot18
Status: NEW → ASSIGNED
Attachment #8560820 - Flags: review?(ted)
Comment on attachment 8560820 [details] [diff] [review]
[mozilla-central] Add the in-tree config for mozmill tests

Er, oops, guess I was too premature on this patch. :-(
Attachment #8560820 - Attachment is obsolete: true
Attachment #8560820 - Flags: review?(ted)
What is the motivation for tracking-thunderbird38? Why is this important?
(In reply to Kent James (:rkent) from comment #6)
> What is the motivation for tracking-thunderbird38? Why is this important?

This reduces the maintenance burden of the buildbot infrastructure for testing. The most important piece for TB38 tracking is the mozilla-central change (to add mozmill to the list of acceptable testsuites); the buildbot changes could be done later and backported as necessary.
OK let's try to push this. You probably have a reasonable chance of getting it aurora I suppose if we miss the cutoff.
tracking-thunderbird38: ? → +
Don't know if there's a better reviewer, but here goes. I've mostly copy-pasted the mozharness stuff from xpcshell.
Attachment #8564611 - Flags: review?(jlund)
This should be the correct change to make to mozilla-central to add the in-tree config stuff.
Attachment #8564612 - Flags: review?(ted)
I don't have the full buildbot accumen to build/test that patch, but once the above two land and go into production, it should be a matter of copying <http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/thunderbird_config.py#129> with an s/xpcshell/mozmill/g, and then adding in the code to make it ride the 38 train.
Flags: needinfo?(bugspam.Callek)
Comment on attachment 8564611 [details] [diff] [review]
[mozharness] Add mozmill to the list of suites [checkin: see comment 15]

Review of attachment 8564611 [details] [diff] [review]:
-----------------------------------------------------------------

looks good so far. Have you tried this out against anything? I'm fine landing and then testing if we roll it out slowly. i.e. on a project branch.

::: scripts/desktop_unittest.py
@@ +509,5 @@
> +                              abs_app_extensions_dir,
> +                              overwrite='overwrite_if_exists')
> +            modules = ['jsbridge', 'mozmill']
> +            for module in modules:
> +                self.install_module(module=os.path.join(dirs['abs_mozmill_dir'],

where is this getting installed? It appears off glance that we are adding them to a current virtualenv and it sounded like you suggested that has conflicting modules: https://bugzilla.mozilla.org/show_bug.cgi?id=1054308#c3 ?
(In reply to Jordan Lund (:jlund) from comment #12)
> looks good so far. Have you tried this out against anything? I'm fine
> landing and then testing if we roll it out slowly. i.e. on a project branch.

I've tested the mozharness in a local semi-reproduction of Linux buildbots in a docker container and on an OS X builder.

> where is this getting installed? It appears off glance that we are adding
> them to a current virtualenv and it sounded like you suggested that has
> conflicting modules: https://bugzilla.mozilla.org/show_bug.cgi?id=1054308#c3
> ?

I fixed the conflicting module issue in bug 1121107.
Comment on attachment 8564611 [details] [diff] [review]
[mozharness] Add mozmill to the list of suites [checkin: see comment 15]

Review of attachment 8564611 [details] [diff] [review]:
-----------------------------------------------------------------

ship it
Attachment #8564611 - Flags: review?(jlund) → review+
Comment on attachment 8564611 [details] [diff] [review]
[mozharness] Add mozmill to the list of suites [checkin: see comment 15]

Checked into the mozharness repository: http://hg.mozilla.org/build/mozharness/
Attachment #8564611 - Attachment description: [mozharness] Add mozmill to the list of suites → [mozharness] Add mozmill to the list of suites [checkin: see comment 15]
Sorry, that last comment should say http://hg.mozilla.org/build/mozharness/rev/08cd66cbfacd
Comment on attachment 8564612 [details] [diff] [review]
[mozilla-central] Add the in-tree config for mozmill tests [checkin: see comment 18]

Review of attachment 8564612 [details] [diff] [review]:
-----------------------------------------------------------------

Sorry for the delay!
Attachment #8564612 - Flags: review?(ted) → review+
Attachment #8564612 - Attachment description: [mozilla-central] Add the in-tree config for mozmill tests → [mozilla-central] Add the in-tree config for mozmill tests [checkin: see comment 18]
Attached patch [buildbot] Switch TB 39+ to mozharness (obsolete) — — Splinter Review 
I've not tested this patch, and I have no idea how.
Flags: needinfo?(bugspam.Callek)
Attachment #8570984 - Flags: review?(bugspam.Callek)
Attachment #8570984 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 8570984 [details] [diff] [review]
[buildbot] Switch TB 39+ to mozharness

Review of attachment 8570984 [details] [diff] [review]:
-----------------------------------------------------------------

Per request over IRC, landed:

https://hg.mozilla.org/build/buildbot-configs/rev/8342d69fe6b7
(In reply to Chris Cooper [:coop] from comment #22)
> In production: https://hg.mozilla.org/build/buildbot-configs/rev/8342d69fe6b7

Err I had to backout due to an issue with buildbot-configs testing (duplicate buildernames), it seems I forgot to comment here...

I'll have to try and help joshua on this one I suspect.
Attachment #8570984 - Attachment is obsolete: true
Attachment #8580167 - Attachment description: Port Thunderbird mozmill tests to mozharness, → [buildbot] Port Thunderbird mozmill tests to mozharness,
Attachment #8580167 - Flags: review?(bugspam.Callek)
Comment on attachment 8580167 [details] [diff] [review]
[buildbot] Port Thunderbird mozmill tests to mozharness,

r+ pending a passing result at:

https://travis-ci.org/Callek/build-buildbot-configs/builds/55140520
Attachment #8580167 - Flags: review?(bugspam.Callek) → review+
(In reply to Joshua Cranmer [:jcranmer] from comment #24)
> Created attachment 8580167 [details] [diff] [review]
> [buildbot] Port Thunderbird mozmill tests to mozharness,

Pushed as https://hg.mozilla.org/build/buildbot-configs/rev/5c63a6b51c55
Since March 23, Mozmill runs in comm-central are failing with:

self.buildbot_config isn't set after running read_buildbot_config!
Running post_fatal callback...
Exiting -1 

Is that because we need the second half of this bug landed?
16:15:26     INFO - Running main action method: read_buildbot_config
16:15:26     INFO - buildbot_json_path is not set.  Skipping...
16:15:26    FATAL - self.buildbot_config isn't set after running read_buildbot_config!
16:15:26    FATAL - Running post_fatal callback...

so it looks like we need to add:

'buildbot_json_path': os.environ.get('PROPERTIES_FILE'),

to the mozharness config for mozmill
jlund, can you look at Callek's comment 29 and see what we can do to get mozmill up again on comm-central? It has been completely failing for days.
Flags: needinfo?(jlund)
apologies for not getting to this yet. I am on point for merge & uplift release work today so I won't be able to look into this until my EOD.