1543220 - Make gTests run in automation

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect)

Description

[Jorg K (CEST = GMT+2)]

Now that M-C is top source directory, we can run gTests. We have two gTests: TestMsgStripRE and TestMailCookie.

The former will land in bug 1466748 and the latter needs to be repaired in bug 1543219 before it will pass.

To run them:

mach gtest TestMailCookie*
mach gtest TestMsgStripRE*

It would be nice to write a whole lot more gTests, so they should become part of the testing suite.

Comment 1

[Jorg K (CEST = GMT+2)]

I found another ancient C++ test which could be converted to a gTest:
https://searchfox.org/comm-central/search?q=TestImapFlagAndUidState&case=false&regexp=false&path=

Comment 2

[Rob Lemley [:rjl]]

Attachment: gtests_in_ci_wip.patch

Work in progress.

Comment 3

[Rob Lemley [:rjl]]

Attachment: gtests_in_ci_wip.patch

Work in progress.

Comment 4

[Jorg K (CEST = GMT+2)]

Nice to see. Note that TestMailCookie will currently fail, bug 1579698, after we spent so much energy on making it work in bug 1543219 :-(

Comment 5

[Rob Lemley [:rjl]]

To narrow down the tests that get run for Thunderbird, we can set the "GTEST_FILTER" environment variable, which is what "mach gtest" does. That would require that we stick to a standard prefix or a few prefixes that can be matched with simple wildcards.

Just now I tried "TestMsg*:TestMail*" and it ran the two tests that are set up.

Would that work? There might be some other ways to do it with some C programming.

Comment 6

[Jorg K (CEST = GMT+2)]

I'm cool with "TestMsg*:TestMail*". That looks "mail-related".

Comment 7

[Rob Lemley [:rjl]]

Attachment: Bug 1543220 - Enable building gtest xul for Thunderbird. r?#firefox-build-system-reviewers

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6e1fbe59c0a3
Enable building gtest xul for Thunderbird. r=nalexander

Comment 9

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/6e1fbe59c0a3

Comment 10

[Jorg K (CEST = GMT+2)]

Hmm, I guess only the M-C part landed here. And I hope we're not busted without the C-C part.

Comment 11

[Rob Lemley [:rjl]]

Thc M-C change just creates the target.gtest.tests.tar.gz artifact during the build so it can be used later for the test.
Unfortunately, setting the environment variables didn't work when I ran my try job late last night. Mozharness saw it just fine and the logs show that GTEST_FILTER is set throughout the setup to run the tests, but then rungtests.py either drops or ignores it, so every test ran. I'll try something else.

Comment 12

[Rob Lemley [:rjl]]

Attachment: thunderbird_gtests.patch

This will get things going. The M-C portion already landed.

I did some sorting as well.

Comment 13

[Geoff Lankow (:darktrojan)]

Comment on attachment 9092247 [details] [diff] [review]
thunderbird_gtests.patch

Review of attachment 9092247 [details] [diff] [review]:
-----------------------------------------------------------------

Okay, but I have some (rhetorical) questions.

::: mozharness/unittests/thunderbird_extra.py
@@ +17,5 @@
>      ],
> +    "all_gtest_suites": {
> +        "gtest": {
> +            'env': {
> +                'GTEST_FILTER': 'TestMail*:TestMsg*'

Switching quote style?

::: taskcluster/ci/test/compiled.yml
@@ +25,5 @@
> +    treeherder-symbol: GTest
> +    instance-size: xlarge
> +    run-on-projects:
> +        by-test-platform:
> +            windows.*-shippable/.*: []  # possible permafail?

What about non-shippable? Does that not also fail?

::: taskcluster/ci/test/test-platforms.yml
@@ +24,3 @@
>          - mozmill-tests
>          - xpcshell-tests
> +        - marionette-tests

Why does marionette move for each of these? Are you sorting the lists? Because if so they're still very unsorted.

Comment 14

[Rob Lemley [:rjl]]

Attachment: thunderbird_gtests.patch

No major changes here, just addressing comments.

I had not run the windows-shippable tests yet, that comment relates only
to Firefox it seems, I suspect because they use PGO on shippable builds
and as of now Thunderbird does not.

However, TestMsgStripRE is failing on OSX. It appears to be a legitimate
fail, not something related to automation.

All tests that are not DISABLED will run with this revision. I can make
a followup to disable macOS if the sheriffs like.

Comment 15

[Jorg K (CEST = GMT+2)]

Yes, let's disable whatever is failing. No idea why TestMsgStripRE is failing on Mac. Some more interference from the Mac address book like we had for Marionette? We also need to disable the MailCookie test, see bug 1579698.

Comment 16

[Rob Lemley [:rjl]]

Attachment: thunderbird_gtests.patch

Disable macOS.
There's a patch on bug 1579698 to disalble MailCookie tests.

Comment 17

[Jorg K (CEST = GMT+2)]

Geoff, if you land this, please also land bug 1579698. Adding NI so the comment doesn't get overlooked.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/fe313409a472
Add gtest tests to Taskcluster. r=darktrojan

Comment 19

[Jorg K (CEST = GMT+2)]

Tests were supposed to be switched off on Mac, but they ran and failed :-(

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/b3245af18949
Follow-up: fix indentation/yaml error. rs=white-space-only

Comment 21

[Rob Lemley [:rjl]]

(In reply to Jorg K (GMT+2) (reduced availability 14-19 of Sept.) from comment #19)

Tests were supposed to be switched off on Mac, but they ran and failed :-(

I will do a followup this afternoon.

Comment 22

[Rob Lemley [:rjl]]

Attachment: thunderbird_gtests_part2.patch

Regular Expressions != Globs

Comment 23

[Jorg K (CEST = GMT+2)]

Comment on attachment 9093412 [details] [diff] [review]
thunderbird_gtests_part2.patch

I've seen a regexp before in my life.

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/c19cbff54c47
Follow-up: Disable gTests on macOS, fix regular expression. r=jorgk