517212 - Run talos tests on release builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Chris AtLee [:catlee]]

We should be running talos tests on our released bits.  They could run all the test suites 5 times or so, and report results to a special "release" branch, e.g. Firefox3.6-Release.

Comment 1

[Chris AtLee [:catlee]]

Attachment: Create generateTalosBranch objects method in buildbotcustom

Moving all the work of creating factories, builders, schedulers, etc. into a method like we're doing for builds/unittests.  The goal is to have this used by talos-pool/talos-staging-pool, and maybe for Try as well.

Comment 2

[Chris AtLee [:catlee]]

Attachment: Use generateBranchObjects from talos configs, and enable release tests on 1.9.2

Comment 3

[alice nodelman [:alice] [:anode]]

Comment on attachment 409121 [details] [diff] [review]
Create generateTalosBranch objects method in buildbotcustom

+                        talosCmd=talosCmd,
+                        fetchSymbols=branch_config['fetch_symbols'],
+                        **extra

That looks suspiciously like it is missing something...

Comment 4

[alice nodelman [:alice] [:anode]]

Comment on attachment 409127 [details] [diff] [review]
Use generateBranchObjects from talos configs, and enable release tests on 1.9.2

For completeness this should also include staging and production talos try configs.

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 409127 [details] [diff] [review]
Use generateBranchObjects from talos configs, and enable release tests on 1.9.2

Catlee, I think you said you're working on new versions of these patches so I'm removing the review request from myself.

Comment 6

[Chris AtLee [:catlee]]

Attachment: Create generateTalosBranch objects method in buildbotcustom

Comment 7

[Chris AtLee [:catlee]]

Attachment: Create generateTalosBranch objects method in buildbotcustom

Comment 8

[Chris AtLee [:catlee]]

Attachment: Use generateBranchObjects from talos configs, and enable release tests on 1.9.2

This will set up the builders, schedulers for testing 1.9.2 release builds.

The changes to mozilla2[-staging]/release_master.py will follow later.

Comment 9

[Chris AtLee [:catlee]]

Attachment: Use generateBranchObjects from talos configs, and enable release tests on 1.9.2

Comment 10

[Chris AtLee [:catlee]]

Note that we won't end up testing *signed* windows builds, for two reasons:

- There's currently no notification from the post-signing steps to trigger testing of said builds

- The Talos code has no way of unpacking and running firefox from a setup.exe package.  For some reason I'm a bit squeamish about running the installer on the talos boxes

Follow-up bug fodder?

Comment 11

[[e-mail] (:bhearsum)]

To be honest, I think testing the zip is fine. The code doesn't differ between the two.

Maybe in the far off future though?

Also, I'll review these patches first thing tomorrow.

Comment 12

[Chris AtLee [:catlee]]

Attachment: Do tests on release builds for mozilla-1.9.2

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 412068 [details] [diff] [review]
Create generateTalosBranch objects method in buildbotcustom


>-        self.addStep(ShellCommand,
>-         name='make_upload',
>+        def get_url(rc, stdout, stderr):
>+            for m in re.findall("^(http://.*?\.(?:tar\.bz2|dmg|zip))", "\n".join([stdout, stderr]), re.M):
>+                if m.endswith("crashreporter-symbols.zip"):
>+                    continue
>+                if m.endswith("tests.tar.bz2"):
>+                    continue
>+                return {'packageUrl': m}
>+            return {}

This extract function or a tiny variation of it is copied around in a bunch of places. We should factor it out somewhere at some point. No need to block on it here, though.

Comment 14

[Chris AtLee [:catlee]]

Comment on attachment 412068 [details] [diff] [review]
Create generateTalosBranch objects method in buildbotcustom

changeset:   487:11c98e6cefa7

Comment 15

[Chris AtLee [:catlee]]

Comment on attachment 412084 [details] [diff] [review]
Use generateBranchObjects from talos configs, and enable release tests on 1.9.2

changeset:   1742:e2f750a4edd3

Comment 16

[Chris AtLee [:catlee]]

Comment on attachment 412134 [details] [diff] [review]
Do tests on release builds for mozilla-1.9.2

changeset:   1743:eaeb6dfa0335

Comment 17

[Chris AtLee [:catlee]]

Attachment: Add Firefox3.6-Release branch to SQL

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

For 3.6b3, there was a bunch of "output not recognized" and general hilarity from having --disable-tests in the mozconfigs (like bug 528759). Bug 457753 will also need that removed.

Bug 460282, and the having a strong package manifest on all platforms, means this should be safe to do for releases.

Comment 19

[Chris AtLee [:catlee]]

We have --enable-tests on the release mozconfigs now, so this should all be working fine.