Closed Bug 517212 Opened 12 years ago Closed 12 years ago

Run talos tests on release builds

Categories

(Release Engineering :: General, defect, P3)

x86
Linux
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: catlee, Assigned: catlee)

References

Details

(Whiteboard: Q4 goal)

Attachments

(4 files, 4 obsolete files)

We should be running talos tests on our released bits.  They could run all the test suites 5 times or so, and report results to a special "release" branch, e.g. Firefox3.6-Release.
Priority: -- → P4
Priority: P4 → P3
Whiteboard: Q4 goal
Moving all the work of creating factories, builders, schedulers, etc. into a method like we're doing for builds/unittests.  The goal is to have this used by talos-pool/talos-staging-pool, and maybe for Try as well.
Attachment #409121 - Flags: review?(bhearsum)
Attachment #409121 - Flags: review?(anodelman)
Attachment #409127 - Flags: review?(bhearsum)
Attachment #409127 - Flags: review?(anodelman)
Comment on attachment 409121 [details] [diff] [review]
Create generateTalosBranch objects method in buildbotcustom

+                        talosCmd=talosCmd,
+                        fetchSymbols=branch_config['fetch_symbols'],
+                        **extra

That looks suspiciously like it is missing something...
Comment on attachment 409127 [details] [diff] [review]
Use generateBranchObjects from talos configs, and enable release tests on 1.9.2

For completeness this should also include staging and production talos try configs.
Comment on attachment 409127 [details] [diff] [review]
Use generateBranchObjects from talos configs, and enable release tests on 1.9.2

Catlee, I think you said you're working on new versions of these patches so I'm removing the review request from myself.
Attachment #409127 - Flags: review?(bhearsum)
Attachment #409121 - Flags: review?(bhearsum)
Attachment #409121 - Attachment is obsolete: true
Attachment #411975 - Flags: review?(bhearsum)
Attachment #411975 - Flags: review?(anodelman)
Attachment #409121 - Flags: review?(anodelman)
Attachment #411975 - Flags: review?(bhearsum)
Attachment #411975 - Flags: review?(anodelman)
Attachment #412068 - Flags: review?(bhearsum)
Attachment #412068 - Flags: review?(anodelman)
This will set up the builders, schedulers for testing 1.9.2 release builds.

The changes to mozilla2[-staging]/release_master.py will follow later.
Attachment #409127 - Attachment is obsolete: true
Attachment #412080 - Flags: review?(anodelman)
Attachment #409127 - Flags: review?(anodelman)
Attachment #411975 - Attachment is obsolete: true
Attachment #412080 - Attachment is obsolete: true
Attachment #412084 - Flags: review?(anodelman)
Attachment #412080 - Flags: review?(anodelman)
Attachment #412084 - Flags: review?(anodelman) → review+
Attachment #412068 - Flags: review?(anodelman) → review+
Note that we won't end up testing *signed* windows builds, for two reasons:

- There's currently no notification from the post-signing steps to trigger testing of said builds

- The Talos code has no way of unpacking and running firefox from a setup.exe package.  For some reason I'm a bit squeamish about running the installer on the talos boxes

Follow-up bug fodder?
To be honest, I think testing the zip is fine. The code doesn't differ between the two.

Maybe in the far off future though?

Also, I'll review these patches first thing tomorrow.
Attachment #412134 - Flags: review?(bhearsum)
Comment on attachment 412068 [details] [diff] [review]
Create generateTalosBranch objects method in buildbotcustom


>-        self.addStep(ShellCommand,
>-         name='make_upload',
>+        def get_url(rc, stdout, stderr):
>+            for m in re.findall("^(http://.*?\.(?:tar\.bz2|dmg|zip))", "\n".join([stdout, stderr]), re.M):
>+                if m.endswith("crashreporter-symbols.zip"):
>+                    continue
>+                if m.endswith("tests.tar.bz2"):
>+                    continue
>+                return {'packageUrl': m}
>+            return {}

This extract function or a tiny variation of it is copied around in a bunch of places. We should factor it out somewhere at some point. No need to block on it here, though.
Attachment #412068 - Flags: review?(bhearsum) → review+
Attachment #412134 - Flags: review?(bhearsum) → review+
Comment on attachment 412068 [details] [diff] [review]
Create generateTalosBranch objects method in buildbotcustom

changeset:   487:11c98e6cefa7
Attachment #412068 - Flags: checked-in+
Comment on attachment 412084 [details] [diff] [review]
Use generateBranchObjects from talos configs, and enable release tests on 1.9.2

changeset:   1742:e2f750a4edd3
Attachment #412084 - Flags: checked-in+
Comment on attachment 412134 [details] [diff] [review]
Do tests on release builds for mozilla-1.9.2

changeset:   1743:eaeb6dfa0335
Attachment #412134 - Flags: checked-in+
Attachment #412214 - Flags: review?(anodelman)
Attachment #412214 - Flags: review?(anodelman) → review+
For 3.6b3, there was a bunch of "output not recognized" and general hilarity from having --disable-tests in the mozconfigs (like bug 528759). Bug 457753 will also need that removed.

Bug 460282, and the having a strong package manifest on all platforms, means this should be safe to do for releases.
We have --enable-tests on the release mozconfigs now, so this should all be working fine.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.