589251 - Parallelize update_verify

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris AtLee [:catlee]]

update_verify runs as a single job per platform.  We could speed this up by splitting it up into different jobs per locale.

Comment 3

[[e-mail] (:bhearsum)]

I have a few ideas about how to do this, only one of them seems realistic, though:
- Stick with 1 Builder per platform and create a Scheduler that creates one, unmergeable BuildSet for each previous version, per platform. Doing this would make it impossible to detect doneness, and is generally dirty.
- Create 1 Builder per platform per previous version. We'd be able to detect doneness through a Dependent Scheduler. This would require that release.py is aware of the current version and previous versions, which is reconfig-dependent, and thus bad.

The only other idea I came up with is mochitest-style chunking. Doing this would give us a fixed number of update verify builders per platform, which means we can detect doneness, and it doesn't require release.py to know anything about the current release, so we're not reconfig dependent.

Within the last idea, there's a couple variations. The simplest route is to just split the testing by version. For example, if we have 3 chunks and 4 previous versions, chunk #1 would test 2 of them, and chunks 2 and 3 would only test one. This could cause chunk #1 to take significantly longer than the others.

The slightly more complicated route is to split by version and locales. This would let us give all chunks more or less the same amount of work, and lower the overall time it takes to run update verify. I'm going to shoot for this unless it becomes more pain than its worth.

Comment 4

[[e-mail] (:bhearsum)]

Working on this here: https://github.com/bhearsum/tools/commits/parallel-verify/

Comment 5

[[e-mail] (:bhearsum)]

Attachment: first pass on chunked update verify, partially tested

This patch does a few things:
- Puts chunking logic in a generic function
- Adds an UpdateVerifyConfig class, capable of reading, writing, and generating chunks of a verify config.
- Adds a basic update verify wrapping script that does chunking

I spent a decent amount of time on UpdateVerifyConfig in hopes that it will be useful elsewhere in the future. It should be easier to write additional update verify tests in the future, for example (eg, to verify that PPC users don't get updates).

I've done some basic testing on this to verify that the correct config gets spit out -- at least on Linux.

Still needs more robust testing, and buildbotcustom integration.

Comment 6

[[e-mail] (:bhearsum)]

Attachment: second pass, bug fixes + enhancements

New in this version (bugfixes excluded):
- Use newly invented _RUNTIME tag to clone tools repo for update verify configs
- Write wrapper for python script

I explicitly decided not to run clobberer or purge builds because they aren't required AFAICT. Happy to add them back in if folks feel it's important.

This version has been through Buildbot on mac, windows, and 32-bit Linux, on the 1.9.2 branch.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: chunked verify, v3

This patch has been tested for normal update verify and major update cases across all 1.9.2 platforms. Changes since the last patch:
- Drop reclone of tools, because script_repo_revision is now set to _RUNTIME tag
- Grab release tag from different property, also because of the above
- Use mkstemp instead of NamedTemporaryFile because the latter doesn't support delete=False on versions we care about

Comment 8

[[e-mail] (:bhearsum)]

Attachment: buildbotcustom integration

- Tag tools repo with _RUNTIME tags during updates.
- Generalize chunking default to DEFAULT_PARALLELIZATION; l10n and update verify chunks are independently controllable through release config overrides
- Generalize function that creates ||ized builder names
- Use ScriptFactory to run update verify / major update

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 523644 [details] [diff] [review]
chunked verify, v3

Obsolete this because it needs a small bit of change to actually work with bug 408453.

Comment 10

[[e-mail] (:bhearsum)]

Attachment: v4

This version is the same as before, but uses build_id + from_path to identify a release, to make bug 408453 work. Adds a couple more tests, too.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 523672 [details] [diff] [review]
buildbotcustom integration

This looked fine from 20000ft.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 526088 [details] [diff] [review]
v4

Sorry Ben, I won't be able to look at this anytime soon.

Comment 13

[[e-mail] (:bhearsum)]

Attachment: v5, fix "None" vs. None bug

I found a bug in my final tests of this patch that caused a Traceback in any case where "from" was set to None.

Comment 14

[[e-mail] (:bhearsum)]

Comment on attachment 531699 [details] [diff] [review]
v5, fix "None" vs. None bug

Damn, found another bug

Comment 15

[[e-mail] (:bhearsum)]

Attachment: v6, more bugs fixed

The following bugs have been fixed since the last version:
- Make chunking work with filenames with spaces
- Greatly improve the chunking algorithm to make chunks more even in terms of work.

I did full update verify runs on the latest 1.9.1, 1.9.2, and 2.0. Most chunks were green, with the red ones fully explainable:
- 1.9.1 Mac, 3.5.18 - kn, ml, and te updates receiving updates to 3.5.19build1 (tracked here: https://bugzilla.mozilla.org/show_bug.cgi?id=629256#c47)
- 1.9.2 - All OK
- 2.0 - en-US "failed" because they point to 5.0b1 on betatest, that's OK because that's where they're supposed to point.

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 523672 [details] [diff] [review]
buildbotcustom integration

Landed on default of buildbotcustom: changeset:   1550:e00aed00ca16

Comment 17

[[e-mail] (:bhearsum)]

Comment on attachment 532955 [details] [diff] [review]
v6, more bugs fixed

Landed: changeset:   1447:edac506701d0

Comment 18

[[e-mail] (:bhearsum)]

This hit production.