571199 - Run verify_l10n.sh per platform

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Rail Aliiev [:rail]]

l10n verification may be split and run for each platform separately after landing the tools patch in Bug 559057. This will speed up the release automation and will reduce required disk space.

Comment 1

[Rail Aliiev [:rail]]

Attachment: tools changes

Comment 2

[Rail Aliiev [:rail]]

Attachment: buildbot-configs changes

Comment 3

[Rail Aliiev [:rail]]

Attachment: buildbotcustom changes

> +                  '%s:/home/ftp/pub/%s/nightly/%s-candidates/build%s/%s' %

Absence of trailing slash will create the same directory hierarchy as before, but for the current platform only.

Comment 4

[Rail Aliiev [:rail]]

Attachment: buildbot-configs changes

Added missing schedulers.

Comment 5

[Rail Aliiev [:rail]]

I just tested the patches in staging. win32, macosx and linux passed the test. 

* Metadiffs created for all platforms are the same (http://pastebin.mozilla.org/736115)

* Disk usage:
2.7G	macosx_l10n_verification/
1.2G	win32_l10n_verification/
1.4G	linux_l10n_verification/

Looks like we may reduce buildSpace from 14G to 5G.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 451011 [details] [diff] [review]
buildbotcustom changes

>diff --git a/process/factory.py b/process/factory.py
> class L10nVerifyFactory(ReleaseFactory):
>     def __init__(self, cvsroot, stagingServer, productName, version,
>                  buildNumber, oldVersion, oldBuildNumber,
>-                 l10nPlatforms, verifyDir='verify', linuxExtension='bz2',
>+                 platform, verifyDir='verify', linuxExtension='bz2',
>                  buildSpace=14, **kwargs):

Go ahead and reduce the buildSpace to 4, and CC some T'bird, SeaMonkey and Calendar folks so they know how to make matching changes to their release_master.py's.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 451012 [details] [diff] [review]
buildbot-configs changes

>diff --git a/mozilla2/release_master.py b/mozilla2/release_master.py
>+    builders.append({
>+        'name': '%s_l10n_verification' % platform,
>+        'slavenames': branchConfig['platforms']['macosx']['slaves'],
>+        'category': 'release',
>+        'builddir': '%s_l10n_verification' % platform,
>+        'factory': l10n_verification_factory,
>+        'nextSlave': _nextFastSlave,
>+    })

If this ends up gobbling up fast mac slaves then we can come back later and switch it run on the native platforms.

Comment 8

[Rail Aliiev [:rail]]

(In reply to comment #7)
> If this ends up gobbling up fast mac slaves then we can come back later and
> switch it run on the native platforms.

+1. The only thing which is my mind, we have to have 7z.exe on win32 slaves in PATH.

> >+                 platform, verifyDir='verify', linuxExtension='bz2',
> >                  buildSpace=14, **kwargs):
> 
> Go ahead and reduce the buildSpace to 4, and CC some T'bird, SeaMonkey and
> Calendar folks so they know how to make matching changes to their
> release_master.py's.

I'll fix tb, sm, calendar configs.

CCing Calendar, Thunderbird and Seamonkey people.

Comment 9

[Rail Aliiev [:rail]]

Attachment: buildbotcustom changes

One liner:
-                 buildSpace=14, **kwargs):
+                 buildSpace=4, **kwargs):

Keeping r+.

Comment 10

[Rail Aliiev [:rail]]

Attachment: buildbot-configs changes

The same changes applied to seamonkey, calendar and thunderbird directories.

Nick, could you review this version (interdiff is here: http://diff.pastebin.mozilla.org/736775)?

1. L10nVerifyFactory requires platform instead of l10nPlatforms
2. 'l10n_verification' -> '%s_l10n_verification' % platform

Comment 11

[Philipp Kewisch [:Fallen][📱📆]]

Comment on attachment 451997 [details] [diff] [review]
buildbot-configs changes

r=philipp for the calendar part of this patch.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 451997 [details] [diff] [review]
buildbot-configs changes

Forwarding r+ on the Firefox parts. Setting r? for SeaMonkey and Thunderbird

Comment 13

[Rail Aliiev [:rail]]

Attachment: buildbot-configs changes

Refreshed bitrotten patch. Keeping r+.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 451009 [details] [diff] [review]
tools changes

http://hg.mozilla.org/build/tools/rev/3a23c12b9e18

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 451988 [details] [diff] [review]
buildbotcustom changes

http://hg.mozilla.org/build/buildbotcustom/rev/e34f1d3d2477

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 455115 [details] [diff] [review]
buildbot-configs changes

http://hg.mozilla.org/build/buildbot-configs/rev/a5b471701909

Comment 17

[Rail Aliiev [:rail]]

Tested during 3.6.7 build1 and 3.5.11 build1

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

The matching clobberer change got done in bug 584620.