465352 - no nightly scheduler for linux-64

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[[e-mail] (:bhearsum)]

Right now we're in a state where only after *all* en-US nightlies finish do they l10n builds get triggered. This is because we have a single Nightly scheduler which triggers all OS' for a branch. We'll have to separate this out into a scheduler per OS.

Comment 1

[[e-mail] (:bhearsum)]

Attachment: fix the nightly schedulers

This patch simply creates a nightly scheduler for each nightly builder, and sets up a Dependent scheduler for each l10n builder. I gave this a go on staging-master and it worked exactly as intended.

Comment 2

[[e-mail] (:bhearsum)]

Comment on attachment 348584 [details] [diff] [review]
fix the nightly schedulers

changeset:   516:c387e1ae4dff

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

The linux64 builder doesn't have a nightly scheduler.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #3)
> The linux64 builder doesn't have a nightly scheduler.

Perhaps this is unrelated, but the linux64 nightly I forced at 2:06 PST doesn't appear on the Firefox tree either. This is after 20 minutes, and builds that started later are appearing.

Comment 5

[Armen [:armenzg]]

> The linux64 builder doesn't have a nightly scheduler.
    66             if platform in ('linux','win32','macosx'):
    67                 nightlyBuilders.append('%s nightly' % \
    68                          branch['platforms'][platform]['base_name'])
    69                 l10nNightlyBuilders.append('%s l10n nightly' % \
    70                          branch['platforms'][platform]['base_name'])
Most likely, right?(In reply to comment #3)

If so, do we want l10n nightly builds on linux-64 as well, right?

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

That looks like a smoking gun to me. BTW, the snippet you pasted is not the current tip, which has the l10nNightlyBuilders lines commented out. If it is to be uncommented at some point in future then we'll want to special case it to skip linux64.

> If so, do we want l10n nightly builds on linux-64 as well, right?

We most certainly do not.

Comment 7

[[e-mail] (:bhearsum)]

I had a quick look at this and it's going to be tricky to get right. I'm happy to do it after branching/beta2 - don't have time right now. I don't think linux64 nightlies are _that_ critical, though.

Comment 8

[Chris Cooper [:coop] (he/him)]

This is only about linux-64 now, so it no longer blocks bug 460791.

Comment 9

[[e-mail] (:bhearsum)]

Attachment: fix the linux 64-bit schedulers

Alright, this patch fixes the linux 64 bit schedulers _without_ breaking anything else (wow!). Mostly it's just a refactor of the nightly schedulers. I've turned l10nNightlyBuilders into a mapping between the en-US builder names and l10n builder names - this makes it easier to know later on whether or not to add an l10 scheduler.

I also fixed up the main TinderboxMailNotifier to stop sending l10n builds to the build tree - this would've become a problem in production.

Comment 10

[Axel Hecht [:Pike]]

You're double checking branch['enable_l10n'] in this line

+        if branch['enable_l10n'] and builder in l10nNightlyBuilders:

as you already checked for enable_l10n before setting l10nNightlyBuilders.

Comment 12

[Armen [:armenzg]]

Comment on attachment 351409 [details] [diff] [review]
fix the linux 64-bit schedulers

>+            builder = '%s nightly' % branch['platforms'][platform]['base_name']
>+            nightlyBuilders.append(builder)
>+            if branch['enable_l10n'] and platform in ('linux','win32','macosx'):
>+                l10nNightlyBuilders[builder] = \
>+                  '%s l10n nightly' % branch['platforms'][platform]['base_name']
Why not:
l10nNightlyBuilders[builder] = '%s l10n nightly' % builder

>-            builders=l10nNightlyBuilders,
>+            builders=l10nNightlyBuilders.values(),
Can you add this change to line 94 as well?
There is 2 TboxMailNotifiers one for Mozilla-l10n and the other for Mozilla-l10n-%locale%

Don't tell John that I am checking my bugmail! :)

Comment 13

[[e-mail] (:bhearsum)]

(In reply to comment #12)
> (From update of attachment 351409 [details] [diff] [review])
> >+            builder = '%s nightly' % branch['platforms'][platform]['base_name']
> >+            nightlyBuilders.append(builder)
> >+            if branch['enable_l10n'] and platform in ('linux','win32','macosx'):
> >+                l10nNightlyBuilders[builder] = \
> >+                  '%s l10n nightly' % branch['platforms'][platform]['base_name']
> Why not:
> l10nNightlyBuilders[builder] = '%s l10n nightly' % builder
> 

Because I didn't want to change the names of of the l10n nightly schedulers.

> >-            builders=l10nNightlyBuilders,
> >+            builders=l10nNightlyBuilders.values(),
> Can you add this change to line 94 as well?
> There is 2 TboxMailNotifiers one for Mozilla-l10n and the other for
> Mozilla-l10n-%locale%

Yeah, that's a good catch.

Comment 14

[Chris Cooper [:coop] (he/him)]

Comment on attachment 351409 [details] [diff] [review]
fix the linux 64-bit schedulers

r+ with Armen's suggested change to the other Notifier.

Comment 15

[[e-mail] (:bhearsum)]

Attachment: same as before w/ addition TinderboxMailNotifier fixed

landed on staging
changeset:   583:8db395f8b23e

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 351578 [details] [diff] [review]
same as before w/ addition TinderboxMailNotifier fixed

Just landed this in production, too: changeset:   585:3e9c440d83d5

The linux 64-bit schedulers are showing pending builds for 2am tomorrow, hooray.

Comment 17

[Chris Cooper [:coop] (he/him)]

bhearsum: we tried to enable l10n on m-c again tonight, but the builders started grabbing slaves and not giving them back. nthomas found the following in the logs:

2008/12/05 17:30 PST [Broker,219,10.2.71.249] <Build Linux mozilla-central l10n nightly>.startBuild
2008/12/05 17:30 PST [Broker,219,10.2.71.249] acquireLocks(step <Build Linux mozilla-central l10n nightly>, locks [])
2008/12/05 17:30 PST [Broker,219,10.2.71.249] sending mail (363 bytes) to ['[e-mail]']
2008/12/05 17:30 PST [Broker,219,10.2.71.249] Starting factory <twisted.mail.smtp.SMTPSenderFactory instance at 0xb54576c>
2008/12/05 17:30 PST [Broker,219,10.2.71.249] Unhandled error in Deferred:
2008/12/05 17:30 PST [Broker,219,10.2.71.249] Traceback (most recent call last):
          File "/tools/buildbot-079/lib/python2.5/site-packages/buildbot/process/builder.py", line 637, in _startBuild_2
            d = build.startBuild(bs, self.expectations, sb)
          File "/tools/buildbot/lib/python2.5/site-packages/buildbot/process/base.py", line 354, in startBuild
            self.acquireLocks().addCallback(self._startBuild_2)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/defer.py", line 191, in addCallback
            callbackKeywords=kw)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/defer.py", line 182, in addCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/defer.py", line 307, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/tools/buildbot/lib/python2.5/site-packages/buildbot/process/base.py", line 373, in _startBuild_2
            self.build_status.buildStarted(self)
          File "/tools/buildbot-079/lib/python2.5/site-packages/buildbot/status/builder.py", line 1149, in buildStarted
            self.builder.buildStarted(self)
          File "/tools/buildbot-079/lib/python2.5/site-packages/buildbot/status/builder.py", line 1678, in buildStarted
            receiver = w.buildStarted(self.getName(), s)
          File "/tools/buildbot-079/lib/python2.5/site-packages/buildbot/status/tinderbox.py", line 135, in buildStarted
            self.buildMessage(name, build, "building")
          File "/tools/buildbot-079/lib/python2.5/site-packages/buildbot/status/tinderbox.py", line 148, in buildMessage
            text += "%s tree: %s\n" % (t, self.tree.render(build))
          File "/tools/buildbot-079/lib/python2.5/site-packages/buildbot/process/properties.py", line 156, in render
            s = self.fmtstring % pmap
        <type 'exceptions.AttributeError'>: BuildStatus instance has no attribute '__getitem__'

2008/12/05 17:30 PST [SMTPSender,client] Stopping factory <twisted.mail.smtp.SMTPSenderFactory instance at 0xb54576c>
2008/12/05 17:30 PST [SMTPSender,client] Stopping factory <twisted.mail.smtp.SMTPSenderFactory instance at 0xae3942c>

It seems to be dying while trying to send mail. Your patch made a change to the notifiers, but we're not sure if that actually got triggered in your staging run this afternoon. Any thoughts?

Comment 18

[Chris Cooper [:coop] (he/him)]

Possibly easier to read the error log here: http://pastebin.mozilla.org/588091

Comment 19

[Axel Hecht [:Pike]]

That seems to be bug 464151. The patches in this bug don't have anything to do with that bustage, resolving this one FIXED again.