Closed Bug 467196 Opened 11 years ago Closed 11 years ago

Fx3.0 l10n nightlies are busted

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nthomas, Assigned: coop)

References

Details

I tried to deploy bug 462177 today and found that production-1.9-master had been left in a state where l10n builds are broken after reconfig or restart. With or without the two patches from that bug you get this in the twistd.log whenever the L10nNightly trigger fires

2008/11/29 19:25 PST [-] Traceback (most recent call last):
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/posixbase.py", line 226, in mainLoop
            self.runUntilCurrent()
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/base.py", line 555, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/builds/buildbot/trunk-automation-master/buildbotcustom/l10n/scheduler.py", line 173, in doPeriodicBuild
            self.helper.createL10nBuilds()
          File "/builds/buildbot/trunk-automation-master/buildbotcustom/l10n/scheduler.py", line 133, in createL10nBuilds
            d = defer.maybeDeferred(self.getLocales)
        --- <exception caught here> ---
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/defer.py", line 107, in maybeDeferred
            result = f(*args, **kw)
          File "/builds/buildbot/trunk-automation-master/buildbotcustom/l10n/scheduler.py", line 114, in getLocales
            args = ['cvs', '-q', '-d', self.CVSROOT, 'co', '-p', self.localesURL]
        <type 'exceptions.AttributeError'>: 'L10nMixin' object has no attribute 'localesURL'

At this point the changes in bug 462177 are applied, but I've touched nothing else (or reverted anything I tried to change).
s/trigger/Scheduler/.
Blocks: 462177
Glancing at the code, this would happen if the repotype isn't set. The codepath I see on tip should ensure that, though.

Is http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&file=scheduler.py&branch=&root=/cvsroot&subdir=mozilla/tools/buildbotcustom/l10n&command=DIFF_FRAMESET&rev1=1.3&rev2=1.4 deployed?
It's not on tip actually, it has rev 1.2 of buildbotcustom/l10n/scheduler.py (amongst others). Perhaps this is just fallout from a partially completed upgrade to 0.7.9 (see bug 464151), but it'd be great if config/code on disk matched what the buildbot instance was running.
Dupe of bug 465415? catlee is planning the upgrade there to get things working again AIUI.
(In reply to comment #4)
> Dupe of bug 465415? catlee is planning the upgrade there to get things working
> again AIUI.

Not a dupe. I have closed the previous bug since it was for another issue.

The current issue was to add the repoType parameter

Can someone check-in the patch mentioned in bug 464151#c12 which adds repoType='cvs'?

At some point, I will need to do a rewrite of the scheduler to forcefully force certain parameters depending on the repoType. By doing this, we would catch more incorrect parameters on reconfig rather than on twistd.log

NOTE: That the parameters are right now optional and set to None :(
(In reply to comment #5)
> (In reply to comment #4)
> > Dupe of bug 465415? catlee is planning the upgrade there to get things working
> > again AIUI.
> 
> Not a dupe. I have closed the previous bug since it was for another issue.
> 
> The current issue was to add the repoType parameter
> 
> Can someone check-in the patch mentioned in bug 464151#c12 which adds
> repoType='cvs'?

Whats left to do here?

1) The patch in bug 464151#c12 has landed. 
2) bug#464151 notes staging-1.9-master has been upgraded.
3) bug#464151 notes production-1.9-master cannot be upgraded until after FF3.0.5 ships on Tuesday.

Is that it? Do we expect the l10n nightlies to start working again once the buildbot upgrade is complete on production-1.9-master, or is there some other stuff that we need to do in the meanwhile?
Now that production-1.9-master is updated to buildbot 0.7.9, with the latest buildbotcustom, I've pulled in the patch from bug 464151 and re-enabled the l10n scheduler. Builds are due to start in 2hrs 40 mins.
Assignee: nobody → nthomas
Priority: -- → P2
Whadaya you know, it's still broken:

2008/12/09 17:00 PST [-] Traceback (most recent call last):
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/posixbase.py", line 226, in mainLoop
            self.runUntilCurrent()
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/base.py", line 555, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/builds/buildbot/trunk-automation-master/buildbotcustom/l10n/scheduler.py", line 191, in doPeriodicBuild
            self.helper.createL10nBuilds()
          File "/builds/buildbot/trunk-automation-master/buildbotcustom/l10n/scheduler.py", line 147, in createL10nBuilds
            d = defer.maybeDeferred(self.getLocales)
        --- <exception caught here> ---
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/defer.py", line 107, in maybeDeferred
            result = f(*args, **kw)
          File "/builds/buildbot/trunk-automation-master/buildbotcustom/l10n/scheduler.py", line 124, in getLocales
            args = ['cvs', '-q', '-d', self.cvsRoot, 'co', '-p', self.localesURL]
        <type 'exceptions.AttributeError'>: 'L10nMixin' object has no attribute 'cvsRoot'
Can you add a self.cvsRoot = cvsRoot line in here?

http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/l10n/scheduler.py
42   def __init__(self, scheduler,
43                repoPath    = None,
44                repoType    = 'cvs', baseTag = 'tip',
45                localesFile = None,
46                cvsRoot     = ':pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot',
47                locales     = None):
48       self.scheduler = scheduler
49       self.repoType = repoType
50       # set a default localesURL accordingly to the repoType if none has been set 
51       if repoType.find('hg') >= 0:
52         if not localesFile:
53           localesFile = "browser/locales/all-locales"          
54         self.localesURL = "http://hg.mozilla.org/%s/raw-file/%s/%s" \
55                           % (repoPath, baseTag, localesFile)
56       elif repoType.find('cvs') >= 0:
57         if not localesFile:
58           self.localesURL = "mozilla/browser/locales/all-locales"
59         else:
60           self.localesURL = localesFile
61           
62       # we are going to have a queue per builder attached to the scheduler
63       self.queue = {} 
64       for builderName in scheduler.builderNames:
65           self.queue[builderName] = []
66       # if the user wants to use something different than all locales
67       if locales:
68           self.locales = locales[:]
69       else:
70           self.locales = None

I can deal with this on Friday when I am back online if you prefer
I added the self.cvsRoot line as suggested by Armen, but we're now hitting a new error:

2008/12/10 16:01 PST [HTTPChannel,0,10.2.72.11] Upgrading buildbot.status.buildeMTPSenderFactory instance at 0x13c1d0ec>
2008/12/10 16:01 PST [Broker,0,10.2.71.10] Unhandled error in Deferred:
2008/12/10 16:01 PST [Broker,0,10.2.71.10] Traceback (most recent call last):
          File "/tools/python/lib/python2.5/site-packages/buildbot/process/build
er.py", line 637, in _startBuild_2
            d = build.startBuild(bs, self.expectations, sb)
          File "/tools/python/lib/python2.5/site-packages/buildbot/process/base.
py", line 354, in startBuild
            self.acquireLocks().addCallback(self._startBuild_2)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/interne
t/defer.py", line 191, in addCallback
            callbackKeywords=kw)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/interne
t/defer.py", line 182, in addCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/interne
t/defer.py", line 307, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/tools/python/lib/python2.5/site-packages/buildbot/process/base.
py", line 373, in _startBuild_2
            self.build_status.buildStarted(self)
          File "/tools/python/lib/python2.5/site-packages/buildbot/status/builde
r.py", line 1149, in buildStarted
            self.builder.buildStarted(self)
          File "/tools/python/lib/python2.5/site-packages/buildbot/status/builde
r.py", line 1678, in buildStarted
            receiver = w.buildStarted(self.getName(), s)
          File "/tools/python/lib/python2.5/site-packages/buildbot/status/tinder
box.py", line 135, in buildStarted
            self.buildMessage(name, build, "building")
          File "/tools/python/lib/python2.5/site-packages/buildbot/status/tinder
box.py", line 148, in buildMessage
            text += "%s tree: %s\n" % (t, build.getProperties().render(self.tree
))
          File "/tools/python/lib/python2.5/site-packages/buildbot/process/prope
rties.py", line 86, in render
            return value.render(self.pmap)
          File "/tools/python/lib/python2.5/site-packages/buildbot/process/prope
rties.py", line 156, in render
            s = self.fmtstring % pmap
          File "/tools/python/lib/python2.5/site-packages/buildbot/process/prope
rties.py", line 131, in __getitem__
            rv = properties[key]
          File "/tools/python/lib/python2.5/site-packages/buildbot/process/prope
rties.py", line 42, in __getitem__
            rv = self.properties[name][0]
        <type 'exceptions.KeyError'>: 'locale'
From the waterfall it looks like coop figured out a fix for this. Mind if we re-assign this to you coop ?
linux and mac seem to be in a funky state on the master?
(In reply to comment #12)
> linux and mac seem to be in a funky state on the master?

That's the error I posted above, Axel. It only appears in the twistd.log.
Assignee: nthomas → ccooper
Status: NEW → ASSIGNED
(In reply to comment #10)
>         <type 'exceptions.KeyError'>: 'locale'

This error appears when we try to force builds because we don't supply a list of locales by default. This locks up the slave, requiring a restart.

Regular nightlies seem to work fine, as evidenced by the Windows builder.
That makes me curious, how do you force l10n builds?
Currently we don't have a way to trigger them. That's the impetus behind bug 466498.
All 3 platforms generated their scheduled nightlies. Will follow-up the triggering issue in bug 466498.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.