467196 - Fx3.0 l10n nightlies are busted

Reporter

Description

•

16 years ago

I tried to deploy bug 462177 today and found that production-1.9-master had been left in a state where l10n builds are broken after reconfig or restart. With or without the two patches from that bug you get this in the twistd.log whenever the L10nNightly trigger fires

2008/11/29 19:25 PST [-] Traceback (most recent call last):
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/posixbase.py", line 226, in mainLoop
            self.runUntilCurrent()
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/base.py", line 555, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/builds/buildbot/trunk-automation-master/buildbotcustom/l10n/scheduler.py", line 173, in doPeriodicBuild
            self.helper.createL10nBuilds()
          File "/builds/buildbot/trunk-automation-master/buildbotcustom/l10n/scheduler.py", line 133, in createL10nBuilds
            d = defer.maybeDeferred(self.getLocales)
        --- <exception caught here> ---
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/defer.py", line 107, in maybeDeferred
            result = f(*args, **kw)
          File "/builds/buildbot/trunk-automation-master/buildbotcustom/l10n/scheduler.py", line 114, in getLocales
            args = ['cvs', '-q', '-d', self.CVSROOT, 'co', '-p', self.localesURL]
        <type 'exceptions.AttributeError'>: 'L10nMixin' object has no attribute 'localesURL'

At this point the changes in bug 462177 are applied, but I've touched nothing else (or reverted anything I tried to change).

Nick Thomas [:nthomas] (UTC+12)

Reporter

Comment 1

•

16 years ago

s/trigger/Scheduler/.

Nick Thomas [:nthomas] (UTC+12)

Reporter

Updated

•

16 years ago

Blocks: 462177

Axel Hecht [:Pike]

Comment 2

•

16 years ago

Glancing at the code, this would happen if the repotype isn't set. The codepath I see on tip should ensure that, though.

Is http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&file=scheduler.py&branch=&root=/cvsroot&subdir=mozilla/tools/buildbotcustom/l10n&command=DIFF_FRAMESET&rev1=1.3&rev2=1.4 deployed?

Nick Thomas [:nthomas] (UTC+12)

Reporter

Comment 3

•

16 years ago

It's not on tip actually, it has rev 1.2 of buildbotcustom/l10n/scheduler.py (amongst others). Perhaps this is just fallout from a partially completed upgrade to 0.7.9 (see bug 464151), but it'd be great if config/code on disk matched what the buildbot instance was running.

Chris Cooper [:coop] (he/him)

Assignee

Comment 4

•

16 years ago

Dupe of bug 465415? catlee is planning the upgrade there to get things working again AIUI.

Armen [:armenzg]

Comment 5

•

16 years ago

(In reply to comment #4)
> Dupe of bug 465415? catlee is planning the upgrade there to get things working
> again AIUI.

Not a dupe. I have closed the previous bug since it was for another issue.

The current issue was to add the repoType parameter

Can someone check-in the patch mentioned in bug 464151#c12 which adds repoType='cvs'?

At some point, I will need to do a rewrite of the scheduler to forcefully force certain parameters depending on the repoType. By doing this, we would catch more incorrect parameters on reconfig rather than on twistd.log

NOTE: That the parameters are right now optional and set to None :(

John O'Duinn [:joduinn] (please use "needinfo?" flag)

Updated

•

16 years ago

Depends on: 464151

John O'Duinn [:joduinn] (please use "needinfo?" flag)

Updated

•

16 years ago

Blocks: 458153

John O'Duinn [:joduinn] (please use "needinfo?" flag)

Comment 6

•

16 years ago

(In reply to comment #5)
> (In reply to comment #4)
> > Dupe of bug 465415? catlee is planning the upgrade there to get things working
> > again AIUI.
> 
> Not a dupe. I have closed the previous bug since it was for another issue.
> 
> The current issue was to add the repoType parameter
> 
> Can someone check-in the patch mentioned in bug 464151#c12 which adds
> repoType='cvs'?

Whats left to do here?

1) The patch in bug 464151#c12 has landed. 
2) bug#464151 notes staging-1.9-master has been upgraded.
3) bug#464151 notes production-1.9-master cannot be upgraded until after FF3.0.5 ships on Tuesday.

Is that it? Do we expect the l10n nightlies to start working again once the buildbot upgrade is complete on production-1.9-master, or is there some other stuff that we need to do in the meanwhile?

Nick Thomas [:nthomas] (UTC+12)

Reporter

Comment 7

•

16 years ago

Now that production-1.9-master is updated to buildbot 0.7.9, with the latest buildbotcustom, I've pulled in the patch from bug 464151 and re-enabled the l10n scheduler. Builds are due to start in 2hrs 40 mins.

Assignee: nobody → nthomas

Priority: -- → P2

Nick Thomas [:nthomas] (UTC+12)

Reporter

Comment 8

•

16 years ago

Whadaya you know, it's still broken:

2008/12/09 17:00 PST [-] Traceback (most recent call last):
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/posixbase.py", line 226, in mainLoop
            self.runUntilCurrent()
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/base.py", line 555, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/builds/buildbot/trunk-automation-master/buildbotcustom/l10n/scheduler.py", line 191, in doPeriodicBuild
            self.helper.createL10nBuilds()
          File "/builds/buildbot/trunk-automation-master/buildbotcustom/l10n/scheduler.py", line 147, in createL10nBuilds
            d = defer.maybeDeferred(self.getLocales)
        --- <exception caught here> ---
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/defer.py", line 107, in maybeDeferred
            result = f(*args, **kw)
          File "/builds/buildbot/trunk-automation-master/buildbotcustom/l10n/scheduler.py", line 124, in getLocales
            args = ['cvs', '-q', '-d', self.cvsRoot, 'co', '-p', self.localesURL]
        <type 'exceptions.AttributeError'>: 'L10nMixin' object has no attribute 'cvsRoot'

Armen [:armenzg]

Comment 9

•

16 years ago

Can you add a self.cvsRoot = cvsRoot line in here?

http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/l10n/scheduler.py
42   def __init__(self, scheduler,
43                repoPath    = None,
44                repoType    = 'cvs', baseTag = 'tip',
45                localesFile = None,
46                cvsRoot     = ':pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot',
47                locales     = None):
48       self.scheduler = scheduler
49       self.repoType = repoType
50       # set a default localesURL accordingly to the repoType if none has been set 
51       if repoType.find('hg') >= 0:
52         if not localesFile:
53           localesFile = "browser/locales/all-locales"          
54         self.localesURL = "http://hg.mozilla.org/%s/raw-file/%s/%s" \
55                           % (repoPath, baseTag, localesFile)
56       elif repoType.find('cvs') >= 0:
57         if not localesFile:
58           self.localesURL = "mozilla/browser/locales/all-locales"
59         else:
60           self.localesURL = localesFile
61           
62       # we are going to have a queue per builder attached to the scheduler
63       self.queue = {} 
64       for builderName in scheduler.builderNames:
65           self.queue[builderName] = []
66       # if the user wants to use something different than all locales
67       if locales:
68           self.locales = locales[:]
69       else:
70           self.locales = None

I can deal with this on Friday when I am back online if you prefer

Chris Cooper [:coop] (he/him)

Assignee

Comment 10

•

16 years ago

I added the self.cvsRoot line as suggested by Armen, but we're now hitting a new error:

2008/12/10 16:01 PST [HTTPChannel,0,10.2.72.11] Upgrading buildbot.status.buildeMTPSenderFactory instance at 0x13c1d0ec>
2008/12/10 16:01 PST [Broker,0,10.2.71.10] Unhandled error in Deferred:
2008/12/10 16:01 PST [Broker,0,10.2.71.10] Traceback (most recent call last):
          File "/tools/python/lib/python2.5/site-packages/buildbot/process/build
er.py", line 637, in _startBuild_2
            d = build.startBuild(bs, self.expectations, sb)
          File "/tools/python/lib/python2.5/site-packages/buildbot/process/base.
py", line 354, in startBuild
            self.acquireLocks().addCallback(self._startBuild_2)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/interne
t/defer.py", line 191, in addCallback
            callbackKeywords=kw)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/interne
t/defer.py", line 182, in addCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/interne
t/defer.py", line 307, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/tools/python/lib/python2.5/site-packages/buildbot/process/base.
py", line 373, in _startBuild_2
            self.build_status.buildStarted(self)
          File "/tools/python/lib/python2.5/site-packages/buildbot/status/builde
r.py", line 1149, in buildStarted
            self.builder.buildStarted(self)
          File "/tools/python/lib/python2.5/site-packages/buildbot/status/builde
r.py", line 1678, in buildStarted
            receiver = w.buildStarted(self.getName(), s)
          File "/tools/python/lib/python2.5/site-packages/buildbot/status/tinder
box.py", line 135, in buildStarted
            self.buildMessage(name, build, "building")
          File "/tools/python/lib/python2.5/site-packages/buildbot/status/tinder
box.py", line 148, in buildMessage
            text += "%s tree: %s\n" % (t, build.getProperties().render(self.tree
))
          File "/tools/python/lib/python2.5/site-packages/buildbot/process/prope
rties.py", line 86, in render
            return value.render(self.pmap)
          File "/tools/python/lib/python2.5/site-packages/buildbot/process/prope
rties.py", line 156, in render
            s = self.fmtstring % pmap
          File "/tools/python/lib/python2.5/site-packages/buildbot/process/prope
rties.py", line 131, in __getitem__
            rv = properties[key]
          File "/tools/python/lib/python2.5/site-packages/buildbot/process/prope
rties.py", line 42, in __getitem__
            rv = self.properties[name][0]
        <type 'exceptions.KeyError'>: 'locale'

Nick Thomas [:nthomas] (UTC+12)

Reporter

Comment 11

•

16 years ago

From the waterfall it looks like coop figured out a fix for this. Mind if we re-assign this to you coop ?

Axel Hecht [:Pike]

Comment 12

•

16 years ago

linux and mac seem to be in a funky state on the master?

Chris Cooper [:coop] (he/him)

Assignee

Comment 13

•

16 years ago

(In reply to comment #12)
> linux and mac seem to be in a funky state on the master?

That's the error I posted above, Axel. It only appears in the twistd.log.

Assignee: nthomas → ccooper

Status: NEW → ASSIGNED

Chris Cooper [:coop] (he/him)

Assignee

Comment 14

•

16 years ago

(In reply to comment #10)
>         <type 'exceptions.KeyError'>: 'locale'

This error appears when we try to force builds because we don't supply a list of locales by default. This locks up the slave, requiring a restart.

Regular nightlies seem to work fine, as evidenced by the Windows builder.

Axel Hecht [:Pike]

Comment 15

•

16 years ago

That makes me curious, how do you force l10n builds?

Chris Cooper [:coop] (he/him)

Assignee

Comment 16

•

16 years ago

Currently we don't have a way to trigger them. That's the impetus behind bug 466498.

Chris Cooper [:coop] (he/him)

Assignee

Comment 17

•

16 years ago

All 3 platforms generated their scheduled nightlies. Will follow-up the triggering issue in bug 466498.

Status: ASSIGNED → RESOLVED

Closed: 16 years ago

Resolution: --- → FIXED

Nobody; OK to take it and work on it

Updated

•

11 years ago

Product: mozilla.org → Release Engineering

Bugzilla

Quick Search

Fx3.0 l10n nightlies are busted

Categories

(Release Engineering :: General, defect, P2)

Tracking

(Not tracked)

People

(Reporter: nthomas, Assigned: coop)

References

Details

Crash Data

Security

(public)

User Story

Description

Comment 1

Updated

Comment 2

Comment 3

Comment 4

Comment 5

Updated

Updated

Comment 6

Comment 7

Comment 8

Comment 9

Comment 10

Comment 11

Comment 12

Comment 13

Comment 14

Comment 15

Comment 16

Comment 17

Updated