555590 - Assertions on pm02 in buildbotcustom/steps/transfer.py for Fennec Desktop builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

2010-03-28 02:21:24-0700 [Broker,2063,10.2.90.29] <Build Win32 Fennec Desktop lorentz l10n>.startBuild
2010-03-28 02:21:24-0700 [Broker,2063,10.2.90.29] error while creating step, factory=buildbotcustom.steps.transfer.MozillaStageUpload, args={'username': 'ffxbld', 'workdir': 'w32mob-lorentz-l10n/firefox-lor
entz/dist', 'sshKey': 'ffxbld_dsa', 'remoteBasePath': '/home/ftp/pub/mobile', 'remoteCandidatesPath': '/home/ftp/pub/mobile', 'releaseToTinderboxBuilds': True, 'releaseToDated': False, 'milestone': 'mobile-lorentz-l10n',
 'uploadCompleteMar': False, 'packageGlob': <buildbot.process.properties.WithProperties instance at 0xd7a9f2c>, 'group': None, 'name': 'upload', 'remoteHost': 'stage.mozilla.org', 'tinderboxBuildsDir': 'mobile-lorentz-l1
0n', 'platform': '', 'releaseToLatest': True, 'dependToDated': False, 'uploadLangPacks': False, 'objdir': 'firefox-lorentz/dist'}
2010-03-28 02:21:24-0700 [Broker,2063,10.2.90.29] Build.setupBuild failed
2010-03-28 02:21:24-0700 [Broker,2063,10.2.90.29] Unhandled Error
  Traceback (most recent call last):
    File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/internet/defer.py", line 242, in callback
      self._startRunCallbacks(result)
    File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/internet/defer.py", line 307, in _startRunCallbacks
      self._runCallbacks()
    File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/internet/defer.py", line 323, in _runCallbacks
      self.result = callback(self.result, *args, **kw)
    File "/tools/buildbot-production/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg/buildbot/process/builder.py", line 802, in _startBuild_2
      d = build.startBuild(bs, self.expectations, sb)
  --- <exception caught here> ---
    File "/tools/buildbot-production/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg/buildbot/process/base.py", line 355, in startBuild
      self.setupBuild(expectations) # create .steps
    File "/tools/buildbot-production/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg/buildbot/process/base.py", line 405, in setupBuild
      step = factory(**args)
    File "/tools/buildbotcustom/buildbotcustom/steps/transfer.py", line 140, in __init__
      assert platform in ('win32', 'linux', 'macosx')
  exceptions.AssertionError: 

Note the 'platform': '' in the args for buildbotcustom.steps.transfer.MozillaStageUpload.

We're getting about 200 of these a day, distributed like this:
$ grep -B3 'Build.setupBuild failed' twistd.log.?? | grep startBuild | cut -d' ' -f4- | sort | uniq -c
     42 <Build Linux Fennec Desktop lorentz l10n>.startBuild
     42 <Build Linux Fennec Desktop mozilla-1.9.2 l10n>.startBuild
     41 <Build Linux Fennec Desktop mozilla-central l10n>.startBuild
     42 <Build Win32 Fennec Desktop lorentz l10n>.startBuild
     42 <Build Win32 Fennec Desktop mozilla-1.9.2 l10n>.startBuild

Betting it's fallout from bug 545539.

Comment 1

[[e-mail] (:bhearsum)]

It certainly seems likely that the mac 10.6 work busted it, although we didn't land any production buildbot-configs changes the other day, so the consumers of this class shouldn't have changed because of us.

Comment 2

[[e-mail] (:bhearsum)]

So, it looks to me like BaseRepackFactory is overriding what MobileNightlyRepackFactory sets as platform, since it's not passed along.

This would've started happening in http://hg.mozilla.org/build/buildbotcustom/rev/5979e1bb7c9a, which is all the way back on Feb 10.

I'm testing this theory, and a fix, in staging.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: pass platform along to BaseRepackFactory

This patch gets us passing platform to BaseRepackFactory, which means it will stop overriding it with an empty string. I also found that 'verbose' is passed to processCommand which is invalid here, because BaseRepackFactory's processCommand doesn't support it.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: pass platform to mobile repack factories, now required

Now that 'platform' is being passed to BaseRepackFactory we need to pass it to all of the child classes. Seems to me that since we got by without this before, that this these classes probably need some refactoring. That's another bug, though!

I managed to get a clean Linux desktop repack in staging with this + the buildbotcustom patch.

Comment 5

[Armen [:armenzg]]

Comment on attachment 435608 [details] [diff] [review]
pass platform along to BaseRepackFactory

r=me

It adds one more property "eval" but that is fine. Ben got a green run all through. I will file a follow up bug.

This patch fixes desktop L10n builds which I doubt they have worked in a long time.

Comment 6

[Armen [:armenzg]]

Comment on attachment 435609 [details] [diff] [review]
pass platform to mobile repack factories, now required

r=me

Comment 8

[[e-mail] (:bhearsum)]

I forced a build along, which went fine:
http://production-master02.build.mozilla.org:8010/builders/Linux%20Fennec%20Desktop%20mozilla-1.9.2%20l10n/builds/126
...now I'm paranoid that it's only broken through the Scheduler, though, so I forced an en-US nightly and we'll see.

Comment 9

[[e-mail] (:bhearsum)]

triggered builds worked fine too.

Comment 10

[[e-mail] (:bhearsum)]

I should note that we're still not sure why this was tripped on the 27th. Both Armen and I think this should've been happening for a long time before that.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Perhaps we only started noticing since bug 526149 ?

Comment 12

[[e-mail] (:bhearsum)]

Perhaps, but the twistd.log's didn't show any exceptions for the 25th and the 26th.

Comment 13

[Armen [:armenzg]]

This fixed the l10n desktop repacks (bug 555534)