544035 - mobile localization repacks broken since feb3

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Aki Sasaki (not active)]

This prevents them from being updated on reconfig.  Not sure if this is just for Fennec or for all repacks.

> aki|buildduty: is the en-US binary url something we have to restart for?
> Pike: no idea
> Pike: aki|buildduty: yes, looks like it
> aki|buildduty: =P
> Pike: aki|buildduty: the compare_attrs of the base build factory aren't extended by the custom props
> Pike: aki|buildduty: for none of our factories, for that matter, so any factory change requires a restart
> aki|buildduty: k
> Pike: aki|buildduty: background, anything that buildbot compares on reconfig that implements util.ComparableMixin needs to have the stuff to trigger a new instance on reconfig in their compare_attrs, either as a ComparableMixin or as primitive type that compares as intended on ==

Comment 1

[Axel Hecht [:Pike]]

This is for all factories, repack and not. compare_attrs isn't found once in buildbotcustom.process.factory.

Comment 2

[Chris AtLee [:catlee]]

I believe that reloading the factories causes the builders to be reconstructed.  In process/builder.py compareToSetup compares the old and new factory classes using !=, which will return True (i.e. the factories are different).

>>> import buildbot.process.factory
>>> f = buildbot.process.factory.BuildFactory()
>>> g = buildbot.process.factory.BuildFactory()
>>> f == g
True
>>> reload(buildbot.process.factory)
<module 'buildbot.process.factory' from 'buildbot/process/factory.pyc'>
>>> h = buildbot.process.factory.BuildFactory()
>>> f == h
False
>>> f != h
True

Comment 3

[Axel Hecht [:Pike]]

See the base class, http://hg.mozilla.org/build/buildbot/file/bd20b289b0f0/buildbot/util.py#l79.

It only differs if the attributes in compare_attrs are equal.

Comment 4

[Chris AtLee [:catlee]]

The comparison on line 95 would make the factories different post-reconfig.

Comment 5

[Axel Hecht [:Pike]]

Any idea why a reconfig didn't pick up the config change in the factory for mobile then?

Comment 6

[Aki Sasaki (not active)]

Debs have been broken for much longer (7th of Jan?) but linux desktop stopped being triggered on Feb 3. We've had a restart of pm02 in between, no improvement.

Comment 7

[Aki Sasaki (not active)]

Feb 3 was when the talos sendchanges to mv.m.c started breaking. ...

Comment 8

[Aki Sasaki (not active)]

mozconfig and objdir landed for repacks as well.

Comment 9

[Armen [:armenzg]]

Aki from my understanding this bug depends on bug 538699 and bug 506989.
Is this correct?

In my work for separating the multi-locale from the maemo build factory I have some goodies (repackaging from within scratchbox) that I can use to fix both bugs. I would still need the patch of bug 538699 to be landed.

I can try to fix this in the beginning of next week (I will be buildonduty though) and get some work off your shoulders if you want me to.

Comment 10

[Aki Sasaki (not active)]

Armen: that's correct.  However, those don't explain why linux desktop fennec repacks aren't triggering, but if we get those two fixed it'll fix most of this bug.

Comment 11

[Aki Sasaki (not active)]

This may be related:

(catlee says):
> I was poking around the build data, trying to figure out why we had 13,000
> jobs on january 29th, and found that we have a TON of maemo l10n builds that run.
>
> e.g. on the 29th we had:
> 3072 linux-fennec-1.9.2-l10n-dep
> 3072 w32mob-1.9.2-l10n-dep
> 3074 maemo-1.9.2-l10n-dep
>
> it's not limited to january 29th either.  On february 20th we had:
> 1903 maemo-1.9.2-l10n-dep
>
> Is this expected?  Any ideas why we have so many?

Comment 12

[Armen [:armenzg]]

The builds can now green but we have the mobile l10n jobs disabled in bug 548178.
We have a re-spawning job for the "Maemo mozilla-1.9.2 l10n build" builder.

Trying to reproduce it on staging is going to be hard and the fastest way to get mobile l10n builds back is to just enable nightly builds without having repackages on-change. We can re-enable repackages-on-change once we figure out what is going on.

Comment 13

[Armen [:armenzg]]

Attachment: disable mobile repacks-on-change but re-enable the nightly ones

I have disabled the code but also moved it after mobile_master is actually loaded. We were currently loading the 1.9.2 poller before the builders were actually maemo 1.9.2 l10n builders actually existed. Not sure if when we re-enable repacks-on-change this will actually fixes it. I will investigate next week with time. Let's now just get the nightly l10n builds back.

I fixed the pollInterval to be 15*60 in both places.

Comment 14

[Aki Sasaki (not active)]

Comment on attachment 430711 [details] [diff] [review]
disable mobile repacks-on-change but re-enable the nightly ones

This polling could be the cause; we're making educated guesses here.

Either way, commenting out will help this weekend, and moving+increasing the poll interval can possibly help fix the problem.

Comment 15

[Armen [:armenzg]]

Comment on attachment 430711 [details] [diff] [review]
disable mobile repacks-on-change but re-enable the nightly ones

Committed.
http://hg.mozilla.org/build/buildbot-configs/rev/89c3546bfef1
Backed out.
http://hg.mozilla.org/build/buildbot-configs/rev/deaf1e760376

We need a way to reproduce this on staging or finding the problem on the twistd logs.

Comment 16

[Armen [:armenzg]]

In bug 550940 the nightly repacks have been re-added and the repacks-on-change for mobile have been disabled.

We will spend time later to try to get a staging master into the same stage as production was to determine the actual root cause. For now, we are back to a normal state (without repacks-on-change for mobile) where we are producing the nightly repacks.