Closed Bug 544035 Opened 10 years ago Closed 10 years ago

mobile localization repacks broken since feb3

Categories

(Release Engineering :: General, defect, P1)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: aki, Assigned: armenzg)

References

Details

(Whiteboard: [l10n][fennec l10n])

Attachments

(1 file)

This prevents them from being updated on reconfig.  Not sure if this is just for Fennec or for all repacks.

> aki|buildduty: is the en-US binary url something we have to restart for?
> Pike: no idea
> Pike: aki|buildduty: yes, looks like it
> aki|buildduty: =P
> Pike: aki|buildduty: the compare_attrs of the base build factory aren't extended by the custom props
> Pike: aki|buildduty: for none of our factories, for that matter, so any factory change requires a restart
> aki|buildduty: k
> Pike: aki|buildduty: background, anything that buildbot compares on reconfig that implements util.ComparableMixin needs to have the stuff to trigger a new instance on reconfig in their compare_attrs, either as a ComparableMixin or as primitive type that compares as intended on ==
This is for all factories, repack and not. compare_attrs isn't found once in buildbotcustom.process.factory.
I believe that reloading the factories causes the builders to be reconstructed.  In process/builder.py compareToSetup compares the old and new factory classes using !=, which will return True (i.e. the factories are different).

>>> import buildbot.process.factory
>>> f = buildbot.process.factory.BuildFactory()
>>> g = buildbot.process.factory.BuildFactory()
>>> f == g
True
>>> reload(buildbot.process.factory)
<module 'buildbot.process.factory' from 'buildbot/process/factory.pyc'>
>>> h = buildbot.process.factory.BuildFactory()
>>> f == h
False
>>> f != h
True
See the base class, http://hg.mozilla.org/build/buildbot/file/bd20b289b0f0/buildbot/util.py#l79.

It only differs if the attributes in compare_attrs are equal.
The comparison on line 95 would make the factories different post-reconfig.
Any idea why a reconfig didn't pick up the config change in the factory for mobile then?
Debs have been broken for much longer (7th of Jan?) but linux desktop stopped being triggered on Feb 3. We've had a restart of pm02 in between, no improvement.
Summary: Repacks missing compare_attrs → mobile localization repacks broken since feb3
Feb 3 was when the talos sendchanges to mv.m.c started breaking. ...
mozconfig and objdir landed for repacks as well.
Aki from my understanding this bug depends on bug 538699 and bug 506989.
Is this correct?

In my work for separating the multi-locale from the maemo build factory I have some goodies (repackaging from within scratchbox) that I can use to fix both bugs. I would still need the patch of bug 538699 to be landed.

I can try to fix this in the beginning of next week (I will be buildonduty though) and get some work off your shoulders if you want me to.
Depends on: 538699, 506989
Whiteboard: [l10n][fennec l10n]
Armen: that's correct.  However, those don't explain why linux desktop fennec repacks aren't triggering, but if we get those two fixed it'll fix most of this bug.
Assignee: nobody → armenzg
Status: NEW → ASSIGNED
Priority: -- → P1
Blocks: 550023
Blocks: 548178
This may be related:

(catlee says):
> I was poking around the build data, trying to figure out why we had 13,000
> jobs on january 29th, and found that we have a TON of maemo l10n builds that run.
>
> e.g. on the 29th we had:
> 3072 linux-fennec-1.9.2-l10n-dep
> 3072 w32mob-1.9.2-l10n-dep
> 3074 maemo-1.9.2-l10n-dep
>
> it's not limited to january 29th either.  On february 20th we had:
> 1903 maemo-1.9.2-l10n-dep
>
> Is this expected?  Any ideas why we have so many?
The builds can now green but we have the mobile l10n jobs disabled in bug 548178.
We have a re-spawning job for the "Maemo mozilla-1.9.2 l10n build" builder.

Trying to reproduce it on staging is going to be hard and the fastest way to get mobile l10n builds back is to just enable nightly builds without having repackages on-change. We can re-enable repackages-on-change once we figure out what is going on.
I have disabled the code but also moved it after mobile_master is actually loaded. We were currently loading the 1.9.2 poller before the builders were actually maemo 1.9.2 l10n builders actually existed. Not sure if when we re-enable repacks-on-change this will actually fixes it. I will investigate next week with time. Let's now just get the nightly l10n builds back.

I fixed the pollInterval to be 15*60 in both places.
Attachment #430711 - Flags: review?(aki)
Comment on attachment 430711 [details] [diff] [review]
disable mobile repacks-on-change but re-enable the nightly ones

This polling could be the cause; we're making educated guesses here.

Either way, commenting out will help this weekend, and moving+increasing the poll interval can possibly help fix the problem.
Attachment #430711 - Flags: review?(aki) → review+
Comment on attachment 430711 [details] [diff] [review]
disable mobile repacks-on-change but re-enable the nightly ones

Committed.
http://hg.mozilla.org/build/buildbot-configs/rev/89c3546bfef1
Backed out.
http://hg.mozilla.org/build/buildbot-configs/rev/deaf1e760376

We need a way to reproduce this on staging or finding the problem on the twistd logs.
Attachment #430711 - Flags: checked-in-
Blocks: 550940
In bug 550940 the nightly repacks have been re-added and the repacks-on-change for mobile have been disabled.

We will spend time later to try to get a staging master into the same stage as production was to determine the actual root cause. For now, we are back to a normal state (without repacks-on-change for mobile) where we are producing the nightly repacks.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.