1366729 - Android Nightlys broken since may 19th

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, defect)

Description

[Carsten Book [:Tomcat]]

Seems the Android Trunk nightlys are broken somehow since sunday with https://treeherder.mozilla.org/logviewer.html#?job_id=100871919&repo=mozilla-central&lineNumber=56590

[task 2017-05-22T10:48:14.955679Z] 10:48:14     INFO -  10:48:14     INFO -  /home/worker/workspace/build/src/toolkit/mozapps/installer/packager.mk:39: recipe for target 'stage-package' failed

Catlee: do you know who could take a look ?

Comment 1

[Axel Hecht [:Pike]]

Maybe bug 1366169? That introduced an error message that shows up in the log often. That landed on the 19th, though.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

:pascalc also filed a bug on this under bug 1366743. Looks like nightlies were broken ever since 19th since thats when we last got artifacts under https://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-api-15/.

Comment 4

[Bill Gianopoulos [:WG9s]]

I don't get how my bug is a dupe of this if it is then please explain why did I not get a 5/20 update????

Comment 5

[Bill Gianopoulos [:WG9s]]

(In reply to Axel Hecht [:Pike] from comment #1)
> Maybe bug 1366169? That introduced an error message that shows up in the log
> often. That landed on the 19th, though.

Like I said no nightly updated from 5/19 builds so no idea why this bug says issue started
 on 5/21.

Comment 6

[Bill Gianopoulos [:WG9s]]

I also find it just funny that things glandium is fond of calling footguns, his fixes for end up being a footgun.  Just sayin! ;-)

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

At first glance, I suspect :Pike's right. We do get lots of "[task 2017-05-22T10:48:12.861816Z] 10:48:12     INFO -  10:48:12     INFO -  Error: /home/worker/workspace/build/src/obj-firefox/dist/bin/chrome/en-US.manifest:1: "locale alerts en-US en-US/locale/en-US/alerts/" overrides "locale alerts en-US en-US/locale/en-US/alerts/"" which seem pretty much related to https://hg.mozilla.org/mozilla-central/rev/1100dbb56818#l2.63

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Bill Gianopoulos [:WG9s] from comment #4)
> I don't get how my bug is a dupe of this if it is then please explain why
> did I not get a 5/20 update????

Hey Bill,

Which bug are you talking about? If you're talking about bug 1366743, my reasoning to close that as a dupe was purely chronological (this bug was filed before that by ~2h) + the fact that lots of people were already roped in here. Not sure if this is best practice or not but I usually consider for refence the ones that arrive first in a chronological order. I could be wrong, it's just a matter of my own preference. Fwiw, I have also amended the title to reflect the since-5/19 brokage reality.

Comment 9

[Bill Gianopoulos [:WG9s]]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #8)
> (In reply to Bill Gianopoulos [:WG9s] from comment #4)
> > I don't get how my bug is a dupe of this if it is then please explain why
> > did I not get a 5/20 update????
> 
> Hey Bill,
> 
> Which bug are you talking about? If you're talking about bug 1366743, my
> reasoning to close that as a dupe was purely chronological (this bug was
> filed before that by ~2h) + the fact that lots of people were already roped
> in here. Not sure if this is best practice or not but I usually consider for
> refence the ones that arrive first in a chronological order. I could be
> wrong, it's just a matter of my own preference. Fwiw, I have also amended
> the title to reflect the since-5/19 brokage reality.

NO now that the the title is amended I have no issue on it being duped.  Was A bug about and issue that started on 5/19 being duped to a bug saying it started 2 days later that was the issue.  I thought perhaps these were NOT the same issue because of the different start dates.

Comment 10

[Gregory Szorc [:gps]]

This is fallout from changes that were tracked in bug 1366169. The packaging code now detects some scenarios that could lead to files being overwritten when you probably didn't intend for them to be. The fix is either to loosen the overwriting restrictions or to restructure the packaging manifests so they don't trigger the overrides. Preferably the latter.

With only looking at the error message, it looks like there are redundant entries for "locale alerts en-US en-US/locale/en-US/alerts/". You'll want to either remove one or restrict one to a tighter scope so it isn't exactly redundant.

I'll needinfo glandium just in case. But I think this can be fixed with a trivial change to packaging manifests.

Comment 11

[Axel Hecht [:Pike]]

Actually, this might be two problems:

One is that with multi-locale packaging, we add en-US a second time to the temporary manifest, so it feels like we're packaging something as en-US we already have. Not too bad, if it was the same?

The other is, we're packaging locales over and over, and they're actually complaining about one locale overwriting the other. That sounds like a bug in the check, which should distinguish overrides in zh-TW from those on en-US or kk.

Clearing needinfo on catlee, as this is a build issue.

Tentatively moving it to Firefox Android, but I also think that there are aspects here that are Core Build.

Comment 12

[Julien Cristau [:jcristau]]

Do we need to backout 1366169 to get nightlies back?

Comment 13

[Mike Hommey [:glandium]]

Attachment: Bug 1366729 - Properly handle "multi-content" manifest entries after bug 1366169.

Some manifest entries (e.g. skin or locale) have an attached identifier,
and there can be different entries with different identifiers for the
same chrome name. The change from bug 1366169 would consider those as
errors, while they are the expected configuration.

Review commit: https://reviewboard.mozilla.org/r/141638/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/141638/

Comment 14

[Mike Hommey [:glandium]]

Attachment: Bug 1366729 - Skip manifest entries that are packaged twice.

Review commit: https://reviewboard.mozilla.org/r/141640/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/141640/

Comment 15

[Gregory Szorc [:gps]]

Comment on attachment 8870193 [details]
Bug 1366729 - Properly handle "multi-content" manifest entries after bug 1366169.

https://reviewboard.mozilla.org/r/141638/#review145596

Comment 16

[Gregory Szorc [:gps]]

Comment on attachment 8870194 [details]
Bug 1366729 - Skip manifest entries that are packaged twice.

https://reviewboard.mozilla.org/r/141640/#review145598

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/aec720895e5e
Properly handle "multi-content" manifest entries after bug 1366169. r=gps
https://hg.mozilla.org/integration/autoland/rev/249495cc2ea5
Skip manifest entries that are packaged twice. r=gps

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/aec720895e5e
https://hg.mozilla.org/mozilla-central/rev/249495cc2ea5