Closed Bug 1313640 Opened 4 years ago Closed 4 years ago

Android Nightly Fail with 04:17:24 FATAL - Halting on failure while running ['mock_mozilla', '-r', 'mozilla-centos6-x86_64-android', '-q', '--cwd', '/builds/slave/m-cen-and-api-15-ntly-00000000', '--unpriv', '--shell', '/usr/bin/env MOZ_AUTOMATION=1 "LES

Categories

(Release Engineering :: General, defect)

defect
Not set
blocker

Tracking

(firefox52 fixed)

RESOLVED FIXED
Tracking Status
firefox52 --- fixed

People

(Reporter: cbook, Assigned: catlee)

References

()

Details

Attachments

(1 file)

https://treeherder.mozilla.org/logviewer.html#?job_id=5424148&repo=mozilla-central#L33682

not sure where it fails but if affect both android nightlys it seems
Callek: i will retrigger android nightlys in the hope that fix this, but in any case could you take a look, thanks!
Flags: needinfo?(bugspam.Callek)
 04:17:24     INFO -  07:17:24     INFO -  WARNING: Found 50 duplicated files taking 347909 bytes (219451 compressed)
 04:17:24     INFO -  07:17:24     INFO -  ERROR: The following duplicated files are not allowed:
 04:17:24     INFO -  07:17:24     INFO -  chrome/en-US/locale/en-US/browser/devicePrompt.properties
 04:17:24     INFO -  07:17:24     INFO -  chrome/pt-PT/locale/pt-PT/browser/devicePrompt.properties
04:17:24 INFO - 07:17:24 INFO - chrome/en-US/locale/branding/brand.properties

...

Lots of entries like that, which partly worries me either in that we're packaging far more than <locale> *or* that the checker script is bad when used with l10n.

I suspect the retrigger will not solve anything.

:catlee, is there an easy fix in your mind, or should we backout Bug 1303184
Depends on: 1303184
Flags: needinfo?(bugspam.Callek) → needinfo?(catlee)
Richard, are there plans to remove the duplicate files as part of multilocale builds somehow?
Flags: needinfo?(catlee) → needinfo?(rnewman)
(In reply to Justin Wood (:Callek) from comment #2)
> Lots of entries like that, which partly worries me either in that we're
> packaging far more than <locale> *or* that the checker script is bad when
> used with l10n.

Admittedly when I first looked at this I didn't realize these were multi-locale builds. So it is indeed (and expectedly) packaging more than one locale in one go. I'm still waiting on results for other sets of l10n with regard to this.
Android builds are indeed multilocale.

I don't know if that dupe checker is spotting actual duplicate file contents, rather than repeated path subtrees; if so, I'd be happy to see a bug -- with analysis of space savings when packaged -- flattening those down to en-US.

(Obviously the checker would still have to not warn for peer locale dupes that don't dupe to the root locale…)
Flags: needinfo?(rnewman)
It's checking for actual duplicated content. Most of these are already present in en-US, but not all. From the log (https://treeherder.mozilla.org/logviewer.html#?job_id=5424148&repo=mozilla-central#L33514), it looks like there's a total of 347909 bytes (219451 compressed) that are duplicated.
(In reply to Richard Newman [:rnewman] from comment #6)
> I don't know if that dupe checker is spotting actual duplicate file
> contents, rather than repeated path subtrees; if so, I'd be happy to see a
> bug -- with analysis of space savings when packaged -- flattening those down
> to en-US.

If you search for logs from a completed build that runs find-dupes.py, it prints said statistics. e.g. "04:11:59     INFO -  WARNING: Found 34 duplicated files taking 87540 bytes (30893 compressed)" (this was from an Android Nightly from this week).
Comment on attachment 8805555 [details]
Bug 1313640 - Add support for allowing duplicate files in multilocale builds.

https://reviewboard.mozilla.org/r/89322/#review88484

::: toolkit/mozapps/installer/find-dupes.py:285
(Diff revision 1)
>  
> +def normalize_l10n_path(p):
> +    '''
> +    Normalizes localized paths to en-US
> +
> +    >>> normalize_osx_path('chrome/es-ES/locale/branding/brand.properties')

I think this wants to be `normalize_osx_path`

::: toolkit/mozapps/installer/find-dupes.py:288
(Diff revision 1)
> +    m = re.search(r'chrome/(\S+)/locale', p)
> +    if m:
> +        locale = m.group(1)
> +        p = p.replace(locale, 'en-US')
> +    return p

This can be written as:

return re.sub(r'chrome/(\S+)/locale',
              'chrome/en-US/locale',
              p)
              
This is safer than what's written because it won't replace all occurrences of the locale with `en-US`: only the ones in the `chrome/.../locale` pattern.
Attachment #8805555 - Flags: review?(gps) → review-
Comment on attachment 8805555 [details]
Bug 1313640 - Add support for allowing duplicate files in multilocale builds.

https://reviewboard.mozilla.org/r/89322/#review88484

> This can be written as:
> 
> return re.sub(r'chrome/(\S+)/locale',
>               'chrome/en-US/locale',
>               p)
>               
> This is safer than what's written because it won't replace all occurrences of the locale with `en-US`: only the ones in the `chrome/.../locale` pattern.

That doesn't catch all occurrences of locale strings in paths. There are many of the pattern 'chrome/$AB_CD/locale/$AB_CD' as well.
See Also: → 1313702
I filed Bug 1313702 to investigate skipping/stripping some of these duplicates at build time.
Attachment #8805555 - Flags: review?(bugspam.Callek)
I can't get mozreview to re-request review. gps, can you please look when you get a chance?
Flags: needinfo?(gps)
can we fix this asap since this is hitting nightly user and so crash-data etc :(
Severity: normal → blocker
Just to clarify: we have no crash reports for the Oct 28, 29, 30 Fennec builds. This is bad :(
Assignee: nobody → catlee
Comment on attachment 8805555 [details]
Bug 1313640 - Add support for allowing duplicate files in multilocale builds.

https://reviewboard.mozilla.org/r/89322/#review88906

I tested this out locally by making a dummy directory with the duplicate files that are failing the Android nightly builds and running find-dupes.py manually. With this fix applied, the error goes away.
Attachment #8805555 - Flags: review+
Flags: needinfo?(gps)
Comment on attachment 8805555 [details]
Bug 1313640 - Add support for allowing duplicate files in multilocale builds.

https://reviewboard.mozilla.org/r/89322/#review88940
Attachment #8805555 - Flags: review?(gps) → review+
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a352e8d4b5d9
Add support for allowing duplicate files in multilocale builds. r=gps,mshal
OK well this is ridiculous.  we go days at a time with no Android nightly builds because of this.  Staring this to this bug is NOT an acceptable way to deal with this.  we need to get nightly builds created in a timely manner.
Also I have no idea what the certificate issue that has been starred as a duplicate of this issue has to do with it.
https://hg.mozilla.org/mozilla-central/rev/a352e8d4b5d9
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Depends on: 1374773
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.