1315309 - Error when building firefox from source using custom --with-app-name

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[andré]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Build ID: 20161024062541

Steps to reproduce:

Build firefox from source using changeset: 320921:9c6b3b3a119b

Enviroment: Ubuntu 16.04 64-bit

./mach build
./mach package (fails)


Actual results:

Fails to create a package

(...)
 0:08.44 Duplicates 126280 bytes:
 0:08.44   firefox-nightly
 0:08.44   firefox-nightly-bin
 0:08.44 
 0:08.44 WARNING: Found 35 duplicated files taking 208714 bytes (uncompressed)
 0:08.44 ERROR: The following duplicated files are not allowed:
 0:08.44 firefox-nightly
 0:08.44 firefox-nightly-bin
 0:08.44 /home/averissimo/no-backup/firefox-wrapper/firefox/toolkit/mozapps/installer/packager.mk:41: recipe for target 'stage-package' failed
 0:08.44 make[3]: *** [stage-package] Error 1
 0:08.44 make[3]: Leaving directory '/home/averissimo/no-backup/firefox-wrapper/firefox/obj-x86_64-pc-linux-gnu/browser/installer'
 0:08.44 /home/averissimo/no-backup/firefox-wrapper/firefox/toolkit/mozapps/installer/packager.mk:97: recipe for target 'make-package' failed
 0:08.44 make[2]: *** [make-package] Error 2
 0:08.44 make[2]: Leaving directory '/home/averissimo/no-backup/firefox-wrapper/firefox/obj-x86_64-pc-linux-gnu/browser/installer'
 0:08.44 /home/averissimo/no-backup/firefox-wrapper/firefox/config/rules.mk:523: recipe for target 'default' failed
 0:08.44 make[1]: *** [default] Error 2
 0:08.44 make[1]: Leaving directory '/home/averissimo/no-backup/firefox-wrapper/firefox/obj-x86_64-pc-linux-gnu/browser/installer'
 0:08.44 /home/averissimo/no-backup/firefox-wrapper/firefox/browser/build.mk:9: recipe for target 'package' failed
 0:08.44 make: *** [package] Error 2
 0:08.44 make: Leaving directory '/home/averissimo/no-backup/firefox-wrapper/firefox/obj-x86_64-pc-linux-gnu'


Expected results:

Create a tar package for firefox nightly with custom binary.

It works without:

--with-app-name=firefox-nightly

Comment 1

[Gregory Szorc [:gps]]

This is a regression from bug 1303184.

catlee: can you please take a look? Somewhere you'll need to plug in MOZ_APP_NAME for the paths to the binaries.

Comment 2

[andré]

Thanks for the tip on the right direction.

This is solved by diff below. I suck at python and even more in firefox, so this solution is by no means perfect or global.

diff -r f13e90d496cf toolkit/mozapps/installer/find-dupes.py
--- a/toolkit/mozapps/installer/find-dupes.py	Mon Nov 07 20:38:29 2016 -0800
+++ b/toolkit/mozapps/installer/find-dupes.py	Tue Nov 08 14:51:32 2016 +0000
@@ -9,6 +9,7 @@
 from mozpack.files import DeflatedFile
 from collections import OrderedDict
 import argparse
+import buildconfig
 
 '''
 Find files duplicated in a given packaged directory, independently of its
@@ -110,7 +111,14 @@
     for filename in args.dupes_files:
         with open(filename) as dupes_file:
             allowed_dupes.extend([line.partition('#')[0].rstrip() for line in dupes_file])
-
+    '''
+    Add --app-name to duplicates
+    '''
+    allowed_dupes.extend([buildconfig.substs['MOZ_APP_NAME'].rstrip()])
+    allowed_dupes.extend([buildconfig.substs['MOZ_APP_NAME'].rstrip() + '-bin'])
+    '''
+    call on find_dupes
+    '''
     find_dupes(args.directory, bail=not args.warning, allowed_dupes=allowed_dupes)
 
 if __name__ == "__main__":

Comment 3

[Chris AtLee [:catlee]]

I've been looking at a very similar approach. I havent' been able to verify if it actually works or not though :)

diff --git a/browser/installer/Makefile.in b/browser/installer/Makefile.in
--- a/browser/installer/Makefile.in
+++ b/browser/installer/Makefile.in
@@ -5,17 +5,17 @@
 STANDALONE_MAKEFILE := 1
 DIST_SUBDIR := browser
 
 include $(topsrcdir)/config/rules.mk
 
 MOZ_PKG_REMOVALS = $(srcdir)/removed-files.in
 
 MOZ_PKG_MANIFEST = $(srcdir)/package-manifest.in
-MOZ_PKG_DUPEFLAGS = -f $(srcdir)/allowed-dupes.mn
+MOZ_PKG_DUPEFLAGS = -f $(srcdir)/allowed-dupes.mn --app-name $(MOZ_APP_NAME)
 
 # Some files have been already bundled with xulrunner
 ifndef MOZ_MULET
 MOZ_PKG_FATAL_WARNINGS = 1
 else
 DEFINES += -DMOZ_MULET
 endif
 
diff --git a/toolkit/mozapps/installer/find-dupes.py b/toolkit/mozapps/installer/find-dupes.py
--- a/toolkit/mozapps/installer/find-dupes.py
+++ b/toolkit/mozapps/installer/find-dupes.py
@@ -96,22 +96,28 @@ def find_dupes(source, allowed_dupes, ba
 
 
 def main():
     parser = argparse.ArgumentParser(description='Find duplicate files in directory.')
     parser.add_argument('--warning', '-w', action='store_true',
                         help='Only warn about duplicates, do not exit with an error')
     parser.add_argument('--file', '-f', action='append', dest='dupes_files', default=[],
                         help='Add exceptions to the duplicate list from this file')
+    parser.add_argument('--app-name', dest='app_name',
+                        help='Application name')
     parser.add_argument('directory',
                         help='The directory to check for duplicates in')
 
     args = parser.parse_args()
 
     allowed_dupes = []
     for filename in args.dupes_files:
         with open(filename) as dupes_file:
             allowed_dupes.extend([line.partition('#')[0].rstrip() for line in dupes_file])
 
+    if args.app_name:
+        # TODO: Remove firefox/firefox-bin?
+        allowed_dupes.extend([args.app_name, '{}-bin'.format(args.app_name)])
+
     find_dupes(args.directory, bail=not args.warning, allowed_dupes=allowed_dupes)
 
 if __name__ == "__main__":
     main()

Comment 4

[Hiroshi Hatake [:cosmo0920]]

I've also hit this issue on macOS with FIREFOX_BETA_52_BASE tag in mozilla-beta.

Comment 5

[Mike Hommey [:glandium]]

Chris, can you make an actual patch of this and r? a build peer?

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1315309 - Preprocess find-dupes exception list.

Review commit: https://reviewboard.mozilla.org/r/118134/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/118134/

Comment 7

[Michael Shal [:mshal]]

Comment on attachment 8844808 [details]
Bug 1315309 - Preprocess find-dupes exception list.

https://reviewboard.mozilla.org/r/118134/#review120202

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/af4fdc8f073a
Preprocess find-dupes exception list. r=mshal

Comment 9

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/af4fdc8f073a

Comment 10

[Mike Hommey [:glandium]]

Comment on attachment 8844808 [details]
Bug 1315309 - Preprocess find-dupes exception list.

Approval Request Comment
[Feature/Bug causing the regression]: Regression from bug 1303184
[User impact if declined]: Can't build with --with-app-name. This is an option that tends to be used by organisations that prefer to stay on ESR, thus the a? for ESR too.
[Is this code covered by automated tests?]: No
[Has the fix been verified in Nightly?]: No, but verified on Debian firefox-esr packages.
[Needs manual test from QE? If yes, steps to reproduce]: N/A
[List of other uplifts needed for the feature/fix]: N/A
[Is the change risky?]: No
[Why is the change risky/not risky?]: Essentially, it's NPOTB. The only thing that could go wrong with a patch to the exception list is that duplicate files are not marked as allowed anymore, which would break the build, which didn't happen.
[String changes made/needed]: N/A

Comment 11

[Gerry Chang [:gchang]]

Comment on attachment 8844808 [details]
Bug 1315309 - Preprocess find-dupes exception list.

Fix a build issue. Aurora54+ & Beta53+.

Comment 12

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/b1974c6495e8

Comment 13

[Carsten Book [:Tomcat]]

has problems landing in beta like:

grafting 405976:b1974c6495e8 "Bug 1315309 - Preprocess find-dupes exception list. r=mshal a=gchang"
merging browser/installer/allowed-dupes.mn
merging toolkit/mozapps/installer/packager.mk
warning: conflicts while merging toolkit/mozapps/installer/packager.mk! (edit, then use 'hg resolve --mark')
abort: unresolved conflicts, can't continue
(use 'hg resolve' and 'hg graft --continue')

could you take a look, thanks!

Comment 14

[Mike Hommey [:glandium]]

There was a trivial conflict.
https://hg.mozilla.org/releases/mozilla-beta/rev/2c6189b62427a436363233adcd0a43743604fef5

Comment 15

[Julien Cristau [:jcristau]]

Comment on attachment 8844808 [details]
Bug 1315309 - Preprocess find-dupes exception list.

fix packaging issue when using --with-app-name, esr52+

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr52/rev/7ecdd4d4b1b6