Closed Bug 1076954 Opened 10 years ago Closed 7 years ago

Contents/MacOS/firefox is always added in partial mars

Categories

(Release Engineering :: Release Requests, defect)

Product:
Release Engineering
Component:
Release Requests
Platform:
x86_64
Windows 8.1
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: robert.strong.bugs, Unassigned)

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2530] ) 

While going over the update logs on Mac I have noticed that the Contents/MacOS/firefox file is always added and never patched in partial mar files. I haven't looked into why as of yet.
The log has it being forced, which is not surprising I suppose:
make -C tools/update-packaging partial-patch STAGE_DIR=../../dist/update SRC_BUILD=../../previous SRC_BUILD_ID=20141002030202 DST_BUILD=../../current DST_BUILD_ID=20141002093155
 in dir /builds/slave/m-cen-osx64-ntly-0000000000000/build/obj-firefox/i386 (timeout 1200 secs)
 watching logfiles {}
 argv: ['make', '-C', 'tools/update-packaging', 'partial-patch', 'STAGE_DIR=../../dist/update', 'SRC_BUILD=../../previous', 'SRC_BUILD_ID=20141002030202', 'DST_BUILD=../../current', 'DST_BUILD_ID=20141002093155']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-EurNGy/Render
  CCACHE_COMPRESS=1
  CCACHE_DIR=/builds/ccache
  CCACHE_UMASK=002
  CHOWN_REVERT=~/bin/chown_revert
  CHOWN_ROOT=~/bin/chown_root
  DISPLAY=/tmp/launch-Az2JDb/org.x:0
  HG_SHARE_BASE_DIR=/builds/hg-shared
  HOME=/Users/cltbld
  IS_NIGHTLY=yes
  LC_ALL=C
  LOGNAME=cltbld
  MAR=../dist/host/bin/mar
  MBSDIFF=../dist/host/bin/mbsdiff
  MOZ_AUTOMATION=1
  MOZ_CRASHREPORTER_NO_REPORT=1
  MOZ_OBJDIR=obj-firefox
  MOZ_SIGN_CMD=python /builds/slave/m-cen-osx64-ntly-0000000000000/tools/release/signing/signtool.py --cachedir /builds/slave/m-cen-osx64-ntly-0000000000000/signing_cache -t /builds/slave/m-cen-osx64-ntly-0000000000000/token -n /builds/slave/m-cen-osx64-ntly-0000000000000/nonce -c /builds/slave/m-cen-osx64-ntly-0000000000000/tools/release/signing/host.cert -H gpg:signcode:mar:jar:b2gmar:signing4.srv.releng.scl3.mozilla.com:9100 -H gpg:signcode:mar:jar:b2gmar:signing5.srv.releng.scl3.mozilla.com:9100 -H gpg:signcode:mar:jar:b2gmar:signing6.srv.releng.scl3.mozilla.com:9100 -H dmg:mac-signing2.srv.releng.scl3.mozilla.com:9100 -H dmg:mac-signing3.srv.releng.scl3.mozilla.com:9100 -H dmg:mac-signing4.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9100
  MOZ_SYMBOLS_EXTRA_BUILDID=macosx64
  MOZ_UPDATE_CHANNEL=nightly
  PATH=/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py
  PWD=/builds/slave/m-cen-osx64-ntly-0000000000000/build/obj-firefox/i386
  SHELL=/bin/bash
  SHLVL=0
  SSH_AUTH_SOCK=/tmp/launch-CekfZt/Listeners
  SYMBOL_SERVER_HOST=symbolpush.mozilla.org
  SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_ffx/
  SYMBOL_SERVER_SSH_KEY=/Users/cltbld/.ssh/ffxbld_dsa
  SYMBOL_SERVER_USER=ffxbld
  TINDERBOX_OUTPUT=1
  TMPDIR=/var/folders/x9/1l75yxsd5g90ps0h9fhwhzm800000w/T/
  TOOLTOOL_CACHE=/builds/tooltool_cache
  TOOLTOOL_HOME=/builds
  USER=cltbld
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.7
  __CF_USER_TEXT_ENCODING=0x1C:0:0
 using PTY: False
MAR=/builds/slave/m-cen-osx64-ntly-0000000000000/build/obj-firefox/i386/dist/host/bin/mar \
	MBSDIFF=/builds/slave/m-cen-osx64-ntly-0000000000000/build/obj-firefox/i386/dist/host/bin/mbsdiff \
	  /builds/slave/m-cen-osx64-ntly-0000000000000/build/tools/update-packaging/make_incremental_update.sh \
	  '../../dist/update/firefox-35.0a1.en-US.mac.partial.20141002030202-20141002093155.mar' \
	  '../../previous' \
	  '../../current'
/builds/slave/m-cen-osx64-ntly-0000000000000/build/obj-firefox/i386/previous /builds/slave/m-cen-osx64-ntly-0000000000000/build/obj-firefox/i386/tools/update-packaging
/builds/slave/m-cen-osx64-ntly-0000000000000/build/obj-firefox/i386/tools/update-packaging
/builds/slave/m-cen-osx64-ntly-0000000000000/build/obj-firefox/i386/current /builds/slave/m-cen-osx64-ntly-0000000000000/build/obj-firefox/i386/tools/update-packaging
/builds/slave/m-cen-osx64-ntly-0000000000000/build/obj-firefox/i386/tools/update-packaging

Adding type instruction to update manifests
       type partial

Adding file patch and add instructions to update manifests
diffing "Contents/_CodeSignature/CodeResources"
      patch "Contents/_CodeSignature/CodeResources.patch" "Contents/_CodeSignature/CodeResources"
diffing "Contents/Resources/webapprt/webapprt.ini"
      patch "Contents/Resources/webapprt/webapprt.ini.patch" "Contents/Resources/webapprt/webapprt.ini"
 add-if-not "Contents/Resources/update-settings.ini" "Contents/Resources/update-settings.ini"
        add "Contents/Resources/removed-files" (forced)
        add "Contents/Resources/precomplete" (forced)
diffing "Contents/Resources/platform.ini"
      patch "Contents/Resources/platform.ini.patch" "Contents/Resources/platform.ini"
diffing "Contents/Resources/omni.ja"
      patch "Contents/Resources/omni.ja.patch" "Contents/Resources/omni.ja"
 add-if-not "Contents/Resources/defaults/pref/channel-prefs.js" "Contents/Resources/defaults/pref/channel-prefs.js"
diffing "Contents/Resources/browser/components/libbrowsercomps.dylib"
      patch "Contents/Resources/browser/components/libbrowsercomps.dylib.patch" "Contents/Resources/browser/components/libbrowsercomps.dylib"
diffing "Contents/Resources/application.ini"
      patch "Contents/Resources/application.ini.patch" "Contents/Resources/application.ini"
diffing "Contents/MacOS/webapprt-stub"
      patch "Contents/MacOS/webapprt-stub.patch" "Contents/MacOS/webapprt-stub"
diffing "Contents/MacOS/updater.app/Contents/MacOS/updater"
      patch "Contents/MacOS/updater.app/Contents/MacOS/updater.patch" "Contents/MacOS/updater.app/Contents/MacOS/updater"
diffing "Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container"
      patch "Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container.patch" "Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container"
diffing "Contents/MacOS/libsoftokn3.dylib"
      patch "Contents/MacOS/libsoftokn3.dylib.patch" "Contents/MacOS/libsoftokn3.dylib"
diffing "Contents/MacOS/libplugin_child_interpose.dylib"
      patch "Contents/MacOS/libplugin_child_interpose.dylib.patch" "Contents/MacOS/libplugin_child_interpose.dylib"
diffing "Contents/MacOS/libnssdbm3.dylib"
      patch "Contents/MacOS/libnssdbm3.dylib.patch" "Contents/MacOS/libnssdbm3.dylib"
diffing "Contents/MacOS/libnssckbi.dylib"
      patch "Contents/MacOS/libnssckbi.dylib.patch" "Contents/MacOS/libnssckbi.dylib"
diffing "Contents/MacOS/libnss3.dylib"
      patch "Contents/MacOS/libnss3.dylib.patch" "Contents/MacOS/libnss3.dylib"
diffing "Contents/MacOS/libmozglue.dylib"
      patch "Contents/MacOS/libmozglue.dylib.patch" "Contents/MacOS/libmozglue.dylib"
diffing "Contents/MacOS/libmozalloc.dylib"
      patch "Contents/MacOS/libmozalloc.dylib.patch" "Contents/MacOS/libmozalloc.dylib"
diffing "Contents/MacOS/libfreebl3.dylib"
      patch "Contents/MacOS/libfreebl3.dylib.patch" "Contents/MacOS/libfreebl3.dylib"
diffing "Contents/MacOS/firefox-bin"
      patch "Contents/MacOS/firefox-bin.patch" "Contents/MacOS/firefox-bin"
        add "Contents/MacOS/firefox" (forced)
diffing "Contents/MacOS/crashreporter.app/Contents/MacOS/crashreporter"
      patch "Contents/MacOS/crashreporter.app/Contents/MacOS/crashreporter.patch" "Contents/MacOS/crashreporter.app/Contents/MacOS/crashreporter"
diffing "Contents/MacOS/XUL"
      patch "Contents/MacOS/XUL.patch" "Contents/MacOS/XUL"

Adding file add instructions to update manifests

Adding file remove instructions to update manifests

Adding file and directory remove instructions from file 'removed-files'
     remove "Contents/Resources/chrome.manifest"
    rmrfdir "Contents/Plug-Ins/PrintPDE.plugin/"
      rmdir "Contents/MacOS/webapprt/components/"
    rmrfdir "Contents/MacOS/updates/"
     remove "Contents/MacOS/updates.xml"
     remove "Contents/MacOS/update-settings.ini"
    rmrfdir "Contents/MacOS/searchplugins/"
    rmrfdir "Contents/MacOS/plugins/MRJPlugin.plugin/"
    rmrfdir "Contents/MacOS/plugins/JavaEmbeddingPlugin.bundle/"
    rmrfdir "Contents/MacOS/plugins/Default Plugin.plugin/"
      rmdir "Contents/MacOS/modules/"
      rmdir "Contents/MacOS/jssubloader/"
      rmdir "Contents/MacOS/greprefs/"
    rmrfdir "Contents/MacOS/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/"
    rmrfdir "Contents/MacOS/extensions/{641d8d09-7dda-4850-8228-ac0ab65e2ac9}/"
    rmrfdir "Contents/MacOS/extensions/testpilot@labs.mozilla.com/"
    rmrfdir "Contents/MacOS/extensions/talkback@mozilla.org/"
    rmrfdir "Contents/MacOS/extensions/reporter@mozilla.org/"
    rmrfdir "Contents/MacOS/extensions/inspector@mozilla.org/"
      rmdir "Contents/MacOS/extensions/"
     remove "Contents/MacOS/distribution/extensions/testpilot@labs.mozilla.com.xpi"
      rmdir "Contents/MacOS/distribution/extensions/"
      rmdir "Contents/MacOS/distribution/"
    rmrfdir "Contents/MacOS/defaults/profile/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}/"
      rmdir "Contents/MacOS/defaults/profile/extensions/"
      rmdir "Contents/MacOS/defaults/profile/chrome/"
    rmrfdir "Contents/MacOS/defaults/profile/US/"
      rmdir "Contents/MacOS/defaults/profile/"
      rmdir "Contents/MacOS/defaults/autoconfig/"
    rmrfdir "Contents/MacOS/defaults/"
      rmdir "Contents/MacOS/components/"
      rmdir "Contents/MacOS/chrome/overlayinfo/"
      rmdir "Contents/MacOS/chrome/"
     remove "Contents/MacOS/chrome.manifest"
     remove "Contents/MacOS/active-update.xml"

Adding directory remove instructions for directories that no longer exist

Finished
Looks like make_incremental_updates.sh is requesting it: http://mxr.mozilla.org/mozilla-central/source/tools/update-packaging/make_incremental_update.sh#74

Which is from bug 770996, which was to fix a regression after we started signing Mac builds. I think we didn't need this for a long time, but we do know because we'll be signing partner builds again because of v2 signing...
With v2 signing all executable binaries are signed though it isn't clear this will be a problem with v2 signing since we first sign the binaries and then sign the bundle... right?

Also, it would be a good thing if there is a change to make_incremental_update.sh that it is also made to make_incremental_updates.py otherwise we'll end up in an inconsistent state again as happened with the removed-files file.
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #3)
> With v2 signing all executable binaries are signed though it isn't clear
> this will be a problem with v2 signing since we first sign the binaries and
> then sign the bundle... right?

I don't think so. We have to resign the partner builds because they add new files to the bundle. Resigning causes the "firefox" binary to change (I think it embeds some sort of hash that is calculated based on the other files in the bundle) - it doesn't just add signatures to the new files :(.

> Also, it would be a good thing if there is a change to
> make_incremental_update.sh that it is also made to
> make_incremental_updates.py otherwise we'll end up in an inconsistent state
> again as happened with the removed-files file.

Yeah, good point. Though, I don't think we actually use the .py anymore...not at the moment, at least.
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2530]
Robert, is there anything that we should do with regards to this bug?
Flags: needinfo?(robert.strong.bugs)
Yes, the firefox bin shouldn't be a forced add.
Flags: needinfo?(robert.strong.bugs)
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #6)
> Yes, the firefox bin shouldn't be a forced add.

If we don't force add, I _think_ that partial updates will break for Mac partner builds (because their firefox-bin is not the same as vanilla).
I am concerned that with Mac partner builds signed separately and using the same update as normal builds if the bundle's signing has to be verified for any reason after an update it will fail. This will fail if keychain support is added and there might be other cases. Off the top of my head the possible solutions are to rewrite the distribution code to use files outside of the Firefox bundle along with creating a Mac installer that can handle installing files outside of the bundle or to create separate updates for the partner bundles.

In summary, I'm less concerned about the fact that the firefox binary on Mac is always added as I am about the potential larger ramification of the decision to handle partner builds in this manner so go ahead and close this bug if you want.
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #8)
> I am concerned that with Mac partner builds signed separately and using the
> same update as normal builds if the bundle's signing has to be verified for
> any reason after an update it will fail. This will fail if keychain support
> is added and there might be other cases. Off the top of my head the possible
> solutions are to rewrite the distribution code to use files outside of the
> Firefox bundle along with creating a Mac installer that can handle
> installing files outside of the bundle or to create separate updates for the
> partner bundles.
> 
> In summary, I'm less concerned about the fact that the firefox binary on Mac
> is always added as I am about the potential larger ramification of the
> decision to handle partner builds in this manner so go ahead and close this
> bug if you want.

I don't think this is the ideal way to handle partner builds, but until we have a way to do them without modifying files inside of the bundle, the choice is between this, and not signing them -- which makes them unrunnable on an out of box 10.9.5 or 10.10 install. I thought we had a bug on file for making such a change, but I can't find it now...
I"m going to close this as INCOMPLETE. The current approach works fine and we have no other way to let partner repacks work.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.