Closed Bug 1885985 Opened 2 years ago Closed 2 years ago

macOS framework changes yield artifact builds with `nightly` channel pref that can and do apply updates

Categories

(Toolkit :: Application Update, defect)

Product:
Toolkit
Component:
Application Update
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
127 Branch
Tracking Flags:
Tracking Status
firefox-esr115 --- unaffected
firefox124 --- unaffected
firefox125 --- disabled
firefox126 --- wontfix
firefox127 --- fixed

People

(Reporter: nalexander, Assigned: nalexander)

References

(Regression)

Details

(Keywords: regression, Whiteboard: [fidedi-ope])

Attachments

(2 external files)

Bug 1848414 moved the old channel-prefs.js file (and update-settings.ini) to two macOS frameworks, ChannelPrefs.framework and UpdateSettings.framework. The artifacts.py mechanism was updated at that time: https://searchfox.org/mozilla-central/rev/529f04f4cd2ae68a0f729ba91cf8985edb23e9d3/python/mozbuild/mozbuild/artifacts.py#660,669.

Unfortunately, that results in an artifact build with update channel nightly, further resulting in updates being downloaded and (mostly) failing to apply.

A local developer build has update channel default and accepted update channels ``, i.e., the empty string. This ticket tracks arranging frameworks that encode these values for artifact builds.

For artifact builds, we locally resign, and the resigning already handles .framework contents appropriately. So there's no particular issue with signing in automation; as long as the build produces the frameworks, we can incorporate them in the local artifact builds.

Bug 1666491 introduced a generic mechanism for producing files in the build in order for artifact builds to consume them. It appears that these "extra archives" are processed with the signing writer (see https://searchfox.org/mozilla-central/rev/529f04f4cd2ae68a0f729ba91cf8985edb23e9d3/python/mozbuild/mozbuild/artifacts.py#419), so there's no extra work required to handle the frameworks. So I expect that we can follow Bug 1666491 to produce an "extra archive" containing these artifact-specific frameworks without a great deal of labour.

:bytesized, since you are the author of the regressor, bug 1848414, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Flags: needinfo?(bytesized)
Severity: -- → S3
Flags: needinfo?(bytesized)

Set release status flags based on info from the regressing bug 1848414

status-firefox124: --- → unaffected
status-firefox125: --- → affected
status-firefox126: --- → affected
status-firefox-esr115: --- → unaffected

Can we bump this up from S3? This is seriously disrupting the frontend team's work, as they basically can't depend on their local builds actually being what they built (except by rebasing very frequently).

Flags: needinfo?(bytesized)
Severity: S3 → S2
Flags: needinfo?(bytesized)

@nalexander I'm trying to understand the mechanism that you laid out for fixing this. I believe that you are proposing that we generate both the "real" channelprefs/updatesettings artifacts and also the "local build" versions of the artifacts. Should we make a different local artifact for each channel, even though they will all build the same values into the frameworks? Also, do we want to only generate the local build artifact via a specific build task (I think this might be what is happening here) rather than building two frameworks on every build?

I'm struggling to understand the patches on Bug 1666491. Are they also creating two versions of the artifacts and choosing between them based on whether the build is a local build? I'm having trouble figuring that part out.

Flags: needinfo?(nalexander)
Whiteboard: [fidedi-ope]

I talked to nalexander about this in-person, and I am going to go ahead and write a patch for the frameworks that is compiled as part of the regular build process that has the correct channel for a local build. Once I have that, I'll still need to come back to how to get the artifact in the right place and how to use artifact during a local build. But this part I can at least get started on without help.

Flags: needinfo?(nalexander)

This patch only builds the necessary frameworks, it does not include the necessary logic to distribute them as artifacts.

Assignee: nobody → bytesized
Status: NEW → ASSIGNED
Attachment #9393649 - Attachment description: Bug 1885985 - Build macOS frameworks for use with artifact builds r=nalexander!,spohl! → Bug 1885985 - Produce and consume macOS frameworks for artifact builds. r?bytesized!,spohl!,#build!
Assignee: bytesized → nalexander

To verify this, I pushed to try at https://treeherder.mozilla.org/jobs?repo=try&revision=f9eb933226013825dc6f1541f9856c57fa2b8fff.

The OS X Cross Compiled Shippable DMG repack here yields (after allowing the application in Security & Privacy):

Update Channel 	nightly-try

in about:support, and:

$ ~/Applications/Firefox\ Nightly.app/Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater --channels-allowed                                                                                    Channels Allowed: firefox-mozilla-central

That looks as I expect, and certainly not the local build variants introduced by these commits.

In addition,

Depends on: 1848414
Pushed by nalexander@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/9f185bc37baf Produce and consume macOS frameworks for artifact builds. r=spohl,application-update-reviewers,bytesized,firefox-build-system-reviewers,glandium

Backed out for causing failures on UpdateSettings

[task 2024-04-17T19:30:14.929Z] 19:30:14     INFO -  /builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/build/bin/python /builds/worker/checkouts/gecko/config/nsinstall.py ../../../../../../dist/update_framework_artifacts/UpdateSettings-localbuild.framework/UpdateSettings ../../../../../../dist/update_framework_artifacts/UpdateSettings-localbuild.framework
[task 2024-04-17T19:30:14.929Z] 19:30:14    ERROR -  Traceback (most recent call last):
[task 2024-04-17T19:30:14.929Z] 19:30:14     INFO -    File "/builds/worker/checkouts/gecko/config/nsinstall.py", line 172, in <module>
[task 2024-04-17T19:30:14.929Z] 19:30:14     INFO -      sys.exit(_nsinstall_internal(sys.argv[1:]))
[task 2024-04-17T19:30:14.929Z] 19:30:14     INFO -    File "/builds/worker/checkouts/gecko/config/nsinstall.py", line 160, in _nsinstall_internal
[task 2024-04-17T19:30:14.930Z] 19:30:14     INFO -      copy_all_entries(args, target)
[task 2024-04-17T19:30:14.930Z] 19:30:14     INFO -    File "/builds/worker/checkouts/gecko/config/nsinstall.py", line 119, in copy_all_entries
[task 2024-04-17T19:30:14.930Z] 19:30:14     INFO -      handleTarget(e, dest)
[task 2024-04-17T19:30:14.930Z] 19:30:14     INFO -    File "/builds/worker/checkouts/gecko/config/nsinstall.py", line 150, in handleTarget
[task 2024-04-17T19:30:14.930Z] 19:30:14     INFO -      shutil.copy(srcpath, targetpath)
[task 2024-04-17T19:30:14.930Z] 19:30:14     INFO -    File "/builds/worker/fetches/python/lib/python3.8/shutil.py", line 418, in copy
[task 2024-04-17T19:30:14.931Z] 19:30:14     INFO -      copyfile(src, dst, follow_symlinks=follow_symlinks)
[task 2024-04-17T19:30:14.931Z] 19:30:14     INFO -    File "/builds/worker/fetches/python/lib/python3.8/shutil.py", line 264, in copyfile
[task 2024-04-17T19:30:14.931Z] 19:30:14     INFO -      with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
[task 2024-04-17T19:30:14.931Z] 19:30:14     INFO -  FileNotFoundError: [Errno 2] No such file or directory: '/builds/worker/workspace/obj-build/dist/update_framework_artifacts/UpdateSettings-localbuild.framework/UpdateSettings'
[task 2024-04-17T19:30:14.931Z] 19:30:14    ERROR -  gmake[4]: *** [Makefile:20: libs] Error 1
[task 2024-04-17T19:30:14.931Z] 19:30:14     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/toolkit/mozapps/update/updater/macos-frameworks/UpdateSettings-localbuild'
[task 2024-04-17T19:30:14.932Z] 19:30:14    ERROR -  gmake[3]: *** [/builds/worker/checkouts/gecko/config/recurse.mk:94: toolkit/mozapps/update/updater/macos-frameworks/UpdateSettings-localbuild/libs] Error 2
[task 2024-04-17T19:30:14.932Z] 19:30:14     INFO -  gmake[3]: Target 'recurse_libs' not remade because of errors.
[task 2024-04-17T19:30:14.932Z] 19:30:14    ERROR -  gmake[2]: *** [/builds/worker/checkouts/gecko/config/recurse.mk:34: libs] Error 2
[task 2024-04-17T19:30:14.932Z] 19:30:14    ERROR -  gmake[1]: *** [/builds/worker/checkouts/gecko/config/rules.mk:361: default] Error 2
[task 2024-04-17T19:30:14.932Z] 19:30:14     INFO -  gmake: *** [client.mk:60: build] Error 2
[task 2024-04-17T19:30:14.950Z] 19:30:14     INFO -  0 compiler warnings present.
[task 2024-04-17T19:30:15.191Z] 19:30:15     INFO - Return code: 2
[task 2024-04-17T19:30:15.191Z] 19:30:15  WARNING - setting return code to 2
Flags: needinfo?(nalexander)

I had a legitimate logic error in the backed out patch: after review I shifted a directory name, nhttps://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/S-0Pv1HuRWSn1JHKMY3Xxg/runs/0/artifacts/public/build/target.update_framework_artifacts.zipot realizing that I was moving from and to the same location. I've addressed this in https://treeherder.mozilla.org/jobs?repo=try&revision=d4739a3016d220d130aeb1130263d17fd56a396d.

I've tested by checking the .framework files in the DMG and the archive are not the same in the B job.
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TqU67PqXTta3NEHdE4IvTg/runs/0/artifacts/public/build/target.update_framework_artifacts.zip
Looking instead at the AB job, I've tested that the .framework files in the DMG and the archive are the same.

That is:

$ shasum B_DMG/*.framework/*
36b228fcccae477a76cf4d5542730a921ec8655a  B_DMG/ChannelPrefs.framework/ChannelPrefs
shasum: B_DMG/ChannelPrefs.framework/Resources: Is a directory
shasum: B_DMG/UpdateSettings.framework/Resources: Is a directory
db69ae4009fb3e33adb9e77dceeec7bf75aaa6df  B_DMG/UpdateSettings.framework/UpdateSettings
$ shasum B_archive/*.framework/*
0466c0c551db45fb23eaad7672d0a19e214d7ad0  B_archive/ChannelPrefs-localbuild.framework/ChannelPrefs
shasum: B_archive/ChannelPrefs-localbuild.framework/Resources: Is a directory
shasum: B_archive/UpdateSettings-localbuild.framework/Resources: Is a directory
32b74551518ce72411c91cbd6d31130f1c892df7  B_archive/UpdateSettings-localbuild.framework/UpdateSettings
$ shasum AB_DMG/*.framework/*
0466c0c551db45fb23eaad7672d0a19e214d7ad0  AB_DMG/ChannelPrefs.framework/ChannelPrefs
shasum: AB_DMG/ChannelPrefs.framework/Resources: Is a directory
shasum: AB_DMG/UpdateSettings.framework/Resources: Is a directory
32b74551518ce72411c91cbd6d31130f1c892df7  AB_DMG/UpdateSettings.framework/UpdateSettings
$ shasum AB_archive/*.framework/*
0466c0c551db45fb23eaad7672d0a19e214d7ad0  AB_archive/ChannelPrefs-localbuild.framework/ChannelPrefs
shasum: AB_archive/ChannelPrefs-localbuild.framework/Resources: Is a directory
shasum: AB_archive/UpdateSettings-localbuild.framework/Resources: Is a directory
32b74551518ce72411c91cbd6d31130f1c892df7  AB_archive/UpdateSettings-localbuild.framework/UpdateSettings

Note that the B DMG is different from all the others, which are identical. That's what I expect: real, then 3 copies of the localbuild variants.

I've also successfully built and run locally with MOZ_ARTIFACT_TASK=S-0Pv1HuRWSn1JHKMY3Xxg, finding an update channel of default (as expected) and empty accepted MAR channel IDs.

I'm going to reland shortly.

Flags: needinfo?(nalexander)
Pushed by nalexander@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/a6309c1903e7 Produce and consume macOS frameworks for artifact builds. r=spohl,application-update-reviewers,bytesized,firefox-build-system-reviewers,glandium
Regressions: 1892379

https://hg.mozilla.org/mozilla-central/rev/a6309c1903e7

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
status-firefox127: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 127 Branch
Status: RESOLVED → REOPENED
status-firefox127: fixed → ---
Resolution: FIXED → ---
Target Milestone: 127 Branch → ---

I missed 2 things (at least):

  1. I didn't pass through the artifacts in the macOS shippable "build" unifier;
  2. I didn't accommodate these changes in the artifact-specific diffoscope definitions.

Here's a macOS shippable B (the build-unifier thing) that passes through the update_framework_artifacts.zip as expected: https://treeherder.mozilla.org/jobs?repo=try&revision=0a58c6de4af8c31bc99b54d5f7437efad3a01c27&selectedTaskRun=Bs3tZtOAQW2SX10JagGItA.0 (a separate push due to some typos in the relevant .yml file).

Here's a try push with AB succeeding, and the diffoscope bustage addressed: https://treeherder.mozilla.org/jobs?repo=try&revision=824f9d79bb9757e7490b677dc75183518873e8ba.

I think this is ready to re-land now.

Pushed by nalexander@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/ceddfae53011 Produce and consume macOS frameworks for artifact builds. r=spohl,application-update-reviewers,bytesized,firefox-build-system-reviewers,glandium
Regressions: 1893365

https://hg.mozilla.org/mozilla-central/rev/ceddfae53011

Status: REOPENED → RESOLVED
Closed: 2 years ago → 2 years ago
status-firefox127: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → 127 Branch

(In reply to Aron Cseh from comment #18)

https://hg.mozilla.org/mozilla-central/rev/ceddfae53011

A clobber build on central from this morning yields artifact builds that produce nightly-autoland as the app.update.channel. Is that expected?

(I also think I saw it update but I can't quite be sure if that was because it had queued something from the previous build somehow.)

Flags: needinfo?(nalexander)
See Also: → 1893518

I just verified this locally. First:

$ git rev-parse HEAD
7249a43524fe55f71736514715aca6e9d71cc18f
$ git cinnabar git2hg 7249a43524fe55f71736514715aca6e9d71cc18f
792c9892791495e8f13e77e5a78bfda83ab7a455

i.e., I'm on https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=792c9892791495e8f13e77e5a78bfda83ab7a455, which has artifacts built and available.

My configuration:

$ cat mozconfig-browser-artifact                                                                                                                                                    
mk_add_options AUTOCLOBBER=1

ac_add_options --enable-application=browser
ac_add_options --enable-debug
ac_add_options --enable-artifact-builds
# ac_add_options --enable-artifact-build-symbols=full

mk_add_options MOZ_OBJDIR="../objdirs/objdir-browser-artifact"

Now, let's start clean:

$ rm -rf ~/.mozbuild/package-frontend
$ ./mach clobber

This shouldn't be necessary, but it's possible there are interactions with old files; let's avoid them.

Now, let's build:

$ env MOZCONFIG=mozconfig-browser-artifact ./mach build
...

Now, let's verify in 2 ways. First:

$ env MOZCONFIG=mozconfig-browser-artifact ./mach run
/Users/nalexander/Mozilla/objdirs/objdir-browser-artifact/dist/NightlyDebug.app/Contents/MacOS/firefox -no-remote -foreground -profile /Users/nalexander/Mozilla/objdirs/objdir-browser-artifact/tmp/profile-default
...

Navigate to about:support, and witness Update Channel default, as expected. Modifying the command line above, execute:

$ env MOZCONFIG=mozconfig-browser-artifact /Users/nalexander/Mozilla/objdirs/objdir-browser-artifact/dist/NightlyDebug.app/Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater --channels-allowed
Channels Allowed:

Witness the empty string for channels allowed. (Delimiters would improve this, but that's follow-up.)

Finally, let's witness another thing for follow-up:

$ find ../objdirs/objdir-browser-artifact -iname 'UpdateSettings' | xargs shasum
f09e644e4083e04cc64c73dc4f495983341a0985  ../objdirs/objdir-browser-artifact/_tests/xpcshell/toolkit/mozapps/update/tests/data/updater.app/Contents/Frameworks/UpdateSettings.framework/UpdateSettings
f09e644e4083e04cc64c73dc4f495983341a0985  ../objdirs/objdir-browser-artifact/_tests/testing/mochitest/browser/toolkit/mozapps/update/tests/browser/updater.app/Contents/Frameworks/UpdateSettings.framework/UpdateSettings
fdfb0100fdb78499f1c8db4cf7a1209a15d80696  ../objdirs/objdir-browser-artifact/dist/update_framework_artifacts/UpdateSettings-localbuild.framework/UpdateSettings
f2307db16fe8e5e23d799786678205e6dcf65d95  ../objdirs/objdir-browser-artifact/dist/bin/UpdateSettings
fdfb0100fdb78499f1c8db4cf7a1209a15d80696  ../objdirs/objdir-browser-artifact/dist/bin/updater.app/Contents/Frameworks/UpdateSettings.framework/UpdateSettings
fdfb0100fdb78499f1c8db4cf7a1209a15d80696  ../objdirs/objdir-browser-artifact/dist/NightlyDebug.app/Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/UpdateSettings
f2307db16fe8e5e23d799786678205e6dcf65d95  ../objdirs/objdir-browser-artifact/dist/NightlyDebug.app/Contents/Resources/UpdateSettings
shasum: ../objdirs/objdir-browser-artifact/toolkit/mozapps/update/updater/macos-frameworks/UpdateSettings: Is a directory
nalexander@proto ~/M/gecko.git (bookmarks/central) [1]> find ../objdirs/objdir-browser-artifact -iname 'ChannelPrefs' | xargs shasum
bf790ce034cb97c15aa0de873953c16ecf66e2c2  ../objdirs/objdir-browser-artifact/dist/update_framework_artifacts/ChannelPrefs-localbuild.framework/ChannelPrefs
ca6e36fad63e9d3c03f28540c67abb4a20d1e25b  ../objdirs/objdir-browser-artifact/dist/bin/ChannelPrefs
bf790ce034cb97c15aa0de873953c16ecf66e2c2  ../objdirs/objdir-browser-artifact/dist/bin/ChannelPrefs.framework/ChannelPrefs
ca6e36fad63e9d3c03f28540c67abb4a20d1e25b  ../objdirs/objdir-browser-artifact/dist/NightlyDebug.app/Contents/Resources/ChannelPrefs
bf790ce034cb97c15aa0de873953c16ecf66e2c2  ../objdirs/objdir-browser-artifact/dist/NightlyDebug.app/Contents/Frameworks/ChannelPrefs.framework/ChannelPrefs
shasum: ../objdirs/objdir-browser-artifact/toolkit/mozapps/macos-frameworks/ChannelPrefs: Is a directory

Note that there are (outside of tests) 2 versions of each, but that the ones in dist/update_framework_artifacts make it into the .framework location that is actually used.

I'm confident this is working as expected, but would (of course) appreciate detailed reproductions showing where these steps area breaking down for folks in the wild.

Flags: needinfo?(nalexander)

I'm on https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=2069eada79d9 which currently doesn't appear to have artifacts available.

My mozconfig:

mk_add_options MOZ_OBJDIR=/Users/dave/mozilla/build/trunk/obj-browser-opt-artifact
mk_add_options AUTOCONF=/opt/local/bin/autoconf213
mk_add_options RUN_AUTOCONF_LOCALLY=1
mk_add_options AUTOCLOBBER=1
ac_add_options --enable-application=browser
ac_add_options --enable-artifact-builds
ac_add_options --enable-bootstrap

I did a clobber build and in about:support the update channel shows as nightly-autoland.

$ /Users/dave/mozilla/build/trunk/obj-browser-opt-artifact/dist/Nightly.app/Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater --channels-allowed
Error: 38

Err?

It looks like because there are no artifacts for the revision in mozilla-central it is downloading artifacts from autoland:

trunk$ mach artifact install -v
 0:00.48 D hg suggested 500 candidate revisions
 0:00.48 D hg suggested candidate revision: 2069eada79d9b4095e8799aaa8bd7659d5017ddb
 0:00.48 D hg suggested candidate revision: ed2448d27fe2e034cd5804dc7310c3f72ac2eaa5
 0:00.48 D hg suggested candidate revision: 83e1bb4d563fb463e439465b2384c45696917682
 0:00.48 D hg suggested candidate revision: 7139810235147a4159299e10f84a3cba9f53a0af
 0:00.48 D hg suggested candidate revision: 1bfb1774ed031054098f840557e452455c9e6e7b
 0:00.48 D hg suggested candidate revision: b89431ee0f24bcb481d894ad21288e4f09e1960d
 0:00.48 D hg suggested candidate revision: 09e856b07e9415a7b04b4b30507772309282c508
 0:00.48 D hg suggested candidate revision: f8b91c7dc43be081fbbc44688b532e3a06a201ad
 0:00.48 D hg suggested candidate revision: 405fd04905cf4f39ee92486d120d58a829ad3052
 0:00.48 D hg suggested candidate revision: fa89588b0f91ed1a549744df0ba623fbf316f100
 0:00.48 D hg suggested candidate revision: 37aaa704c1d0ea8007d7a4d89a10bd9848d7cc08
 0:00.48 D hg suggested candidate revision: 9fef88f56156b0aac7a8f73e812114f98e63a94f
 0:00.48 D hg suggested candidate revision: 1f2bdb707f167eed5d98d8da88f511e00ab4c55d
 0:00.48 D hg suggested candidate revision: 8278ed498d0273ad666cc644639dbeffd0b763eb
 0:00.48 D hg suggested candidate revision: b6ed0b385f4160d2d2ba5254a378a8ae5367e8e0
 0:00.48 D hg suggested candidate revision: bcbdf6e884506c99219936aa36878b22457a52b9
 0:00.48 D hg suggested candidate revision: 4122b658394088c5305b8f0805335476e0c4a82b
 0:00.48 D hg suggested candidate revision: dc05314ddc0f4b37cc50bbdc1a2629548fa1103f
 0:00.48 D hg suggested candidate revision: 7b6284ac44a7a1b2ba1fa14618791f65693f12d3
 0:00.48 D hg suggested candidate revision: 95c1da74b3218589c43b30c5809907e10a05545b
 0:00.48 D hg suggested candidate revision: and 480 more
 0:00.48 D Attempting to find a pushhead containing 2069eada79d9b4095e8799aaa8bd7659d5017ddb on mozilla-central.
 0:00.48 D Attempting to find a pushhead containing 2069eada79d9b4095e8799aaa8bd7659d5017ddb on integration/autoland.
 0:00.48 D Retrieving the last 50 pushheads starting with id 41765 on mozilla-central
 0:00.48 D Retrieving the last 50 pushheads starting with id 204572 on integration/autoland
 0:00.49 D Trying to find artifacts for hg revision 2069eada79d9b4095e8799aaa8bd7659d5017ddb on tree mozilla-central.
 0:00.49 D Searching Taskcluster index with namespace: gecko.v2.mozilla-central.shippable.revision.2069eada79d9b4095e8799aaa8bd7659d5017ddb.firefox.macosx64-opt
 0:00.72 D Trying to find artifacts for hg revision 2069eada79d9b4095e8799aaa8bd7659d5017ddb on tree integration/autoland.
 0:00.72 D Searching Taskcluster index with namespace: gecko.v2.autoland.shippable.revision.2069eada79d9b4095e8799aaa8bd7659d5017ddb.firefox.macosx64-opt
 0:00.88 D Trying to find artifacts for hg revision ed2448d27fe2e034cd5804dc7310c3f72ac2eaa5 on tree integration/autoland.
 0:00.88 D Not yielding artifact named public/build/buildhub.json as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/host/bin/mar as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/host/bin/mbsdiff as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/mozharness.zip as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.awsy.tests.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.condprof.tests.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.cppunittest.tests.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.crashreporter-symbols.zip as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.generated-files.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.gtest.tests.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target_info.txt as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.jittest.tests.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.json as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.jsreftest.tests.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.jsshell.zip as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.langpack.xpi as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.mochitest.tests.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.mozinfo.json as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.perftests.tests.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.raptor.tests.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.reftest.tests.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.talos.tests.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.test_packages.json as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.txt as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.updater-dep.tests.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.web-platform.tests.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/build/target.xpcshell.tests.tar.gz as a candidate artifact
 0:00.88 D Not yielding artifact named public/chain-of-trust.json as a candidate artifact
 0:00.88 D Not yielding artifact named public/chain-of-trust.json.sig as a candidate artifact
 0:00.88 D Not yielding artifact named public/cidata/sccache.log as a candidate artifact
 0:00.88 D Not yielding artifact named public/cidata/sccache-stats.json as a candidate artifact
 0:00.88 D Not yielding artifact named public/cidata/target.crashreporter-symbols-full.tar.zst as a candidate artifact
 0:00.88 D Not yielding artifact named public/logs as a candidate artifact
 0:00.88 D Not yielding artifact named public/logs/certified.log as a candidate artifact
 0:00.88 D Not yielding artifact named public/logs/live_backing.log as a candidate artifact
 0:00.88 D Not yielding artifact named public/logs/live.log as a candidate artifact
 0:00.88 D Installing from remote pushhead ed2448d27fe2e034cd5804dc7310c3f72ac2eaa5 on integration/autoland

The end of that trail is the index entry https://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.v2.autoland.shippable.revision.ed2448d27fe2e034cd5804dc7310c3f72ac2eaa5.firefox/macosx64-opt, which leads to https://firefox-ci-tc.services.mozilla.com/tasks/Ya1A-DJERImEM29cvhqpIw. That task produced the relevant artifacts (https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.autoland.shippable.revision.ed2448d27fe2e034cd5804dc7310c3f72ac2eaa5.firefox.macosx64-opt/artifacts/public%2Fbuild%2Ftarget.update_framework_artifacts.zip), so I can't explain why your build didn't fetch them, since I see that your head (https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=2069eada79d9) definitely has the extra artifacts change at https://hg.mozilla.org/mozilla-central/file/2069eada79d9b4095e8799aaa8bd7659d5017ddb/python/mozbuild/mozbuild/artifacts.py#l678.

$ /Users/dave/mozilla/build/trunk/obj-browser-opt-artifact/dist/Nightly.app/Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater --channels-allowed
Error: 38

This means that the updater couldn't load UpdateSettings.framework. That's expected since it shouldn't have the correct artifacts for this to work.

If you continue to see this, I'll need more logs just like the ones you've provided. At this moment, I can't understand why you don't see the update framework artifacts being processed and installed locally.

Regressions: 1893552

I didn't see the development build apply a pending update this morning, but the development build is showing the update channel as nightly. Is that expected? It does look like the UpdateSettings framework is in the build updater but still seeing the same error when running with --channels-allowed.

(In reply to Dave Townsend [:mossop] from comment #23)

I didn't see the development build apply a pending update this morning, but the development build is showing the update channel as nightly.

No -- I would expect default, since the underlying .framework comes from https://searchfox.org/mozilla-central/rev/2e7374599f4020d0490ec8c5e877c179c9e153c9/toolkit/mozapps/macos-frameworks/ChannelPrefs-localbuild/moz.build#13.

Is that expected? It does look like the UpdateSettings framework is in the build updater but still seeing the same error when running with --channels-allowed.

That's also surprising. Can you give me the find | shasum outputs for your tree, please?

Flags: needinfo?(dtownsend)

(In reply to Nick Alexander :nalexander [he/him] from comment #24)

(In reply to Dave Townsend [:mossop] from comment #23)

I didn't see the development build apply a pending update this morning, but the development build is showing the update channel as nightly.

No -- I would expect default, since the underlying .framework comes from https://searchfox.org/mozilla-central/rev/2e7374599f4020d0490ec8c5e877c179c9e153c9/toolkit/mozapps/macos-frameworks/ChannelPrefs-localbuild/moz.build#13.

Is that expected? It does look like the UpdateSettings framework is in the build updater but still seeing the same error when running with --channels-allowed.

That's also surprising. Can you give me the find | shasum outputs for your tree, please?

Sorry -- from your object directory, like my:

$ find ../objdirs/objdir-browser-artifact -iname 'UpdateSettings' | xargs shasum
...
$ find ../objdirs/objdir-browser-artifact -iname 'ChannelPrefs' | xargs shasum
...

above.

Building revision 04a668ef9c90
It installed the artifacts from https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dQijG8QHS467PIrL0HaJcA/artifacts/public/build/target.dmg and https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dQijG8QHS467PIrL0HaJcA/artifacts/public/build/target.update_framework_artifacts.zip.

f31ba6453b6e5426fdb58f3c705763f9d41fadcb  ./_tests/xpcshell/toolkit/mozapps/update/tests/data/updater.app/Contents/Frameworks/UpdateSettings.framework/UpdateSettings
f31ba6453b6e5426fdb58f3c705763f9d41fadcb  ./_tests/testing/mochitest/browser/toolkit/mozapps/update/tests/browser/updater.app/Contents/Frameworks/UpdateSettings.framework/UpdateSettings
b0ff8c87de23a433ebc6f1ab02861bfd7bac6d3e  ./dist/update_framework_artifacts/UpdateSettings-localbuild.framework/UpdateSettings
b0ff8c87de23a433ebc6f1ab02861bfd7bac6d3e  ./dist/bin/updater.app/Contents/Frameworks/UpdateSettings.framework/UpdateSettings
b0ff8c87de23a433ebc6f1ab02861bfd7bac6d3e  ./dist/Nightly.app/Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/UpdateSettings

818d79d5c63989aae0a1d0255903f293105e3295  ./dist/update_framework_artifacts/ChannelPrefs-localbuild.framework/ChannelPrefs
818d79d5c63989aae0a1d0255903f293105e3295  ./dist/bin/ChannelPrefs.framework/ChannelPrefs
818d79d5c63989aae0a1d0255903f293105e3295  ./dist/Nightly.app/Contents/Frameworks/ChannelPrefs.framework/ChannelPrefs
Flags: needinfo?(dtownsend)

@nalexander Are you still investigating this?

Flags: needinfo?(nalexander)

(In reply to Robin Steuber (they/them) [:bytesized] from comment #27)

@nalexander Are you still investigating this?

No, because I haven't had any further reports. I see :gijs suggested on Slack that it's ongoing; if it is, I'd like to run it to ground.

Flags: needinfo?(nalexander)
Regressions: 1898637
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: