Latest braindump: - regression is within https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d6f61c448b906c1e68cdc66920d227f008cc2db9&tochange=6f5601bcf5ef9fa22b46e44e1d5e34893a04fc9e which is ranging from the last green cron job to the first green build with PGO disabled - some of those changes fails to apply current native messaging patch - the patch that started to break patch application was https://hg.mozilla.org/mozilla-central/rev/c04135b4f2515623b31c4b2db9eb5e971754f5be - I have tested updating snap's build to use LLVM 17.0.6, and WASI 20 to no luck - I have tested disabling Utility process sandbox in https://searchfox.org/mozilla-central/rev/c003be8b9727672e7d30972983b375f4c200233f/build/pgo/profileserver.py#148-154 to no luck - My bisections were not 100% reliable, some build failing one day and succeeding the other ; it is highly likely one needs to `snap run snapcraft clean` to make sure (this made builds working failing again for me) STR for local repro: - follow setup on https://snapcraft.io/docs/build-providers to make sure you have lxc - (make sure you dont run docker it might break with networking) - git clone https://github.com/canonical/firefox-snap/, checkout `nightly` branch, revert the change that disables `MOZ_PGO=1` - `snap run snapcraft --verbosity verbose --use-lxd --debug` STR for try: - take m-c ``` diff --git a/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch b/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch new file mode 100644 index 0000000000000..169b626fd05ad --- /dev/null +++ b/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch @@ -0,0 +1,15 @@ +diff --git a/snapcraft.yaml b/snapcraft.yaml +index 5ee8164..77d1529 100644 +--- a/snapcraft.yaml ++++ b/snapcraft.yaml +@@ -366,9 +366,7 @@ parts: + # Linking with gold fails on armhf (error: undefined reference to '__aeabi_uldivmod') and would need to be + # investigated further, and running PGO on arm64 takes forever (> 4 days in the Launchpad build environment!). + echo "ac_add_options --enable-linker=lld" >> $MOZCONFIG +- # Temporarily disable MOZ_PGO due to build failures: +- # https://bugzilla.mozilla.org/1869010 +- # echo "ac_add_options MOZ_PGO=1" >> $MOZCONFIG ++ echo "ac_add_options MOZ_PGO=1" >> $MOZCONFIG + fi + if [ $CRAFT_TARGET_ARCH != "armhf" ]; then + echo "ac_add_options --enable-rust-simd" >> $MOZCONFIG ``` - `./mach try fuzzy --full -q "'snap 'build 'try 'opt"`
Bug 1869010 Comment 10 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Latest braindump: - regression is within https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d6f61c448b906c1e68cdc66920d227f008cc2db9&tochange=6f5601bcf5ef9fa22b46e44e1d5e34893a04fc9e which is ranging from the last green cron job to the first green build with PGO disabled - some of those changes fails to apply current native messaging patch - the patch that started to break patch application was https://hg.mozilla.org/mozilla-central/rev/c04135b4f2515623b31c4b2db9eb5e971754f5be - I have tested updating snap's build to use LLVM 17.0.6, and WASI 20 to no luck - I have tested disabling Utility process sandbox in https://searchfox.org/mozilla-central/rev/c003be8b9727672e7d30972983b375f4c200233f/build/pgo/profileserver.py#148-154 to no luck - My bisections were not 100% reliable, some build failing one day and succeeding the other ; it is highly likely one needs to `snap run snapcraft clean` to make sure (this made builds working failing again for me) - Tried to update https://github.com/canonical/firefox-snap/pull/22 but PGO was still failing STR for local repro: - follow setup on https://snapcraft.io/docs/build-providers to make sure you have lxc - (make sure you dont run docker it might break with networking) - git clone https://github.com/canonical/firefox-snap/, checkout `nightly` branch, revert the change that disables `MOZ_PGO=1` - `snap run snapcraft --verbosity verbose --use-lxd --debug` STR for try: - take m-c ``` diff --git a/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch b/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch new file mode 100644 index 0000000000000..169b626fd05ad --- /dev/null +++ b/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch @@ -0,0 +1,15 @@ +diff --git a/snapcraft.yaml b/snapcraft.yaml +index 5ee8164..77d1529 100644 +--- a/snapcraft.yaml ++++ b/snapcraft.yaml +@@ -366,9 +366,7 @@ parts: + # Linking with gold fails on armhf (error: undefined reference to '__aeabi_uldivmod') and would need to be + # investigated further, and running PGO on arm64 takes forever (> 4 days in the Launchpad build environment!). + echo "ac_add_options --enable-linker=lld" >> $MOZCONFIG +- # Temporarily disable MOZ_PGO due to build failures: +- # https://bugzilla.mozilla.org/1869010 +- # echo "ac_add_options MOZ_PGO=1" >> $MOZCONFIG ++ echo "ac_add_options MOZ_PGO=1" >> $MOZCONFIG + fi + if [ $CRAFT_TARGET_ARCH != "armhf" ]; then + echo "ac_add_options --enable-rust-simd" >> $MOZCONFIG ``` - `./mach try fuzzy --full -q "'snap 'build 'try 'opt"` Triggered for a few days ; can be monitored on https://treeherder.mozilla.org/jobs?repo=try&searchStr=snap%2Camd64%2Copt&author=alissy%40mozilla.com&fromchange=3d1e32cd02cf18b0e75b0da291dcacf23534c227 ``` for HG_COMMIT in 31cf2cbea5939b2dc1639d5e011baee375fbf5f0 fd253d27e64ec2f0a321ef83bffbcf28392e9f88 b75cedb105e1b56ef1e3923c067100b6873ae53f 65a0553b296ae1432e5d18c3d2d607cacbb0a35e 556c8b22cbbbcc165c42b2c52ebdfd01be3cee6b 0ce8fcf63e627cf9d314fa8a471c3c2ad148ecd8 45383edb447de3f2958ed375a0b23b834aa60140 20cc1c7fc2f60de0739cc71c9d7bdc14ab12a55d a998d7d64649b68a9a7d81f3ea60c20da709fcc3 4028ba678039a3dd035da464b91632270e28a10b d56a4ca0375444e4e7ad2d4c1b1a2c2eca416b87 e7df49cb67e7b69c365956231cacc4ad6c9e2ef8 ac99cb1291098db9161c56ba4e184dd55edb0fb2 e6f456e40dda9f799c2fef84edc219d26631e268 bf2921de1f1d57eaf1a448425f112af0512c7612 0bd4f7d97fbb5edcc7b53fc0a05780c585109060 5d77812091445baad76c5d7a00d1ed102739c3c0 8493fb875960b426cb370c157c238bf6f607145f 407e629a0c77071d4f5bdc1560d8ce1530d5e3f6 cbfee6d8a0626dc4cea99ff66f3c41fff48c85ff 7d5aa4b996a1781850ab8f50a8846fa176288e26 69baaa4f31721fbc09b490c3cb1cdb465564e79e e901b86e5d894b8b40093d72036208cf8837e098 8e959a7ded5f111e711c06a6728f76d3bd660699 f198e511d598e9041ab3f540bf54ebe6103b88b1 ; do git checkout -b test_hg_${HG_COMMIT} $(git cinnabar hg2git ${HG_COMMIT}) git cherry-pick 859bf24171fe1f0b0371174c90ec8e9488063e1a git commit --amend --message "[SNAP] MOZ_PGO=1 on ${HG_COMMIT}" ./mach try fuzzy --push-to-lando --full -q "'snap 'build 'try 'opt" done; ```
Latest braindump: - regression is within https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d6f61c448b906c1e68cdc66920d227f008cc2db9&tochange=6f5601bcf5ef9fa22b46e44e1d5e34893a04fc9e which is ranging from the last green cron job to the first green build with PGO disabled - some of those changes fails to apply current native messaging patch - the patch that started to break patch application was https://hg.mozilla.org/mozilla-central/rev/c04135b4f2515623b31c4b2db9eb5e971754f5be - I have tested updating snap's build to use LLVM 17.0.6, and WASI 20 to no luck - I have tested disabling Utility process sandbox in https://searchfox.org/mozilla-central/rev/c003be8b9727672e7d30972983b375f4c200233f/build/pgo/profileserver.py#148-154 to no luck - My bisections were not 100% reliable, some build failing one day and succeeding the other ; it is highly likely one needs to `snap run snapcraft clean` to make sure (this made builds working failing again for me) - Tried to update https://github.com/canonical/firefox-snap/pull/22 but PGO was still failing STR for local repro: - follow setup on https://snapcraft.io/docs/build-providers to make sure you have lxc - (make sure you dont run docker it might break with networking) - git clone https://github.com/canonical/firefox-snap/, checkout `nightly` branch, revert the change that disables `MOZ_PGO=1` - `snap run snapcraft --verbosity verbose --use-lxd --debug` STR for try: - take m-c ``` diff --git a/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch b/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch new file mode 100644 index 0000000000000..169b626fd05ad --- /dev/null +++ b/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch @@ -0,0 +1,15 @@ +diff --git a/snapcraft.yaml b/snapcraft.yaml +index 5ee8164..77d1529 100644 +--- a/snapcraft.yaml ++++ b/snapcraft.yaml +@@ -366,9 +366,7 @@ parts: + # Linking with gold fails on armhf (error: undefined reference to '__aeabi_uldivmod') and would need to be + # investigated further, and running PGO on arm64 takes forever (> 4 days in the Launchpad build environment!). + echo "ac_add_options --enable-linker=lld" >> $MOZCONFIG +- # Temporarily disable MOZ_PGO due to build failures: +- # https://bugzilla.mozilla.org/1869010 +- # echo "ac_add_options MOZ_PGO=1" >> $MOZCONFIG ++ echo "ac_add_options MOZ_PGO=1" >> $MOZCONFIG + fi + if [ $CRAFT_TARGET_ARCH != "armhf" ]; then + echo "ac_add_options --enable-rust-simd" >> $MOZCONFIG ``` - `./mach try fuzzy --full -q "'snap 'build 'try 'opt"` Triggered for a few days ; can be monitored on https://treeherder.mozilla.org/jobs?repo=try&searchStr=snap%2Camd64%2Copt&author=alissy%40mozilla.com&fromchange=3d1e32cd02cf18b0e75b0da291dcacf23534c227 ``` for HG_COMMIT in 31cf2cbea5939b2dc1639d5e011baee375fbf5f0 fd253d27e64ec2f0a321ef83bffbcf28392e9f88 b75cedb105e1b56ef1e3923c067100b6873ae53f 65a0553b296ae1432e5d18c3d2d607cacbb0a35e 556c8b22cbbbcc165c42b2c52ebdfd01be3cee6b 0ce8fcf63e627cf9d314fa8a471c3c2ad148ecd8 45383edb447de3f2958ed375a0b23b834aa60140 20cc1c7fc2f60de0739cc71c9d7bdc14ab12a55d a998d7d64649b68a9a7d81f3ea60c20da709fcc3 4028ba678039a3dd035da464b91632270e28a10b d56a4ca0375444e4e7ad2d4c1b1a2c2eca416b87 e7df49cb67e7b69c365956231cacc4ad6c9e2ef8 ac99cb1291098db9161c56ba4e184dd55edb0fb2 e6f456e40dda9f799c2fef84edc219d26631e268 bf2921de1f1d57eaf1a448425f112af0512c7612 0bd4f7d97fbb5edcc7b53fc0a05780c585109060 5d77812091445baad76c5d7a00d1ed102739c3c0 8493fb875960b426cb370c157c238bf6f607145f 407e629a0c77071d4f5bdc1560d8ce1530d5e3f6 cbfee6d8a0626dc4cea99ff66f3c41fff48c85ff 7d5aa4b996a1781850ab8f50a8846fa176288e26 69baaa4f31721fbc09b490c3cb1cdb465564e79e e901b86e5d894b8b40093d72036208cf8837e098 8e959a7ded5f111e711c06a6728f76d3bd660699 f198e511d598e9041ab3f540bf54ebe6103b88b1 ; do git checkout -b test_hg_${HG_COMMIT} $(git cinnabar hg2git ${HG_COMMIT}) git cherry-pick 859bf24171fe1f0b0371174c90ec8e9488063e1a git commit --amend --message "[SNAP] MOZ_PGO=1 on ${HG_COMMIT}" ./mach try fuzzy --push-to-lando --full -q "'snap 'build 'try 'opt" done; ```
Latest braindump: - regression is within https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d6f61c448b906c1e68cdc66920d227f008cc2db9&tochange=6f5601bcf5ef9fa22b46e44e1d5e34893a04fc9e which is ranging from the last green cron job to the first green build with PGO disabled - some of those changes fails to apply current native messaging patch - the patch that started to break patch application was https://hg.mozilla.org/mozilla-central/rev/c04135b4f2515623b31c4b2db9eb5e971754f5be - I have tested updating snap's build to use LLVM 17.0.6, and WASI 20 to no luck - I have tested disabling Utility process sandbox in https://searchfox.org/mozilla-central/rev/c003be8b9727672e7d30972983b375f4c200233f/build/pgo/profileserver.py#148-154 to no luck - My bisections were not 100% reliable, some build failing one day and succeeding the other ; it is highly likely one needs to `snap run snapcraft clean` to make sure (this made builds working failing again for me) - Tried to update https://github.com/canonical/firefox-snap/pull/22 but PGO was still failing - local build fails but do not seem to generate any crash minidump (`find / -type f -name "*.dmp"` yields nothing) STR for local repro: - follow setup on https://snapcraft.io/docs/build-providers to make sure you have lxc - (make sure you dont run docker it might break with networking) - git clone https://github.com/canonical/firefox-snap/, checkout `nightly` branch, revert the change that disables `MOZ_PGO=1` - `snap run snapcraft --verbosity verbose --use-lxd --debug` STR for try: - take m-c ``` diff --git a/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch b/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch new file mode 100644 index 0000000000000..169b626fd05ad --- /dev/null +++ b/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch @@ -0,0 +1,15 @@ +diff --git a/snapcraft.yaml b/snapcraft.yaml +index 5ee8164..77d1529 100644 +--- a/snapcraft.yaml ++++ b/snapcraft.yaml +@@ -366,9 +366,7 @@ parts: + # Linking with gold fails on armhf (error: undefined reference to '__aeabi_uldivmod') and would need to be + # investigated further, and running PGO on arm64 takes forever (> 4 days in the Launchpad build environment!). + echo "ac_add_options --enable-linker=lld" >> $MOZCONFIG +- # Temporarily disable MOZ_PGO due to build failures: +- # https://bugzilla.mozilla.org/1869010 +- # echo "ac_add_options MOZ_PGO=1" >> $MOZCONFIG ++ echo "ac_add_options MOZ_PGO=1" >> $MOZCONFIG + fi + if [ $CRAFT_TARGET_ARCH != "armhf" ]; then + echo "ac_add_options --enable-rust-simd" >> $MOZCONFIG ``` - `./mach try fuzzy --full -q "'snap 'build 'try 'opt"` Triggered for a few days ; can be monitored on https://treeherder.mozilla.org/jobs?repo=try&searchStr=snap%2Camd64%2Copt&author=alissy%40mozilla.com&fromchange=3d1e32cd02cf18b0e75b0da291dcacf23534c227 ``` for HG_COMMIT in 31cf2cbea5939b2dc1639d5e011baee375fbf5f0 fd253d27e64ec2f0a321ef83bffbcf28392e9f88 b75cedb105e1b56ef1e3923c067100b6873ae53f 65a0553b296ae1432e5d18c3d2d607cacbb0a35e 556c8b22cbbbcc165c42b2c52ebdfd01be3cee6b 0ce8fcf63e627cf9d314fa8a471c3c2ad148ecd8 45383edb447de3f2958ed375a0b23b834aa60140 20cc1c7fc2f60de0739cc71c9d7bdc14ab12a55d a998d7d64649b68a9a7d81f3ea60c20da709fcc3 4028ba678039a3dd035da464b91632270e28a10b d56a4ca0375444e4e7ad2d4c1b1a2c2eca416b87 e7df49cb67e7b69c365956231cacc4ad6c9e2ef8 ac99cb1291098db9161c56ba4e184dd55edb0fb2 e6f456e40dda9f799c2fef84edc219d26631e268 bf2921de1f1d57eaf1a448425f112af0512c7612 0bd4f7d97fbb5edcc7b53fc0a05780c585109060 5d77812091445baad76c5d7a00d1ed102739c3c0 8493fb875960b426cb370c157c238bf6f607145f 407e629a0c77071d4f5bdc1560d8ce1530d5e3f6 cbfee6d8a0626dc4cea99ff66f3c41fff48c85ff 7d5aa4b996a1781850ab8f50a8846fa176288e26 69baaa4f31721fbc09b490c3cb1cdb465564e79e e901b86e5d894b8b40093d72036208cf8837e098 8e959a7ded5f111e711c06a6728f76d3bd660699 f198e511d598e9041ab3f540bf54ebe6103b88b1 ; do git checkout -b test_hg_${HG_COMMIT} $(git cinnabar hg2git ${HG_COMMIT}) git cherry-pick 859bf24171fe1f0b0371174c90ec8e9488063e1a git commit --amend --message "[SNAP] MOZ_PGO=1 on ${HG_COMMIT}" ./mach try fuzzy --push-to-lando --full -q "'snap 'build 'try 'opt" done; ```
Latest braindump: - regression is within https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d6f61c448b906c1e68cdc66920d227f008cc2db9&tochange=6f5601bcf5ef9fa22b46e44e1d5e34893a04fc9e which is ranging from the last green cron job to the first green build with PGO disabled - some of those changes fails to apply current native messaging patch - the patch that started to break patch application was https://hg.mozilla.org/mozilla-central/rev/c04135b4f2515623b31c4b2db9eb5e971754f5be - I have tested updating snap's build to use LLVM 17.0.6, and WASI 20 to no luck - I have tested disabling Utility process sandbox in https://searchfox.org/mozilla-central/rev/c003be8b9727672e7d30972983b375f4c200233f/build/pgo/profileserver.py#148-154 to no luck - My bisections were not 100% reliable, some build failing one day and succeeding the other ; it is highly likely one needs to `snap run snapcraft clean` to make sure (this made builds working failing again for me) - Tried to update https://github.com/canonical/firefox-snap/pull/22 but PGO was still failing - local build fails but do not seem to generate any crash minidump (`find / -type f -name "*.dmp"` yields nothing, within the `LXC` container) STR for local repro: - follow setup on https://snapcraft.io/docs/build-providers to make sure you have lxc - (make sure you dont run docker it might break with networking) - git clone https://github.com/canonical/firefox-snap/, checkout `nightly` branch, revert the change that disables `MOZ_PGO=1` - `snap run snapcraft --verbosity verbose --use-lxd --debug` STR for try: - take m-c ``` diff --git a/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch b/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch new file mode 100644 index 0000000000000..169b626fd05ad --- /dev/null +++ b/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch @@ -0,0 +1,15 @@ +diff --git a/snapcraft.yaml b/snapcraft.yaml +index 5ee8164..77d1529 100644 +--- a/snapcraft.yaml ++++ b/snapcraft.yaml +@@ -366,9 +366,7 @@ parts: + # Linking with gold fails on armhf (error: undefined reference to '__aeabi_uldivmod') and would need to be + # investigated further, and running PGO on arm64 takes forever (> 4 days in the Launchpad build environment!). + echo "ac_add_options --enable-linker=lld" >> $MOZCONFIG +- # Temporarily disable MOZ_PGO due to build failures: +- # https://bugzilla.mozilla.org/1869010 +- # echo "ac_add_options MOZ_PGO=1" >> $MOZCONFIG ++ echo "ac_add_options MOZ_PGO=1" >> $MOZCONFIG + fi + if [ $CRAFT_TARGET_ARCH != "armhf" ]; then + echo "ac_add_options --enable-rust-simd" >> $MOZCONFIG ``` - `./mach try fuzzy --full -q "'snap 'build 'try 'opt"` Triggered for a few days ; can be monitored on https://treeherder.mozilla.org/jobs?repo=try&searchStr=snap%2Camd64%2Copt&author=alissy%40mozilla.com&fromchange=3d1e32cd02cf18b0e75b0da291dcacf23534c227 ``` for HG_COMMIT in 31cf2cbea5939b2dc1639d5e011baee375fbf5f0 fd253d27e64ec2f0a321ef83bffbcf28392e9f88 b75cedb105e1b56ef1e3923c067100b6873ae53f 65a0553b296ae1432e5d18c3d2d607cacbb0a35e 556c8b22cbbbcc165c42b2c52ebdfd01be3cee6b 0ce8fcf63e627cf9d314fa8a471c3c2ad148ecd8 45383edb447de3f2958ed375a0b23b834aa60140 20cc1c7fc2f60de0739cc71c9d7bdc14ab12a55d a998d7d64649b68a9a7d81f3ea60c20da709fcc3 4028ba678039a3dd035da464b91632270e28a10b d56a4ca0375444e4e7ad2d4c1b1a2c2eca416b87 e7df49cb67e7b69c365956231cacc4ad6c9e2ef8 ac99cb1291098db9161c56ba4e184dd55edb0fb2 e6f456e40dda9f799c2fef84edc219d26631e268 bf2921de1f1d57eaf1a448425f112af0512c7612 0bd4f7d97fbb5edcc7b53fc0a05780c585109060 5d77812091445baad76c5d7a00d1ed102739c3c0 8493fb875960b426cb370c157c238bf6f607145f 407e629a0c77071d4f5bdc1560d8ce1530d5e3f6 cbfee6d8a0626dc4cea99ff66f3c41fff48c85ff 7d5aa4b996a1781850ab8f50a8846fa176288e26 69baaa4f31721fbc09b490c3cb1cdb465564e79e e901b86e5d894b8b40093d72036208cf8837e098 8e959a7ded5f111e711c06a6728f76d3bd660699 f198e511d598e9041ab3f540bf54ebe6103b88b1 ; do git checkout -b test_hg_${HG_COMMIT} $(git cinnabar hg2git ${HG_COMMIT}) git cherry-pick 859bf24171fe1f0b0371174c90ec8e9488063e1a git commit --amend --message "[SNAP] MOZ_PGO=1 on ${HG_COMMIT}" ./mach try fuzzy --push-to-lando --full -q "'snap 'build 'try 'opt" done; ```
Latest braindump: - regression is within https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d6f61c448b906c1e68cdc66920d227f008cc2db9&tochange=6f5601bcf5ef9fa22b46e44e1d5e34893a04fc9e which is ranging from the last green cron job to the first green build with PGO disabled - some of those changes fails to apply current native messaging patch - the patch that started to break patch application was https://hg.mozilla.org/mozilla-central/rev/c04135b4f2515623b31c4b2db9eb5e971754f5be - I have tested updating snap's build to use LLVM 17.0.6, and WASI 20 to no luck - I have tested disabling Utility process sandbox in https://searchfox.org/mozilla-central/rev/c003be8b9727672e7d30972983b375f4c200233f/build/pgo/profileserver.py#148-154 to no luck - My bisections were not 100% reliable, some build failing one day and succeeding the other ; it is highly likely one needs to `snap run snapcraft clean` to make sure (this made builds working failing again for me) - Tried to update https://github.com/canonical/firefox-snap/pull/22 but PGO was still failing - local build fails but do not seem to generate any crash minidump (`find / -type f -name "*.dmp"` yields nothing, within the `LXC` container) STR for local repro: - follow setup on https://snapcraft.io/docs/build-providers to make sure you have lxc - (make sure you dont run docker it might break with networking) - git clone https://github.com/canonical/firefox-snap/, checkout `nightly` branch, revert the change that disables `MOZ_PGO=1` - make sure you force `REVISION=` for the `hg clone` step - `snap run snapcraft --verbosity verbose --use-lxd --debug` STR for try: - take m-c ``` diff --git a/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch b/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch new file mode 100644 index 0000000000000..169b626fd05ad --- /dev/null +++ b/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch @@ -0,0 +1,15 @@ +diff --git a/snapcraft.yaml b/snapcraft.yaml +index 5ee8164..77d1529 100644 +--- a/snapcraft.yaml ++++ b/snapcraft.yaml +@@ -366,9 +366,7 @@ parts: + # Linking with gold fails on armhf (error: undefined reference to '__aeabi_uldivmod') and would need to be + # investigated further, and running PGO on arm64 takes forever (> 4 days in the Launchpad build environment!). + echo "ac_add_options --enable-linker=lld" >> $MOZCONFIG +- # Temporarily disable MOZ_PGO due to build failures: +- # https://bugzilla.mozilla.org/1869010 +- # echo "ac_add_options MOZ_PGO=1" >> $MOZCONFIG ++ echo "ac_add_options MOZ_PGO=1" >> $MOZCONFIG + fi + if [ $CRAFT_TARGET_ARCH != "armhf" ]; then + echo "ac_add_options --enable-rust-simd" >> $MOZCONFIG ``` - `./mach try fuzzy --full -q "'snap 'build 'try 'opt"` Triggered for a few days ; can be monitored on https://treeherder.mozilla.org/jobs?repo=try&searchStr=snap%2Camd64%2Copt&author=alissy%40mozilla.com&fromchange=3d1e32cd02cf18b0e75b0da291dcacf23534c227 ``` for HG_COMMIT in 31cf2cbea5939b2dc1639d5e011baee375fbf5f0 fd253d27e64ec2f0a321ef83bffbcf28392e9f88 b75cedb105e1b56ef1e3923c067100b6873ae53f 65a0553b296ae1432e5d18c3d2d607cacbb0a35e 556c8b22cbbbcc165c42b2c52ebdfd01be3cee6b 0ce8fcf63e627cf9d314fa8a471c3c2ad148ecd8 45383edb447de3f2958ed375a0b23b834aa60140 20cc1c7fc2f60de0739cc71c9d7bdc14ab12a55d a998d7d64649b68a9a7d81f3ea60c20da709fcc3 4028ba678039a3dd035da464b91632270e28a10b d56a4ca0375444e4e7ad2d4c1b1a2c2eca416b87 e7df49cb67e7b69c365956231cacc4ad6c9e2ef8 ac99cb1291098db9161c56ba4e184dd55edb0fb2 e6f456e40dda9f799c2fef84edc219d26631e268 bf2921de1f1d57eaf1a448425f112af0512c7612 0bd4f7d97fbb5edcc7b53fc0a05780c585109060 5d77812091445baad76c5d7a00d1ed102739c3c0 8493fb875960b426cb370c157c238bf6f607145f 407e629a0c77071d4f5bdc1560d8ce1530d5e3f6 cbfee6d8a0626dc4cea99ff66f3c41fff48c85ff 7d5aa4b996a1781850ab8f50a8846fa176288e26 69baaa4f31721fbc09b490c3cb1cdb465564e79e e901b86e5d894b8b40093d72036208cf8837e098 8e959a7ded5f111e711c06a6728f76d3bd660699 f198e511d598e9041ab3f540bf54ebe6103b88b1 ; do git checkout -b test_hg_${HG_COMMIT} $(git cinnabar hg2git ${HG_COMMIT}) git cherry-pick 859bf24171fe1f0b0371174c90ec8e9488063e1a git commit --amend --message "[SNAP] MOZ_PGO=1 on ${HG_COMMIT}" ./mach try fuzzy --push-to-lando --full -q "'snap 'build 'try 'opt" done; ```
Latest braindump: - regression is within https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d6f61c448b906c1e68cdc66920d227f008cc2db9&tochange=6f5601bcf5ef9fa22b46e44e1d5e34893a04fc9e which is ranging from the last green cron job to the first green build with PGO disabled - some of those changes fails to apply current native messaging patch - the patch that started to break patch application was https://hg.mozilla.org/mozilla-central/rev/c04135b4f2515623b31c4b2db9eb5e971754f5be - renaming of files in https://hg.mozilla.org/mozilla-central/rev/f3aeb9958460a3831b9f5a225d322074641f6242 is also required to apply the patch - I have tested updating snap's build to use LLVM 17.0.6, and WASI 20 to no luck - I have tested disabling Utility process sandbox in https://searchfox.org/mozilla-central/rev/c003be8b9727672e7d30972983b375f4c200233f/build/pgo/profileserver.py#148-154 to no luck - My bisections were not 100% reliable, some build failing one day and succeeding the other ; it is highly likely one needs to `snap run snapcraft clean` to make sure (this made builds working failing again for me) - Tried to update https://github.com/canonical/firefox-snap/pull/22 but PGO was still failing - local build fails but do not seem to generate any crash minidump (`find / -type f -name "*.dmp"` yields nothing, within the `LXC` container) STR for local repro: - follow setup on https://snapcraft.io/docs/build-providers to make sure you have lxc - (make sure you dont run docker it might break with networking) - git clone https://github.com/canonical/firefox-snap/, checkout `nightly` branch, revert the change that disables `MOZ_PGO=1` - make sure you force `REVISION=` for the `hg clone` step - `snap run snapcraft --verbosity verbose --use-lxd --debug` STR for try: - take m-c ``` diff --git a/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch b/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch new file mode 100644 index 0000000000000..169b626fd05ad --- /dev/null +++ b/taskcluster/docker/snap-coreXX-build/patches/nightly/re-add-pgo.patch @@ -0,0 +1,15 @@ +diff --git a/snapcraft.yaml b/snapcraft.yaml +index 5ee8164..77d1529 100644 +--- a/snapcraft.yaml ++++ b/snapcraft.yaml +@@ -366,9 +366,7 @@ parts: + # Linking with gold fails on armhf (error: undefined reference to '__aeabi_uldivmod') and would need to be + # investigated further, and running PGO on arm64 takes forever (> 4 days in the Launchpad build environment!). + echo "ac_add_options --enable-linker=lld" >> $MOZCONFIG +- # Temporarily disable MOZ_PGO due to build failures: +- # https://bugzilla.mozilla.org/1869010 +- # echo "ac_add_options MOZ_PGO=1" >> $MOZCONFIG ++ echo "ac_add_options MOZ_PGO=1" >> $MOZCONFIG + fi + if [ $CRAFT_TARGET_ARCH != "armhf" ]; then + echo "ac_add_options --enable-rust-simd" >> $MOZCONFIG ``` - `./mach try fuzzy --full -q "'snap 'build 'try 'opt"` Triggered for a few days ; can be monitored on https://treeherder.mozilla.org/jobs?repo=try&searchStr=snap%2Camd64%2Copt&author=alissy%40mozilla.com&fromchange=3d1e32cd02cf18b0e75b0da291dcacf23534c227 ``` for HG_COMMIT in 31cf2cbea5939b2dc1639d5e011baee375fbf5f0 fd253d27e64ec2f0a321ef83bffbcf28392e9f88 b75cedb105e1b56ef1e3923c067100b6873ae53f 65a0553b296ae1432e5d18c3d2d607cacbb0a35e 556c8b22cbbbcc165c42b2c52ebdfd01be3cee6b 0ce8fcf63e627cf9d314fa8a471c3c2ad148ecd8 45383edb447de3f2958ed375a0b23b834aa60140 20cc1c7fc2f60de0739cc71c9d7bdc14ab12a55d a998d7d64649b68a9a7d81f3ea60c20da709fcc3 4028ba678039a3dd035da464b91632270e28a10b d56a4ca0375444e4e7ad2d4c1b1a2c2eca416b87 e7df49cb67e7b69c365956231cacc4ad6c9e2ef8 ac99cb1291098db9161c56ba4e184dd55edb0fb2 e6f456e40dda9f799c2fef84edc219d26631e268 bf2921de1f1d57eaf1a448425f112af0512c7612 0bd4f7d97fbb5edcc7b53fc0a05780c585109060 5d77812091445baad76c5d7a00d1ed102739c3c0 8493fb875960b426cb370c157c238bf6f607145f 407e629a0c77071d4f5bdc1560d8ce1530d5e3f6 cbfee6d8a0626dc4cea99ff66f3c41fff48c85ff 7d5aa4b996a1781850ab8f50a8846fa176288e26 69baaa4f31721fbc09b490c3cb1cdb465564e79e e901b86e5d894b8b40093d72036208cf8837e098 8e959a7ded5f111e711c06a6728f76d3bd660699 f198e511d598e9041ab3f540bf54ebe6103b88b1 ; do git checkout -b test_hg_${HG_COMMIT} $(git cinnabar hg2git ${HG_COMMIT}) git cherry-pick 859bf24171fe1f0b0371174c90ec8e9488063e1a git commit --amend --message "[SNAP] MOZ_PGO=1 on ${HG_COMMIT}" ./mach try fuzzy --push-to-lando --full -q "'snap 'build 'try 'opt" done; ```