Closed Bug 1342383 Opened 3 years ago Closed 2 years ago

Failed release-comm-beta-thunderbird_source

Categories

(Thunderbird :: Build Config, defect, P1)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: wsmwk, Unassigned)

References

Details

(In reply to Ryan VanderMeulen [:RyanVM] from comment #19)
> https://hg.mozilla.org/releases/mozilla-beta/rev/484c20783a1b

(I assume this issue is blocking our release)

https://archive.mozilla.org/pub/thunderbird/candidates/52.0b4-candidates/build1/logs/release-comm-beta-thunderbird_source-bm73-build1-build3.txt.gz
nthomas points to this bug for a failed thunderbird_source beta build step yielding...

builder: release-comm-beta-thunderbird_source
slave: bld-linux64-spot-327
starttime: 1487904997.44
results: failure (2)
buildid: 20170223185626
builduid: 004bf4ba9da24d8e8477ec142f559c4e
revision: e3e36f9e7694

========= Started make source-package failed (results: 2, elapsed: 0 secs) (at 2017-02-23 19:06:26.675059) =========
mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave/tb-rel-c-beta-tb_source-000000/build/comm-beta/objdir-tb --unpriv --shell '/usr/bin/env HG_SHARE_BASE_DIR="/builds/hg-shared" LC_ALL="C" no_tooltool="1" MOZILLA_REV="THUNDERBIRD_52_0b4_RELEASE" MOZ_AUTOMATION="1" MOZ_SIGN_CMD="python /builds/slave/tb-rel-c-beta-tb_source-000000/tools/release/signing/signtool.py --cachedir /builds/slave/tb-rel-c-beta-tb_source-000000/signing_cache -t /builds/slave/tb-rel-c-beta-tb_source-000000/token -n /builds/slave/tb-rel-c-beta-tb_source-000000/nonce -c /builds/slave/tb-rel-c-beta-tb_source-000000/tools/release/signing/host.cert -H
...
# Make sure to have repository information available and then generate the
# sourcestamp file.
make -C ../.. 'source-repo.h' 'buildid.h'
source-repo.h
make[2]: Entering directory `/builds/slave/tb-rel-c-beta-tb_source-000000/build/comm-beta/objdir-tb'
/builds/slave/tb-rel-c-beta-tb_source-000000/build/comm-beta/objdir-tb/_virtualenv/bin/python -m mozbuild.action.file_generate /builds/slave/tb-rel-c-beta-tb_source-000000/comm-beta/mozilla/build/variables.py source_repo_header source-repo.h .deps/source-repo.h.pp
buildid.h
/builds/slave/tb-rel-c-beta-tb_source-000000/build/comm-beta/objdir-tb/_virtualenv/bin/python -m mozbuild.action.file_generate /builds/slave/tb-rel-c-beta-tb_source-000000/comm-beta/mozilla/build/variables.py buildid_header buildid.h .deps/buildid.h.pp
make[2]: Leaving directory `/builds/slave/tb-rel-c-beta-tb_source-000000/build/comm-beta/objdir-tb'
make make-sourcestamp-file
make[2]: Entering directory `/builds/slave/tb-rel-c-beta-tb_source-000000/build/comm-beta/objdir-tb/mail/installer'
/bin/sh: ../../dist/linux-x86_64/en-US//thunderbird-52.0b4.txt: No such file or directory
make[2]: *** [make-sourcestamp-file] Error 1
make[2]: Leaving directory `/builds/slave/tb-rel-c-beta-tb_source-000000/build/comm-beta/objdir-tb/mail/installer'
make[1]: *** [source-package] Error 2
make[1]: Leaving directory `/builds/slave/tb-rel-c-beta-tb_source-000000/build/comm-beta/objdir-tb/mail/installer'
make: *** [source-package] Error 2
State Changed: unlock buildroot
program finished with exit code 2
elapsedTime=0.840202
I guess this is a regression from the removal of "$(NSINSTALL) -D $(DIST)/$(PKG_PATH)".

Maybe it can be replaced by "$(MKDIR) -p $(DIST)/$(PKG_PATH)"? The source-package rule already uses "$(MKDIR) -p".
Also see on https://treeherder.mozilla.org/#/jobs?repo=comm-beta in the build oranges:

# Package JavaScript Shell
Packaging JavaScript Shell...
rm -f ../../dist/jsshell-linux-i686.zip
/builds/slave/tb-c-beta-lx-00000000000000000/build/objdir-tb/_virtualenv/bin/python -m mozbuild.action.zip -C ../../dist/bin /builds/slave/tb-c-beta-lx-00000000000000000/build/objdir-tb/dist/jsshell-linux-i686.zip js libmozglue.so  libnspr4.so libplds4.so libplc4.so 
/bin/sh: ../../dist/linux-i686/en-US//earlybird-52.0.txt: No such file or directory
Component: Release Automation → Build Config
Priority: -- → P1
Product: Release Engineering → Thunderbird
QA Contact: rail
aleth, can you propose a patch?
Flags: needinfo?(aleth)
(In reply to Jan Steffens from comment #1)
> I guess this is a regression from the removal of "$(NSINSTALL) -D
> $(DIST)/$(PKG_PATH)".
> 
> Maybe it can be replaced by "$(MKDIR) -p $(DIST)/$(PKG_PATH)"? The
> source-package rule already uses "$(MKDIR) -p".

I think that would make sense, as per bug 1338099 comment 9 (I'm not a build system expert though).

I wonder: why is this only happening with TB and not on the FF tree?
It looks like PKG_PATH is getting set to a non-empty value for these builds. I don't see anywhere in m-c or c-c where that actually happens, am I missing something?
(In reply to Jorg K (GMT+1) from comment #2)
> Also see on https://treeherder.mozilla.org/#/jobs?repo=comm-beta in the
> build oranges:
Also causing build oranges on C-A: https://treeherder.mozilla.org/#/jobs?repo=comm-aurora
looks to be a blocker for fennec beta now too:

make[1]: Entering directory `/builds/slave/rel-m-beta-fennec_source-00000/build/mozilla-beta/obj-firefox/browser/installer'
Generate the sourcestamp file
# Make sure to have repository information available and then generate the
# sourcestamp file.
make -C ../.. 'source-repo.h' 'buildid.h'
source-repo.h
make[2]: Entering directory `/builds/slave/rel-m-beta-fennec_source-00000/build/mozilla-beta/obj-firefox'
/builds/slave/rel-m-beta-fennec_source-00000/build/mozilla-beta/obj-firefox/_virtualenv/bin/python -m mozbuild.action.file_generate /builds/slave/rel-m-beta-fennec_source-00000/mozilla-beta/build/variables.py source_repo_header source-repo.h .deps/source-repo.h.pp
buildid.h
/builds/slave/rel-m-beta-fennec_source-00000/build/mozilla-beta/obj-firefox/_virtualenv/bin/python -m mozbuild.action.file_generate /builds/slave/rel-m-beta-fennec_source-00000/mozilla-beta/build/variables.py buildid_header buildid.h .deps/buildid.h.pp
make[2]: Leaving directory `/builds/slave/rel-m-beta-fennec_source-00000/build/mozilla-beta/obj-firefox'
make make-sourcestamp-file
make[2]: Entering directory `/builds/slave/rel-m-beta-fennec_source-00000/build/mozilla-beta/obj-firefox/browser/installer'
/bin/sh: ../../dist/linux-x86_64/en-US//fennec-52.0b10.txt: No such file or directory
make[2]: *** [make-sourcestamp-file] Error 1
make[2]: Leaving directory `/builds/slave/rel-m-beta-fennec_source-00000/build/mozilla-beta/obj-firefox/browser/installer'
make[1]: *** [source-package] Error 2
make[1]: Leaving directory `/builds/slave/rel-m-beta-fennec_source-00000/build/mozilla-beta/obj-firefox/browser/installer'
make: *** [source-package] Error 2
State Changed: unlock buildroot
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> It looks like PKG_PATH is getting set to a non-empty value for these builds.
> I don't see anywhere in m-c or c-c where that actually happens, am I missing
> something?

iiuc, it was removed with this cset: https://hg.mozilla.org/mozilla-central/rev/318f1bcd336e

specifically:
-PKG_PATH = $(MOZ_PKG_PLATFORM)/$(AB_CD)/

furthermore, I was wondering if it's actually necessary to have the
trailing /, since that seems to be added to the /thunderbird...
(In reply to Jordan Lund (:jlund) from comment #7)
> looks to be a blocker for fennec beta now too:
> 


as per irc, can we just back out this one rev: https://hg.mozilla.org/releases/mozilla-beta/rev/484c20783a1b while tb fixes issue? I will backout if I don't hear from you by EOD.
Flags: needinfo?(vseerror)
Fwiw & per irc discussion, the (probably) offending bug (#1338099) might be backed out.
As I said in comment #6, Aurora is also affected, for example:

https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-aurora-win32/1488386181/comm-aurora-win32-bm70-build1-build0.txt.gz

C:/Users/cltbld/AppData/Local/Temp/make1168-6.sh: line 1: ../../dist/win32/en-US//earlybird-53.0a2.txt: No such file or directory

What's the plan to fix that?
thanks all!
can you rerun the 52.0b4 source build step?
Flags: needinfo?(vseerror) → needinfo?(jlund)
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #15)
> can you rerun the 52.0b4 source build step?

hm, are you sure you want to do that? iirc, rail force pushed to 52.0b4 to cdns because this failed back on feb 24th and blocked remaining release automation jobs. If we I trigger this now, I wonder if this will cause a cascading affect of releasing other jobs that were previously blocked.

Also, hasn't this release shipped? I would be hesitant to run release automation jobs weeks after shipping.

Please needinfo again as I will likely miss reply
Flags: needinfo?(jlund)
Thanks for the info. To be concise, normally I wouldn't want rerun it if there is risk to messing up other areas of 52.0b

However, Bug 1346129 involves 52.0 failed thunderbird_source. So we are in the same boat with 52.0. We'll either need rerun the source step, or rerun 52.0 build - which we may well need to do do anyway because of 52.0 repack issues.
I assume that this is fixed, since this release has been made.
Status: NEW → RESOLVED
Closed: 2 years ago
Flags: needinfo?(aleth)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.