Thunderbird 54.0b1 build1: build step failed on win32 (and all OS) caused by removing MOZ_PKG_PRETTYNAMES

RESOLVED FIXED

Status

defect
P2
major
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: wsmwk, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Reporter

Description

2 years ago
(I don't know whether this is automation fail or Thunderbird build fail)

from https://archive.mozilla.org/pub/thunderbird/candidates/54.0b1-candidates/build1/logs/release-comm-beta-win32_build-bm91-build1-build4.txt.gz

https://pastebin.mozilla.org/9020242

========= Started find filepath failed (results: 2, elapsed: 0 secs) (at 2017-04-28 11:40:39.456958) =========
'bash' '-c' 'find build/objdir-tb/dist/update/win32/en-US -maxdepth 1 -type f -name thunderbird-54.0b1.complete.mar'
 in dir c:\builds\moz2_slave\tb-rel-c-beta-w32_bld-00000000\. (timeout 1200 secs)
 watching logfiles {}
 argv: ['bash', '-c', 'find build/objdir-tb/dist/update/win32/en-US -maxdepth 1 -type f -name thunderbird-54.0b1.complete.mar']
 environment:
  ALLUSERSPROFILE=C:\ProgramData
  APPDATA=C:\Users\cltbld\AppData\Roaming
...
  WINDIR=C:\Windows
  WINDOWS_TRACING_FLAGS=3
  WINDOWS_TRACING_LOGFILE=C:\BVTBin\Tests\installpackage\csilogfile.log
  WIX_351728_PATH=c:/mozilla-build/wix-351728
  _=C:\mozilla-build\buildbotve\Scripts\python
 using PTY: False
find: build/objdir-tb/dist/update/win32/en-US: No such file or directory
Reporter

Comment 1

2 years ago
https://archive.mozilla.org/pub/thunderbird/candidates/54.0b1-candidates/build1/logs/release-comm-beta-macosx64_build-bm82-build1-build16.txt.gz

 find: build/objdir-tb/dist/update/mac/en-US: No such file or directory


https://archive.mozilla.org/pub/thunderbird/candidates/54.0b1-candidates/build1/logs/release-comm-beta-linux_build-bm91-build1-build9.txt.gz

 find: `build/objdir-tb/dist/update/linux-i686/en-US': No such file or directory
program finished with exit code 1
When I check the log, I see the signtool.py output is in ../../dist/update/ and not in ../../dist/update/linux-i686/en-US. With this it's clear the following find fails. Is signtool.py wrong or should the path be adapted?

python /builds/slave/tb-rel-c-beta-lx_bld-000000000/tools/release/signing/signtool.py --cachedir /builds/slave/tb-rel-c-beta-lx_bld-000000000/signing_cache -t /builds/slave/tb-rel-c-beta-lx_bld-000000000/token -n /builds/slave/tb-rel-c-beta-lx_bld-000000000/nonce -c /builds/slave/tb-rel-c-beta-lx_bld-000000000/tools/release/signing/host.cert -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120 -H macapp:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120 -f mar '../../dist/update/thunderbird-54.0b1.en-US.linux-i686.complete.mar'

As reference the 53beta:

python /builds/slave/tb-rel-c-beta-lx_bld-000000000/tools/release/signing/signtool.py --cachedir /builds/slave/tb-rel-c-beta-lx_bld-000000000/signing_cache -t /builds/slave/tb-rel-c-beta-lx_bld-000000000/token -n /builds/slave/tb-rel-c-beta-lx_bld-000000000/nonce -c /builds/slave/tb-rel-c-beta-lx_bld-000000000/tools/release/signing/host.cert -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120 -H macapp:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120 -f mar '../../dist/update/linux-i686/en-US/thunderbird-53.0b2.complete.mar'
# quick investigation on the 54.0b1 build machine
[cltbld@bld-lion-r5-071]$ pwd
/builds/slave/tb-rel-c-beta-m64_bld-00000000

# a somewhat broader find than the log to locate the complete mar
[cltbld@bld-lion-r5-071]$ find . -type f -name thunderbird-54.0b1.complete.mar

# huh, do we have any complete.mar file at all ?
[cltbld@bld-lion-r5-071]$ find . -type f -name '*.complete.mar'
./build/objdir-tb/dist/update/thunderbird-54.0b1.en-US.mac.complete.mar

For comparision 53.0b2 found 
  build/objdir-tb/dist/update/mac/en-US/thunderbird-53.0b2.complete.mar

So looks like a pretty naming issue, and is fallout from bug 1329355 removing MOZ_PKG_PRETTYNAMES.
Reporter

Comment 4

2 years ago
A yes, bug 1329355 comment 11, bug 1329355 comment 18. I guess we never opened a bug for it :(
Reporter

Comment 5

2 years ago
Richard, are you able to provide a fix?

This blocks our progress to reenabling updates from TB45, so => major
Blocks: 1329355
Severity: normal → major
Flags: needinfo?(richard.marti)
Summary: Thunderbird 54.0b1 build1: build step failed on win32 (and all OS) → Thunderbird 54.0b1 build1: build step failed on win32 (and all OS) caused by removing MOZ_PKG_PRETTYNAMES
I looked into the changes from bug 1329355 and can't say what changes are needed on our side, sorry.
Flags: needinfo?(richard.marti)
(In reply to Richard Marti (:Paenglab) from comment #6)
> I looked into the changes from bug 1329355 and can't say what changes are
> needed on our side, sorry.

Oh.  This has finally hit beta..  :/  

I'll have a look at it.

Updated

2 years ago
Priority: -- → P2
It seems to me you have two realistic options here
* petition for the global backout of bug 1329355 (just https://reviewboard.mozilla.org/r/104620/diff/2#index_header may be sufficient)
* set up a verbranch again and apply the backout there, either temporarily to buy some time, or permanently (ie always have a verbranch going forward, with all that entails with merging etc)

Otherwise you're left with adjusting the release automation to the non-pretty naming, or implement a renaming somehow. As this is likely require quite a lot of work, and RelEng won't be able to support that with development time, I'd strongly recommend either of the two bullet points.
Reporter

Comment 9

2 years ago
Thanks Nick.

Fallen and ewong, what is the preference?
Flags: needinfo?(philipp)
Flags: needinfo?(ewong)

Comment 10

2 years ago
I can back stuff out on the beta release branch for TB if necessary.
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #9)
> Thanks Nick.
> 
> Fallen and ewong, what is the preference?

I am doubting they're going to agree with the global backout and this
bug is a tad bit more involved than I had hoped (currently having
less time to devote to this as I had thought).

In the meantime, I think the best way forward is to backout on a relbranch and
build beta off that.
Flags: needinfo?(ewong)
I would also opt for fixing it in our release automation. It may be more work, but just a little bit of renaming doesn't sound very complicated to me?

If you want to get 54.0b1 to build I'd also suggest a temporary relbranch. I wouldn't use that as a permanent solution.
Flags: needinfo?(philipp)
Reporter

Comment 14

2 years ago
Agh!  I totally missed Jorg's backout.  So we are good to build?  Why then are we doing 52.1.1?

Comment 15

2 years ago
Umm, someone is confused here. We thought we were good to build, and you *did* trigger a build (build 2) on 2017-05-05, the day of the backout. Sadly that didn't go very far an ran into bug 1362611. We are doing 52.1.1 since TB 54 beta did indeed fail on the first attempt due to this bug here and then on the second attempt due to bug 1362611.
Reporter

Updated

2 years ago
See Also: → 1362611

Comment 16

2 years ago
Tom, as you might have noticed by now, we have a terrible back-log of build & release issues. Here is today's issue of the day:

To build TB 54 beta I had to create a release branch and back out two changesets from M-B, see comment #13. Today is branch day, so TB 55 moved to beta. That means, to build another beta, we'd have to create a branch and back out stuff, etc.

So could you please look at the "pretty names" issue from bug 1329355 to see what needs to be adapted in TB.
Flags: needinfo?(tom.prince)
I think backing out bug 1329355 globally is probably the best short-term solution. Looking at those patches, most of it is behind if-defs anyway. 

Looking at that bug, there isn't nearly enough information to determine what needs to be adapted. It looks like that is just removing dead (from the point of view of firefox) code. That code became dead, because firefox releng under took a several month long process to change how they did releases.

I haven't been able to quickly find a description of thunderbird's release process, so I'm not sure where go looking for what would need to change in it to be able to make the corresponding changes in thunderbird's release process.
Flags: needinfo?(tom.prince)

Comment 18

2 years ago
The bad news is that bug 1329355 cannot be backed out from Mozilla Beta at version 55 any more due to merge conflicts.

If I understand it correctly, we need to add back in the "pretty stuff", but the file(s) where to add it back in have changed meanwhile. Also, under these circumstances it's hard to tell whether any manually "nurtured" backout will be correct.

Since this was a dead end to start with, I suggest we abandon this approach and fix the bug for real.

Comment 19

2 years ago
Let's close this bug since all the TB 54 betas are done. We'll open a new bug for failures to build TB 55 beta.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Reporter

Updated

2 years ago
See Also: → 1373284
You need to log in before you can comment on or make changes to this bug.