1360700 - Thunderbird 54.0b1 build1: build step failed on win32 (and all OS) caused by removing MOZ_PKG_PRETTYNAMES

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect, P2)

Description

[Wayne Mery (:wsmwk)]

(I don't know whether this is automation fail or Thunderbird build fail)

from https://archive.mozilla.org/pub/thunderbird/candidates/54.0b1-candidates/build1/logs/release-comm-beta-win32_build-bm91-build1-build4.txt.gz

https://pastebin.mozilla.org/9020242

========= Started find filepath failed (results: 2, elapsed: 0 secs) (at 2017-04-28 11:40:39.456958) =========
'bash' '-c' 'find build/objdir-tb/dist/update/win32/en-US -maxdepth 1 -type f -name thunderbird-54.0b1.complete.mar'
 in dir c:\builds\moz2_slave\tb-rel-c-beta-w32_bld-00000000\. (timeout 1200 secs)
 watching logfiles {}
 argv: ['bash', '-c', 'find build/objdir-tb/dist/update/win32/en-US -maxdepth 1 -type f -name thunderbird-54.0b1.complete.mar']
 environment:
  ALLUSERSPROFILE=C:\ProgramData
  APPDATA=C:\Users\cltbld\AppData\Roaming
...
  WINDIR=C:\Windows
  WINDOWS_TRACING_FLAGS=3
  WINDOWS_TRACING_LOGFILE=C:\BVTBin\Tests\installpackage\csilogfile.log
  WIX_351728_PATH=c:/mozilla-build/wix-351728
  _=C:\mozilla-build\buildbotve\Scripts\python
 using PTY: False
find: build/objdir-tb/dist/update/win32/en-US: No such file or directory

Comment 1

[Wayne Mery (:wsmwk)]

https://archive.mozilla.org/pub/thunderbird/candidates/54.0b1-candidates/build1/logs/release-comm-beta-macosx64_build-bm82-build1-build16.txt.gz

 find: build/objdir-tb/dist/update/mac/en-US: No such file or directory


https://archive.mozilla.org/pub/thunderbird/candidates/54.0b1-candidates/build1/logs/release-comm-beta-linux_build-bm91-build1-build9.txt.gz

 find: `build/objdir-tb/dist/update/linux-i686/en-US': No such file or directory
program finished with exit code 1

Comment 2

[Richard Marti (:Paenglab)]

When I check the log, I see the signtool.py output is in ../../dist/update/ and not in ../../dist/update/linux-i686/en-US. With this it's clear the following find fails. Is signtool.py wrong or should the path be adapted?

python /builds/slave/tb-rel-c-beta-lx_bld-000000000/tools/release/signing/signtool.py --cachedir /builds/slave/tb-rel-c-beta-lx_bld-000000000/signing_cache -t /builds/slave/tb-rel-c-beta-lx_bld-000000000/token -n /builds/slave/tb-rel-c-beta-lx_bld-000000000/nonce -c /builds/slave/tb-rel-c-beta-lx_bld-000000000/tools/release/signing/host.cert -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120 -H macapp:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120 -f mar '../../dist/update/thunderbird-54.0b1.en-US.linux-i686.complete.mar'

As reference the 53beta:

python /builds/slave/tb-rel-c-beta-lx_bld-000000000/tools/release/signing/signtool.py --cachedir /builds/slave/tb-rel-c-beta-lx_bld-000000000/signing_cache -t /builds/slave/tb-rel-c-beta-lx_bld-000000000/token -n /builds/slave/tb-rel-c-beta-lx_bld-000000000/nonce -c /builds/slave/tb-rel-c-beta-lx_bld-000000000/tools/release/signing/host.cert -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:sha2signcodestub:osslsigncode:signcode:mar:mar_sha384:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120 -H macapp:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120 -f mar '../../dist/update/linux-i686/en-US/thunderbird-53.0b2.complete.mar'

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

# quick investigation on the 54.0b1 build machine
[cltbld@bld-lion-r5-071]$ pwd
/builds/slave/tb-rel-c-beta-m64_bld-00000000

# a somewhat broader find than the log to locate the complete mar
[cltbld@bld-lion-r5-071]$ find . -type f -name thunderbird-54.0b1.complete.mar

# huh, do we have any complete.mar file at all ?
[cltbld@bld-lion-r5-071]$ find . -type f -name '*.complete.mar'
./build/objdir-tb/dist/update/thunderbird-54.0b1.en-US.mac.complete.mar

For comparision 53.0b2 found 
  build/objdir-tb/dist/update/mac/en-US/thunderbird-53.0b2.complete.mar

So looks like a pretty naming issue, and is fallout from bug 1329355 removing MOZ_PKG_PRETTYNAMES.

Comment 4

[Wayne Mery (:wsmwk)]

A yes, bug 1329355 comment 11, bug 1329355 comment 18. I guess we never opened a bug for it :(

Comment 5

[Wayne Mery (:wsmwk)]

Richard, are you able to provide a fix?

This blocks our progress to reenabling updates from TB45, so => major

Comment 6

[Richard Marti (:Paenglab)]

I looked into the changes from bug 1329355 and can't say what changes are needed on our side, sorry.

Comment 7

[Edmund Wong (:ewong)]

(In reply to Richard Marti (:Paenglab) from comment #6)
> I looked into the changes from bug 1329355 and can't say what changes are
> needed on our side, sorry.

Oh.  This has finally hit beta..  :/  

I'll have a look at it.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

It seems to me you have two realistic options here
* petition for the global backout of bug 1329355 (just https://reviewboard.mozilla.org/r/104620/diff/2#index_header may be sufficient)
* set up a verbranch again and apply the backout there, either temporarily to buy some time, or permanently (ie always have a verbranch going forward, with all that entails with merging etc)

Otherwise you're left with adjusting the release automation to the non-pretty naming, or implement a renaming somehow. As this is likely require quite a lot of work, and RelEng won't be able to support that with development time, I'd strongly recommend either of the two bullet points.

Comment 9

[Wayne Mery (:wsmwk)]

Thanks Nick.

Fallen and ewong, what is the preference?

Comment 10

[Jorg K (CEST = GMT+2)]

I can back stuff out on the beta release branch for TB if necessary.

Comment 11

[Edmund Wong (:ewong)]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #9)
> Thanks Nick.
> 
> Fallen and ewong, what is the preference?

I am doubting they're going to agree with the global backout and this
bug is a tad bit more involved than I had hoped (currently having
less time to devote to this as I had thought).

In the meantime, I think the best way forward is to backout on a relbranch and
build beta off that.

Comment 12

[Philipp Kewisch [:Fallen][📱📆]]

I would also opt for fixing it in our release automation. It may be more work, but just a little bit of renaming doesn't sound very complicated to me?

If you want to get 54.0b1 to build I'd also suggest a temporary relbranch. I wouldn't use that as a permanent solution.

Comment 13

[Jorg K (CEST = GMT+2)]

Backed out
https://hg.mozilla.org/mozilla-central/rev/68d491c00355
https://hg.mozilla.org/mozilla-central/rev/318f1bcd336e
from bug 1329355 on THUNDERBIRD540b1_2017050501_RELBRANCH:

https://hg.mozilla.org/releases/mozilla-beta/rev/80691907902f284e3057de73902ffb4da92278c4
https://hg.mozilla.org/releases/mozilla-beta/rev/c250f0a735411e1bc514f842d0ee87a98aabf74a

As Philipp said, we need to find a permanent solution at least when TB 55 beta comes around.

Comment 14

[Wayne Mery (:wsmwk)]

Agh!  I totally missed Jorg's backout.  So we are good to build?  Why then are we doing 52.1.1?

Comment 15

[Jorg K (CEST = GMT+2)]

Umm, someone is confused here. We thought we were good to build, and you *did* trigger a build (build 2) on 2017-05-05, the day of the backout. Sadly that didn't go very far an ran into bug 1362611. We are doing 52.1.1 since TB 54 beta did indeed fail on the first attempt due to this bug here and then on the second attempt due to bug 1362611.

Comment 16

[Jorg K (CEST = GMT+2)]

Tom, as you might have noticed by now, we have a terrible back-log of build & release issues. Here is today's issue of the day:

To build TB 54 beta I had to create a release branch and back out two changesets from M-B, see comment #13. Today is branch day, so TB 55 moved to beta. That means, to build another beta, we'd have to create a branch and back out stuff, etc.

So could you please look at the "pretty names" issue from bug 1329355 to see what needs to be adapted in TB.

Comment 17

[Tom Prince [:tomprince]]

I think backing out bug 1329355 globally is probably the best short-term solution. Looking at those patches, most of it is behind if-defs anyway. 

Looking at that bug, there isn't nearly enough information to determine what needs to be adapted. It looks like that is just removing dead (from the point of view of firefox) code. That code became dead, because firefox releng under took a several month long process to change how they did releases.

I haven't been able to quickly find a description of thunderbird's release process, so I'm not sure where go looking for what would need to change in it to be able to make the corresponding changes in thunderbird's release process.

Comment 18

[Jorg K (CEST = GMT+2)]

The bad news is that bug 1329355 cannot be backed out from Mozilla Beta at version 55 any more due to merge conflicts.

If I understand it correctly, we need to add back in the "pretty stuff", but the file(s) where to add it back in have changed meanwhile. Also, under these circumstances it's hard to tell whether any manually "nurtured" backout will be correct.

Since this was a dead end to start with, I suggest we abandon this approach and fix the bug for real.

Comment 19

[Jorg K (CEST = GMT+2)]

Let's close this bug since all the TB 54 betas are done. We'll open a new bug for failures to build TB 55 beta.