thunderbird version 51 beta repacks fail

VERIFIED FIXED

Status

defect
VERIFIED FIXED
3 years ago
Last year

People

(Reporter: wsmwk, Assigned: aleth)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

Reporter

Description

3 years ago
all repacks failed for all OS. I looked at one win32 and one linux repack

---------------------

https://archive.mozilla.org/pub/thunderbird/candidates/51.0b1-candidates/build1/logs/release-comm-beta-linux_repack_2-bm74-build1-build5.txt.gz

 ['compare-locales', '--merge-dir', '/builds/slave/tb-rel-c-beta-lx_rpk_2-0000000/comm-beta/obj-l10n/mail/locales/merged', '--l10n-ini', '/builds/slave/tb-rel-c-beta-lx_rpk_2-0000000/comm-beta/calendar/locales/l10n.ini', 'ca']

--------------------

https://archive.mozilla.org/pub/thunderbird/candidates/51.0b1-candidates/build1/logs/release-comm-beta-win32_repack_6-bm70-build1-build2.txt.gz

      xbl.properties
          -GTK2Conflict
          +GTK2Conflict2
          -WinConflict
          +WinConflict2
    netErrorApp.dtd
        ERROR: Unparsed content "

<!-- This file exists to allow applications to override one or more messages
     from netError.dtd; Applications which want to do this should override
     this file with their own version of netErrorApp.dtd -->

<!-- An example (from Firefox):" at 211-458
        ERROR: Unparsed content "
-->
" at 1161-1166
        +_junk_1_211-458
        +_junk_2_1161-1166



mozmake.exe[1]: Entering directory 'c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/comm-beta/obj-l10n/calendar/lightning'
rm -f -rf c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_6-000000\\comm-beta\\obj-l10n\\mail\\locales\\merged/calendar
c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/comm-beta/mozilla/mach compare-locales \
    --merge-dir c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_6-000000\\comm-beta\\obj-l10n\\mail\\locales\\merged \
    --l10n-ini c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/comm-beta/calendar/locales/l10n.ini \
    lt
Ambiguous object directory detected. We detected that both c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/comm-beta/obj-l10n and c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/comm-beta/mozilla/obj-l10n could be object directories. This is typically caused by having a mozconfig pointing to a different object directory from the current working directory. To solve this problem, ensure you do not have a default mozconfig in searched paths.
c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/comm-beta/calendar/lightning/lightning-packager.mk:102: recipe for target 'merge-lt' failed
mozmake.exe[1]: Leaving directory 'c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/comm-beta/obj-l10n/calendar/lightning'
Makefile:226: recipe for target 'calendar-merge-lt' failed
command: ERROR
Traceback (most recent call last):
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_6-000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_6-000000\\comm-beta/build/pymake/make.py', 'installers-lt']' returned non-zero exit status 2
command: END (1.42s elapsed)

mozmake.exe[1]: *** [merge-lt] Error 1
pymake\..\..\mozmake.exe: *** [calendar-merge-lt] Error 2
Traceback (most recent call last):
  File "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/scripts/scripts/l10n/create-release-repacks.py", line 142, in createRepacks
    mozillaDir=mozillaDir, mozillaSrcDir=mozillaSrcDir)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_6-000000\scripts\lib\python\build\l10n.py", line 188, in repackLocale
    run_cmd(make + ["installers-%s" % locale], cwd=localeSrcDir, env=env)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_6-000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_6-000000\\comm-beta/build/pymake/make.py', 'installers-is']' returned non-zero exit status 2
Fallen, could you take a look at this ?
Flags: needinfo?(philipp)
Assignee

Comment 2

3 years ago
On Linux the error is that comm-beta/mozilla/obj-l10n doesn't exist, same code path.

Possible cause: MOZ_OBJDIR is set to a relative path here, where it is absolute on c-a.
Assignee

Updated

3 years ago
Component: Build Config → General Automation
Product: Thunderbird → Release Engineering
QA Contact: catlee
Version: 51 Branch → unspecified
Reporter

Comment 3

3 years ago
xref bug 1297786?
Posted file Fix - v1 (obsolete) β€”
Something like this should work. I hope there are no downsides to using an absolute MOZ_OBJDIR?
Assignee: nobody → philipp
Status: NEW → ASSIGNED
Flags: needinfo?(philipp)
Attachment #8816450 - Flags: review?(rail)
Comment on attachment 8816450 [details]
Fix - v1

this is hard to follow. But it looks like you will be changing this value for a lot of places where we call 'make $target' within create-release-repacks.py

I imagine that this script is fickle. e.g. we probably have parts that expect a relative path and it will try to join an abs path to it. In other words, I hope we do a `if abspath(env["MOZ_OBJDIR"]): # leave path alone` in all these places.

Even the error in question:
`OSError: [Errno 2] No such file or directory: '/builds/slave/tb-rel-c-beta-lx_rpk_2-0000000/comm-beta/mozilla/obj-l10n'`

makes me think we could possibly end up with a path like:
`/builds/slave/tb-rel-c-beta-lx_rpk_2-0000000/comm-beta/mozilla//builds/slave/tb-rel-c-beta-lx_rpk_2-0000000/comm-beta/$objdir` :)

Ideally we set this up in staging but I imagine that's difficult. I'm pretty sure only tb is using this code now so the scope is pretty harmless to try this as a hotfix.

iow - give it a whirl!
Attachment #8816450 - Flags: review?(rail) → review+
We'll see how it goes and back it out if needed.

https://hg.mozilla.org/build/tools/rev/633092dc002e790c8869e93bb6819d6edd5958c6
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
From our current build at http://ftp.mozilla.org/pub/thunderbird/candidates/51.0b1-candidates/build2/logs/release-comm-beta-linux_repack_10-bm72-build1-build18.txt.gz
Not sure of the significance of this error, but certainly different from the last errors.

InsecurePlatformWarning
Caught HTTPError: ["Couldn't find locale identified by: Thunderbird-51.0b1-build2, Linux_x86-gcc3, zh-TW"]
The following tracebacks were detected during repacks:
zh-CN:
Traceback (most recent call last):
  File "/builds/slave/tb-rel-c-beta-lx_rpk_10-000000/scripts/scripts/l10n/create-release-repacks.py", line 146, in createRepacks
    checksums = parseChecksumsFile(open(checksums_file).read())
IOError: [Errno 2] No such file or directory: '/builds/slave/tb-rel-c-beta-lx_rpk_10-000000/comm-beta/obj-l10n/mail/locales/dist/linux-i686/zh-CN/thunderbird-51.0b1.checksums'
Backing out because this also breaks Thunderbird 45.x builds. Hope we can find a more targeted fix for this. See also bug 1323107

https://hg.mozilla.org/build/tools/rev/a30e18094041f4267e70eff3711f202515d41216
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Reporter

Comment 9

3 years ago
Fallen, thanks for the backout.

What's our way forward?
And do we do it here or in bug 1323107?
See Also: → 1323107
Reporter

Comment 10

3 years ago
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #9)
> Fallen, thanks for the backout.
> 
> What's our way forward?
> And do we do it here or in bug 1323107?

Thinking out loud, do we have enough info to accomplish it this week? 
Or does it need a groupthink on IRC or here in the bug?

I'm hoping we can have the intial 51 beta out soon enough so we can do two, before 52 would be done near the end of January.
Flags: needinfo?(philipp)
Off the top of my head I am not sure how we can fix this easily. One version requires absolute paths, the other requires not to do so because of windows path issues. I guess a reasonable approach would be to fix the windows path issues on 45.x. Releng might not be happy about this, but given there are just 2-3 versions of 45.x left maybe we can just push and back out accordingly.
Flags: needinfo?(philipp)
Reporter

Comment 12

3 years ago
Thanks Fallen.  We'd certainly do 45.7, 45.8, 45.9 ~end of April.  Assuming 52.0 comes out reasonably on time that gives ~2 months overlap of 45 and 52.

Comment 11 sound reasonable to releng?  (nthomas shows PTO)
Preferred alternatives?
Flags: needinfo?(rail)
Flags: needinfo?(jlund)
Assignee

Comment 13

3 years ago
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #10)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #9)
> > Fallen, thanks for the backout.
> > 
> > What's our way forward?
> > And do we do it here or in bug 1323107?
> 
> Thinking out loud, do we have enough info to accomplish it this week? 
> Or does it need a groupthink on IRC or here in the bug?
> 
> I'm hoping we can have the intial 51 beta out soon enough so we can do two,
> before 52 would be done near the end of January.

I am travelling and have some laptop trouble, so unfortunately I don't
think I 'll be able to take a look until the 6th.

> Off the top of my head I am not sure how we can fix this easily. One version requires absolute paths, the other requires not to 
> do so because of windows path issues. I guess a reasonable approach would be to fix the windows path issues on 45.x. Releng 
> might not be happy about this, but given there are just 2-3 versions of 45.x left maybe we can just push and back out accordingly.

Maybe I'm misunderstanding, but do we know the Windows path issue won't affect 51 as well?
I'll let jlund (current releaseduty) decide this.
Flags: needinfo?(rail)
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #12)
> Thanks Fallen.  We'd certainly do 45.7, 45.8, 45.9 ~end of April.  Assuming
> 52.0 comes out reasonably on time that gives ~2 months overlap of 45 and 52.
> 
> Comment 11 sound reasonable to releng?  (nthomas shows PTO)
> Preferred alternatives?

I have two assumptions on this:

1) you are talking about landing a fix for one release channel than backing it out so it doesn't break another release channel and this 'juggling' will persist for ~2 months?

2) scanning the code, this patch is only used by thunderbird. Given the isolation, I have no problem with this if someone from tb manages and coordinates the landing+backout. One thing that would help releaseduty would be to keep this bug open and update it with the current state throughout the releases in case there is a failure and we end up investigating.
Flags: needinfo?(jlund)
Reporter

Comment 16

3 years ago
(In reply to Jordan Lund (:jlund) from comment #15)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #12)
> > Thanks Fallen.  We'd certainly do 45.7, 45.8, 45.9 ~end of April.  Assuming
> > 52.0 comes out reasonably on time that gives ~2 months overlap of 45 and 52.
> > 
> > Comment 11 sound reasonable to releng?  (nthomas shows PTO)
> > Preferred alternatives?
> 
> I have two assumptions on this:
> 
> 1) you are talking about landing a fix for one release channel than backing
> it out so it doesn't break another release channel and this 'juggling' will
> persist for ~2 months?

yes.

> 2) scanning the code, this patch is only used by thunderbird. Given the
> isolation, I have no problem with this if someone from tb manages and
> coordinates the landing+backout. One thing that would help releaseduty would
> be to keep this bug open and update it with the current state throughout the
> releases in case there is a failure and we end up investigating.

Thanks for the suggestion.  Perhaps we can use whiteboard, like
[MOZ_OBJDIR-state: relative path for TB45]
[MOZ_OBJDIR-state: absolute path for TBbetas]

Fallen, what do you think?  
We can plan it out and minimize the patch churn to some extent, because building TB45 has been reliable. The X factor is beta build failures which might cause us to deviate from the sequence I envision as something like:
TB51b1, TB51b2, 45.7, TB52b1, TB52b2, TB52b3, 45.8, TB53b1, TB53b2, 45.9
Flags: needinfo?(philipp)
Whiteboard: [MOZ_OBJDIR-state: relative path for TB45]
Seems sensible to me. We should also post the changesets as usual of course.
Flags: needinfo?(philipp)
Reporter

Comment 18

3 years ago
(In reply to Philipp Kewisch [:Fallen] from comment #17)
> Seems sensible to me. We should also post the changesets as usual of course.

excellent. Can you push the patch? Then we can get beta going again.
And is there anyone else on TB team who can push and backout for us?
Flags: needinfo?(philipp)
Reporter

Comment 19

3 years ago
Does the following expanded sequence look correct, ie. once 45.9 is done we no longer need relative path? 
(assuming 45.9 is the last 45.x ESR)  
Note, another X factor may be unscheduled security releases and chemspills, which will hopefully be none.

<push patch for absolute path>
TB51.0b1, TB51.0b2, <backout patch = relative path>, 45.7.0, <push patch for absolute path>
TB52.0b1, TB52.0b2, TB52.0b3, <backout patch = relative path>, 45.8.0, <push patch for absolute path>, 52.0, (52.0.1?)
TB53.0b1, TB53.0b2, <backout patch = relative path>, 45.9.0, <push patch for absolute path>, 52.1.0
TB54.0b1, ...
Reporter

Comment 21

3 years ago
(In reply to Richard Marti (:Paenglab) from comment #20)
> https://hg.mozilla.org/build/tools/rev/
> 569c5fbf6581935ec16a7671a99c337d7b90e958

Thanks Richard.
Keywords: leave-open
Whiteboard: [MOZ_OBJDIR-state: relative path for TB45] → [MOZ_OBJDIR-state: absolute path for TBbetas]
Reporter

Comment 23

3 years ago
summarizing where we are and how we got here ...

The absolute path patch didn't help any of the builds.  The chronology of 51.0beta from https://public.etherpad-mozilla.org/p/thunderbird-release-51.0b1

build5 2017-01-06
* win32 repacks fail, with bug 1317863 absolute path patch (after having backed it out for 45.6.0)
build4 2016-12-12 - with updated l10n
* win32 repacks fail, with bug 1317863 absolute path patch
build3 
* abandoned 
build2 - 2016-12-03  (with absolute path patch from Bug 1317863)
* ALL repacks fail b/c reading checksums file - Bug 1321994
build1 - 2016-11-15
* ALL repacks fail - eventually patch developed for absolute path in Bug 1317863
Assignee

Comment 24

3 years ago
Looking at the original failure in this bug, it's only the make installers step (and in particular mach) which seems to require the absolute objdir, so we can make the fix more specific and hopefully avoid the Windows problems that way.
Attachment #8825135 - Flags: review?(jlund)
Assignee

Updated

3 years ago
Assignee: philipp → aleth
Status: REOPENED → ASSIGNED
Assignee

Comment 25

3 years ago
(In reply to aleth [:aleth] from comment #24)
> Created attachment 8825135 [details] [diff] [review]
> Use an absolute path in MOZ_OBJDIR only for 'make installers' in repacks

If this gets r+, I propose we back out the currently landed patch first and rebase, so that the final patch only touches l10n.py, to avoid confusion in the future.
Comment on attachment 8825135 [details] [diff] [review]
Use an absolute path in MOZ_OBJDIR only for 'make installers' in repacks

Review of attachment 8825135 [details] [diff] [review]:
-----------------------------------------------------------------

no problem targeting the specific target that's failing.

In the long term, it would be great to set up a staging bbot release env for this so you have faster turn around and not forced to dev live in prod.

Short term, I believe these files are only used by TB now as ff has moved to mozharness for l10n. You should have the power to make decisions here and not have folks like me block you by being slow to review. Feel free to 'own' this code and get review without releng. side-note: maybe as a joint exercise, it would be useful to come up with a list of source files and code in bbot/tools that are only used by TB.

::: lib/python/build/l10n.py
@@ +185,5 @@
>  
>      compareLocales(compareLocalesRepo, locale, l10nRepoDir, localeSrcDir,
>                     l10nIni, revision=revision, merge=merge)
> +
> +    make_installers_env = env.copy()

in case this env is nested, it would be safer to deepcopy here
Attachment #8825135 - Flags: review?(jlund) → review+
Assignee

Comment 27

3 years ago
https://hg.mozilla.org/build/tools/rev/879c0fb645b7b260fb94327d0fc7a8a980d17bdb
Backed out changeset 569c5fbf6581 (bug 1317863).

https://hg.mozilla.org/build/tools/rev/8ec00c563fd0ae6e0e5597e98b10296b12b579b2
Bug 1317863 - Use an absolute path in MOZ_OBJDIR only for 'make installers' in repacks. r=jlund
Assignee

Comment 28

3 years ago
(In reply to Jordan Lund (:jlund) from comment #26)
> In the long term, it would be great to set up a staging bbot release env for
> this so you have faster turn around and not forced to dev live in prod.
> 
> Short term, I believe these files are only used by TB now as ff has moved to
> mozharness for l10n. You should have the power to make decisions here and
> not have folks like me block you by being slow to review. Feel free to 'own'
> this code and get review without releng. side-note: maybe as a joint
> exercise, it would be useful to come up with a list of source files and code
> in bbot/tools that are only used by TB.

Assembling such a list would be a useful taking of stock and speed up reviews. The problem with fully owning the code, however, is that I'm not sure any of the c-c contributors knows it well enough to be able to confidently review it.

> ::: lib/python/build/l10n.py
> > +    make_installers_env = env.copy()
> 
> in case this env is nested, it would be safer to deepcopy here

Fixed on checkin.
Flags: needinfo?(philipp)
Assignee

Updated

3 years ago
Whiteboard: [MOZ_OBJDIR-state: absolute path for TBbetas]
Assignee

Updated

3 years ago
Keywords: leave-open
Assignee

Updated

3 years ago
Attachment #8816450 - Attachment is obsolete: true
Reporter

Comment 29

3 years ago
all repacks are failing build6

br:
Traceback (most recent call last):
  File "/builds/slave/tb-rel-c-beta-l64_rpk_1-000000/scripts/scripts/l10n/create-release-repacks.py", line 142, in createRepacks
    mozillaDir=mozillaDir, mozillaSrcDir=mozillaSrcDir)
  File "/builds/slave/tb-rel-c-beta-l64_rpk_1-000000/scripts/lib/python/build/l10n.py", line 189, in repackLocale
    make_installers_env = env.deepcopy()
AttributeError: 'dict' object has no attribute 'deepcopy'
Assignee

Comment 30

3 years ago
https://hg.mozilla.org/build/tools/rev/c1ced0cfc40dc54b11d8927415a20dfb60dfc95d
Bug 1317863 - Followup to not use deepcopy() as it's not available on dicts. rs=bustage-fix
Assignee

Updated

3 years ago
Status: ASSIGNED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
Reporter

Comment 31

3 years ago
Thanks everyone!  
Two beta builds and no repack problems!
Status: RESOLVED → VERIFIED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.