1317863 - thunderbird version 51 beta repacks fail

Status: VERIFIED FIXED

(Release Engineering :: General, defect)

Description

[Wayne Mery (:wsmwk)]

all repacks failed for all OS. I looked at one win32 and one linux repack

---------------------

https://archive.mozilla.org/pub/thunderbird/candidates/51.0b1-candidates/build1/logs/release-comm-beta-linux_repack_2-bm74-build1-build5.txt.gz

 ['compare-locales', '--merge-dir', '/builds/slave/tb-rel-c-beta-lx_rpk_2-0000000/comm-beta/obj-l10n/mail/locales/merged', '--l10n-ini', '/builds/slave/tb-rel-c-beta-lx_rpk_2-0000000/comm-beta/calendar/locales/l10n.ini', 'ca']

--------------------

https://archive.mozilla.org/pub/thunderbird/candidates/51.0b1-candidates/build1/logs/release-comm-beta-win32_repack_6-bm70-build1-build2.txt.gz

      xbl.properties
          -GTK2Conflict
          +GTK2Conflict2
          -WinConflict
          +WinConflict2
    netErrorApp.dtd
        ERROR: Unparsed content "

<!-- This file exists to allow applications to override one or more messages
     from netError.dtd; Applications which want to do this should override
     this file with their own version of netErrorApp.dtd -->

<!-- An example (from Firefox):" at 211-458
        ERROR: Unparsed content "
-->
" at 1161-1166
        +_junk_1_211-458
        +_junk_2_1161-1166



mozmake.exe[1]: Entering directory 'c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/comm-beta/obj-l10n/calendar/lightning'
rm -f -rf c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_6-000000\\comm-beta\\obj-l10n\\mail\\locales\\merged/calendar
c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/comm-beta/mozilla/mach compare-locales \
    --merge-dir c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_6-000000\\comm-beta\\obj-l10n\\mail\\locales\\merged \
    --l10n-ini c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/comm-beta/calendar/locales/l10n.ini \
    lt
Ambiguous object directory detected. We detected that both c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/comm-beta/obj-l10n and c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/comm-beta/mozilla/obj-l10n could be object directories. This is typically caused by having a mozconfig pointing to a different object directory from the current working directory. To solve this problem, ensure you do not have a default mozconfig in searched paths.
c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/comm-beta/calendar/lightning/lightning-packager.mk:102: recipe for target 'merge-lt' failed
mozmake.exe[1]: Leaving directory 'c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/comm-beta/obj-l10n/calendar/lightning'
Makefile:226: recipe for target 'calendar-merge-lt' failed
command: ERROR
Traceback (most recent call last):
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_6-000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_6-000000\\comm-beta/build/pymake/make.py', 'installers-lt']' returned non-zero exit status 2
command: END (1.42s elapsed)

mozmake.exe[1]: *** [merge-lt] Error 1
pymake\..\..\mozmake.exe: *** [calendar-merge-lt] Error 2
Traceback (most recent call last):
  File "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_6-000000/scripts/scripts/l10n/create-release-repacks.py", line 142, in createRepacks
    mozillaDir=mozillaDir, mozillaSrcDir=mozillaSrcDir)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_6-000000\scripts\lib\python\build\l10n.py", line 188, in repackLocale
    run_cmd(make + ["installers-%s" % locale], cwd=localeSrcDir, env=env)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_6-000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-beta-w32_rpk_6-000000\\comm-beta/build/pymake/make.py', 'installers-is']' returned non-zero exit status 2

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Fallen, could you take a look at this ?

Comment 2

[aleth [:aleth]]

On Linux the error is that comm-beta/mozilla/obj-l10n doesn't exist, same code path.

Possible cause: MOZ_OBJDIR is set to a relative path here, where it is absolute on c-a.

Comment 3

[Wayne Mery (:wsmwk)]

xref bug 1297786?

Comment 4

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Attachment: Fix - v1

Something like this should work. I hope there are no downsides to using an absolute MOZ_OBJDIR?

Comment 5

[Jordan Lund (:jlund)]

Comment on attachment 8816450 [details]
Fix - v1

this is hard to follow. But it looks like you will be changing this value for a lot of places where we call 'make $target' within create-release-repacks.py

I imagine that this script is fickle. e.g. we probably have parts that expect a relative path and it will try to join an abs path to it. In other words, I hope we do a `if abspath(env["MOZ_OBJDIR"]): # leave path alone` in all these places.

Even the error in question:
`OSError: [Errno 2] No such file or directory: '/builds/slave/tb-rel-c-beta-lx_rpk_2-0000000/comm-beta/mozilla/obj-l10n'`

makes me think we could possibly end up with a path like:
`/builds/slave/tb-rel-c-beta-lx_rpk_2-0000000/comm-beta/mozilla//builds/slave/tb-rel-c-beta-lx_rpk_2-0000000/comm-beta/$objdir` :)

Ideally we set this up in staging but I imagine that's difficult. I'm pretty sure only tb is using this code now so the scope is pretty harmless to try this as a hotfix.

iow - give it a whirl!

Comment 6

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

We'll see how it goes and back it out if needed.

https://hg.mozilla.org/build/tools/rev/633092dc002e790c8869e93bb6819d6edd5958c6

Comment 7

[Joe Sabash [:JoeS1]]

From our current build at http://ftp.mozilla.org/pub/thunderbird/candidates/51.0b1-candidates/build2/logs/release-comm-beta-linux_repack_10-bm72-build1-build18.txt.gz
Not sure of the significance of this error, but certainly different from the last errors.

InsecurePlatformWarning
Caught HTTPError: ["Couldn't find locale identified by: Thunderbird-51.0b1-build2, Linux_x86-gcc3, zh-TW"]
The following tracebacks were detected during repacks:
zh-CN:
Traceback (most recent call last):
  File "/builds/slave/tb-rel-c-beta-lx_rpk_10-000000/scripts/scripts/l10n/create-release-repacks.py", line 146, in createRepacks
    checksums = parseChecksumsFile(open(checksums_file).read())
IOError: [Errno 2] No such file or directory: '/builds/slave/tb-rel-c-beta-lx_rpk_10-000000/comm-beta/obj-l10n/mail/locales/dist/linux-i686/zh-CN/thunderbird-51.0b1.checksums'

Comment 8

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Backing out because this also breaks Thunderbird 45.x builds. Hope we can find a more targeted fix for this. See also bug 1323107

https://hg.mozilla.org/build/tools/rev/a30e18094041f4267e70eff3711f202515d41216

Comment 9

[Wayne Mery (:wsmwk)]

Fallen, thanks for the backout.

What's our way forward?
And do we do it here or in bug 1323107?

Comment 10

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #9)
> Fallen, thanks for the backout.
> 
> What's our way forward?
> And do we do it here or in bug 1323107?

Thinking out loud, do we have enough info to accomplish it this week? 
Or does it need a groupthink on IRC or here in the bug?

I'm hoping we can have the intial 51 beta out soon enough so we can do two, before 52 would be done near the end of January.

Comment 11

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Off the top of my head I am not sure how we can fix this easily. One version requires absolute paths, the other requires not to do so because of windows path issues. I guess a reasonable approach would be to fix the windows path issues on 45.x. Releng might not be happy about this, but given there are just 2-3 versions of 45.x left maybe we can just push and back out accordingly.

Comment 12

[Wayne Mery (:wsmwk)]

Thanks Fallen.  We'd certainly do 45.7, 45.8, 45.9 ~end of April.  Assuming 52.0 comes out reasonably on time that gives ~2 months overlap of 45 and 52.

Comment 11 sound reasonable to releng?  (nthomas shows PTO)
Preferred alternatives?

Comment 13

[aleth [:aleth]]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #10)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #9)
> > Fallen, thanks for the backout.
> > 
> > What's our way forward?
> > And do we do it here or in bug 1323107?
> 
> Thinking out loud, do we have enough info to accomplish it this week? 
> Or does it need a groupthink on IRC or here in the bug?
> 
> I'm hoping we can have the intial 51 beta out soon enough so we can do two,
> before 52 would be done near the end of January.

I am travelling and have some laptop trouble, so unfortunately I don't
think I 'll be able to take a look until the 6th.

> Off the top of my head I am not sure how we can fix this easily. One version requires absolute paths, the other requires not to 
> do so because of windows path issues. I guess a reasonable approach would be to fix the windows path issues on 45.x. Releng 
> might not be happy about this, but given there are just 2-3 versions of 45.x left maybe we can just push and back out accordingly.

Maybe I'm misunderstanding, but do we know the Windows path issue won't affect 51 as well?

Comment 14

[Rail Aliiev [:rail]]

I'll let jlund (current releaseduty) decide this.

Comment 15

[Jordan Lund (:jlund)]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #12)
> Thanks Fallen.  We'd certainly do 45.7, 45.8, 45.9 ~end of April.  Assuming
> 52.0 comes out reasonably on time that gives ~2 months overlap of 45 and 52.
> 
> Comment 11 sound reasonable to releng?  (nthomas shows PTO)
> Preferred alternatives?

I have two assumptions on this:

1) you are talking about landing a fix for one release channel than backing it out so it doesn't break another release channel and this 'juggling' will persist for ~2 months?

2) scanning the code, this patch is only used by thunderbird. Given the isolation, I have no problem with this if someone from tb manages and coordinates the landing+backout. One thing that would help releaseduty would be to keep this bug open and update it with the current state throughout the releases in case there is a failure and we end up investigating.

Comment 16

[Wayne Mery (:wsmwk)]

(In reply to Jordan Lund (:jlund) from comment #15)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #12)
> > Thanks Fallen.  We'd certainly do 45.7, 45.8, 45.9 ~end of April.  Assuming
> > 52.0 comes out reasonably on time that gives ~2 months overlap of 45 and 52.
> > 
> > Comment 11 sound reasonable to releng?  (nthomas shows PTO)
> > Preferred alternatives?
> 
> I have two assumptions on this:
> 
> 1) you are talking about landing a fix for one release channel than backing
> it out so it doesn't break another release channel and this 'juggling' will
> persist for ~2 months?

yes.

> 2) scanning the code, this patch is only used by thunderbird. Given the
> isolation, I have no problem with this if someone from tb manages and
> coordinates the landing+backout. One thing that would help releaseduty would
> be to keep this bug open and update it with the current state throughout the
> releases in case there is a failure and we end up investigating.

Thanks for the suggestion.  Perhaps we can use whiteboard, like
[MOZ_OBJDIR-state: relative path for TB45]
[MOZ_OBJDIR-state: absolute path for TBbetas]

Fallen, what do you think?  
We can plan it out and minimize the patch churn to some extent, because building TB45 has been reliable. The X factor is beta build failures which might cause us to deviate from the sequence I envision as something like:
TB51b1, TB51b2, 45.7, TB52b1, TB52b2, TB52b3, 45.8, TB53b1, TB53b2, 45.9

Comment 17

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Seems sensible to me. We should also post the changesets as usual of course.

Comment 18

[Wayne Mery (:wsmwk)]

(In reply to Philipp Kewisch [:Fallen] from comment #17)
> Seems sensible to me. We should also post the changesets as usual of course.

excellent. Can you push the patch? Then we can get beta going again.
And is there anyone else on TB team who can push and backout for us?

Comment 19

[Wayne Mery (:wsmwk)]

Does the following expanded sequence look correct, ie. once 45.9 is done we no longer need relative path? 
(assuming 45.9 is the last 45.x ESR)  
Note, another X factor may be unscheduled security releases and chemspills, which will hopefully be none.

<push patch for absolute path>
TB51.0b1, TB51.0b2, <backout patch = relative path>, 45.7.0, <push patch for absolute path>
TB52.0b1, TB52.0b2, TB52.0b3, <backout patch = relative path>, 45.8.0, <push patch for absolute path>, 52.0, (52.0.1?)
TB53.0b1, TB53.0b2, <backout patch = relative path>, 45.9.0, <push patch for absolute path>, 52.1.0
TB54.0b1, ...

Comment 20

[Richard Marti (:Paenglab)]

https://hg.mozilla.org/build/tools/rev/569c5fbf6581935ec16a7671a99c337d7b90e958

Comment 21

[Wayne Mery (:wsmwk)]

(In reply to Richard Marti (:Paenglab) from comment #20)
> https://hg.mozilla.org/build/tools/rev/
> 569c5fbf6581935ec16a7671a99c337d7b90e958

Thanks Richard.

Comment 22

[Wayne Mery (:wsmwk)]

All win32 repacks failing again
https://archive.mozilla.org/pub/thunderbird/candidates/51.0b1-candidates/build5/logs/release-comm-beta-win32_repack_8-bm73-build1-build3.txt.gz

Comment 23

[Wayne Mery (:wsmwk)]

summarizing where we are and how we got here ...

The absolute path patch didn't help any of the builds.  The chronology of 51.0beta from https://public.etherpad-mozilla.org/p/thunderbird-release-51.0b1

build5 2017-01-06
* win32 repacks fail, with bug 1317863 absolute path patch (after having backed it out for 45.6.0)
build4 2016-12-12 - with updated l10n
* win32 repacks fail, with bug 1317863 absolute path patch
build3 
* abandoned 
build2 - 2016-12-03  (with absolute path patch from Bug 1317863)
* ALL repacks fail b/c reading checksums file - Bug 1321994
build1 - 2016-11-15
* ALL repacks fail - eventually patch developed for absolute path in Bug 1317863

Comment 24

[aleth [:aleth]]

Attachment: Use an absolute path in MOZ_OBJDIR only for 'make installers' in repacks

Looking at the original failure in this bug, it's only the make installers step (and in particular mach) which seems to require the absolute objdir, so we can make the fix more specific and hopefully avoid the Windows problems that way.

Comment 25

[aleth [:aleth]]

(In reply to aleth [:aleth] from comment #24)
> Created attachment 8825135 [details] [diff] [review]
> Use an absolute path in MOZ_OBJDIR only for 'make installers' in repacks

If this gets r+, I propose we back out the currently landed patch first and rebase, so that the final patch only touches l10n.py, to avoid confusion in the future.

Comment 26

[Jordan Lund (:jlund)]

Comment on attachment 8825135 [details] [diff] [review]
Use an absolute path in MOZ_OBJDIR only for 'make installers' in repacks

Review of attachment 8825135 [details] [diff] [review]:
-----------------------------------------------------------------

no problem targeting the specific target that's failing.

In the long term, it would be great to set up a staging bbot release env for this so you have faster turn around and not forced to dev live in prod.

Short term, I believe these files are only used by TB now as ff has moved to mozharness for l10n. You should have the power to make decisions here and not have folks like me block you by being slow to review. Feel free to 'own' this code and get review without releng. side-note: maybe as a joint exercise, it would be useful to come up with a list of source files and code in bbot/tools that are only used by TB.

::: lib/python/build/l10n.py
@@ +185,5 @@
>  
>      compareLocales(compareLocalesRepo, locale, l10nRepoDir, localeSrcDir,
>                     l10nIni, revision=revision, merge=merge)
> +
> +    make_installers_env = env.copy()

in case this env is nested, it would be safer to deepcopy here

Comment 27

[aleth [:aleth]]

https://hg.mozilla.org/build/tools/rev/879c0fb645b7b260fb94327d0fc7a8a980d17bdb
Backed out changeset 569c5fbf6581 (bug 1317863).

https://hg.mozilla.org/build/tools/rev/8ec00c563fd0ae6e0e5597e98b10296b12b579b2
Bug 1317863 - Use an absolute path in MOZ_OBJDIR only for 'make installers' in repacks. r=jlund

Comment 28

[aleth [:aleth]]

(In reply to Jordan Lund (:jlund) from comment #26)
> In the long term, it would be great to set up a staging bbot release env for
> this so you have faster turn around and not forced to dev live in prod.
> 
> Short term, I believe these files are only used by TB now as ff has moved to
> mozharness for l10n. You should have the power to make decisions here and
> not have folks like me block you by being slow to review. Feel free to 'own'
> this code and get review without releng. side-note: maybe as a joint
> exercise, it would be useful to come up with a list of source files and code
> in bbot/tools that are only used by TB.

Assembling such a list would be a useful taking of stock and speed up reviews. The problem with fully owning the code, however, is that I'm not sure any of the c-c contributors knows it well enough to be able to confidently review it.

> ::: lib/python/build/l10n.py
> > +    make_installers_env = env.copy()
> 
> in case this env is nested, it would be safer to deepcopy here

Fixed on checkin.

Comment 29

[Wayne Mery (:wsmwk)]

all repacks are failing build6

br:
Traceback (most recent call last):
  File "/builds/slave/tb-rel-c-beta-l64_rpk_1-000000/scripts/scripts/l10n/create-release-repacks.py", line 142, in createRepacks
    mozillaDir=mozillaDir, mozillaSrcDir=mozillaSrcDir)
  File "/builds/slave/tb-rel-c-beta-l64_rpk_1-000000/scripts/lib/python/build/l10n.py", line 189, in repackLocale
    make_installers_env = env.deepcopy()
AttributeError: 'dict' object has no attribute 'deepcopy'

Comment 30

[aleth [:aleth]]

https://hg.mozilla.org/build/tools/rev/c1ced0cfc40dc54b11d8927415a20dfb60dfc95d
Bug 1317863 - Followup to not use deepcopy() as it's not available on dicts. rs=bustage-fix

Comment 31

[Wayne Mery (:wsmwk)]

Thanks everyone!  
Two beta builds and no repack problems!