Missing runtime dll when generating partial updates for Thunderbird on Windows

RESOLVED FIXED

Status

Release Engineering
Release Automation
RESOLVED FIXED
a year ago
a year ago

People

(Reporter: nthomas, Assigned: nthomas)

Tracking

unspecified

Firefox Tracking Flags

(firefox49 fixed, firefox50 fixed, firefox51 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(3 attachments)

(Assignee)

Description

a year ago
eg https://archive.mozilla.org/pub/thunderbird/candidates/49.0b1-candidates/build2/logs/release-comm-beta-win32_build-bm71-build1-build3.txt.gz

Adding file patch and add instructions to update manifests
diffing "xul.dll"

command timed out: 1200 seconds without output running ['bash', 'c:/builds/moz2_slave/tb-rel-c-beta-w32_bld-00000000/build/mozilla/tools/update-packaging/make_incremental_update.sh', 'update/win32/en-US/thunderbird-45.2b1-49.0b1.partial.mar', '../previous', '../current'], attempting to kill
SIGKILL failed to kill process
using fake rc=-1
program finished with exit code -1

A partial spanning 4 release cycles might be part of it, and windows is slower in general. We can try bumping up the timeout as a workaround.
Ship-it was specified with only 3 partials : 44.0b1build1,47.0b2build1,45.2b1build2
I can skip the older partials if needed - I estimate only 1-2k users on each of the two older versions
Note - we skipped betas 46 and 48.
(Assignee)

Comment 2

a year ago
Dropping the older partials would be simpler than playing whack-a-mole fixing first en-US and then repacks, but won't you hit the same problem when you do a release build and the most recent version is 45.2.0 ?
(In reply to Nick Thomas [:nthomas] from comment #2)
> Dropping the older partials would be simpler than playing whack-a-mole
> fixing first en-US and then repacks, but won't you hit the same problem when
> you do a release build and the most recent version is 45.2.0 ?

I would assume so.  Any downside to being more generous with the timeout, which is already 20 minutes?
(Assignee)

Comment 4

a year ago
It's a useful first step, but we don't know if it just takes a bit longer or something pathological is going on. Firefox wouldn't hit because it's generating partials with funsize (external to the slave).

Taking, since this is set in buildbot code.
Assignee: nobody → nthomas
(Assignee)

Comment 5

a year ago
Created attachment 8782685 [details] [diff] [review]
[buildbotcustom] Increase timeout to 40 minutes from 20

This increases the timeout to 40 minutes for en-US builds (thunderbird releases only). We already have that timeout for repacks.

In the event that 40 minutes is not long enough, we'd also have to modify these two lines:
http://hg.mozilla.org/build/buildbotcustom/file/default/process/release.py#l737
http://hg.mozilla.org/build/buildbotcustom/file/default/process/release.py#l834
Attachment #8782685 - Flags: review?(jlund)
Attachment #8782685 - Flags: review?(jlund) → review+
In production: https://hg.mozilla.org/build/buildbotcustom/rev/68ec5535f7af
win32 failed again, so we need another patch per comment 5


https://archive.mozilla.org/pub/thunderbird/candidates/49.0b1-candidates/build3/logs/release-comm-beta-win32_build-bm77-build1-build1.txt.gz

Adding type instruction to update manifests
       type partial
Adding file patch and add instructions to update manifests
diffing "xul.dll"
command timed out: 2400 seconds without output running ['bash', 'c:/builds/moz2_slave/tb-rel-c-beta-w32_bld-00000000/build/mozilla/tools/update-packaging/make_incremental_update.sh', 'update/win32/en-US/thunderbird-45.2b1-49.0b1.partial.mar', '../previous', '../current'], attempting to kill

Updated

a year ago
Duplicate of this bug: 1293401
see comment 7
Flags: needinfo?(rail)
Flags: needinfo?(nthomas)
(Assignee)

Updated

a year ago
Flags: needinfo?(nthomas)
Summary: Timeouts generating partial updates for Thunderbird on Windows → Missing runtime dll when generating partial updates for Thunderbird on Windows
(Assignee)

Comment 10

a year ago
Created attachment 8783385 [details]
[screenshot] mbsdiff.exe needs vcruntime140.dll

Not really a timeout. Looks more like fallout from swapping to vc2015, with the mbsdiff utility needing needs a MS runtime.

Possible fixes
* adjust PATH (dll is present on MOZ_OBJDIR/dist/bin, MOZ_OBJDIR/dist/thunderbird
* install runtime package on builders
* look for differences between mar.exe and mbsdiff.exe; perhaps mar.exe is statically linked
(Assignee)

Comment 11

a year ago
(In reply to Nick Thomas [:nthomas] from comment #10)
> Possible fixes
> * adjust PATH (dll is present on MOZ_OBJDIR/dist/bin, MOZ_OBJDIR/dist/thunderbird

Prepending either of these to PATH allow mbsdiff to emit their usage messages, wasn't able to verify partial mar generation.
Using vs2015u2\VC\redist\x64\Microsoft.VC140.CRT\ does not work, still needs an api-ms-blah-blah-blah.dll

> * install runtime package on builders

Possible, but might take a while.

> * look for differences between mar.exe and mbsdiff.exe; perhaps mar.exe is
> statically linked

mar is statically linked, via https://dxr.mozilla.org/mozilla-beta/source/modules/libmar/tool/moz.build#41
This try push tests doing the same for mbsdiff - https://treeherder.mozilla.org/#/jobs?repo=try&revision=26c6139d84ed
Comment hidden (mozreview-request)
(Assignee)

Updated

a year ago
Flags: needinfo?(rail)

Comment 13

a year ago
mozreview-review
Comment on attachment 8783397 [details]
Bug 1296449, statically link mbsdiff.exe, like we do for mar.exe

https://reviewboard.mozilla.org/r/73198/#review71302
Attachment #8783397 - Flags: review?(mh+mozilla) → review+

Comment 14

a year ago
Pushed by nthomas@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c01a9c030fd0
statically link mbsdiff.exe, like we do for mar.exe r=glandium
(Assignee)

Comment 15

a year ago
Comment on attachment 8783397 [details]
Bug 1296449, statically link mbsdiff.exe, like we do for mar.exe

Approval Request Comment
[Feature/regressing bug #]: Swapping to Visual Studio 2015 regressed this
[User impact if declined]: No partial updates for Thunderbird releases, currently blocking their 49.0b1
[Describe test coverage new/current, TreeHerder]: None
[Risks and why]: Firefox unaffected as it uses different partial update generation system
[String/UUID change made/needed]: None
Attachment #8783397 - Flags: approval-mozilla-beta?
Attachment #8783397 - Flags: approval-mozilla-aurora?
Comment on attachment 8783397 [details]
Bug 1296449, statically link mbsdiff.exe, like we do for mar.exe

sure, happy to help the tb folks!
Attachment #8783397 - Flags: approval-mozilla-beta?
Attachment #8783397 - Flags: approval-mozilla-beta+
Attachment #8783397 - Flags: approval-mozilla-aurora?
Attachment #8783397 - Flags: approval-mozilla-aurora+
nthomas can you land?  (no one else has picked it up)
Flags: needinfo?(nthomas)

Comment 18

a year ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/c01a9c030fd0
Status: NEW → RESOLVED
Last Resolved: a year ago
status-firefox51: --- → fixed
Resolution: --- → FIXED
(Assignee)

Comment 19

a year ago
Waiting for trees to reopen ...
ah - I didn't realize there were issues.
(Assignee)

Comment 21

a year ago
https://hg.mozilla.org/releases/mozilla-aurora/rev/4808dc6e844f
https://hg.mozilla.org/releases/mozilla-beta/rev/922311bede46
status-firefox49: --- → fixed
status-firefox50: --- → fixed
Flags: needinfo?(nthomas)

Updated

a year ago
Blocks: 1297787
(Assignee)

Comment 22

a year ago
Comment on attachment 8782685 [details] [diff] [review]
[buildbotcustom] Increase timeout to 40 minutes from 20

Backed this out because 49.0b1 build4 worked with the statically linked mbsdiff.exe, and partials were generated in ~ 4 minutes.

https://hg.mozilla.org/build/buildbotcustom/rev/e03bbd8376e3
Attachment #8782685 - Flags: checked-in-
In production: https://hg.mozilla.org/build/buildbotcustom/rev/e03bbd8376e3
You need to log in before you can comment on or make changes to this bug.