1266382 - many update verify failures for Firefox 45.1.0esr build1 due to 'binary files found in diff'

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

+++ This bug was initially created as a clone of Bug #1263721 +++

Full list of failures:
failed at update_verify_esr_6/6 on win32
failed at update_verify_esr_4/6 on macosx64
failed at update_verify_esr_4/6 on linux
failed at update_verify_esr_5/6 on macosx64
failed at update_verify_esr_4/6 on linux64
failed at update_verify_esr_6/6 on macosx64
failed at update_verify_esr_6/6 on linux64
failed at update_verify_esr_5/6 on linux
failed at update_verify_esr_5/6 on linux64	
failed at update_verify_esr_6/6 on linux

Example from http://buildbot-master72.bb.releng.usw2.mozilla.com:8001/builders/release-mozilla-esr45-linux_update_verify_esr_6%2F6/builds/1/steps/run_script/logs/stdio:


Got this response:
<?xml version="1.0"?>
<updates>
    <update type="minor" displayVersion="38.8.0esr" appVersion="38.8.0" platformVersion="38.8.0" buildID="20160420141331" detailsURL="https://www.mozilla.org/zh-TW/firefox/38.8.0/releasenotes/">
        <patch type="complete" URL="http://archive.mozilla.org/pub/firefox/candidates/38.8.0esr-candidates/build1/update/linux-i686/zh-TW/firefox-38.8.0esr.complete.mar" hashFunction="sha512" hashValue="b94644d8bde2b44e7e5fc7c61cc99ca923bc2b2212da56779bcb36c3a22e2504426475288df366ae2a3eb6dc73cdf909331048f46e97f86a250fc64a79313249" size="46515769"/>
    </update>
</updates>

Retrieving 'http://archive.mozilla.org/pub/firefox/candidates/38.8.0esr-candidates/build1/update/linux-i686/zh-TW/firefox-38.8.0esr.complete.mar' from cache...
command: END (1789.51s elapsed)

+ set +x
Scanning log for failures
-------------------------
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.3.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.3.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.3.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.2.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.2.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.2.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.1.1esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.1.1esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.1.1esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.1.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.1.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.1.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-31.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.8.1esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.8.1esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.8.1esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.8.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.8.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.8.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.7.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.7.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.7.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.6.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.6.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.6.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.5.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.5.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.5.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.4.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.4.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.4.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.3.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.3.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.3.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.2.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.2.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.2.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.1.1esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.1.1esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.1.1esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.1.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.1.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.1.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Linux_x86-gcc3 downloads/firefox-24.0esr.tar.bz2 vs. downloads/firefox-45.1.0esr.tar.bz2: 1
-------------------------
This run has failed, see the above log

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

From IRC in #releaseduty

16:04:40 <mtabara> could it be related to the fact that there is 'Unknown' Revision for these jobs in https://secure.pub.build.mozilla.org/buildapi/running ?
16:04:45 <rail> unknown is expected, it's not set in the old style release builders
16:05:12 <rail> and the failures are probably related to the fact that we still have esr38 and have split update paths
16:05:53 <rail> so we don't serve updates to 38.x up to 45
16:06:49 <rail> we can remove all 38 versions from 45's patcher config and regenerate update verify configs
16:07:06 <rail> that should green up the tests
16:07:39 <rail> but we would need to add them back in the next cycle, when we should start serving updates for older users
16:10:15 <kmoir> mtabara: I can fix the patcher configs, I did something similar for tb 45.0

Comment 2

[Kim Moir [:kmoir] ET]

Attachment: bug1266382.patch

Comment 3

[Kim Moir [:kmoir] ET]

Attachment: bug1266382-2.patch

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 8743900 [details] [diff] [review]
bug1266382-2.patch

Review of attachment 8743900 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM!

Can you also file a bug to merge esr38 patcher config into esr45's for the next cycle?

Comment 5

[Kim Moir [:kmoir] ET]

Comment on attachment 8743900 [details] [diff] [review]
bug1266382-2.patch

and tagged with FIREFOX_45_1_0esr_RELEASE_RUNTIME

other bug 1266452 on file for next release cycle

Comment 6

[Kim Moir [:kmoir] ET]

Attachment: bug1266382-3.patch

I specified the release channel instead of esr channel generating them before, this fixes it.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

When we ship 45.2.0esr we'll be removing the restriction in the Balrog rules which prevents 38.x and older update to 45.2. At that point I'd like to have the last few 38.x in the patcher config. Any objections ? If not, could we please have a followup bug (blocking the next merge) which adds data back in.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Oops sorry, didn't read all the comments here.