1241951 - binary files found in diff while update verification in FF 44.0 build2 on beta channel

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

While running the update verification on the beta channel for the FF 44.0build2 we have encountered a suspicious "FAIL: binary files found in diff" error in *all* linux, linux64 and macosx64 tests - again in *all* of them. After further digging and testing we've concluded that it might be intermittent and retriggered all the culprits. Everything went OK after it and tests passed. We went on to release 44.0build 2 on the beta channel.

However, it'd be a good idea to dig more and narrow down to find what happened. The data we have so far indicates that it's way too odd to be an intermittent error, hence some more investigation is needed.

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

macosx64 sample: http://buildbot-master86.bb.releng.scl3.mozilla.com:8001/builders/release-mozilla-release-macosx64_update_verify_beta_2%2F6/builds/6

linux sample: http://buildbot-master91.bb.releng.usw2.mozilla.com:8001/builders/release-mozilla-release-linux_update_verify_beta_6%2F6/builds/3

linux64 sample: http://buildbot-master91.bb.releng.usw2.mozilla.com:8001/builders/release-mozilla-release-linux64_update_verify_beta_4%2F6/builds/2

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

From the first locale in the mac log linked above,

# simulated 44.0b9 pl update request
Downloading 'https://aus4.mozilla.org/update/3/Firefox/44.0/20160114165817/Darwin_x86_64-gcc3-u-i386-x86_64/pl/beta-localtest/default/default/default/update.xml?force=1' and placing in cache...

# get the files
Downloading 'http://archive.mozilla.org/pub/firefox/candidates/44.0-candidates/build2/update/mac/pl/firefox-44.0.complete.mar' and placing in cache...

Downloading 'http://archive.mozilla.org/pub/mozilla.org/firefox/releases/44.0b9/mac/pl/Firefox 44.0b9.dmg' and placing in cache...

Downloading 'http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/44.0-candidates/build1/mac/pl/Firefox 44.0.dmg' and placing in cache...

The last request is bogus because it should be build2 instead of build1. Turns out the jobs got a slightly stale copy of the tools repo, in particular b574dd8b8dc5 from the updates configs bump for release, instead of 7d14ee1948c5 from the config bump for the beta channel. 

I checked the scheduling in allthethings.json and that's OK (see the downstreams on release-mozilla-release-beta_updates_done). So this might be hg caching/syncing, any changes there recently ?

7d14ee1948c5+cdeae27a76b2 landed at Thu Jan 21 07:54:11 2016 +0000
The mac slave started pulling at Wed Jan 20 23:54:59 2016 PST, and finished at 23:55:04 (assiming all clocks correct).