564202 - Make major update verifications smarter

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect, P3)

Description

[Armen [:armenzg]]

Currently the three different builders always end up red.
The problem is that there are known FAILURES and known WARNINGS [1] that happen over and over again from release to release and they always count the way that we do it now.

Yesterday I had few ideas but I reduced it to this:

* Compare the log from the previous verify.sh and the current one

The following steps modify the downloaded logs to remove firefox versions and make them version-agnostic. The diff between the logs give me an empty file since there are no differences between logs. No differences would mean green (nothing new has appeared) and anything different should be red and have the releng check the logs.
* First substitution removes all the lines from where we download each locale's build (build number and browser version)
* Second substitution flattens the version from remaining lines
 sed -E -e 's/.*http.*//' -e 's/[0-9]+.[0-9]+.[0-9]+//g' -e '/^$/ d' < 5.linux.log > 5.linux.stripped.log
 sed -E -e 's/.*http.*//' -e 's/[0-9]+.[0-9]+.[0-9]+//g' -e '/^$/ d' < 6.linux.log > 6.linux.stripped.log
 diff -pU 8 5.linux.stripped.log 6.linux.stripped.log > linux.diff_between_releases.log


== 1 ==
Known warnings and failures:
* Affects all 3 platforms:
   * failed to get partial/complete mar because we don't have "mn" for 3.6.x 

Using  https://aus2.mozilla.org/update/1/Firefox/3.5.10/20100504085637/Linux_x86-gcc3/mn/betatest/update.xml?force=1
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.5.10/20100504085637/Linux_x86-gcc3/mn/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1
Using  https://aus2.mozilla.org/update/1/Firefox/3.5.10/20100504085637/Linux_x86-gcc3/mn/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.5.10/20100504085637/Linux_x86-gcc3/mn/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1

   * For 'sl' we get these warnings. These were removed between 3.6 and 3.6.2 which is why they're no. No issues here.

Only in source/firefox/searchplugins: iskalnik-gov-si.xml
Only in source/firefox/searchplugins: najdi-si.xml
Only in source/firefox/searchplugins: yahoo.xml
WARN: non-binary files found in diff
WARN: check_updates returned warning for Linux_x86-gcc3 downloads/firefox-3.5.10.tar.bz2 vs. downloads/firefox-3.6.4.tar.bz2: 2

* Linux:
   * No test failure that was only Linux specific 

* MacOSX:
   * For all locales: 
Contents of source/Firefox.app/Contents/Plug-Ins dir only in source or target 379277599        0 drwxr-xr-x    3 cltbld   staff         102 May  4 10:31 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin
379277600        0 drwxr-xr-x    4 cltbld   staff         136 May  5 08:05 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents
379277604        0 drwxr-xr-x    2 cltbld   staff          68 May  5 08:05 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/MacOS
379277605        0 drwxr-xr-x    3 cltbld   staff         102 May  4 10:31 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources
379277607        0 drwxr-xr-x    3 cltbld   staff         102 May  5 08:05 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj
379277611        0 drwxr-xr-x    2 cltbld   staff          68 May  5 08:05 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj/PrintPDE.nib
WARN: non-binary files found in diff
WARN: check_updates returned warning for Darwin_Universal-gcc3 downloads/Firefox 3.5.10.dmg vs. downloads/Firefox 3.6.4.dmg: 2

* Windows:
   * The chk files will go away with bug 489961
   * For all locales: 
Only in source/bin: .autoreg
Files source/bin/freebl3.chk and target/bin/freebl3.chk differ
Files source/bin/nssdbm3.chk and target/bin/nssdbm3.chk differ
Only in source/bin: removed-files
Files source/bin/softokn3.chk and target/bin/softokn3.chk differ
WARN: non-binary files found in diff
WARN: check_updates returned warning for WINNT_x86-msvc downloads/Firefox Setup 3.5.10.exe vs. downloads/Firefox Setup 3.6.4.exe: 2

Comment 1

[Rail Aliiev [:rail]]

Recent MU update verify builders are all green. WFM?