1501141 - Final verify fails but job is green, and only tests mac

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Update verify jobs have been green but not checking anything:

Mon Oct 22 09:46:31 UTC 2018:  Starting stopwatch...
Mon Oct 22 09:46:31 UTC 2018:  
Mon Oct 22 09:46:31 UTC 2018:  Please be aware output will now be buffered up, and only displayed after completion.
Mon Oct 22 09:46:31 UTC 2018:  Therefore do not be alarmed if you see no output for several minutes.
Mon Oct 22 09:46:31 UTC 2018:  See https://bugzilla.mozilla.org/show_bug.cgi?id=863602#c5 for details.
Mon Oct 22 09:46:31 UTC 2018:  
xargs: ../get-update-xml.sh: No such file or directory
xargs: ../test-mar-url.sh: No such file or directory
Mon Oct 22 09:46:32 UTC 2018:  
Mon Oct 22 09:46:32 UTC 2018:  Stopping stopwatch...
Mon Oct 22 09:46:32 UTC 2018:  
Mon Oct 22 09:46:32 UTC 2018:  All tests passed successfully.
Mon Oct 22 09:46:32 UTC 2018:  
Mon Oct 22 09:46:32 UTC 2018:  
Mon Oct 22 09:46:32 UTC 2018:  ====================================
Mon Oct 22 09:46:32 UTC 2018:  KEY STATS
Mon Oct 22 09:46:32 UTC 2018:  ====================================
Mon Oct 22 09:46:32 UTC 2018:  
Mon Oct 22 09:46:32 UTC 2018:  Config files scanned:                       5
Mon Oct 22 09:46:32 UTC 2018:  Update xml files downloaded and parsed:     2036
Mon Oct 22 09:46:32 UTC 2018:  Unique mar urls found:                      0
Mon Oct 22 09:46:32 UTC 2018:  Failures:                                   0
Mon Oct 22 09:46:32 UTC 2018:  Parallel processes used (maximum limit):    48
Mon Oct 22 09:46:32 UTC 2018:  Execution time:                             1 seconds
Mon Oct 22 09:46:32 UTC 2018:  
[taskcluster 2018-10-22 09:46:32.695Z] === Task Finished ===
[taskcluster 2018-10-22 09:46:32.695Z] Successful task run with exit code: 0 completed in 93.276 seconds

This will be fallout from https://hg.mozilla.org/build/tools/rev/d5289f1ceaaa on Sep 23.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

https://hg.mozilla.org/build/tools/rev/f48e358fe285b8973c6300d6592e27d206300d47 to restore those two files. We should also make the script catch the failures and set a sensible exit code.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Test all platforms, and error on missing scripts

If you look at this 64.0b3 log carefully
  https://queue.taskcluster.net/v1/task/XaCmrGGaTs-_V-ODM3ty5g/runs/2/artifacts/public/logs/live_backing.log
you'll see it's only testing mac, which is the first config in the arg list. The bash array for the configs isn't being used properly in the for loop. Test run at https://taskcluster-artifacts.net/FubeIs-zSEyckO0kcMs1yQ/2/public/logs/live_backing.log

This also fixes up not erroring out when the helper scripts are missing, tests:
https://taskcluster-artifacts.net/fK18MaKVRnumZRspcTx25w/0/public/logs/live_backing.log
https://taskcluster-artifacts.net/BZu7joBORB-sv3bDNtev4w/0/public/logs/live_backing.log

I'm going to pre-emptively land this, and rerun 63.0 since that's imminent, but please check it.

https://hg.mozilla.org/build/tools/rev/5403bbf3cffa

Comment 3

[Aki Sasaki (not active)]

Comment on attachment 9019251 [details] [diff] [review]
Test all platforms, and error on missing scripts

TIL ${var[@]}

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Thanks. 

FTR, the checking one platform regression started in 59, now fixed in 63.