527670 - improve update_verify output

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement, P5)

Description

[Chris AtLee [:catlee]]

Our current update_verify output is a mix of output from downloading files, applying updates, and the actual comparison results.

For clarity, we should separate these out so that in general only the comparison between the full install and the updated install are shown.  A better distinction between 'source' and 'target' is also needed.

We also need to have a set of known differences per branch / platform so we can filter them out, or raise a warning if they're not present.

Finally, this improved update_verify output should be sent to release-drivers and/or qa as part of the release process.

Comment 1

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 2

[[e-mail] (:bhearsum)]

It would be really great to make situations like the one in this log better:
http://production-master01.build.mozilla.org:8010/builders/release-mozilla-central-win32_update_verify/builds/4

The step turns green, which is probably valid, but there's some differences to removed-files and search plugins that the release engineer needs to pay attention to. Having those raised in a different log would be great.

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

I'm wondering if anyone else has time to take a look at this as it will likely be missed if it stays assigned to me due to lack of time to work on it.

Comment 4

[Chris Cooper [:coop] (he/him)]

Not too concerned about getting this done immediately vs. other work since it *is* an enhancement. I'm fine with putting this on the backburner for a bit.

Comment 5

[Lukas Blakk [:lsblakk] use ?needinfo]

putting [triagefollowup] on this since I will not get a chance to work on this during this quarter so if anyone else can take it and finish it, great, otherwise I will do it next quarter as my other old bug will be completed as part of my Q4 goals.

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to Lukas Blakk [:lsblakk] from comment #5)
> putting [triagefollowup] on this since I will not get a chance to work on
> this during this quarter so if anyone else can take it and finish it, great,
> otherwise I will do it next quarter as my other old bug will be completed as
> part of my Q4 goals.

Letting this one slip is not a problem.

Comment 7

[[e-mail] (:bhearsum)]

Mass move of bugs to Release Automation component.

Comment 8

[Lukas Blakk [:lsblakk] use ?needinfo]

Throwing this back in the pool as I'm not getting time to work on this.

Comment 10

[[e-mail] (:bhearsum)]

We improved things a little bit by adding a secondary log that summarizes the differences found (https://bugzilla.mozilla.org/show_bug.cgi?id=1446160).

This doesn't help for other types of failures, but it's something.

Comment 11

[[e-mail] (:bhearsum)]

Aki wrote this elsewhere, adding it here as one of the only concrete ideas we have about this:
I think we have failures sprinkled all over the log, without an intuitive way to find them. Gathering them all and outputting them at the bottom of the log and/or in a summary log artifact would be a good improvement. Bonus points for looking at the types of failures and giving an interpretation: "this appears to be an issue with a single locale; is it new?" "This looks like bustage related to an RC"

Comment 12

[Aki Sasaki (not active)]

I don't think there's a great way to do all of that inside the existing bash scripts. However, we could do something like

bash update_verify_script | tee raw.log | python parse_output.py | tee parsed.log

or just python parse_uv_output.py and have that wrap the bash script. Or rewrite the whole thing in python; I think the fact that this is in an ancient legacy bash script discourages people from touching it.

Comment 13

[Aki Sasaki (not active)]

https://gist.github.com/escapewindow/80cc57ee9bb331cdabfee62fe6089aef for a sampling of my gripes about the log output.

Comment 14

[Heitor Neiva [:hneiva]]

Attachment: Bug 527670 - add named arguments to update_verify r=aki

Quality of life improvement to update_verify script and taskgraph cli arguments

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a2531a72488a
add named arguments to update_verify r=aki

Comment 16

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/a2531a72488a

Comment 17

[Heitor Neiva [:hneiva]]

Attachment: Bug 527670 - Refactor Update Verify readme, r?aki

Added instructions to run update-verify locally

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9f94fffefbe6
Refactor Update Verify readme, r=aki

Comment 19

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/9f94fffefbe6

Comment 20

[Heitor Neiva [:hneiva]]

Attachment: Bug 527670: Async download/cache UpdateVerify files and improve logs r?aki

Changed wget output log granularity from :mega: to :giga: (only used in case the python library didn't cover all usecases)
Added python methods to async cache/download all required files for update verify
Modified Dockerfile requirements to include aiohttp python lib

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2160e62ae7c3
Async download/cache UpdateVerify files and improve logs r=aki

Comment 22

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/2160e62ae7c3

Comment 23

[Heitor Neiva [:hneiva]]

Attachment: WIP: Bug 527670: Add documentation to update-verify clear_cache()

Added documentation to methods, missed from last PR

Comment 24

[Heitor Neiva [:hneiva]]

Attachment: Bug 527670: Add documentation to update-verify clear_cache() r?aki

Added documentation to methods, missed from last PR

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c2cfd92e8655
Add documentation to update-verify clear_cache() r=aki

Comment 26

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/c2cfd92e8655

Comment 27

[Heitor Neiva [:hneiva]]

Attachment: Bug 527670 - Fix download failure handling on UV async downloader r?aki

Removing the unlinking shouldn't be a problem, only possible side effect would be removing the partially downloaded file

Comment 28

[Intermittent Failures Robot]

18 failures in 495 pushes (0.036 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 18

Platform and build breakdown:

• linux64-devedition: 5
  • opt: 5
• linux32-devedition: 3
  • opt: 3
• windows2012-32-devedition: 5
  • opt: 5
• windows2012-32-shippable: 1
  • opt: 1
• windows2012-64-shippable: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=527670&startday=2021-11-01&endday=2021-11-01&tree=all

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c88f473c5738
Fix download failure handling on UV async downloader r=aki DONTBUILD

Comment 30

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/c88f473c5738

Comment 31

[Intermittent Failures Robot]

18 failures in 3030 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 18

Platform and build breakdown:

• linux64-devedition: 5
  • opt: 5
• linux32-devedition: 3
  • opt: 3
• windows2012-32-devedition: 5
  • opt: 5
• windows2012-32-shippable: 1
  • opt: 1
• windows2012-64-shippable: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=527670&startday=2021-11-01&endday=2021-11-07&tree=all

Comment 32

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:hneiva, maybe it's time to close this bug?
For more information, please visit BugBot documentation.