1439860 - tools/update-packaging scripts need cleaning up for safety

Status: REOPENED

(Release Engineering :: General, defect)

Description

[Simon Fraser [:sfraser] ⌚️GMT]

shell scripts in tools/update-packaging need cleaning up so that they are safer and less prone to error.

Comment 1

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: Bug 1439860 update-packaging linting fixes and shell style updates

Review commit: https://reviewboard.mozilla.org/r/221914/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/221914/

Comment 2

[[e-mail] (:bhearsum)]

Comment on attachment 8952678 [details]
Bug 1439860 update-packaging linting fixes and shell style updates

https://reviewboard.mozilla.org/r/221914/#review228188

This looks fine, but I think it should be tested on oak first, because we can't verify this with a push to try or anything like that.

Comment 3

[Simon Fraser [:sfraser] ⌚️GMT]

Is it safe to land on oak now, without interrupting any other work?

Comment 4

[Robert Strong (they/them - no direct email)]

Feel free to land on oak. I am fairly certain oak still doesn't create partial mars though.

Comment 5

[Simon Fraser [:sfraser] ⌚️GMT]

Have pushed to oak, so we'll get a general pass even if partials aren't generated.

Comment 6

[Simon Fraser [:sfraser] ⌚️GMT]

Oak does seem to be generating partials, and the tasks are working:

https://public-artifacts.taskcluster.net/Wq6GEMJ1T8eYh0AlCXcZAg/0/public/logs/live_backing.log

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/40237ca354be
update-packaging linting fixes and shell style updates r=bhearsum

Comment 8

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/40237ca354be

Comment 9

[Alice0775 White]

Hey, today's nightly updater(win10 x64) is 39MB (usually less than 10MB).
Is this as expected?

Comment 10

[Simon Fraser [:sfraser] ⌚️GMT]

It's not expected - I have a fix, but it looks like the original is not on central any more, and I can't see a backout message. Trying to work out what's happened

Comment 11

[Jim Jeffery not reading bug-mail 1/2/11]

Info, Partial updates was broken before and 'fixed' in bug https://bugzilla.mozilla.org/show_bug.cgi?id=1438415

Not sure if this work maybe overwrote that change... provided for FYI

Comment 12

[Simon Fraser [:sfraser] ⌚️GMT]

(In reply to Jim Jeffery not reading bug-mail 1/2/11 from comment #11)
> Info, Partial updates was broken before and 'fixed' in bug
> https://bugzilla.mozilla.org/show_bug.cgi?id=1438415
> 
> Not sure if this work maybe overwrote that change... provided for FYI

A different bug, sadly! That one was the cache hook incorrectly reporting success, and this one was confusion between mac/linux versions of stat and their options.

Comment 13

[Tom Prince [:tomprince]]

This has caused a couple of failures: Bug 1446110 and Bug 1448001. I think perhaps this should be reverted. Shell is a subtle enough language that it would be better to rewrite this in python than try to fix the linting errors.

Comment 14

[Simon Fraser [:sfraser] ⌚️GMT]

(In reply to Tom Prince [:tomprince] from comment #13)
> This has caused a couple of failures: Bug 1446110 and Bug 1448001. I think
> perhaps this should be reverted. Shell is a subtle enough language that it
> would be better to rewrite this in python than try to fix the linting errors.

Possibly so, yes. It's a lot of shell commands to fit in python, though - I'll try a conversion and see how it works out.

Comment 15

[Tom Prince [:tomprince]]

I'm not sure how high a priority this is. Can we just revert to the un-linted version, and then incrementally port things over if and when other changes are required?

Comment 16

[Tom Prince [:tomprince]]

Backed out:

https://hg.mozilla.org/integration/mozilla-inbound/rev/e30b39f972f0ae879e1e7290668a817b438b14a4

Comment 17

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/e30b39f972f0

Comment 20

[Chris AtLee [:catlee]]

since the backout we're seeing some shell errors in the logs. e.g. from https://taskcluster-artifacts.net/C-_aGWLrQdSe9GYRHC4S_w/0/public/logs/live_backing.log

2018-04-16 11:24:13,433 - INFO - Generating partial /tmp/tmp2rm5iayl/target.partial-4.mar
2018-04-16 11:28:18,109 - DEBUG - target.partial-3.mar: /tmp/tmpu9f2pr4c/from /tmp/tmpu9f2pr4c
2018-04-16 11:28:18,109 - DEBUG - target.partial-3.mar: /tmp/tmpu9f2pr4c
2018-04-16 11:28:18,109 - DEBUG - target.partial-3.mar: /tmp/tmpu9f2pr4c/to /tmp/tmpu9f2pr4c
2018-04-16 11:28:18,109 - DEBUG - target.partial-3.mar: /tmp/tmpu9f2pr4c
2018-04-16 11:28:18,109 - DEBUG - target.partial-3.mar: 
2018-04-16 11:28:18,109 - DEBUG - target.partial-3.mar: Adding type instruction to update manifests
2018-04-16 11:28:18,109 - DEBUG - target.partial-3.mar:        type partial
2018-04-16 11:28:18,109 - DEBUG - target.partial-3.mar: 
2018-04-16 11:28:18,109 - DEBUG - target.partial-3.mar: Adding file patch and add instructions to update manifests
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar:  add-if-not "update-settings.ini" "update-settings.ini"
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar:         add "removed-files" (forced)
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar:         add "precomplete" (forced)
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: diffing "plugin-container.sig"
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: stat: cannot read file system information for '%z': No such file or directory
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: /home/worker/bin/mbsdiff_hook.sh: line 97: [:   File: "/tmp/tmpu9f2pr4c/to/plugin-container.sig"
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar:     ID: f9d53149184763bf Namelen: 255     Type: overlayfs
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: Block size: 4096       Fundamental block size: 4096
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: Blocks: Total: 30929913   Free: 23101692   Available: 21524784
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: Inodes: Total: 7864320    Free: 6630830: integer expression expected
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: usage: dog metric post [-h] [--host HOST] [--no_host] [--device DEVICE]
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar:                        [--tags TAGS] [--localhostname] [--type TYPE]
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar:                        name value
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: dog metric post: error: argument value: invalid float value: ''
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: stat: cannot read file system information for '%z': No such file or directory
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: /home/worker/bin/mbsdiff_hook.sh: line 65: [:   File: "/tmp/tmpu9f2pr4c/to/plugin-container.sig"
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar:     ID: f9d53149184763bf Namelen: 255     Type: overlayfs
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: Block size: 4096       Fundamental block size: 4096
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: Blocks: Total: 30929913   Free: 23101678   Available: 21524770
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: Inodes: Total: 7864320    Free: 6630828: integer expression expected
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: Completed 380 Bytes/380 Bytes (842 Bytes/s) with 1 file(s) remaining
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: upload: to.work/plugin-container.sig.patch.xz to s3://tc-gp-private-1d-us-east-1/releng/mbsdiff-cache/6cad3532cf880f46323d38b7a0c364d2435a685e32f4c1a977d57e4d2e4ae27fe1669180ae44581a6daf27cac5e76873a316aa60101f48f1b6051c1077111e51/f7552a4b80b3b5d395e548bbe228f9cae31a52193c677a7a5380145036696846d4517f7205972268906f09141aea9e8c52f09a851ce8844fccd74c389427d63e/plugin-container.sig.patch.xz
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar: /tmp/tmpu9f2pr4c/to.work/plugin-container.sig.patch.xz saved on s://tc-gp-private-1d-us-east-1/releng/mbsdiff-cache/
2018-04-16 11:28:18,110 - DEBUG - target.partial-3.mar:       patch "plugin-container.sig.patch" "plugin-container.sig"

Comment 21

[Simon Fraser [:sfraser] ⌚️GMT]

This was fixed in one of the commits, so it looks like the entire set of changes wasn't removed.

Comment 22

[Simon Fraser [:sfraser] ⌚️GMT]

Hm, although it's not that, is it. I wonder why this script was reverted. I'll fail it forward.

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

Some bustage from this bug showed up on Thunderbird 60.0b11, just for Windows. That's because the backout isn't on the THUNDERBIRD_60_VERBRANCH in the mozilla-esr60 repo, Thunderbird has a removed-files.in which is very long (https://hg.mozilla.org/releases/comm-beta/file/ddb54cc8a2a1/mail/installer/removed-files.in), and there's a regression in handling detection rmrfdir on Windows.

Here's the log for the win32 en-US repackage job (Nr) - https://taskcluster-artifacts.net/ahWqFcysRlSyQqj_ubkYxg/0/public/logs/live_backing.log

...
23:43:47     INFO -  Adding file and directory remove instructions from file 'removed-files'
23:43:47     INFO -      rmrfdir "chrome/app-chrome.manifest"
23:43:47     INFO -      rmrfdir "chrome/chrome.rdf"
23:43:47     INFO -      rmrfdir "chrome/en-US-mail.jar"
23:43:47     INFO -      rmrfdir "chrome/en-win.jar"
23:43:47     INFO -      rmrfdir "chrome/installed-chrome.txt"
23:43:47     INFO -      rmrfdir "chrome/mail.jar"
23:43:47     INFO -      rmrfdir "chrome/offline.jar"
23:43:47     INFO -      rmrfdir "chrome/offline.manifest"
23:43:47     INFO -        rmdir "chrome/overlayinfo/"

Those rmrfdir operations should be remove; the rmdir is correct. This only shows up on Windows. (for comparison, linux64 - https://taskcluster-artifacts.net/HON6DrWQRLqqnxfy3Z5AbQ/0/public/logs/live_backing.log; mac - https://queue.taskcluster.net/v1/task/Xj-02v_qTvGSeRmWN9Nw3g/runs/0/artifacts/public/logs/live.log)


Backout state: Done on default for mozilla-central, mozilla-beta, mozilla-release, but not mozilla-esr60. Firefox is probably OK because browser/installer/removed-files.in got emptied after the lzma commpresion change, but it'd be better to keep everything in sync.

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Backout on esr60 THUNDERBIRD_60_VERBRANCH: https://hg.mozilla.org/releases/mozilla-esr60/rev/67a96d088945

Comment 27

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee is inactive on Bugzilla, so the assignee is being reset.