1312518 - XML header not detected correctly in periodic file update script

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris Cooper [:coop] (he/him)]

philor noticed this was broken. At some point the header of the AMO version started including encoding information, and our script fails on that.

Patch incoming.

Comment 1

[Chris Cooper [:coop] (he/him)]

Attachment: [tools] Use xmllint to vlid downloaded xml artifacts

I checked, and we have xmllint available on the linux spot instances, so we might as well use it to give us more confidence about the state of the downloaded XML artifacts. Broke the check out into its own function should we need to change it in the future.

It's a shell patch, so tagging Hal for review.

Comment 2

[Hal Wine [:hwine] use NI!]

Comment on attachment 8804051 [details] [diff] [review]
[tools] Use xmllint to vlid downloaded xml artifacts

Review of attachment 8804051 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm functionally. One minor suggestion, but okay to land without taking.

::: scripts/periodic_file_updates/periodic_file_updates.sh
@@ +388,5 @@
> +   if [ ! -e ${XMLLINT} ]; then
> +       echo "ERROR: xmllint not found in PATH"
> +       exit 60
> +   fi
> +   ${XMLLINT} --format ${xmlfile} 1>/dev/null

No functional issue, but you might consider replacing '--format' with '--nonet --noout' - the first to preclude any entity-expansion-attack, the 2nd to save cpu cycles.

Comment 3

[Chris Cooper [:coop] (he/him)]

https://hg.mozilla.org/build/tools/rev/92c3837580e05636185f7b7e7f104f8f57cf60f4
Bug 1312518 - Use xmllint to validate downloaded xml artifacts - r=hwine

Comment 4

[Chris Cooper [:coop] (he/him)]

(In reply to Hal Wine [:hwine] (use NI) from comment #2) 
> No functional issue, but you might consider replacing '--format' with
> '--nonet --noout' - the first to preclude any entity-expansion-attack, the
> 2nd to save cpu cycles.

Landed with the above suggestion, thanks!

Comment 5

[Phil Ringnalda (:philor)]

"warning: failed to load external entity "blocklist_hg.xml""

Comment 6

[Chris Cooper [:coop] (he/him)]

https://hg.mozilla.org/build/tools/rev/91da3cc1621029fa08aca4da55c9e0f085804ed2
Bug 1312518 - fix variable substitution error - r=bustage

Comment 7

[Chris Cooper [:coop] (he/him)]

Success:

http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64/mozilla-central-linux64-periodicupdate-bm70-build1-build1.txt.gz
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=ca5956d3e4de98ec9e1aff413c7a651757b3c300