630526 - Port |Bug 426214 - Automatically update blocklist.xml in cvs/hg from blocklist service| to SeaMonkey

Status: RESOLVED FIXED

(SeaMonkey :: Release Engineering, defect)

Description

[Serge Gautherie (:sgautherie)]

I'm not very familiar with this feature, but I'm assuming that SeaMonkey blocklist should be (automatically) updated from time to times too, shouldn't it?

NB: And, in the meantime, manually.

Comment 1

[Justin Wood (:Callek)]

taking so I don't forget about it.

Can you also possibly file a two bugs, one blocked by this..

"Update blocklist.xml for seamonkey" and "Ensure blocklist.xml is up to date when we ship SeaMonkey 2.1" the latter will be blocking, the former I want to do "asap". -- Thanks.

Comment 2

[Justin Wood (:Callek)]

Ok, in another bug I noticed our buildbot automation has some code already to do this, though the code runs a build-tools script that is Firefox only. [http://hg.mozilla.org/build/tools/file/2059f9f3f47b/scripts/blocklist]

The solution is to copy the script for Comm-Central apps, and use it here instead.

CC-ing gozer and jhopkins incase one of them want to take on the script part.

We do have two options here, relating to this, #1, do we want each [c-c] app to be responsible for their own blocklist updates, or should we have the script do them all, and only one of us run the builder?

I would prefer each-app separately, mostly due to the option for each app to choose to cancel the process if something is going against plan, etc. but all-in-one might be easier from a script and automation side.

Comment 3

[Robert Kaiser]

(In reply to comment #2)
> The solution is to copy the script for Comm-Central apps, and use it here
> instead.

Actually, I disagree here, it would be much better to ask Mozilla releng if they would agree with us adding parameters to the existing script that make that one works for us and fall back to the Firefox values when the new params don't exist. Actually, they might one day use the additional options for mobile as well.

Comment 4

[Justin Wood (:Callek)]

Attachment: update sync-hg-blocklist.sh

This modifies only the tools script, being sure to keep it working in firefox if the -p param doesn't get added to the buildbotcustom instance before this is deployed.

I ran a dry-run of the script for both instances locally as follows (Note; I am not updating SeaMonkey's blocklist yet, as I want to be sure that once deployed this _all_ works fine on our slaves)

Justin@DELLXPS400-1 ~/Sources/Mozilla/tools/scripts/blocklist
$ sh sync-hg-blocklist.sh -d -b comm-central -u "[e-mail]" -k "~/.ssh/id_dsa" -c -p "s
eamonkey" -n
--23:32:45--  http://hg.mozilla.org/comm-central/raw-file/default/suite/config/version.txt
           => `version.txt'
Resolving hg.mozilla.org... 63.245.208.189
Connecting to hg.mozilla.org|63.245.208.189|:80... connected.
HTTP request sent, awaiting response... 200 Script output follows
Length: 8 [text/plain]

100%[===================================================>] 8             --.--K/s

23:32:45 (378.22 KB/s) - `version.txt' saved [8/8]

--23:32:46--  https://addons.mozilla.org/blocklist/3/%7B92650c4d-4b8e-4d2a-b7eb-24ecf4f6b63a%7
/2.1b3/SeaMonkey/20090105024647/blocklist-sync/en-US/nightly/blocklist-sync/default/default/
           => `blocklist_amo.xml'
Resolving addons.mozilla.org... 63.245.217.40
Connecting to addons.mozilla.org|63.245.217.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2,834 (2.8K) [text/xml]

100%[===================================================>] 2,834         --.--K/s

23:32:47 (24.66 MB/s) - `blocklist_amo.xml' saved [2834/2834]

--23:32:47--  http://hg.mozilla.org/comm-central/raw-file/default/suite/app/blocklist.xml
           => `blocklist_hg.xml'
Resolving hg.mozilla.org... 63.245.208.189
Connecting to hg.mozilla.org|63.245.208.189|:80... connected.
HTTP request sent, awaiting response... 200 Script output follows
Length: 1,426 (1.4K) [text/xml]

100%[===================================================>] 1,426         --.--K/s

23:32:48 (42.28 MB/s) - `blocklist_hg.xml' saved [1426/1426]

Blocklist files differ, but not updating hg in dry-run mode.

Justin@DELLXPS400-1 ~/Sources/Mozilla/tools/scripts/blocklist
$ sh sync-hg-blocklist.sh -d -b mozilla-central -u "[e-mail]" -k "~/.ssh/id_dsa" -c -p
 "firefox" -n
--23:33:30--  http://hg.mozilla.org/mozilla-central/raw-file/default/browser/config/version.tx

           => `version.txt'
Resolving hg.mozilla.org... 63.245.208.189
Connecting to hg.mozilla.org|63.245.208.189|:80... connected.
HTTP request sent, awaiting response... 200 Script output follows
Length: 10 [text/plain]

100%[===================================================>] 10            --.--K/s

23:33:30 (421.30 KB/s) - `version.txt' saved [10/10]

--23:33:30--  https://addons.mozilla.org/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7
/4.0b12/Firefox/20090105024647/blocklist-sync/en-US/nightly/blocklist-sync/default/default/
           => `blocklist_amo.xml'
Resolving addons.mozilla.org... 63.245.209.91
Connecting to addons.mozilla.org|63.245.209.91|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4,872 (4.8K) [text/xml]

Comment 5

[Justin Wood (:Callek)]

Comment on attachment 513023 [details] [diff] [review]
update sync-hg-blocklist.sh

>-HOST=`/bin/hostname -s`
>+#HOST=`/bin/hostname -s`
>+HOST="\"Callek's home computer\""

err this was debugging cruft from windows testing; removed locally.

Comment 6

[Mark Banner (:standard8)]

Comment on attachment 513023 [details] [diff] [review]
update sync-hg-blocklist.sh

>+case "$PRODUCT" in
>+    firefox)
>+        APP_DIR="browser";
>+        APP_ID="%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D";
>+        APP_NAME="Firefox";;
>+    seamonkey)
>+        APP_DIR="suite";
>+        APP_ID="%7B92650c4d-4b8e-4d2a-b7eb-24ecf4f6b63a%7D";
>+        APP_NAME="SeaMonkey";;
>+    *)
>+        echo >&2 "Invalid Product was passed to $0. Passed value was: $PRODUCT";
>+        echo >&2 $USAGE;
>+        exit 1;;
>+esac

Why not just pass all of that data in via arguments? Then it is very configurable for any product that comes along.

Comment 7

[Serge Gautherie (:sgautherie)]

(In reply to comment #1)
> Can you also possibly file a two bugs, one blocked by this..
> 
> "Update blocklist.xml for seamonkey" and "Ensure blocklist.xml is up to date
> when we ship SeaMonkey 2.1" the latter will be blocking, the former I want to
> do "asap". -- Thanks.

Let the former be this very bug, per your comment 4.
Wrt the latter, let's reopen bug 525465 if the current bug is not fixed before release time.

(In reply to comment #4)
> I am
> not updating SeaMonkey's blocklist yet, as I want to be sure that once deployed
> this _all_ works fine on our slaves)

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 513023 [details] [diff] [review]
update sync-hg-blocklist.sh

>-HOST=`/bin/hostname -s`
>+#HOST=`/bin/hostname -s`
>+HOST="\"Callek's home computer\""

Looks fine except for this hunk, r+ with that left out.

Comment 9

[Justin Wood (:Callek)]

Attachment: misc.py changes to support new argument

This is the patch to support this change in misc.py.

This must only be deployed after the script change, otherwise the [current] script will error out on not knowing what |-p| is.

Alternatively I can do this addition without the |if| if desired.

Comment 10

[Justin Wood (:Callek)]

Attachment: misc.py uncomment out blocklist builder (from generateCCBranchObjects)

KaiRo, I if you approve this, I'll also take it as rs+=you to enable blocklist updates in our master config [for trunk]

Comment 11

[Justin Wood (:Callek)]

Comment on attachment 513023 [details] [diff] [review]
update sync-hg-blocklist.sh

http://hg.mozilla.org/build/tools/rev/e1624eaed706

Comment 12

[Robert Kaiser]

Comment on attachment 513358 [details] [diff] [review]
misc.py uncomment out blocklist builder (from generateCCBranchObjects)

Looks good to me.

We possibly should take another look on our blocklist and if something from the Firefox blocklist should also be applied to our side, but that's a different bug.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 513357 [details] [diff] [review]
misc.py changes to support new argument

Looks OK to me.

Thunderbird will have issues with
 http://hg.mozilla.org/build/buildbot-configs/file/default/thunderbird/config.py#l390
if they want to use this stuff too.

Comment 14

[Justin Wood (:Callek)]

(In reply to comment #13)
> Comment on attachment 513357 [details] [diff] [review]
> misc.py changes to support new argument
> Looks OK to me.
> Thunderbird will have issues with

The switch in the script above can easily use Thunderbird here. So I don't anticipate this being a problem yet.

Comment 15

[Justin Wood (:Callek)]

Landed:
https://hg.mozilla.org/build/buildbotcustom/rev/56e21ad2fb5d (main misc.py patch)
https://hg.mozilla.org/build/buildbotcustom/rev/1c993575142b (CC misc.py patch, + comment removal)
https://hg.mozilla.org/build/buildbot-configs/rev/39927ec27bcc (enable for trunk)

I only enabled for trunk since I forgot comm-1.9.1 uses a version-1.9.1.txt file instead of the normal version file (due to the way we used one repo to support two mozilla-* repos) I'll attack the 1.9.1 side in another bug.

Triggered a build, which resulted in Green but failed since I forgot one small piece in the script.

http://hg.mozilla.org/build/tools/rev/6f0cdf2ce441 (got r+=dustin over IRC)

Retriggered the manual update and came out a success:
http://hg.mozilla.org/comm-central/rev/f655ed11a31b

Of note is that the hostname call seems to have resulted in an empty string on our box, will look into that separately as well.