1184381 - Automatic updates of Thunderbird blocklist stopped in 2014

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Kent James (:rkent)]

/mail/app/blocklist.xml stopped receiving automatic updates in June of 2014, so that has not been updated since then.

At the very least we need to manually updated prior to the next point release.

Comment 1

[Mark Banner (:standard8)]

You probably want to talk to releng folks & find out what's going on with their process - http://hg.mozilla.org/mozilla-central/filelog/5df788c56ae7/browser/app/blocklist.xml shows the Firefox version being updated, so it could just be a permissions problem or something.

Comment 2

[Mark Banner (:standard8)]

Note: iirc, the blocklist is updated soon after starting TB, but having a reasonably up to date one straight away, would probably help with dll blocking etc.

Comment 3

[Philip Chee]

http://mxr.mozilla.org/build/source/tools/scripts/blocklist/sync-hg-blocklist.sh

http://hg.mozilla.org/build/tools/filelog/1b7a297d2054/scripts/blocklist/sync-hg-blocklist.sh

The only change in June 2014 is:
Bug 869051 - Race condition between builders that push updates to in-tree files
http://hg.mozilla.org/build/tools/rev/6b39c6438489

Comment 4

[Philipp Kewisch [:Fallen][📱📆]]

Rail, any idea what this might be?

Comment 5

[Rail Aliiev [:rail]]

Attachment: enable-blocklist-updates.diff

Looks like we missed this at some point...

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 8636163 [details] [diff] [review]
enable-blocklist-updates.diff

https://hg.mozilla.org/build/buildbot-configs/rev/ff9c87e5ffd5

Comment 7

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/ff9c87e5ffd5

Comment 8

[Rail Aliiev [:rail]]

I forced one of the builders and it failed with the following error. Can someone adapt the script so it handles TB as well?

bash scripts/scripts/periodic_file_updates/periodic_file_updates.sh -b comm-central --bundle https://ftp-ssl.mozilla.org/pub/mozilla.org/thunderbird/bundles/comm-central.hg -p thunderbird -u tbirdbld -k '~cltbld/.ssh/tbirdbld_dsa' -a --blocklist
 in dir /builds/slave/c-cen-l64-periodicupdate-00000/. (timeout 1200 secs)
 watching logfiles {}
 argv: ['bash', 'scripts/scripts/periodic_file_updates/periodic_file_updates.sh', '-b', 'comm-central', '--bundle', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/thunderbird/bundles/comm-central.hg', '-p', 'thunderbird', '-u', 'tbirdbld', '-k', '~cltbld/.ssh/tbirdbld_dsa', '-a', '--blocklist']
 environment:
  CCACHE_HASHDIR=
  G_BROKEN_FILENAMES=1
  HISTCONTROL=ignoredups
  HISTSIZE=1000
  HOME=/home/cltbld
  HOSTNAME=bld-linux64-spot-314.build.releng.usw2.mozilla.com
  LANG=en_US.UTF-8
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=cltbld
  MAIL=/var/spool/mail/cltbld
  PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin
  PROPERTIES_FILE=/builds/slave/c-cen-l64-periodicupdate-00000/buildprops.json
  PWD=/builds/slave/c-cen-l64-periodicupdate-00000
  SHELL=/bin/bash
  SHLVL=1
  TERM=linux
  TMOUT=86400
  USER=cltbld
  _=/tools/buildbot/bin/python
 using PTY: False
http://hg.mozilla.org/comm-central/raw-file/default/browser/config/version.txt:
2015-07-22 08:20:55 ERROR 404: Not Found.
ERROR: wget exited with a non-zero exit code: 8
INFO: parsed version is 
INFO: wget -nv --no-check-certificate -O blocklist_amo.xml https://blocklist.addons.mozilla.org/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D//Firefox/20090105024647/blocklist-sync/en-US/nightly/blocklist-sync/default/default/
https://blocklist.addons.mozilla.org/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D//Firefox/20090105024647/blocklist-sync/en-US/nightly/blocklist-sync/default/default/:
2015-07-22 08:20:56 ERROR 404: NOT FOUND.
ERROR: wget exited with a non-zero exit code: 8
program finished with exit code 8
elapsedTime=0.957870

Comment 9

[Rail Aliiev [:rail]]

back to the pool

Comment 10

[Wayne Mery (:wsmwk)]

(In reply to Mark Banner (:standard8) from comment #2)
> Note: iirc, the blocklist is updated soon after starting TB, but having a
> reasonably up to date one straight away, would probably help with dll
> blocking etc.

I've looked at the wiki regarding blocklist and I want to be sure I understand the impact of this...

This bug only impacts a user with a new profile (which in most cases will be a new user), until such time that thunderbird can download the lastest blocklist from mozilla?  i.e. this bug won't help prevent crashes for users who may have updated (or reinstalled) and are using an existing profile.

Comment 11

[Kent James (:rkent)]

(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #10)
>
> I've looked at the wiki regarding blocklist and I want to be sure I
> understand the impact of this...
> 
> This bug only impacts a user with a new profile (which in most cases will be
> a new user), until such time that thunderbird can download the lastest
> blocklist from mozilla?  i.e. this bug won't help prevent crashes for users
> who may have updated (or reinstalled) and are using an existing profile.

I think that is correct.

Comment 12

[Philipp Kewisch [:Fallen][📱📆]]

Attachment: Fix periodic updates script to accomodate for Thunderbird - v1

This should do it. I've tested with the dry run mode:

$ scripts/periodic_file_updates/periodic_file_updates.sh -b comm-central --bundle https://ftp-ssl.mozilla.org/pub/mozilla.org/thunderbird/bundles/comm-central.hg -p thunderbird -u tbirdbld -k '~cltbld/.ssh/tbirdbld_dsa' -a --blocklist -n 
2015-08-26 00:21:24 URL:http://hg.mozilla.org/comm-central/raw-file/default/mail/config/version.txt [7/7] -> "version.txt" [1]
INFO: parsed version is 43.0a1
INFO: wget -nv --no-check-certificate -O blocklist_amo.xml https://blocklist.addons.mozilla.org/blocklist/3/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D/43.0a1/Thunderbird/20090105024647/blocklist-sync/en-US/nightly/blocklist-sync/default/default/
2015-08-26 00:21:26 URL:https://blocklist.addons.mozilla.org/blocklist/3/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D/43.0a1/Thunderbird/20090105024647/blocklist-sync/en-US/nightly/blocklist-sync/default/default/ [138281] -> "blocklist_amo.xml" [1]
INFO: wget -nv -O blocklist_hg.xml http://hg.mozilla.org/comm-central/raw-file/default/mail/app/blocklist.xml
2015-08-26 00:21:27 URL:http://hg.mozilla.org/comm-central/raw-file/default/mail/app/blocklist.xml [86396/86396] -> "blocklist_hg.xml" [1]
INFO: diffing in-tree blocklist against the blocklist from AMO...
INFO: AMO blocklist differs from the one in-tree.
INFO: Updates are available, not updating hg in dry-run mode.

Comment 13

[Rail Aliiev [:rail]]

Comment on attachment 8652602 [details] [diff] [review]
Fix periodic updates script to accomodate for Thunderbird - v1

Review of attachment 8652602 [details] [diff] [review]:
-----------------------------------------------------------------

I'm not familiar with this code. I'd ask someone from http://hg.mozilla.org/build/tools/log/dbca4fa3f6a9/scripts/periodic_file_updates/periodic_file_updates.sh to review the patch. My 2 cents below.

::: scripts/periodic_file_updates/periodic_file_updates.sh
@@ +579,5 @@
> +        ;;
> +    firefox)
> +        APP_DIR="browser"
> +        APP_ID="%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D"
> +        APP_NAME="Firefox"

These values are already set for all products globally. I'd either delete the "firefox" block (to be backwards compatible) or delete the global definition and add "*)" case with non zero exit code.

Comment 14

[Philipp Kewisch [:Fallen][📱📆]]

Attachment: Fix periodic updates script to accomodate for Thunderbird - v2

(In reply to Rail Aliiev [:rail] from comment #13)
> > +    firefox)
> > +        APP_DIR="browser"
> > +        APP_ID="%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D"
> > +        APP_NAME="Firefox"
> 
> These values are already set for all products globally. I'd either delete
> the "firefox" block (to be backwards compatible) or delete the global
> definition and add "*)" case with non zero exit code.

I went with deleting the global definition and adding the *) case, but keeping the default for the product at "firefox". This way it should be backwards compatible in case Firefox doesn't use -p but still keeps app config at once place.

This won't work of course if other products use this script, without adding config for that. Let me know if I should add more.

Comment 15

[Philipp Kewisch [:Fallen][📱📆]]

https://hg.mozilla.org/build/tools/rev/baf47eca96df

Does this bug need any further action, or will it just work now?

Comment 16

[Rail Aliiev [:rail]]

It'd be great to verify the changes. I tried to force the builder but the push failed because c-c was closed. At least it was able to get to that point! \o/

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Worked on comm-aurora - https://hg.mozilla.org/releases/comm-aurora/rev/640dce05d60a

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

FTR, this is the list of builders
    "TB Linux x86-64 comm-aurora periodic file update"
    "TB Linux x86-64 comm-beta periodic file update"
    "TB Linux x86-64 comm-central periodic file update"
    "TB Linux x86-64 comm-esr38 periodic file update"

Doesn't look like there are any branches missing.

Comment 19

[Philipp Kewisch [:Fallen][📱📆]]

I noticed we have a new xpcshell failure that might be related to this, filed bug 1200567.