1459891 - Update public suffix list automatically once per cycle (from github)

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Jason Duell]

Right now we're manually updating the public suffix list (most recently in bug 1459882).  

What I'd like to see next is a simple automated system where once per cycle we auto-grab

   https://github.com/publicsuffix/list/blob/master/public_suffix_list.dat

and store it in the tree as 

    /netwerk/dns/effective_tld_names.dat

I talked to :gps about this and he said to file a bug in this component.

Update frequency: we're thinking to just do this update once per nightly cycle, fairly close to merge time.  That way the list will get beta coverage but still not be too stale in release.  

(Note: we have more ambitious plans to stop using a hardcoded-once-at-compile-time approach and use something closer to the phishing list approach where we update the list periodically--see bug 1083971--but that will be a while, so this is a good incremental step).

Comment 2

[Jason Duell]

Just a nudge to see if this work can get on a schedule at some point.

Comment 3

[Gregory Szorc [:gps]]

This feels like something that could now be done with a Taskcluster hook.

That being said, if the frequency is once per cycle, I'm not sure it is worth automating. We may want to add it to the list of tasks performed when we cut new releases.

RyanVM: do you have any opinions?

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Could we add it as another periodic file update job like we do for addons blacklist, HSTS/HPKP pinning updates, etc? That runs twice a week at the moment.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Simon, how easily could this be hooked up as part of the periodic file update?

Comment 6

[Simon Fraser [:sfraser] ⌚️GMT]

I've got a draft patch ready. Does it only need to run against mozilla-central?

Comment 7

[Ryan VanderMeulen [:RyanVM]]

I don't see any reason we couldn't run these on Beta/ESR as well. What do you think, Daniel?

Comment 8

[Daniel Stenberg [:bagder]]

Since the PSL reflects the state of the domains on the Internet *right now* I would agree that it makes sense to have as many as possible of our branches to adapt. I'm all for Beta/ESR too.

Comment 9

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: Bug 1459891 Routinely update in-tree TLD list r=ryanvm

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/af5911ba3ef2
Routinely update in-tree TLD list r=RyanVM,jlorenzo

Comment 11

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/af5911ba3ef2

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/a2063ad6267a

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/9491263a845f