1311105 - Generate and publish a list of built in system addons for releases

Status: NEW

(Release Engineering :: Release Automation: Other, defect, P3)

Description

[Rehan Dalal (Inactive)]

I am working on Morgoth, a tool to help simplify managing and shipping system addons. We need to be able to track which system addons (name and version) are part of each release.

mshal was kind enough to write a script that parses the packaged build and generates a list of addons with their version number for a given release. You can see the script here:
https://github.com/mozilla/morgoth/issues/40#issuecomment-251092886

Comment 1

[Rail Aliiev [:rail]]

I have some questions:

1) is the list platform-agnostic? Do we ship/plan to ship different addons for different platforms?

2) Probably it'd be better to use JSON for the data

3) Do you need this data published to archive.mozilla.org?

Comment 2

[Rail Aliiev [:rail]]

If the data are not platform agnostic, I'd suggest to modify the build system to output these data and publish alongside the builds, similar to http://ftp.mozilla.org/pub/firefox/candidates/50.0b9-candidates/build1/linux-x86_64/en-US/firefox-50.0b9.json or even inside it. This way it would ride the release trains as well.

Comment 3

[Rehan Dalal (Inactive)]

> 
> 1) is the list platform-agnostic? Do we ship/plan to ship different addons for different platforms?
> 
We will not currently be shipping different addons for different platforms.

> 
> 2) Probably it'd be better to use JSON for the data
> 
Sure, that script just pulls the relevant data out. But yes, it probably makes sense to parse that data to a JSON blob.

> 
> 3) Do you need this data published to archive.mozilla.org?
> 
Ideally we should publish this data to Morgoth which currently lives at: https://morgoth.stage.mozaws.net

Eventually there will be a production instance that we will also need to publish to.

If we can POST the data to this API endpoint that would be great:
https://morgoth.stage.mozaws.net/api/v1/addon_group/update_built_in/

:

{
  browser_version: 'x.y.z',
  addons: [
    {
      name: '[e-mail]',
      version: 'x.y',
    },
    {
      name: '[e-mail]',
      version: 'x.y',
    }
  ]
}

Comment 4

[Rail Aliiev [:rail]]

(In reply to Rehan Dalal [:rehan, :rdalal] from comment #3)
> > 
> > 1) is the list platform-agnostic? Do we ship/plan to ship different addons for different platforms?
> > 
> We will not currently be shipping different addons for different platforms.

In this case we can use one of the platforms, let's say linux64, as a base. Sounds sane?
 

> > 
> > 3) Do you need this data published to archive.mozilla.org?
> > 
> Ideally we should publish this data to Morgoth which currently lives at:
> https://morgoth.stage.mozaws.net
> 
> Eventually there will be a production instance that we will also need to
> publish to.


Does it require auth/special routing? I can't load it atm.

> If we can POST the data to this API endpoint that would be great:
> https://morgoth.stage.mozaws.net/api/v1/addon_group/update_built_in/

Shouldn't be hard. In the worst case we can start by publishing the data as a json file.

>       name: '[e-mail]',

Do we need to parse install.rdf to get proper name/version?

Comment 5

[Rail Aliiev [:rail]]

Also, do you care about release candidates http://archive.mozilla.org/pub/firefox/candidates/ or just releases https://archive.mozilla.org/pub/firefox/releases/?

Comment 6

[Benson Wong [:mostlygeek]]

bhearsum mentioned this in irc. rail's idea: publish a json with the built in XPI versions. 

I +1 this idea. I think it would set up a single source of truth (archive.mozilla.org) for builds and their metadata. Then morgoth can pull the file off of archive.m.o directly instead of needing to store a duplicate of it in another database. 

:rail 

- do different builds/architectures have different built in system addons? 
- what would be appropriate path for the XPI json file? 
  - or json files if there are multiple for a build
- what's the best way to auto-create the JSON file with the build?

Comment 7

[Rail Aliiev [:rail]]

(In reply to Benson Wong [:mostlygeek] from comment #6)
> - do different builds/architectures have different built in system addons? 

I'd guess it depends, but let's ask Felipe.

> - what would be appropriate path for the XPI json file? 
>   - or json files if there are multiple for a build

Assuming that these JSON files are generated by the build system, they will be probably generated by platform. They usually go under $platform/en-US/$file under the candidates directory. For example, http://archive.mozilla.org/pub/firefox/candidates/52.0b9-candidates/build2/linux-x86_64/en-US/firefox-52.0b9.json

> - what's the best way to auto-create the JSON file with the build?

We should definitely add those in the build system, similar to other JSON files that the build system generates.

Comment 8

[:Felipe Gomes (needinfo for replies!)]

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #7)
> (In reply to Benson Wong [:mostlygeek] from comment #6)
> > - do different builds/architectures have different built in system addons? 
> 
> I'd guess it depends, but let's ask Felipe.

This hasn't happened yet, and we can try to enforce it with review policy, but that could conceivably happen in the code. The system addons list is specified in a moz.build that would allow that. There's already a bunch of channel-specific conditions:

http://searchfox.org/mozilla-central/source/browser/extensions/moz.build

I don't know what's the purpose that you're looking for here, if it needs to be exact or if an approximation is ok. If it's ok, I suggest using the JSON generated for Windows builds, because if we ever have a platform-specific system add-on, it's more likely to be a Windows-only one.


Note that we're talking here about the built-in system add-ons. We've definitely already shipped new add-ons with GoFaster for specific platforms, like the websense one that only goes out for Windows users.

Comment 9

[Benson Wong [:mostlygeek]]

We(In reply to :Felipe Gomes (needinfo me!) from comment #8)

> I don't know what's the purpose that you're looking for here, if it needs to
> be exact or if an approximation is ok. 

Trying to a few things: 

- create a better UI in morgoth specifically for System addons
- better UI needs builtin system addon version data
- we want that builtin addon data to live with the builds on archive.m.o
- we want the data to be in a json file for easy fetching/parsing 


I suppose the question is, where do we put them on archive.m.o so: 

- their contents are exact?
- they're easy to find?

It doesn't sound like we have customized list based so one json file for all platforms would be suitable for now.

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

@Callek: I don't have too much context of this but I was wondering how/if this changed given recent work in addonscript?

Comment 11

[Justin Wood (:Callek)]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #10)
> @Callek: I don't have too much context of this but I was wondering how/if
> this changed given recent work in addonscript?

I think this is a very different set of work, and I don't have much context on SystemAddons either