456142 - Bandwagon read-only feeds

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Collections, defect, P3)

Description

[Justin Scott [:fligtar]]

From the specs:
    *  The AMO API will be extended to support user-created feed output and input.
    * Every available feed will also have an RSS formatted counterpart with a similar URL structure
    * The feed controller will output read-only feeds for the existing AMO default collections: recommended add-on list, newest add-ons, updated add-ons, each category, each user’s add-ons, and search results.
    * The feed controller will output user-created feeds for each feed ID.
    * The same URL used for outputting feed information will be used for writable responses to publish add-ons. For closed feeds, the key appended to the end of the feed URL will be checked for accuracy to determine whether the add-on can be published to the closed feed.

Comment 1

[Justin Scott [:fligtar]]

Nominating for 4.0.3 as part of second Bandwagon milestone.

Comment 2

[Michael Morgan [:morgamic]]

Why are we duplicating feeds that are already available via the API?

Comment 3

[Les Orchard [:lorchard]]

Just noticed that this bug is in my pile, but have also realized that I don't really know anything about Bandwagon.  The wiki doesn't seem to have much to say about it - are there any specs or suchlike I can read to catch up with the general idea of what's going on here?

Comment 4

[Justin Scott [:fligtar]]

Looks like someone already added you to the BaseCamp/Grouphub. There should be lots of information there, and later today I'll comment on this bug with a lot more detailed information about what we need to add the existing feeds, etc.

Comment 5

[Brian King [:kinger]]

As one of the Add-on authors, we have identified the following information that needs to be added to new feeds that are missing from existing feeds:

- feed: name
- feed: site url
- addon: author
- addon: category
- addon: when added
- addon: comments

Other APIs missing:

- subscribe/unsubscribe api
- publish/share to email api

As Justin said, there is more information in Grouphub, but feel free to ask questions.

Comment 6

[Justin Scott [:fligtar]]

Hey Les, this is the detailed information for read-only feed changes. I will file related bugs for the parts that are worthy of their own bug.

The existing API documentation is here: https://wiki.mozilla.org/Update:Remora_API_Docs

For feeds, we'll need an action similar to the "list" method in those docs. It should be something like

https://services.addons.mozilla.org/api/[version]/feed/[feed id]/[optional:password hash]

The resulting feed can be formatted like the list XML is, except we need to add the properties Brian mentioned above. So for each add-on, we need:

* author name (this should be the same as on the site, so if someone has a nickname, use that, otherwise use first + last name). Should *not* be email address.
* add-on categories. This should likely have an element per category name. For example, <categories><category id="12">Web Development</category></categories>
* date added. This is specific to bandwagon and is the date the add-on was added to the collection/list. This should be the created field in addons_collections.
* comments. When the feed/list author publishes an add-on to the feed, they can add personal comments. That's what this refers to.

Additionally, for the entire feed, we'll need to include the feed metadata. The relevant fields to be included are:

Collection ID (required; non-localizable; never changeable), Collection Name (required; localizable), Collection Description (required; localizable; maxlength of 200 characters), whether the collection is open (anyone can write to) or closed (must have collection password to write to)

The feed ID will be the alphanumeric unique identifier for a feed, such as "bobs-favorites".

There are a number of existing API methods that return lists of add-ons, such as search results and recommend lists. These should have the author and category properties added, but not the others.

The optional password is only necessary when publishing to a feed, which will be covered in a separate bug to be filed after this comment.

So for this bug, only the things mentioned in this comment should be done, not everything listed in the original comment. I'll file separate bugs for those.

Let us know if you have any questions!

Comment 7

[Justin Scott [:fligtar]]

Also since you're new to bandwagon, the following words are synonymous for most purposes here:

bandwagon/collection/feed/list/bundle :)

Comment 8

[Les Orchard [:lorchard]]

Okay, looks like I can try posting a comment here again.  Some questions / issues:

* Can we pick a term and stick with it?  I'll pick "collection" because "feeds"
make me think of RSS and Atom (e.g. http://feedvalidator.org/ )  Seems doubly
confusing because it looks like these collections want RSS feeds along with the
API format.

* Is there a web UI for this feature yet?  The docs I'm finding on the group
hub seem to be for a canned set of addons, and code I see in SVN doesn't seem
to have per-user management functionality.

* If no web UI, how are per-user collections initially created?  Should there
be a POST URL to create new collections?

* How are collections IDs discovered?  Should there be a list-of-collections
URL added to the API?

* POST to append a new addon to a collection - is there also a need to round
out CRUD operations with PUT to edit and DELETE to remove one from the list?

* Does the extension expect to POST new addon details as an XML doc in request
body?  JSON okay?  URL-encoded variables preferred?

* What is a password hash / how is it derived?  Can it be changed?  In the API,
or just in a web UI?  Is the password hash a private or shared thing?

* Is there no auth requirement beyond password hash?  That is, can anonymous
POSTs inject addons into an "open" collection without restriction?  Spam /
abuse concerns here?

* If there is an additional user auth requirement - how should it work?  What
can the extension support?  Basic auth?  Some kind of cookie munging?  OAuth? 
It doesn't look like any other part of the AMO API does auth, so this seems
like new territory.

---

Without a web UI, a POST and GET only API on just addon collections seems to leave a lot of pieces usable only through direct SQL manipulations.

I could just work up what I think is a decent REST API for CRUD on per-user addon collections, with suggestions on request body types and auth schemes...  but I don't want to assume too much too far.

Comment 9

[Brian King [:kinger]]

(In reply to comment #8)

Justin and David are better to answer these questions, but I will try to answer a few of them.

> Okay, looks like I can try posting a comment here again.  Some questions /
> issues:
> 
> * Can we pick a term and stick with it?  I'll pick "collection" because "feeds"
> make me think of RSS and Atom (e.g. http://feedvalidator.org/ )  Seems doubly
> confusing because it looks like these collections want RSS feeds along with the
> API format.

So the current API output does not adhere to a feed standard? My understanding is that Collections is a different thing, more focused on getting users to install a group of add-ons at once. What we are trying to do with the extension, even though that may be one of the features, is something a little different. We are not just asking users to consume, but to share (hence writable feeds, another bug). Technically they are collections, and I agree Feeds might not be the most user-friendly name.

Feeds may not be the best name, esp. if they are not actually feeds. I have no strong opinion on what the name should be, and am not sure if the marketing guys mind if we piggy back on the Collections name. 

> * Is there a web UI for this feature yet?  The docs I'm finding on the group
> hub seem to be for a canned set of addons, and code I see in SVN doesn't seem
> to have per-user management functionality.

No Web UI AFAIK yet.

> * If no web UI, how are per-user collections initially created?  Should there
> be a POST URL to create new collections?

Right now in the extension we are using already existing feeds on AMO and writing some by hand.

Have you had a chance to install the extension yet? Details on where to find it are in recent messages in grouphub.

Comment 10

[Justin Scott [:fligtar]]

(In reply to comment #8)
> Okay, looks like I can try posting a comment here again.  Some questions /
> issues:
> 
> * Can we pick a term and stick with it?  I'll pick "collection" because "feeds"
> make me think of RSS and Atom (e.g. http://feedvalidator.org/ )  Seems doubly
> confusing because it looks like these collections want RSS feeds along with the
> API format.
> 
Collections was going to refer to the marketing program now known as Fashion Your Fox. The main part of Bandwagon was going to be subscribing to extension feeds via this add-on feed reader extension. Since the existing controller is called "collections", I'm fine with sticking with that. I just want to make sure that subscribing to collections via the extension doesn't look like an additional feature in the final product that no one will use, because it was supposed to be a big selling point.

> * Is there a web UI for this feature yet?  The docs I'm finding on the group
> hub seem to be for a canned set of addons, and code I see in SVN doesn't seem
> to have per-user management functionality.
> 
Ryan is working on the web view for normal collections. The URL structure there will eventually be https://preview.addons.mozilla.org/en-US/firefox/collections/display/{collection id}

> * If no web UI, how are per-user collections initially created?  Should there
> be a POST URL to create new collections?
> 
Collections will be created using a web interface that isn't created yet - it's a bug to be done in a later milestone. There should be at least one collection already in the db for the Fashion Your Fox collection.

> * How are collections IDs discovered?  Should there be a list-of-collections
> URL added to the API?
> 
There will be a Collections Directory on the web to be created in a later milestone where users can browse to find the collection they want and subscribe or install from there.

> * POST to append a new addon to a collection - is there also a need to round
> out CRUD operations with PUT to edit and DELETE to remove one from the list?
> 
Nope, only adding can be done from the extension. The rest of the management will be done from a web interface.

> * Does the extension expect to POST new addon details as an XML doc in request
> body?  JSON okay?  URL-encoded variables preferred?
> 
Brian can answer here as to his preference.

> * What is a password hash / how is it derived?  Can it be changed?  In the API,
> or just in a web UI?  Is the password hash a private or shared thing?
> 
When the user creates a closed collection, they will enter a write password that is required to publish to the feed. This password will be entered in the extension and an md5 hash appended to the end of the POST URL. The API should check if the password on file for the collection matches that hash. The password is changeable from the web interface. If you think there's a better way to do this that isn't too much work, I'm interested in suggestions. But I'd really just like to get everything working to start with and then we can make improvements like per-user passwords, etc.

> * Is there no auth requirement beyond password hash?  That is, can anonymous
> POSTs inject addons into an "open" collection without restriction?  Spam /
> abuse concerns here?
> 
I'm expecting very little use of "open" collections, but for those that do choose to be open, yes, anyone could publish add-ons to that collection. Open will not be the default.

> * If there is an additional user auth requirement - how should it work?  What
> can the extension support?  Basic auth?  Some kind of cookie munging?  OAuth? 
> It doesn't look like any other part of the AMO API does auth, so this seems
> like new territory.
> 
Not applicable. We wanted the extension to not be connected to a user's AMO account at all.

Let me know if you have more questions. The Feature Specs doc at https://mozilla.grouphub.com/W1166691 is also a good reference.

Comment 11

[David McNamara [:mackers]]

(In reply to comment #10)
> > * Does the extension expect to POST new addon details as an XML doc in request
> > body?  JSON okay?  URL-encoded variables preferred?

We're left the implementation open in the extension, so we're easy on this, either XML or JSON.

I suggest going with whatever form existing API calls take (if any) and we can go with that.

Comment 12

[Les Orchard [:lorchard]]

I think I've almost got my head straight on this... so:

* Going to go with JSON as the request body format.  XML would make things
symmetric, but JSON is probably the easiest to generate & parse between PHP &
JS.  No other API calls take input like this, so this will be the first.

* If there's no web UI yet to create collections, it seems like a
chicken-and-egg problem in having any collections/feeds available for writing
with the API.  I think that's part of what's got me head-scratching.

* Is that Fashion Your Fox feed meant to be writable if it's part of a marketing
program?  Should I plan to create a few by hand in a SQL dump as part of a
patch for this, so the API's at least usable to develop against?

* So these feeds of extensions are completely standalone and anonymous, just by
default protected with optional password hashes.  Seems a bit unusual, but I'm
probably missing something.

* I've been trying to digest that doc and other material in the grouphub, but
have been lost / overwhelmed.  That's why I'm asking all the questions.

Comment 13

[Les Orchard [:lorchard]]

Attachment: sample proposed feed output

I think I have something working on my laptop, but it needs some more work because I think I'm doing some nasty things to the database.  

In the meantime, I want to be sure I'm on the right track.  So, here's an attachment of the current XML output I've got going.  The data behind it is basically junk, but hopefully the structure makes sense.

The <addons> element gets a new child <feedmeta> with <id>, <name>, <description>, and <closed> children.

The <addon> elements under <addon> each get a new child <feedmeta> with <added> and <comments> children.  I've also added a <categories> element listing <category id="#">...</category> for the addon's tagged categories.

Comment 14

[Les Orchard [:lorchard]]

Comment on attachment 346991 [details]
sample proposed feed output

tossing an r? to [e-mail] just for good measure.

Comment 15

[David McNamara [:mackers]]

(In reply to comment #13)
> Created an attachment (id=346991) [details]
> sample proposed feed output

This looks great: We have meta information for the feed: id, name, description and closed flag. We also have meta information for each addon: author, category, added date and comments.

The only thing that I can see that is missing is the "site url". 

This is the URL that is opened when the user clicks the "View Site" button in the extension. We're not sure yet what this URL will look like - probably a collections URL for AMO collections or some other URL for external collections. It's also possible that we'll be able to construct this URL from the feed id (for AMO collections).

Perhaps for now, it would be a good idea to just have a <site> element in <feedmeta> containing a URL. It can be empty for now, or if no site exists.

One more thing: is the <feedmeta> element under <addon> confusing? Maybe it should be called something like "<addonmeta>", as it doesn't pertain to the feed itself.

Comment 16

[Justin Scott [:fligtar]]

Les, for now let's leave add the URL field Dave mentioned as going to normal AMO/collections/display/{id}

And I agree with Dave that having non-addon elements under <addons> is a bit confusing.

Otherwise, looking good! Thanks

Comment 17

[Brian King [:kinger]]

(In reply to comment #10)
> Collections was going to refer to the marketing program now known as Fashion
> Your Fox. The main part of Bandwagon was going to be subscribing to extension
> feeds via this add-on feed reader extension. Since the existing controller is
> called "collections", I'm fine with sticking with that. I just want to make
> sure that subscribing to collections via the extension doesn't look like an
> additional feature in the final product that no one will use, because it was
> supposed to be a big selling point.

Justin, do you mean we should change all references to 'Feeds' in the extension UI to 'Collections'?

And I'm curious, why do you think such a name change would make it look like an additional feature? Do you mean that you want the extension to be tightly coupled to the 'Fashion your Firefox' web feature?

Comment 18

[Les Orchard [:lorchard]]

Attachment: patch to add read-only collection feed to API

Trying to get this rolling again, here's a patch for what I have so far on generating the read-only feeds.

Comment 19

[Les Orchard [:lorchard]]

Attachment: sample collections to exercise read only feeds

In lieu of a web UI to create new feeds, here's some SQL with a pair of feeds (ids 10 and 11) for use with a recent Remora dump.

Comment 20

[Les Orchard [:lorchard]]

Comment on attachment 348084 [details] [diff] [review]
patch to add read-only collection feed to API

adding r? for laura

Comment 21

[Laura Thomson :laura]

Comment on attachment 348084 [details] [diff] [review]
patch to add read-only collection feed to API

Generally ok.

I would:
- restructure the XML into a <collection> including <addons /> and the other metadata
- do a version bump (1.3), and please start a wiki page documenting that (I'll fill in the earlier versions.)

Comment 22

[Les Orchard [:lorchard]]

Attachment: revised sample collection XML output

checked in as r19855, attached a new sample of the XML

Comment 23

[Les Orchard [:lorchard]]

Hopefully this will show up on preview soon, *but* it will need SQL changes as noted here:

https://wiki.mozilla.org/Update:Developers/Database_Changes#4.0.4

Comment 24

[Brian King [:kinger]]

Has this made it tp preview. ?

Is there any schedule for this going live, i.e. preview.addons... -> addons...
?

Comment 25

[Les Orchard [:lorchard]]

This is on preview, but looks like it needs a DB update before it'll work.

Comment 26

[Les Orchard [:lorchard]]

With the completion of bug 465569, this appears to be working on preview now:

https://preview.addons.mozilla.org/en-US/firefox/api/1.3/feed/1

Comment 27

[Michael Morgan [:morgamic]]

If we can test this we can update SAMO as well.

Comment 28

[Michael Morgan [:morgamic]]

Brian, Fligtar - stephend is on vacation so if either of you would like to test this in the meantime that would help.

Comment 29

[David McNamara [:mackers]]

I'm seeing the new fields in the feed, but they seem to be empty.

<collection>
    <id>1</id>
    <name></name>
    <description></description>
    <closed></closed>
    <site>https://preview.addons.mozilla.org/en-US/firefox/collection/display/1</site>
    <addons>
    <addon>
                    <meta>
                    <added>0000-00-00 00:00:00</added>
                    <comments></comments>
                </meta>
[...]

This is happening for both AMO and preview.AMO

Comment 30

[Les Orchard [:lorchard]]

(In reply to comment #29)
> I'm seeing the new fields in the feed, but they seem to be empty.

I don't think these fields have values in the DB for the Fashion Your Firefox collection

Comment 31

[Justin Scott [:fligtar]]

Yeah, FYF doesn't have anything in those fields. We'll make some collections for testing on preview.

Comment 32

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #31)
> Yeah, FYF doesn't have anything in those fields. We'll make some collections
> for testing on preview.

But, per today's meeting, the absence of those collections shouldn't block this from going out on Thursday's IT push, right?  (In other words, this can be verified as-is, yes?)

Comment 33

[Stephen Donner [:stephend] Not actively reading bugmail]

Reopening;

XML Parsing Error: not well-formed
Location: https://preview.addons.mozilla.org/en-US/firefox/api/1.3/feed/1
Line Number 78, Column 138:based on icons from art.gnome.org. Walnut is a complete skin redesign so that all windows, widgets, panels, etc have a wooden look (&feel). </summary>

Comment 34

[Les Orchard [:lorchard]]

Looks like the fix to bug 466580 broke the feed

Comment 35

[Les Orchard [:lorchard]]

The patch to bug 466580 should fix this now.

Comment 36

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED on preview.