592590 - Build a flexible activity tracker

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Developer Pages, defect, P1)

Description

[Wil Clouser [:clouserw]]

If you look at the developer hub in AMO you'll see an activity feed for each add-on[1].  This includes tons of actions, from "$user added $addon to $collection" to "$user wrote a review of $collection" or "add-ons can now be compatible with Firefox $version."  Right now all these actions are stored in the `eventlog` table in the database.  Due to the structure of this table, this isn't a very flexible solution.

We'd like to use MongoDB to be able to drop in blobs of json that we can query arbitrarily.  Interesting views of this data are:

Show all activity related to $addon, $collection, or $user sorted by date
Show all activity that has happened (across several collections, add-ons, or users) sorted by date

This bug includes:
- defining the schema (please get r? after this step)
- creating convenience methods to log to the schema.  Parity with what we log in remora is fine, but we'll add more later.

This bug does not include:
- pulling data out for any kind of reporting

[1] I can't link you to my feed since it's behind auth, but the RSS feed can be seen:  https://addons.mozilla.org/en-US/developers/feed/all?privaterss=dbae3485-0386-d930-f4e0-e211bd91ecb9

Comment 1

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Here's the schema:

db.addon_log.findOne()
{
        "_id" : ObjectId("4cb62a13cc883c6e7c2c0731"),
        "a" : 3615,
        "args" : [
                {
                        "u" : 1
                },
                {
                        "u" : 2
                },
                {
                        "c" : 3
                }
        ],
        "action" : 1
}
a is the addon_id
action is the type of action - e.g. "%s added %s to collection %s"
args is an array of "typed" arguments to the action.  c=collection u=user.

Smaller keys are better with mongo as this document is stored in its entirety.
Note the timestamp can be extracted from the _id.

http://www.mongodb.org/display/DOCS/Capped+Collections

^^ would be cool to use, but sounds like a pain to maintain as it writes over old entries once you exceept the pre-defined caps.

I'll add some methods to an AddonLog object that will write data to this collection.

Comment 2

[Jeff Balogh (:jbalogh)]

(In reply to comment #1)
> Smaller keys are better with mongo as this document is stored in its entirety.
> Note the timestamp can be extracted from the _id.

Is this a useful optimization?  We need to store pointers to addons and users and collections and licenses and addonusers and versions and files and ..., so we have to maintain a mapping for all of these.  We have 150,000 addonlog rows now, which probably isn't all that much in terms of memory.  We're adding human overhead in parsing these keys to fix a potential hardware problem.

>         "action" : 1
> }

What are we gaining by having symbolic constants instead of strings here?  It's more mental overhead.

Comment 3

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

I assumed the log would be much larger than 150K.

The actions are constants with integer values.  We can change them to unique string values as well.

db.addon_log.findOne()
{
        "_id" : ObjectId("4cb62a13cc883c6e7c2c0731"),
        "addon_id" : 3615,
        "args" : [
                {
                        "users.userprofile" : 1
                },
                {
                        "users.userprofile" : 2
                },
                {
                        "bandwagon.collection" : 3
                }
        ],
        "action" : 'User added to Collection'
}


Then we can have action classes:

class USER_ADDED_TO_COLLECTION():
    pretty='User added to Collection'
    log_entry='%s added %s to collection: %s'

Comment 4

[Jeff Balogh (:jbalogh)]

I've been writing things like this:

AddonLog.log(AddonUser, request, addon=addon,                                 
             action=action, author=author)

AddonLog.log(License, request, addon=addon, license=license)

Then AddonLog would ask AddonUser or License how to format the record.

class License:

    def format_log(addon, license):
        return 'Set %s's license to %s' % (addon.name, license.name)


class AddonLog:

    def log(cls, request, **kw):
        return '%s: %s' % (request.user, cls.format_log(**kw))

Comment 5

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

To support that, So the schema would be:

db.addon_log.findOne()
{
        "_id" : ObjectId("4cb62a13cc883c6e7c2c0731"),
        "addon_id" : 3615,
        "object": {"bandwagon.collection" : 3}
        "args" : [
                {
                        "users.userprofile" : 1
                },
                {
                        "users.userprofile" : 2
                }
        ],
        "action" : 'added_publisher'
}

we'd get the Collection object and give it action=added_publisher and the two users involved as arguments.

AddonUser/License as first arguments to log seem redundant as that can be inferred from the class type.

Comment 6

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

So we'll do this in mysql:

log_users:    id, user id, action, args, date
log_addons: id, addon id, actions, args, date

Comment 7

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

http://github.com/jbalogh/zamboni/commit/3e84bc7