641640 - Ensure VAMO is set up to handle Personas pings

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Code Quality, defect, P1)

Description

[Wil Clouser [:clouserw]]

Right now when you install a persona, the JSON blob that defines the persona also sends an update_check URL.  That is http://www.getpersonas.com/update_check right now.

Since GP is scheduled to be put down, we need to send these update pings somewhere else, the natural spot would be versioncheck.amo.  This bug is to ensure that we're all onboard with that plan and that you can handle the GP ping format to that location.

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

We need to study this.  The biggest problem is that vamo is by far the largest data source that we parse, and the current ETL that is responsible for parsing personas might not be able to handle that volume.  Especially since we are using a different ETL to process the other data in vamo logs.

I'll work with Nelson this week to sort something out on it.  What is the ETA for EOL?

Comment 2

[Wil Clouser [:clouserw]]

Sometime in Q3, but I'd like to get our plan ready well before that.

Comment 3

[Daniel Einspanjer [:dre] [:deinspanjer]]

Okay, we can definitely work something out given a few weeks of breathing room after Fx4. :)

Comment 4

[Wil Clouser [:clouserw]]

Any news?

Comment 5

[Wil Clouser [:clouserw]]

(In reply to comment #4)
> Any news?

Comment 6

[Daniel Einspanjer [:dre] [:deinspanjer]]

Nelson, ping

Comment 7

[Nelson]

We can point personas and lw themes log processing to vamo files, I'll look into that next monday. 

My only concern is with filesizes, VAMO logs are huge and our current etl's performance may get hurt.

Comment 8

[Daniel Einspanjer [:dre] [:deinspanjer]]

Well then I suggest you take the time to integrate the processing into the existing ETL that processes vamo.

Comment 9

[Nelson]

We currently parse requests from getpersonas.com and getpersonas-ssl.com with requests like 

GET\s/([a-zA-Z\-]*)/update_check/(\d*),

where the first field is the locale and the second is the persona id. 

This ping occurs once per day when Firefox checks whether there's an updated version of that specific persona, and this is the ping we use to determine persona usage.

What would the new ping look like? As long as we have a pattern to look for on the vamo logs we can integrate this ping on vamo parsing and phase out our current personas and lightweight themes processing.

Comment 10

[Wil Clouser [:clouserw]]

The plan was just to change the location.  So, I'd expect:

    versioncheck.addons.mozilla.org/:lang/update_check/:id

It would be nice to say something about personas in there though.  I'll see what I can do.

Comment 11

[Wil Clouser [:clouserw]]

Hi Rob.  This is low priority for 6.4.0 but is potentially a quick bug and a good break from other stuff.

The bug is essentially taking all the logic in this short file [1] and putting it in [2] which you've got experience with already.  We just want VAMO to be able to accept persona pings and respond intelligently.  If you throw /personas/ after /:lang/ in the URL in comment 10 I'd be happy. :)

Once this ^^ is done we can resolve this bug and open a new one for metrics to make sure they do whatever process they need on their end.

[1] http://viewvc.svn.mozilla.org/vc/projects/getpersonas.com/trunk/server/update_check.php?view=markup

[2] https://github.com/mozilla/zamboni/blob/master/services/update.py

Comment 12

[Rob Hudson [:robhudson]]

work in progress:
https://github.com/robhudson/zamboni/commit/44fe6902

A couple outstanding questions or issues here...

1) Do we need to continue to escape slashes like the current JSON output does, or is this a PHP artifact? The patch as it is doesn't do this.

2) The patch needs to set a few settings to the persona CDN and site URL.

3) The data URL requires the persona files exist on the same server to calculate the base64 string, so this is flagged as "TODO".

4) There are some migration questions that may affect this code.

Comment 13

[Wil Clouser [:clouserw]]

When you land this can you let us know the URLs requests will be coming in on?  Also we're going to need to be able to 301 stuff from getpersonas.com to here so the URL will need to support that.  We can make that a separate bug if this one is ready to land and can't handle that though.

Comment 14

[Rob Hudson [:robhudson]]

Answer to #1: No, we don't need to escape slashes.

I got a quick review from Andy on Friday and would like to do the following before I push this in:

1. Use more things from services/utils.py that is shared amongst the scripts there.
2. Write tests.

Bumping to 6.4.2

Comment 15

[Wil Clouser [:clouserw]]

bug 847193 gives us a=ops for moving this traffic to VAMO

Comment 16

[Rob Hudson [:robhudson]]

Rebased onto current master and incorporated changes based on Andy's feedback:
https://github.com/robhudson/zamboni/commit/149116baafe25ead7a45e7e5f04c7bcc909eebd4

This depends on a few bugs for certain files to be in place:
* bug 846088
* bug 846957

Comment 17

[Rob Hudson [:robhudson]]

TODO:
* Tests
* Use simpler path structure with ID.

Comment 19

[Christopher Van Wiemeersch [:cvan]]

https://github.com/mozilla/zamboni/commit/73bea26

Thanks, Rob, for doing most of the work here - much appreciated! I shall now work with IT to stand up this wsgi application.

Comment 20

[Christopher Van Wiemeersch [:cvan]]

https://github.com/mozilla/zamboni/commit/6d3d643

Comment 21

[Christopher Van Wiemeersch [:cvan]]

https://github.com/mozilla/zamboni/commit/fa951c5