508972 - [AMO] Process old paypal stats

Status: RESOLVED FIXED

(Mozilla Metrics :: Data/Backend Reports, defect)

Description

[Wil Clouser [:clouserw]]

We integrated paypal into AMO with 5.0.7.  We've got some rudimentary stat gathering built in but you can do it better and faster than us so it would be more appropriate if metrics just did all of it.  This bug is about extracting what we can from our old logs and filling in the stats_contributions table.

Between when 5.0.7 launched (July 15) and now there are URLs like this in the logs:

https://addons.mozilla.org/en-US/firefox/addon/$addon_id?________________

where _____________ is a gigantic pile of data from paypal.  The only thing we care about in the data from paypal is mc_gross.

Steps for this bug:

1) Parse the logs from July 15 -> now for that format, extracting $addon_id and $mc_gross from the URL.  Run something like:

INSERT INTO stats_contributions(addon_id, amount, created) VALUES ($addon_id, $mc_gross, now());

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

Will work on implementing tomorrow and Monday 2009-08-17.

Comment 2

[Nick Nguyen [:osunick]]

Any luck?

Comment 3

[Daniel Einspanjer [:dre] [:deinspanjer]]

The back processing is ready and will be deployed and run tomorrow morning.  I expect that it will take two days of running to complete the month of data from July 15 to Aug 15.

Comment 4

[Daniel Einspanjer [:dre] [:deinspanjer]]

Guys, the backprocessing is failing to find any records that match the format described in comment #0.  Maybe I'm just missing something obvious, but I can't grep the string "mc_gross" in any requests out of the addons.mozilla.org log files from 2009-07-15-00 through 2009-07-20-23.

Comment 5

[Wil Clouser [:clouserw]]

I thought we had access logs mounted on cm-vpn01 but I don't see them.  Without the logs I can't do anything with this.  CCing oremj

Comment 6

[Daniel Einspanjer [:dre] [:deinspanjer]]

Actually, I take it back, my grep had not gotten all the way through
2009-07-20-23 yet.  I found my first hit on 2009-07-20-18.  A donation of
$3.00.
I'm re-running the grep just to make sure, but what level of traffic are we
talking about here?  I assumed when you said that you wouldn't be able to
handle logging the Paypal redirect requests to the DB as they happened that we
were dealing with a lot of traffic.  Could it actually be one hit in a week or
am I missing some other format or some other location for the traffic?

Comment 7

[Wil Clouser [:clouserw]]

(In reply to comment #6)
> Actually, I take it back, my grep had not gotten all the way through
> 2009-07-20-23 yet.  I found my first hit on 2009-07-20-18.  A donation of
> $3.00.
> I'm re-running the grep just to make sure, but what level of traffic are we
> talking about here?  I assumed when you said that you wouldn't be able to
> handle logging the Paypal redirect requests to the DB as they happened that we
> were dealing with a lot of traffic.  Could it actually be one hit in a week or
> am I missing some other format or some other location for the traffic?

One hit a week is low but I don't think it is a lot of traffic, no.  I would expect double or triple digits per day right now.

Comment 8

[Daniel Einspanjer [:dre] [:deinspanjer]]

I found a total of twelve donations before 2009-08-06, three to one add-on, and nine to another add-on.  Oddly, four of the nine donations were all for the same amount and approximately the same time of day on each successive day from July 30th to August 4th.

These twelve entries in the stats_contributions table as described above.
Please let me know if you think I could be missing other traffic.

Comment 9

[Wil Clouser [:clouserw]]

(In reply to comment #8)
> I found a total of twelve donations before 2009-08-06, three to one add-on, and
> nine to another add-on.  Oddly, four of the nine donations were all for the
> same amount and approximately the same time of day on each successive day from
> July 30th to August 4th.
> 
> These twelve entries in the stats_contributions table as described above.
> Please let me know if you think I could be missing other traffic.
I can't tell you without access to the logs.  I've filed bug 511906 but if you could send me some logs I can look.