1450108 - Process Dev and Stage AMO nginx logs using EMR Mango

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, enhancement, P1)

Description

[Jason Thomas [:jason]]

AMO stats related changes are difficult to test today because we do not process and import stats data into these environments. With the EMR mango setup we can process these logs and have the available for the environments.

We would need to:

* Start shipping nginx logs on a daily basis to a S3 bucket
* Configure S3 ACLs between environments
* Configure mango log processing job to process the logs in the S3 bucket

Comment 1

[:wezhou]

Hi Jason,

Let us know what we need to do from amo perspective. (maybe create sub tickets of this one and assign to me?)

From amo perspective, we prefer that each our environment ships logs to a different S3 bucket. It's clearer for us to configure that way.

Thanks.

Comment 2

[Jason Thomas [:jason]]

Moving to P2 until bug 1451029 is completed.

Comment 3

[:wezhou]

Add a note to myself to update https://github.com/mozilla/addons-server/issues/5207 when this is done.

Comment 4

[Jason Thomas [:jason]]

I've begun work on this and realized that there are some days when there are no downloads or update pings for these environments.

We should drive some traffic to these environments and simulate add-on download and updates. 

:mat do you have a preference for which add-on and theme we use here? Maybe the qa team has some input here?

Comment 5

[Mathieu Pillard [:mat]]

Ideally, it would be several add-ons.

5-10 add-ons of each type (addontypeid in 1, 3, 5, 10 are the ones that matter the most for extensions) that are public (status = 4, _current_version is NOT NULL) would be ideal. To get you started you could pick the last x ones matching those criteria and ORDER BY average_daily_users DESC, then hardcode their ids (since the average_daily_users value is going to change once you're done)

Comment 6

[Jason Thomas [:jason]]

> Ideally, it would be several add-ons. 

I am going to file another bug for this request.


https://github.com/mozilla/telemetry-airflow/pull/320 for airflow dag addition.
https://github.com/mozilla-services/weblog_processing/pull/9 to add support for multiple amo environments.

Comment 7

[Jason Thomas [:jason]]

https://github.com/mozilla/addons-server/pull/8783 for addons-server changes

https://github.com/mozilla-services/cloudops-deployment/pull/2209 for puppet and config changes.

Comment 8

[Jason Thomas [:jason]]

https://workflow.telemetry.mozilla.org/admin/airflow/tree?dag_id=mango_log_processing_amo_dev_stage for log processing status.

Deployed changes to addons-dev, stats are available https://addons-dev.allizom.org/en-US/firefox/addon/multi-account-containers/statistics/?last=30

Comment 9

[:wezhou]

Hi Jason, I'm getting a 400 error when accessing the telemetry link above.

```
The redirect URI in the request, https://workflow.telemetry.mozilla.org/oauth2callback?next=%2Fadmin%2Fairflow%2Ftree%3Fdag_id%3Dmango_log_processing_amo_dev_stage, does not match the ones authorized for the OAuth client. 
```

I may need READ permission for that URL.

Also could you remind me the dag_id for -prod?

Comment 10

[Jason Thomas [:jason]]

You should already have access. Can you try https://workflow.telemetry.mozilla.org first and then going to that page? mango_log_processing_amo  is prod.

Comment 11

[:wezhou]

Thanks Jason. It works for me now.