1293953 - Schedule the HW survey job (using Airflow?)

Status: RESOLVED FIXED

(Cloud Services :: Metrics: Product Metrics, defect, P1)

Description

[Alessio Placitelli [:Dexter]]

This bug is about scheduling the HW survey data extraction job. I'm filing a bug, rather than scheduling through a.t.m.o, as I want to investigate the possibility of scheduling it using Airflow.

Comment 1

[Alessio Placitelli [:Dexter]]

Can and should new jobs be scheduled using Airflow? If so... how to do that? :)

Comment 2

[Mark Reid [:mreid]]

Yes, if this is something we will rely upon externally, then we should definitely schedule it via Airflow. You'll need to sent a PR to the telemetry-airflow repo at [1].

Does the hardware survey data extraction make use of an existing derived dataset? Or does it query the raw pings directly?

The tl;dr if you're using the raw pings is that you will create a new DAG for the hardware survey similar to the existing ones in the "dags" directory of that repo. You can run a Jupyter notebook directly, or you can run a jar file, or you can run a shell script depending on how your code is organized.

If your code depends on an existing derived dataset, you can insert your job as a task in the DAG for that dataset.

[1] https://github.com/mozilla/telemetry-airflow

Comment 3

[Alessio Placitelli [:Dexter]]

Attachment: Schedule the HW survey job

Comment 4

[Alessio Placitelli [:Dexter]]

Andre, can the game hardware survey repo [1] be made public? Unfortunately, we can't schedule the job until this happens.

There's no data on the repo, just the website skeleton and the python notebook used to extract the data.

[1] - https://github.com/mozilla/firefox-hardware-survey

Comment 5

[Andre Vrignaud [:andre] [Seattle - PST]]

I don't see why not, but I think Rebecca owns it or knows who can make it public. I cannot myself. Rebecca, any concerns? And if not, can you point us in the right direction? Thanks!

Comment 6

[Alessio Placitelli [:Dexter]]

(In reply to Andre Vrignaud [:andre] [Seattle - PST] from comment #5)
> I don't see why not, but I think Rebecca owns it or knows who can make it
> public. I cannot myself. Rebecca, any concerns? And if not, can you point us
> in the right direction? Thanks!

I pinged Rebecca this morning and she said that there's no reason for the repository to be private.
We only need to make sure nothing compromising (WAU, MAU, privacy sensitive stuff) is available in the rev history.

She also mentioned that Ali should have enough privileges to make the repo public.
Looking at [1] it seems like we never committed data files, MAU, WAU or anything potentially sensible.

@Ali, would you kindly confirm and, if nothing was committed, make the repository public?

[1] - https://github.com/mozilla/firefox-hardware-survey/commits/master

Comment 7

[Alessio Placitelli [:Dexter]]

(In reply to Alessio Placitelli [:Dexter] from comment #6)
> We only need to make sure nothing compromising (WAU, MAU, privacy sensitive
> stuff) is available in the rev history.

I obviously meant NOT available in the rev history :-D

Comment 8

[Ali Almossawi]

Sure, done.

Comment 9

[[e-mail]]

I'll clear my NI given Comment #6.

Comment 10

[Alessio Placitelli [:Dexter]]

(In reply to Ali Almossawi from comment #8)
> Sure, done.

Thanks Ali!

Comment 11

[Alessio Placitelli [:Dexter]]

The job is scheduled to run weekly using Airflow.