1664309 - Migrate mobile/desktop mau databricks jobs

Status: RESOLVED FIXED

(Data Platform and Tools :: General, task, P2)

Description

[Ben Wu [:benwu]]

The jobs Mobile Product MAU for 2020 and Desktop MAU & DAU in Tier1 are scheduled on databricks and should be moved to something else if still needed. There's also the FileStore - Embed Images notebook scheduled that may be relevant but I don't have access to it.

Xuan, are these still needed going forward?

If needed, we'll probably do this along with bug 1664234

Comment 1

[Xuan Luo]

How much pain takes to keep them? We still use them from time to time :) If too much trouble, I will then figure out some other alternatives.

Comment 2

[Ben Wu [:benwu]]

I'll add this to the list of things to look at. I don't expect that this will be a lot of work to turn into a script that outputs a static webpage

Comment 3

[Xuan Luo]

Cool, let me know if I can help.

Comment 4

[Ben Wu [:benwu]]

Xuan, do these still need to be migrated now that we're at the end of 2020? I would guess not but I'm working on migrating them to protosaur dashboards using docker-etl.

Dashboards: https://protosaur.dev/desktop-mau-dau-tier1-2020/ https://protosaur.dev/mobile-mau-2020/
WIP code: https://github.com/mozilla/docker-etl/tree/benwu/mau-dashboards/jobs/desktop_mobile_mau_2020/

If these don't need to continue I'd still like to use them as a proof of concept for docker-etl + protosaur and have a big "don't use this data" disclaimer at the top. This could be an interesting pattern in the future.

Comment 5

[Xuan Luo]

Hey Ben, sounds good. Let's not migrate them in the interest of time and effort need to be taken.

Part of the mobile mau data is available in another dashboard already; also we might move away from MAU/DAU for reporting in 2021.

Thanks for checking in :)

Comment 6

[Ben Wu [:benwu]]

Test dashboards now up and updated daily.

Code here: https://github.com/mozilla/docker-etl/tree/main/jobs/desktop-mobile-mau-2020