1683750 - Airflow bqetl_monitoring DAG not up-to-date

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, defect)

Description

[Anna Scholtz [:ascholtz]]

The bqetl_monitoring DAG doesn't list the monitoring__average_ping_sizes__v1 task in the web interface, The task was added last week via bigquery-etl: https://github.com/mozilla/bigquery-etl/blob/b726bf739ebc448031266463fde00776bc8d3e19/dags/bqetl_monitoring.py#L24

Comment 1

[Harold Woo]

Something may be up with the gh-pages branch

error: cannot lock ref 'refs/remotes/origin/gh-pages': Unable to create '/app/pvmount/bigquery-etl/.git/refs/remotes/origin/gh-pages.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
From https://github.com/mozilla/bigquery-etl
! 383e5d53..7b905fb9 gh-pages -> origin/gh-pages (unable to update local ref)
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.

Comment 2

[Harold Woo]

I've fixed the dag file mount git issue for now, but will track root cause in https://bugzilla.mozilla.org/show_bug.cgi?id=1669591. Quick fix is to manually deploy a sidecar container (because needs root), and to rm -rf the bigquery-etl git folder so the next sync will do a clean checkout