1728869 - Probe Scraper failing since Aug 30th 2021

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P1)

Description

[William Lachance (:wlach)]

The probe-scraper run hasn't run successfully since Aug 30th 2021. It appears that part of the problem was an incorrectly applied PR which caused it to silently fail, which is now fixed:

https://github.com/mozilla/probe-scraper/pull/334

However, the most recent re-run has failed (non silently this time, at least!):

https://workflow.telemetry.mozilla.org/log?task_id=probe_scraper&dag_id=probe_scraper&execution_date=2021-09-02T15%3A00%3A05.724549%2B00%3A00

Unfortunately we don't have logs in airflow of what happened due to mozilla/telemetry-airflow#844. IIRC the workaround is to use stackdriver logging, but I don't think I have access to that. :whd, would you be able to retrieve some logs to tell us what might be happening?

Comment 1

[Wesley Dawson [:whd]]

You should be able to see these logs in stackdriver via e.g. https://cloudlogging.app.goo.gl/bDosHSe2HVYJM12b7. It looks like the third attempt of the job is running at with pod name probe-scraper-70738f73c8fd4f8b9aada7ecd34d2e27.

Comment 2

[William Lachance (:wlach)]

Attachment: error trace

Got an error log. It looks like we need to use cp rather than sync:

https://stackoverflow.com/questions/19745050/is-it-possible-to-sync-a-single-file-to-s3

Comment 3

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/probe-scraper/pull/336

Comment 4

[William Lachance (:wlach)]

I've merged a fix but we're basically at my end of day now. I think I'll just wait for the probe scraper to run normally tonight. I expect it to complete successfully.

Comment 5

[Jan-Erik Rediger [:janerik]]

It seems probe-scraper and schema deploys ran tonight. The mps deploy dashboard reports successful runs.
However we did receive an errro email for try 1 out of 3 at 00:14 UTC with Pod Launching failed: Pod Launching failed: There was an error reading the kubernetes API.
Seems try 2 succeeded then I guess.

Comment 6

[William Lachance (:wlach)]

(In reply to Jan-Erik Rediger [:janerik] from comment #5)

It seems probe-scraper and schema deploys ran tonight. The mps deploy dashboard reports successful runs.
However we did receive an errro email for try 1 out of 3 at 00:14 UTC with Pod Launching failed: Pod Launching failed: There was an error reading the kubernetes API.
Seems try 2 succeeded then I guess.

I checked in stackdriver and didn't see anything unusual happen in the first run, it seems likely that was an intermittent problem with the kubernetes <-> airflow component. Looking at mps-deploys I suspect this sort of thing happens occasionally looking at the logs (likely we're seeing this problem in the entries where there is a delay in the graph):

https://protosaur.dev/mps-deploys/

Comment 7

[William Lachance (:wlach)]

I checked in stackdriver and didn't see anything unusual happen in the first run, it seems likely that was an intermittent problem with the kubernetes <-> airflow component. Looking at mps-deploys I suspect this sort of thing happens occasionally looking at the logs (likely we're seeing this problem in the entries where there is a delay in the graph):

https://protosaur.dev/mps-deploys/

I just checked and the probe scraper job actually finished on schedule in the last few months (i.e. delays in deploys must have been due to other reasons).

I'm going to resolve this for now-- we can open a new bug if the problem recurs.