1764332 - Airflow task probe_scraper.probe_scraper failing on 2022-04-12

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[anicholson]

The Airflow task probe_scraper.probe_scraper failed on 2022-04-12

Comment 1

[anicholson]

Stackdriver: link.

Here's the error log:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/app/probe_scraper/runner.py", line 647, in <module>
    main(
  File "/app/probe_scraper/runner.py", line 540, in main
    load_glean_metrics(
  File "/app/probe_scraper/runner.py", line 295, in load_glean_metrics
    commit_timestamps, repos_metrics_data, emails = git_scraper.scrape(
  File "/app/probe_scraper/scrapers/git_scraper.py", line 191, in scrape
    ts, commits = retrieve_files(repo_info, folder)
  File "/app/probe_scraper/scrapers/git_scraper.py", line 129, in retrieve_files
    hashes = get_commits(repo, rel_path)
  File "/app/probe_scraper/scrapers/git_scraper.py", line 89, in get_commits
    commit, timestamp = entry.strip('"').split(sep)
ValueError: not enough values to unpack (expected 2, got 1)

Comment 2

[Jan-Erik Rediger [:janerik]]

Fails after Cloning https://github.com/mozilla-extensions/bergamot-browser-extension into /app/probe_cache/bergamot.git

Running it locally reproduces it:

python -m probe_scraper.runner --out-dir tmp/out --cache-dir tmp/cache --glean --glean-repo bergamot --dry-run

https://github.com/mozilla-extensions/bergamot-browser-extension now redirects to https://github.com/mozilla-extensions/firefox-translations
I think it didn't before?

Comment 3

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/probe-scraper/pull/413

Comment 4

[Jan-Erik Rediger [:janerik]]

See PR for a potential quick fix.
What we really should do is stop scanning deprecated repositories (like bergamot is). See #401

Comment 5

[Daniel Thorn [:relud]]

mozilla schema generator is now failing with:

items removed from the base
-org-mozilla-bergamot.custom.1.txt

Comment 6

[Alessio Placitelli [:Dexter]]

(In reply to Daniel Thorn [:relud] from comment #5)

mozilla schema generator is now failing with:

items removed from the base
-org-mozilla-bergamot.custom.1.txt

We should probably have an end to end look at this machinery.
Does the above message imply we cannot stop scraping products? If so, that seems rather a strong assumption from the generator side :-D

Jeff, I figured you should probably be involved in this, given that probe-scraper is such a critial piece of our infrastructure that deals both with legacy and glean data.

Comment 7

[Daniel Thorn [:relud]]

Does the above message imply we cannot stop scraping products?

correct, we do not support removing products from probe info service.

Comment 8

[Wesley Dawson [:whd]]

This was also discussed when the replacement for the bergamot app was introduced in https://github.com/mozilla/probe-scraper/pull/400. I believe we have plans to replace probe scraper with a push model but I don't think that has any effect on the invariant "we do not support removing products from probe info service". I think we can soften this invariant a bit by saying that we support removing schemas for products as long as those datasets are being deleted, see bug #1689513 / https://github.com/mozilla/mozilla-schema-generator/blob/main/incompatibility-allowlist#L10 for an example where we did this.

Since this invariant is long-standing and schemas deploys are blocked until the invariant is satisfied, I don't think it's reasonable to have resolution of this issue be predicated on changing the invariant. Instead, we should investigate the following options:

1. Figure out why the ping is now missing and how to modify probe scraper configuration to re-introduce it
   Looking at repositories.yaml, I see the URL for bergamot is at https://github.com/mozilla-extensions/bergamot-browser-extension and its ping files are located src/core/ts/background-scripts/background.js/telemetry/pings.yaml. Visiting this URL redirects me to https://github.com/mozilla-extensions/firefox-translations, which does not appear to have the relevant path in its history at all. I strongly suspect therefore that in the last few days, this repo was renamed or replaced in some way, and that the original (deprecated) bergamot extension version that included the custom ping is now missing. This hypothesis is strengthened by finding an old fork of the repo from Evgeny that has the file: https://github.com/eu9ene/bergamot-browser-extension/blob/main/src/core/ts/background-scripts/background.js/telemetry/pings.yaml.

Therefore I think one solution here might be to reintroduce a public archive of the older version of the code, based on e.g. Evgeny's fork, and point repositories.yaml at that. NI :evgeny and :anatal to see if the above representations about the github repo changing are accurate and if setting up a public archive is reasonable.

2. Proceed with deleting the source data for the old bergamot version

See https://mana.mozilla.org/wiki/pages/viewpage.action?spaceKey=SRE&title=Deleting+Source+Data and the google doc it links to. This procedure has a couple of steps to it and is somewhat involved and I think we should probably investigate (1) first, even if we decided to delete the old app's data.

Comment 9

[Alessio Placitelli [:Dexter]]

(In reply to Wesley Dawson [:whd] from comment #8)

This was also discussed when the replacement for the bergamot app was introduced in https://github.com/mozilla/probe-scraper/pull/400. I believe we have plans to replace probe scraper with a push model but I don't think that has any effect on the invariant "we do not support removing products from probe info service".

Yes, I agree with that.

I think we can soften this invariant a bit by saying that we support removing schemas for products as long as those datasets are being deleted, see bug #1689513 / https://github.com/mozilla/mozilla-schema-generator/blob/main/incompatibility-allowlist#L10 for an example where we did this.

I filed bug 1764701 to discuss the general problem about this invariant.

Since this invariant is long-standing and schemas deploys are blocked until the invariant is satisfied, I don't think it's reasonable to have resolution of this issue be predicated on changing the invariant. Instead, we should investigate the following options:

1. Figure out why the ping is now missing and how to modify probe scraper configuration to re-introduce it

The repo has a legacy branch that contains the old version of the data. I'll file a PR to the probe-scraper to point it to that branch instead of main. Clearing the ni? for the other folks, I'll flag them on the PR for review.

Comment 10

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/probe-scraper/pull/415

Comment 11

[Wesley Dawson [:whd]]

The PR :Dexter filed introduced a regression but :relud fixed it in https://github.com/mozilla/probe-scraper/pull/417. I've re-run the probe scraper DAG and MSG succeeded, so this issue is resolved.

Comment 12

[Evgeny Pavlov]

Hey folks, just to clarify what repos we have: https://github.com/mozilla/firefox-translations is the only relevant repo of project bergamot. https://github.com/mozilla-extensions/firefox-translations is a mirror, just for the deployment of the web extension, it is a rename of https://github.com/mozilla-extensions/bergamot-browser-extension and yes, it used to contain a legacy version of the extension that was referenced by the first version of telemetry. bergamot-browser-extension should be eventually removed from the probe scrapper as soon as we have such capability.