1735512 - telemetry-probes bot unexpectedly stopped creating bugs for expiring telemetry probes

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The telemetry-probes bot stopped warning about telemetry probes which will expire in upcoming Firefox versions. It was handy because I didn't have to create the bugs (but only did so for probes whose expiry broke tests).

Did Ben kick this task off manually or was it bound to his credentials? Frank had been the reviewer for recent changes.

:relud, could you investigate this, please?

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

:glob mentioned the account's last login was yesterday.

Comment 2

[Daniel Thorn [:relud]]

the scheduled job for this is completing without error, but it says creating 13 new bugs and then later says The following bugs were filed for the above probes: (and doesn't list any). I'll see if I can get it working again.

Comment 3

[Daniel Thorn [:relud]]

bugs are only created on "wednesday" runs (which don't happen until UTC thursday) https://github.com/mozilla/probe-scraper/blob/main/probe_scraper/probe_expiry_alert.py#L397-L398

    # Only send create bugs on Wednesdays, run the rest for debugging/error detection
    dryrun = dryrun or current_date.weekday() != 2

The run for October 6th found ['PRINT_BACKGROUND_TASK_TIME_MS', 'PRINT_BACKGROUND_TASK_ROUND_TRIP_TIME_MS'] to be expiring, and already had bugs.

:aryx is this a misunderstanding about when bugs are created, or are you also expecting the run tonight to not create any bugs?

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

There are more expiring probes, e.g. pictureinpicture.most_concurrent_players and they likely will get reported tomorrow. There was an issue updating https://product-details.mozilla.org/ last week but I was under the impression this only affected the data for releases.

I will close this bug on Friday if the expected bugs get created.

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

No bugs have been created during the last run. There are probes which expire once the Gecko version number gets increased to 96 on 2021-11-01, e.g. cache.integrity_check_count. Scalars.yaml and Histograms.json have more expiring probes, nothing in Events.yaml expires.

Comment 6

[Daniel Thorn [:relud]]

I will look into it, but unfortunately I'll be working on a different pressing issue for much of this week before I can get to this.

Comment 7

[Daniel Thorn [:relud]]

For my later reference: this is the error that bug creation is failing with:

creating 13 new bugs
Created bug {'id': 1736627} for HTTP*
Created bug {'id': 1736628} for CLIENT_CERTIFICATE_SCAN_TIME
Created bug {'id': 1736629} for HTMLEDITORS_WITH_*
Created bug {'id': 1736630} for ENCODING_*
Created bug {'id': 1736631} for BROWSER_SET_DEFAULT_USER_CHOICE_RESULT
Created bug {'id': 1736632} for PWMGR_NUM_FORM_HAS_POSSIBLE_USERNAME_EVENT_PER_DOC
Failed to create bugs with arguments: {'product': 'Core', 'component': 'Audio/Video: Playback', 'summary': 'Remove or update probes expiring in Firefox 96: WEB_AUDIO_AUTOPLAY', 'description': "\nThe following Firefox probes will expire in the next major Firefox nightly release: version 96 [1].\n\n```\nWEB_AUDIO_AUTOPLAY\n```\n\n\n\nWhat to do about this:\n1. If one, some, or all of the metrics are no longer needed, please remove them from their definitions files (Histograms.json, Scalars.yaml, Events.yaml).\n2. If one, some, or all of the metrics are still required, please submit a Data Collection Review [2] and patch to extend their expiry.  There is a shorter form for data collection renewal [3].\n\nIf you have any problems, please ask for help on the #data-help Slack channel or the #telemetry Matrix room at https://chat.mozilla.org/#/room/#telemetry:mozilla.org. We'll give you a hand.\n\nYour Friendly, Neighborhood Telemetry Team\n\n[1] https://wiki.mozilla.org/Release_Management/Calendar\n[2] https://wiki.mozilla.org/Firefox/Data_Collection\n[3] https://github.com/mozilla/data-review/blob/master/renewal_request.md\n\nThis is an automated message sent from probe-scraper.  See https://github.com/mozilla/probe-scraper for details.\n", 'version': 'unspecified', 'type': 'task', 'whiteboard': '[probe-expiry-alert]', 'see_also': [1700828], 'flags': [{'name': 'needinfo', 'type_id': 800, 'status': '?', 'requestee': '[e-mail]'}, {'name': 'needinfo', 'type_id': 800, 'status': '?', 'requestee': '[e-mail]'}], 'cc': []}
Error response: {"code":32000,"error":true,"message":"Alastor Wu [:alwu][PTO until Nov.1] <[e-mail]> is not currently accepting \"needinfo\" requests.","documentation":"https://bmo.readthedocs.io/en/latest/api/"}
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/app/probe_scraper/probe_expiry_alert.py", line 457, in <module>
    main(args.date, args.dry_run, args.bugzilla_api_key)
  File "/app/probe_scraper/probe_expiry_alert.py", line 438, in main
    probe_to_bug_id = file_bugs(
  File "/app/probe_scraper/probe_expiry_alert.py", line 389, in file_bugs
    bug_id = create_bug(probe_group, version, bugzilla_api_key)
  File "/app/probe_scraper/probe_expiry_alert.py", line 232, in create_bug
    create_response.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://bugzilla.mozilla.org/rest/bug

Comment 8

[Daniel Thorn [:relud]]

I believe https://github.com/mozilla/probe-scraper/pull/351 should fix this

Comment 9

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Thank you. A heads-up the expiring probes which will cause test failures will get manually created bugs to buy some time to get them fixed by end of next week (before the version increases). I will set the automatic ones as duplicate or into See Also if they are covered by such a bug.

Comment 10

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/probe-scraper/pull/352

Comment 11

[Daniel Thorn [:relud]]

it took a second PR, but this is completing successfully now, and the remaining bugs were created.