1903320 - Create dedicated task routes for Pernosco

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Andrew Halberstadt [:ahal]]

The Pernosco service interfaces with Firefox-CI via pulse exchanges. Currently, they listen to all tasks on the taskcluster-queue/v1/task-completed exchange, grab the task definition and filter out to only the tasks that are supported then run their analysis on them.

However, this means their consumers need to process ~10k tasks per hour and if their consumers ever fall over or there's some network problems preventing messages from being delivered, it starts to cause problems in our RabbitMQ instances which can bring the whole Firefox-CI Taskcluster instance to a halt. We're planning some longer term improvements to make sure this won't happen.

But in the shorter term, a simpler thing we can do is to create a dedicated pernosco route on only the tasks that are supported, and only when the PERNOSCO env is enabled. This should reduce the volume of tasks from many tens of thousands of tasks per day, to merely dozens. In effect, it would be moving the logic around which tasks to run the analysis on from the Pernosco side over to the Mozilla side.

The Pernosco pulse consumers would need to be updated to use this new exchange rather than the queue service's exchange. Though we can leave the PERNOSCO env in place for now to preserve backwards compatibility such that the Pernosco consumers can be updated at a later time.

Comment 1

[Andrew Halberstadt [:ahal]]

This is blocked on https://github.com/taskcluster/taskcluster/pull/7069/files being deployed to fxci (currently scheduled June 27th, 2024).

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1903320 - Create a dedicated task route for pernosco, r?#taskgraph-reviewers!

This sets up a route on Pernosco tasks such that they will emit pulse messages
over the notify service's exchange with a dedicated routing key. This will allow
the Pernosco pulse consumer to receive only tasks that should be recorded.

Comment 3

[Andrew Halberstadt [:ahal]]

Once this lands, I'm going to need to uplift this to all branches.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/730762ad2098
Create a dedicated task route for pernosco, r=taskgraph-reviewers,jcristau

Comment 5

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/730762ad2098

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1903320 - Create a dedicated task route for pernosco,

This sets up a route on Pernosco tasks such that they will emit pulse messages
over the notify service's exchange with a dedicated routing key. This will allow
the Pernosco pulse consumer to receive only tasks that should be recorded.

Original Revision: https://phabricator.services.mozilla.com/D215967

Comment 7

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: Higher risk of Taskcluster outages
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: N/A
• Risk associated with taking this patch: None
• Explanation of risk level: automation only
• String changes made/needed: N/A
• Is Android affected?: no

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1903320 - Create a dedicated task route for pernosco,

This sets up a route on Pernosco tasks such that they will emit pulse messages
over the notify service's exchange with a dedicated routing key. This will allow
the Pernosco pulse consumer to receive only tasks that should be recorded.

Original Revision: https://phabricator.services.mozilla.com/D215967

Comment 9

[Phabricator Automation]

release Uplift Approval Request

• User impact if declined: Higher risk of Taskcluster outages
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: N/A
• Risk associated with taking this patch: None
• Explanation of risk level: automation only
• String changes made/needed: N/A
• Is Android affected?: no

Comment 10

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1903320 - Create a dedicated task route for pernosco,

This sets up a route on Pernosco tasks such that they will emit pulse messages
over the notify service's exchange with a dedicated routing key. This will allow
the Pernosco pulse consumer to receive only tasks that should be recorded.

Original Revision: https://phabricator.services.mozilla.com/D215967

Comment 11

[Phabricator Automation]

esr128 Uplift Approval Request

• User impact if declined: Higher risk of Taskcluster outage
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: N/A
• Risk associated with taking this patch: None
• Explanation of risk level: automation only
• String changes made/needed: N/A
• Is Android affected?: no

Comment 12

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1903320 - Create a dedicated task route for pernosco,

This sets up a route on Pernosco tasks such that they will emit pulse messages
over the notify service's exchange with a dedicated routing key. This will allow
the Pernosco pulse consumer to receive only tasks that should be recorded.

Original Revision: https://phabricator.services.mozilla.com/D215967

Comment 13

[Phabricator Automation]

esr115 Uplift Approval Request

• User impact if declined: Higher risk of Taskcluster outage
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: N/A
• Risk associated with taking this patch: None
• Explanation of risk level: automation only
• String changes made/needed: N/A
• Is Android affected?: no

Comment 14

[Andrew Halberstadt [:ahal]]

This bug is changing the interface between mozilla-central and Pernosco. Because the Pernosco consumer lives out of tree, we need this change to land to all branches.

If we don't land to all branches, then the Pernosco consumer will need to continue using the old interface (which has been responsible for some TC outages due to causing OOMs in our RabbitMQ instance for pulse). Alternatively, not taking these patches would mean developers would no longer be able to use Pernosco on older branches.

Comment 15

[Donal Meehan [:dmeehan]]

Are you planning on attaching more work here, wondering why it's set to leave-open?

Comment 16

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/ce80cd3264b8

Comment 17

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/0c2c5365d46c

Comment 18

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/d952083a9064

Comment 19

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/00fe75ebb47e

Comment 20

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr128/rev/d760a32dbbef

Comment 21

[Andrew Halberstadt [:ahal]]

Nope, guess I set it to wait for the uplifts to land.. there's work that needs to block on this landing everywhere.

Thanks for landing!