1364421 - unable to backfill or add new jobs for buildbot bridge job (linux64 talos, OSX *)

Status: RESOLVED FIXED

(Taskcluster :: General, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

I have been trying for the last hour to 'add new jobs' and 'backfill' for some missing talos data on:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&group_state=expanded&filter-searchStr=linux%20talos%20e10s&tochange=0515ebda07af3263ab124ba1a6eabe212e9e1b89&fromchange=a5e5a6e086f8689b1a481af2393a52deeca25e27

this is frustrating as performance is a priority item.  I would like to request that we close the trees until this is fixed.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:garndt, this looks to be a taskcluster issue, can you get someone on the taskcluster team to look into this?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

as a note, I can 'backfill' and 'add new jobs' for windows talos tests- so this looks to be exclusively related to taskcluster

Comment 3

[Greg Arndt [:garndt]]

So far this seems to be limited to buildbot-bridge jobs.

For the OS X and Linux jobs that were requested to be backfilled (both of which are BBB jobs), these errors appear within pulse_actions:
https://tc-gp-public-31d.s3-us-west-2.amazonaws.com/ateam/pulse-action-dev/9ea6fed3-63ab-402e-9e8b-1e9679e7d73d

There is the chance that backfilling is having trouble tracing a job back to the builder.

Investigation is ongoing and involves looking into : https://github.com/mozilla/mozilla_ci_tools/blob/master/mozci/platforms.py#L172

Comment 4

[Greg Arndt [:garndt]]

So far the investigation is pointing to the fact that builder schedulers are no longer defined for OS X and Linux because they either run in TC entirely (linux) or scheduled via BBB (OS X).

Linux has been this way for quite some time, and OS X was changed on May 4th.

There are some possibilities found in mozci and pulse_actions of things to change, but not 100% certain.

Comment 5

[Kris Maglione [:kmag]]

That would seem to suggest that this is not a new issue, so can we get trees reopened while it's being fixed?

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I would suggest turning SETA off and opening the trees until this is fixed.  It will greatly increase our load, but allow us to not depend on backfilling or adding arbitrary jobs.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: temporarily disable seta

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0f8c033cd3e9
temporarily disable SETA. r=bstack, a=CLOSED TREE

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/fec1331f50b8
temporarily disable SETA for BBB only. r=bstack, a=CLOSED TREE

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/96b36c5f527d
fix flake8 errors. r=me a=CLOSED TREE

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/0f8c033cd3e9
https://hg.mozilla.org/mozilla-central/rev/fec1331f50b8
https://hg.mozilla.org/mozilla-central/rev/96b36c5f527d

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/85477a7a8e1a956fd01acbe39ab09a5a5c625e77
Bug 1364421: actually disable SETA, instead of never running talos; r=bstack

Comment 13

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The last checkin has browser-chrome jobs running on OSX which didn't ran for the 24 hours before: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=85477a7a8e1a956fd01acbe39ab09a5a5c625e77&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/73b3fc64525b
actually disable SETA, instead of never running talos; r=bstack a=infra-fix

Comment 15

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/85477a7a8e1a

Comment 16

[Brian Stack [:bstack]]

Attachment: Bug 1364421 - Allow BBB jobs to be backfilled

Review commit: https://reviewboard.mozilla.org/r/141128/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/141128/

Comment 17

[GitHub Autolander Bot]

Attachment: [treeherder] imbstack:bug-1364421 > mozilla:master

Comment 18

[Greg Arndt [:garndt]]

Comment on attachment 8869544 [details]
Bug 1364421 - Allow BBB jobs to be backfilled

https://reviewboard.mozilla.org/r/141128/#review144710

Comment 19

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/3f5c3ff3db835616806cfc40b5c5b43466716901
Bug 1364421 - Allow bbb jobs to be backfilled (#2495) r=camd

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1cd72f93f155
Allow BBB jobs to be backfilled r=garndt

Comment 21

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

So, once the gecko patch merges to mozilla-central (sometime over the weekend or monday morning, most likely) and the Treeherder patch gets deployed to production (probably sometime on Monday), I think something can land to re-enable SETA for the affected jobs.

Another patch to allow BBB jobs to be triggered via the "Add New Jobs" feature in Treeherder would probably be good at some point, too. Jmaher can probably speak to whether backfilling alone is sufficient for reenabling SETA.

Comment 22

[Joel Maher ( :jmaher ) (UTC -8)]

we can work around add new jobs by using backfilling- so Monday I will get this enabled!

Comment 23

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/1cd72f93f155

Comment 24

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Can you verify that backfilling works from Treeherder stage? You'll need to be on a branch/push with that gecko commit on it (and probably need to have had it landed a few pushes earlier).

Comment 25

[Brian Stack [:bstack]]

I don't think we ever made backfilling work on stage. There's another patch for it floating around that I can try to land again. I broke everything the last time I tried to land it though. Might be bitrotted so I can try to make it work on Monday.

Comment 26

[GitHub Autolander Bot]

Attachment: [treeherder] imbstack:bug-1364421-pt-2 > mozilla:master

Comment 27

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/2281ec3d4b8028d9fe571d6bd8c7fc2d25cfb909
Bug 1364421 - Allow add-new-jobs to work for bbb

Comment 28

[Joel Maher ( :jmaher ) (UTC -8)]

should we go forward and enable SETA again?  I am not sure if all the pieces we know about are landed and fully deployed.

Comment 29

[Brian Stack [:bstack]]

afaict we should be good to re-enable seta. I believe the backfill patch for treeherder is in production and the add-new-jobs one is landed in master. As far as the in-tree patch is concerned, once it has been merged around into all of the branches it is done.

* This all assumes that there are no new bugs introduced by the changes of course.

But I think the next steps would be to have somebody with the permissions to backfill these jobs to try one out in the real world and then turn seta back on if it works!

Comment 30

[Greg Arndt [:garndt]]

(In reply to Joel Maher ( :jmaher) from comment #28)
> should we go forward and enable SETA again?  I am not sure if all the pieces
> we know about are landed and fully deployed.

SETA could be enabled again I think.  We've done about as much testing as I think we could at this point.  SETA/backfilling is pretty hard to test in a non-live environment.

Comment 31

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: re enable seta now that we have BBB backfill capabilities

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8b19b5019913
renable seta for BBB jobs. r=dustin

Comment 33

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/8b19b5019913

Comment 34

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.