1541917 - Linux 32 and Mac shippable test schedules not SETA optimized, high test loads

Status: RESOLVED FIXED

(Testing :: Performance, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

I'm guessing this is fallout from the shippable builds work. Can't be good for our backlogs.

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Linux 32 and macOS are not SETA optimized which causes this for all tests on those platforms. The macOS pool can't keep up with the load and autoland will be closed to reduce the backlog.

Comment 2

[Justin Wood (:Callek)]

So, it looks like there are zero linux32 opt tests in LOW VALUE SETA right now, which I'm not sure how that is possible...

https://treeherder.mozilla.org/api/project/mozilla-inbound/seta/job-priorities/?build_system_type=taskcluster&priority=5

So there is no way that https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/util/seta.py#98 ends up matching it.

OSX similarly has no macosx64/opt in low value seta.

neither low value seta nor high value seta have any shippable specified. -- I'm passing this over the wall to Joel for this part of the investigation.

:jmaher, can you help identify what's going on here?

Comment 3

[Justin Wood (:Callek)]

Attachment: Bug 1541917 - Mac Talos tests are running on every push to integration branches. r=jmaher

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

we need to hack the seta database inside of treeherder, there is a table there you can query from redash(select the treeherder database):
select * from seta_jobpriority where expiration_date<'2019-05-01';

if we remove all these entries, I believe we will be good- I am not 100% sure. The bulk of the shippable tests expire on April 13th and would then default to low value, that is a long time from now :)

Once removed, there is a 1 hour cache inside of treeherder, so changes will not be immediate and can take up to 1 hour to show up.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

:kthiessen, while I have pinged you in irc, this is the bug where we need to solve the problem, I would like to solve this today so we can go into next week with cleaned up jobs.

Comment 6

[Karl Thiessen [:kthiessen, he/him]]

I can look at this sometime late morning Pacific today, but I'd recommend pulling :camd in as back-up, in case I can't make progress right away. I'm also uncomfortable with making direct database queries on prod without some sort of "what if" plan.

Comment 7

[Cameron Dawson [:camd]]

Yeah, I'll run the cleanup on Treeherder. We came up with this process:

1. Use db query to select jobs that will be deleted:
   SELECT * FROM seta_jobpriority WHERE expiration_date < '2019-05-01';
2. Verify results with jmaher
3. Create delete query based off that:
   DELETE from seta_jobpriority WHERE expiration_date < '2019-05-01';
4. Test deleting from prototype
   a. Run select query
   b. Save data off to .csv in MySQLWorkbench
   c. Run delete query
   d. Test Re-import records from .csv
5. Test deleting from stage
6. Verify with jmaher that stage is working as expected
7. Save entries locally from production to .csv
8. Make a snapshot on RDS for production data
9. Delete records from production
10. Monitor

Comment 8

[Cameron Dawson [:camd]]

These records have been deleted. I have them saved locally, so they can be restored if need be. I'll attach them here, too.

Comment 9

[Cameron Dawson [:camd]]

Attachment: jobpriority_prod.csv

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

it appears that we still have these job names in the jobpriority field. This looks to be a system that takes all unknown jobs which are not in the table and adds a expiration_date of 2 weeks; I assumed that deleting the rows would solve the problem.

Unfortunately I don't know the best solutions for this- I would bet that we could do something like:
update jobpriority set expiration_date=null, priority=5 where expiration_date<'2019-05-01'

:camd, do you think we can chat today and do this work cycle one more time?

Comment 11

[Cameron Dawson [:camd]]

Joel and I met over vidyo, reset the expiration date to null on those jobs and then reran analyze_failures. It appears to have fixed the issue.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a024c4370f28
Mac Talos tests are running on every push to integration branches. r=jmaher

Comment 13

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/a024c4370f28