1275589 - Treeherder shows old builder on runnable apis

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect)

Description

[Armen [:armenzg]]

I will investigate why.

May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 New jobs requested by [e-mail] for https://treeherder.mozilla.org/#/jobs?repo=try&revision=bf8885abac03d7698d819975b53a3c19fb300aed 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 List of builders: 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test cppunit 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test gtest 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test marionette 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test jittest 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test web-platform-tests-reftests 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test xpcshell 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Firefox try win64 l10n 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-1 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-2 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-3 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-4 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-5 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-jetpack 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-browser-chrome-1 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-browser-chrome-2 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-browser-chrome-3 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-browser-chrome-4 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-browser-chrome-5 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-browser-chrome-6 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-browser-chrome-7 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-clipboard 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-devtools-chrome-1 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-devtools-chrome-2 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-devtools-chrome-3 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-devtools-chrome-4 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-devtools-chrome-5 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-devtools-chrome-6 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-devtools-chrome-7 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-devtools-chrome-8 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-gl 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-gpu 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-media 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-other 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-browser-screenshots 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-e10s-browser-chrome-1 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-e10s-browser-chrome-2 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-e10s-browser-chrome-3 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-e10s-browser-chrome-4 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-e10s-browser-chrome-5 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-e10s-browser-chrome-6 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-e10s-browser-chrome-7 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-clipboard-e10s 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test mochitest-gpu-e10s 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test crashtest 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test jsreftest 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test reftest 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try opt test reftest-no-accel 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - WINNT 6.1 x86-64 try spidermonkey_try-compacting build 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - WINNT 6.1 x86-64 try spidermonkey_try-plain build 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try debug test mochitest-gpu-e10s 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try debug test mochitest-clipboard-e10s 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try debug test web-platform-tests-1 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try debug test web-platform-tests-2 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try debug test web-platform-tests-3 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try debug test web-platform-tests-5 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try debug test web-platform-tests-4 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try debug test web-platform-tests-6 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try debug test web-platform-tests-7 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try debug test web-platform-tests-8 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try debug test web-platform-tests-9 
May 25 07:37:43 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_runnable	 INFO:	 - Windows 8 64-bit try debug test web-platform-tests-10 
May 25 07:37:55 pulse-actions app/worker2.1: mozci	 ERROR:	 Buildername Windows 8 64-bit try debug test mochitest-gpu-e10s is *NOT* valid.

Comment 1

[Kalpesh Krishna [:martianwars]]

That's not a valid buildername right?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

the valid buildername is:
 Windows 8 64-bit try opt test mochitest-gpu-e10s

we do not have a debug version- there was one temporarily and there is one on the Ash branch, but only opt for now.

Comment 3

[Kalpesh Krishna [:martianwars]]

So the bug isn't with pulse_actions right? Should I check Treeherder or something?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

it could be a bad timing thing- it was a valid job and now it is not- maybe a job was triggered based on old data. Can we hold off and see if this shows up again?  maybe we end up resolving this in a few days as worksforme!

Comment 5

[Armen [:armenzg]]

The builder was removed on the 19th (6 days ago), thus, it should not show up on the allthethings.json of the last few days.
The problem is that the builder still exists in the runnable api [2][3]

camd: why would 'Windows 8 64-bit try opt test mochitest-gpu-e10s' still show up on the runnable api?

[1] https://secure.pub.build.mozilla.org/builddata/reports/allthethings/allthethings.20160519110001._069074312b20_d3c55d9b6b9f_30db9839b8b5.differences.txt
[2] https://treeherder.mozilla.org/api/project/try/runnable_jobs/?format=json
[3]
    {
      "build_system_type": "buildbot",
      "job_group_symbol": "M-e10s",
      "job_group_name": "Mochitest e10s",
      "platform_option": "opt",
      "job_type_description": "",
      "result": "runnable",
      "ref_data_name": "Windows 8 64-bit try opt test mochitest-gpu-e10s",
      "machine_platform_architecture": "x86_64",
      "job_type_id": 6661,
      "build_platform": "windows8-64",
      "job_type_name": "Mochitest e10s GPU",
      "platform": "windows8-64",
      "state": "runnable",
      "machine_platform_id": 45,
      "build_os": "win",
      "option_collection_hash": "102210fe594ee9b33d82058545b1ed14f4c8206e",
      "job_type_symbol": "gpu",
      "job_group_description": "fill me",
      "job_coalesced_to_guid": null,
      "machine_platform_os": "win",
      "build_architecture": "x86_64",
      "build_platform_id": 45,
      "job_group_id": 9
    },

Comment 6

[Kalpesh Krishna [:martianwars]]

Yeah this might be because TH's database is not updated instantly, but once every day or something like that

Comment 7

[Cameron Dawson [:camd]]

Hmm, rechecking this morning and it's still there.  I'm wondering if we are not updating our cache properly where we store this data...  I'll check into this.

Comment 8

[Armen [:armenzg]]

Thank you Cameron!

Comment 9

[Armen [:armenzg]]

FYI I can help mentor a system that will generate a new allthethings.json every time there is a change that would cause it.
If this would help please let me know.
I could even send you a pulse message notifying you of the new one!

Comment 10

[Cameron Dawson [:camd]]

I can see the entry in the ``runnable_job`` table for this buildername.  So, for some reason, we're not actually pruning it when it doesn't exist in ``allthethings``.  Looking at the code, it doesn't appear that we do pruning.  But I can see that it DID ``touch`` the record today, during the process.  Not sure why it did that.  Still investigating.

Comment 11

[Cameron Dawson [:camd]]

ahh, so that buildername DOES show up in the list we fetch:

We use:
ALLTHETHINGS_URL = "https://secure.pub.build.mozilla.org/builddata/reports/allthethings.json"

And that buildername exists in that list.  Is that URL no longer valid?  Or not getting updated for some reason?

Comment 12

[Armen [:armenzg]]

camd, are you loading the URL on your browser? Could it be cached?
I did a search again and the builder does not show up.

The URL is valid. The builder is not showing up for me and should have not for the last week.

This returns that it is not there:
MacAir ~ $ wget -O- -q https://secure.pub.build.mozilla.org/builddata/reports/allthethings.json | grep "Windows 8 64-bit try debug test mochitest-gpu-e10s"

Comment 13

[Cameron Dawson [:camd]]

Ah hah!  You were searching for
Windows 8 64-bit try debug test mochitest-gpu-e10s

But I was searching for:
Windows 8 64-bit try opt test mochitest-gpu-e10s

So, yeah.  The debug one doesn't show for me, but the opt one does.

Comment 14

[Cameron Dawson [:camd]]

I got that from comment 5, fwiw.

Comment 15

[Armen [:armenzg]]

My apologies about that mistake!
Let me know what comes up.

Comment 16

[Cameron Dawson [:camd]]

Sorry, I sounded much more jerk-ish than I meant to in comment 13.  

But I believe this was just a timing thing, as jmaher said.  If I search the JSON from:
https://treeherder.mozilla.org/api/project/try/runnable_jobs/?format=json

for:

Windows 8 64-bit try debug test mochitest-gpu-e10s

It is not there, which is correct.  The "opt" version is, which is also correct.  So I think the problem was that the request to trigger came between the debug buildername being removed and the runnable_jobs table being updated.  At this point, the data is correct, wrt that buildername.

The best we could do would be to shorten the interval between running that process to update the runnable_jobs table.  Perhaps once every 6 hours?  There will always be a possibility of it being out of date unless we update every minute or something (not recommended).  Even then, I guess we could get unlucky in that minute...  :)

Or we can accept that this will occasionally happen and not worry about it.

Which one seems best to you?  It's probably pretty safe to at least shorten the interval to every 4 or 6 hours.

Comment 17

[Armen [:armenzg]]

You never sound jerkish!

Let's start with 4 hours.

However, would it help if we notified you through a pulse message that a new allthethings.json is available?

Comment 18

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:fetch-allthethings-sooner > mozilla:master

Comment 19

[Cameron Dawson [:camd]]

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #17)
> You never sound jerkish!
> 
> Let's start with 4 hours.
> 
> However, would it help if we notified you through a pulse message that a new
> allthethings.json is available?

Oh good.  :)  OK, cool.  Here's a PR to change it to every 4 hours.  I don't think that'll be too much stress on our system.  :)

Comment 20

[Armen [:armenzg]]

camd: I'm also interested to know why it was still lingering after 24 hours.

I'm still getting too many errors for other builders in the last day [1]

Instead of a schedule, would you like me to provide you with a pulse message and point you to the latest allthethings.json?

[1]
May 27 20:40:49 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try debug test mochitest-media is *NOT* valid.

May 27 17:47:35 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try opt test web-platform-tests-5 is *NOT* valid.

May 27 17:47:35 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try opt test marionette is *NOT* valid.

May 27 17:47:36 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try opt test web-platform-tests-4 is *NOT* valid.

May 27 17:47:36 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try opt test web-platform-tests-3 is *NOT* valid.

May 27 17:47:37 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try opt test web-platform-tests-2 is *NOT* valid.

May 27 17:47:37 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try opt test web-platform-tests-1 is *NOT* valid.

May 27 17:47:38 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try opt test mochitest-media is *NOT* valid.

May 27 17:47:40 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try opt test mochitest-devtools-chrome-8 is *NOT* valid.

May 27 17:47:40 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try opt test mochitest-devtools-chrome-7 is *NOT* valid.

May 27 17:47:40 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try opt test mochitest-devtools-chrome-6 is *NOT* valid.

May 27 17:47:41 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try opt test mochitest-devtools-chrome-5 is *NOT* valid.

May 27 17:47:41 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try opt test mochitest-devtools-chrome-4 is *NOT* valid.

May 27 17:47:42 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try opt test mochitest-devtools-chrome-3 is *NOT* valid.

May 27 17:47:42 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try opt test mochitest-devtools-chrome-2 is *NOT* valid.

May 27 17:47:43 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try opt test mochitest-devtools-chrome-1 is *NOT* valid.

May 27 17:47:52 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try opt test web-platform-tests-reftests is *NOT* valid.

May 27 17:47:52 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try debug test web-platform-tests-10 is *NOT* valid.

May 27 17:47:52 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try debug test web-platform-tests-8 is *NOT* valid.

May 27 17:47:53 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try debug test web-platform-tests-9 is *NOT* valid.

May 27 17:47:53 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try debug test web-platform-tests-7 is *NOT* valid.

May 27 17:47:53 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try debug test web-platform-tests-6 is *NOT* valid.

May 27 17:47:53 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try debug test web-platform-tests-4 is *NOT* valid.

May 27 17:47:54 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try debug test web-platform-tests-5 is *NOT* valid.

May 27 17:47:54 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try debug test web-platform-tests-3 is *NOT* valid.

May 27 17:47:54 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try debug test web-platform-tests-2 is *NOT* valid.

May 27 17:47:54 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try debug test web-platform-tests-1 is *NOT* valid.

May 27 17:47:56 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try debug test mochitest-media is *NOT* valid.

May 27 17:47:57 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try debug test mochitest-devtools-chrome-8 is *NOT* valid.

May 27 17:47:57 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try debug test mochitest-devtools-chrome-7 is *NOT* valid.

May 27 17:47:58 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try debug test mochitest-devtools-chrome-5 is *NOT* valid.

May 27 17:47:58 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try debug test mochitest-devtools-chrome-6 is *NOT* valid.

May 27 17:47:59 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try debug test mochitest-devtools-chrome-4 is *NOT* valid.

May 27 17:47:59 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try debug test mochitest-devtools-chrome-3 is *NOT* valid.

May 27 17:48:00 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try debug test mochitest-devtools-chrome-2 is *NOT* valid.

May 27 17:48:01 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 VM-GFX 32-bit try debug test mochitest-devtools-chrome-1 is *NOT* valid.

May 27 17:48:09 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try debug test web-platform-tests-reftests is *NOT* valid.

May 27 17:48:10 pulse-actions app/worker2.1: mozci ERROR: Buildername Windows 7 32-bit try debug test marionette is *NOT* valid.

Comment 21

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/416086c75ccbc9e423ecf35ad65cd9766bd60b02
Bug 1275589 - Shorten refresh duration of runnable jobs list to 4 hours

Comment 22

[Armen [:armenzg]]

camd: here's few other builders that should not show up on runnable:
Windows 7 32-bit try opt test web-platform-tests-*
Linux x86-64 mozilla-beta leak test spidermonkey_tier_1-arm64-sim build
Linux x86-64 mozilla-beta leak test spidermonkey_tier_1-rootanalysis build

Just to be clear anyone reading along, this is not an issue of frequency of imports but rather an issue of old builders still showing up.

Do we have any logic to drop old builders from the runnable apis?

Comment 23

[Cameron Dawson [:camd]]

Armen: ok, sounds like we need another approach.  n-i'ing myself for visibility.  I'll check the code on monday as to how it does the pruning of that list.  Something's obviously not right there...

Comment 24

[Cameron Dawson [:camd]]

Armen--  OK, this is what's happening:

https://github.com/mozilla/treeherder/blob/5d9e430cacd912a41d0a887f6ac66a53874cab81/treeherder/webapp/api/runnable_jobs.py#L26-L26

Based on this, it will show all the builders that existing within the last week.  So that explains why the old ones aren't being expunged...

There's no comment as to WHY it's done this way.  Perhaps there was some thought that if you wanted to run an older job for an older push that you should be allowed to do so.  But it won't work that way.  If a buildername isn't in allthethings, you can't run it at all, even for an older push.  or... correct me if I'm wrong...  :)

So I think I can:

1. get the current time prior to ``touch``ing any buildernames  in the RunnableJobsProcess
2. go ahead and ``touch`` or create buildernames that exist in allthethings
3. do a delete on any buildername with last_touched older than the time from #1
4. remove that time filter from the api

Comment 25

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:runnable-jobs-prune > mozilla:master

Comment 26

[Cameron Dawson [:camd]]

My patch isn't quite working yet in the unit test.  But I'll finish it Monday.

Comment 27

[Cameron Dawson [:camd]]

Comment on attachment 8762201 [details] [review]
[treeherder] mozilla:runnable-jobs-prune > mozilla:master

Finished the unit test for this.  I think this should fix the problem.  We may want to see if we want to run this process more often.  Or, optionally, we could schedule this task every time we encounter a new buildername.  Though that may be overkill.  I figure we can see if this addresses things adequately.  

Should we consider refreshing the list every hour or 1/2 hour though?

Comment 28

[Ed Morley [:emorley]]

Comment on attachment 8762201 [details] [review]
[treeherder] mozilla:runnable-jobs-prune > mozilla:master

> Should we consider refreshing the list every hour or 1/2 hour though?

The task currently takes over 8 mins to run (likely longer with the deletes added by this PR), I think every hour might be too much?

Comment 29

[Cameron Dawson [:camd]]

Cool cool.  Yeah, it's currently at every 4 hours.  Let's see if that's sufficient.  :)

Comment 30

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/7e26d1251003c0d8fe26009013c4cd0c5a4ae2bf
Bug 1275589 - Prune runnable jobs (#1587)

* Bug 1275589 - Prune runnable jobs

Prior to this fix, the list of runnable jobs would be pruned only when
the normal cycle-data process ran.  This is not fast enough, however.

Now, defunct buildernames are removed every time the runnable jobs are
updated.

* test fix

* fix

Comment 31

[Cameron Dawson [:camd]]

Oops, I didn't realize it would keep the commit text from the squashed commits.  I'll do that differently next time...  Or try the autosquash syntax...

This issue should be fixed now.

Comment 32

[Ed Morley [:emorley]]

I think the web UI lets you edit the squashed messages? So may still be able to use that feature when actually squashing :-)

Comment 33

[Armen [:armenzg]]

Thanks camd. I will look at this carefuly by end of the week if that works out.

Comment 34

[Armen [:armenzg]]

Comment on attachment 8762201 [details] [review]
[treeherder] mozilla:runnable-jobs-prune > mozilla:master

My apologies for the delay. It's hard to concentrate at the all hands.

I don't really understand the code, however, I've checked few things.

"Linux x86-64 mozilla-beta leak test spidermonkey_tier_1-arm64-sim build" does not show anymore in here [1]

Papertrail does not show anymore "Buildername %(buildername)s is *NOT* valid" in the logs. If it appears again I will let you know.

4 hours should be sufficient.

Thank you so much for fixing this!

[1] https://treeherder.mozilla.org/api/project/mozilla-beta/runnable_jobs/