810374 - try_spidermonkey jobs are running on non-try slaves

Status: RESOLVED INVALID

(Release Engineering :: General, defect, P2)

Description

[Chris AtLee [:catlee]]

"Linux x86-64 try leak test spidermonkey_try-rootanalysis build" is happening on non-try slaves...also on non-try masters.

Comment 1

[Chris AtLee [:catlee]]

Oh, and the jobs from different try pushes are getting merged together...which makes the results pretty meaningless I think.

Comment 2

[Steve Fink [:sfink] [:s:]]

Attachment: Cross-check builders and slaves to make sure the security levels do not get mixed up

This should really be carved up in the opposite order, but this adds a (failing) test to check that no access level 1 jobs are going to level 3 slaves.

Comment 3

[Steve Fink [:sfink] [:s:]]

Attachment: Record the access_level required for builders, for cross-checking security settings

buildbotcustom portion of the access checks

Comment 4

[Steve Fink [:sfink] [:s:]]

Attachment: Use the spidermonkey slaves passed in as a parameter rather than pulling them from the platform_config

Fix spidermonkey builds to use appropriate slaves, part 1.

Comment 5

[Steve Fink [:sfink] [:s:]]

Attachment: Pass in the appropriate slaves to the spidermonkey builds

Comment 6

[Steve Fink [:sfink] [:s:]]

(In reply to Chris AtLee [:catlee] from comment #0)
> "Linux x86-64 try leak test spidermonkey_try-rootanalysis build" is
> happening on non-try slaves...also on non-try masters.

non-try masters? Oh, the masters are segregated too?

Hm... so my access_level stuff is probably junk.

How are things separated across masters? Are production_localconfig.py and try_localconfig.py supposed to only generate the relevant builders, but both go through scheduler_master.cfg?

Then how are the non-spidermonkey projects avoiding the try scheduler master? Ugh, maybe I don't want to know.

Well, assuming my suppositions are correct, patch upcoming...

Comment 7

[Steve Fink [:sfink] [:s:]]

Attachment: split up projects by try and non-try

Thanks to nthomas for explaining the various master types to me, and which config file corresponds to each. With my shiny new understanding, this patch seems reasonable.

In brief, so I have it written down somewhere electronically:

buildbot starts up a master using buildbot.tac
 -> loads master.cfg
    - one of 3 roles: builder, scheduler, tests
    - symlink to {builder,scheduler,tests}_master.cfg
 -> loads master_localconfig.py
    - for builder|scheduler: either try or non-try
      - symlink {try,build}_localconfig.py
    - for tests: always symlink to tests_localconfig.py
 -> loads config.py
    - either mozilla/config.py (for builders + schedulers)
      or mozilla-tests.py (for tests)
 -> loads localconfig.py
    - symlink to {production,preproduction,staging}_config.py
    - (separate for tests and build/scheduler)

...except that master.cfg actually loads config.py *before* master_localconfig.py, which means that you can use the stuff defined in config.py within the code of master_localconfig.py (important for this patch!), even though the import of config.py from master_localconfig.py would suggest otherwise (and that import doesn't really do anything.)

I will note that neither "local" nor "config" really adds much when you're already within a module named "buildbot-configs", and using "build" in both {try,build}_localconfig.py is rather confusing when there's also {builder,scheduler,tests}_master.cfg. </whine>

Comment 8

[Steve Fink [:sfink] [:s:]]

(In reply to Chris AtLee [:catlee] from comment #1)
> Oh, and the jobs from different try pushes are getting merged
> together...which makes the results pretty meaningless I think.

I haven't addressed this yet. Ok, I haven't even looked to see what this means, given that I know the jobs are failing due to yet another $PATH problem anyway.

Comment 9

[Steve Fink [:sfink] [:s:]]

Comment on attachment 680859 [details] [diff] [review]
Cross-check builders and slaves to make sure the security levels do not get mixed up

Argh, I keep naming my patches the same and bzexport makes them clobber each other.

Comment 10

[Steve Fink [:sfink] [:s:]]

http://hg.mozilla.org/build/buildbot-configs/rev/f24e3b0a81ba

Comment 11

[Steve Fink [:sfink] [:s:]]

Followup bustage fix -
http://hg.mozilla.org/build/buildbot-configs/rev/40fd8b9ca7b0

Comment 12

[Steve Fink [:sfink] [:s:]]

Turns out that last patch cannot be landed separately, because it triggers the try build master to start generating jobs for production slaves, which it doesn't know about. I guess all of the non-test patches need to land together.

http://hg.mozilla.org/build/buildbot-configs/rev/18de55f7ec08 - backed out f24e3b0a81ba
http://hg.mozilla.org/build/buildbot-configs/rev/531a24d02699 - backed out changeset 40fd8b9ca7b0 (bug 810374)

Comment 13

[Chris AtLee [:catlee]]

Disabled the jobs completely until we can get this fixed up:

http://hg.mozilla.org/build/buildbot-configs/rev/b6405077b019

Comment 14

[Steve Fink [:sfink] [:s:]]

Comment on attachment 680891 [details] [diff] [review]
split up projects by try and non-try

I backed out the patch (and the followup fix), so now nothing is checked in.

aki clued me into test-masters.sh, which passes with all of these patches applied.

Comment 15

[Steve Fink [:sfink] [:s:]]

I think everything in this bug has been handled in separate bugs now.

Comment 16

[Steve Fink [:sfink] [:s:]]

Comment on attachment 680859 [details] [diff] [review]
Cross-check builders and slaves to make sure the security levels do not get mixed up

Garbage collecting dead review requests.