432883 - talos config file unification and automation

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[alice nodelman [:alice] [:anode]]

We currently have talos production, stage and try config files checked with the associated buildbot code along with a standard config file in the talos code base.  Adding a single new pref can involve editing and updating upwards of 7 or 8 files along with then doing reconfigs on 3 different buildbots.

To stop the chaos we need:
1 - less config files.  All of the talos configs are nearly identical, it should be possible to have a base config file that is then altered as necessary
2 - automatic config file updates.  If a changes to a config file is checked in it should be automatically applied to the next talos run.

Comment 1

[alice nodelman [:alice] [:anode]]

I've got an initial patch put together here for having PerfConfigurator.py fully configure the checked in talos config template provided in the talos cvs directory (sample.config).

This would change the order of operations a little bit.  Instead of each talos machine downloading a full config file it would work from a checked out template that would be altered per test run.  This would have several benefits:

- make an alteration to the talos config template and have it propagate immediately with no further intervention 
- no more need to have 8 or 9 different talos config files checked in resulting in a simplification of the talos buildbot directory structure along with less likelyhood of copy/paste errors

Comment 2

[alice nodelman [:alice] [:anode]]

Attachment: [Checked in]PerfConfigurator additional options for graph server/controlling tests

Comment 3

[alice nodelman [:alice] [:anode]]

Attachment: [Checked in]have talos stage/production use the new PerfConfigurator options

Comment 4

[alice nodelman [:alice] [:anode]]

The idea behind this is to have a single copy of the talos config file checked into the talos cvs directory.  Each talos factory then just customizes that single config file on a per-test run basis (setting graph server values, turning on and off tests, setting chrome, etc).  In this way we could change the overall preferences that are tested with in a single check in.  This also eliminates a lot of risk from cut and paste errors.

As a phase 2 I'd like to rewrite how talos uses config files to try and make it simpler so that we wouldn't have to go through so many steps to get to a working configuration. 

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 331214 [details] [diff] [review]
[Checked in]PerfConfigurator additional options for graph server/controlling tests

stamp...

Comment 6

[alice nodelman [:alice] [:anode]]

Comment on attachment 331214 [details] [diff] [review]
[Checked in]PerfConfigurator additional options for graph server/controlling tests

Checking in PerfConfigurator.py;
/cvsroot/mozilla/testing/performance/talos/PerfConfigurator.py,v  <--  PerfConfigurator.py
new revision: 1.5; previous revision: 1.4
done

Comment 7

[alice nodelman [:alice] [:anode]]

Comment on attachment 331215 [details] [diff] [review]
[Checked in]have talos stage/production use the new PerfConfigurator options

Checking in perfmaster2/perfrunner.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/perfrunner.py,v  <--  perfrunner.py
new revision: 1.22; previous revision: 1.21
done
Checking in perfmaster2/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/master.cfg,v  <--  master.cfg
new revision: 1.75; previous revision: 1.74
done
Checking in perf-staging/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perf-staging/master.cfg,v  <--  master.cfg
new revision: 1.19; previous revision: 1.18
done

Comment 8

[alice nodelman [:alice] [:anode]]

Attachment: [Checked in]remove sample.configs for talos production/stage

No longer necessary now that the other patches have landed.  If you're keeping count that is 6 less checked in talos config files.

Comment 9

[alice nodelman [:alice] [:anode]]

Attachment: [Checked in]same as before, but for talos try (update to master.cfg/perfrunner)

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 332811 [details] [diff] [review]
[Checked in]remove sample.configs for talos production/stage

awesome.

Comment 11

[alice nodelman [:alice] [:anode]]

Comment on attachment 332811 [details] [diff] [review]
[Checked in]remove sample.configs for talos production/stage

Removing perfmaster2/configs/fast.production.sample.config;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/fast.production.sample.config,v  <--  fast.production.sample.config
new revision: delete; previous revision: 1.7
done
Removing perfmaster2/configs/jss.production.sample.config;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/jss.production.sample.config,v  <--  jss.production.sample.config
new revision: delete; previous revision: 1.6
done
Removing perfmaster2/configs/production.sample.config;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/production.sample.config,v  <--  production.sample.config
new revision: delete; previous revision: 1.9
done
Removing perfmaster2/configs/production.sample.config.nochrome;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/production.sample.config.nochrome,v  <--  production.sample.config.nochrome
new revision: delete; previous revision: 1.6
done
Removing perfmaster2/configs/production.sample.config.nogfx;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/production.sample.config.nogfx,v  <--  production.sample.config.nogfx
new revision: delete; previous revision: 1.9
done
Removing perf-staging/configs/sample.config;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perf-staging/configs/sample.config,v  <--  sample.config
new revision: delete; previous revision: 1.6
done

Comment 12

[alice nodelman [:alice] [:anode]]

Comment on attachment 332829 [details] [diff] [review]
[Checked in]same as before, but for talos try (update to master.cfg/perfrunner)

cvs commit: Examining configs
Checking in perfrunner.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/tryperfmaster/perfrunner.py,v  <--  perfrunner.py
new revision: 1.9; previous revision: 1.8
done
Checking in master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/tryperfmaster/master.cfg,v  <--  master.cfg
new revision: 1.6; previous revision: 1.5
done
Removing configs/sample.config;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/tryperfmaster/configs/sample.config,v  <--  sample.config
new revision: delete; previous revision: 1.6
done

Comment 13

[alice nodelman [:alice] [:anode]]

Applied cleanly to the production try server.

That would be a total of 7 config files removed.  Down to one centralized talos config file that all talos factories use.  Should be very easy in future to make alterations (ie, additional prefs, more tests, etc) and have them take effect immediately.

Comment 14

[alice nodelman [:alice] [:anode]]

This caused regression bug 450401 due to an inadvertant change from tp-cycles 10 to tp-cycles 5.  Fixed in bug 457582.