589167 - Run ts only once per changeset (remove ts either from the cold or the dirty test suite)

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Shawn Wilsher :sdwilsh]

It seems, at least on the places branch, and maybe mozilla-central, that I get more than one run of "Ts" for non-windows platforms.  At least, the graph server has more than one results stored for it.

Comment 1

[[e-mail] (:bhearsum)]

Alice, any idea what's going on here? Catlee noticed that both the "cold" and "dirty" builders report some of the same tests. This "dirty" log reports:
TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":64,"branch":1,"machine":566}]'>ts_cold_generated_min: 7987.16</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":61,"branch":1,"machine":566}]'>ts_cold_generated_min_shutdown: 414.89</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":66,"branch":1,"machine":566}]'>ts_cold_generated_max: 8258.37</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":63,"branch":1,"machine":566}]'>ts_cold_generated_max_shutdown: 558.42</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":16,"branch":1,"machine":566}]'>ts: 475.05</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":36,"branch":1,"machine":566}]'>ts_shutdown: 400.11</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":65,"branch":1,"machine":566}]'>ts_cold_generated_med: 8289.05</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":62,"branch":1,"machine":566}]'>ts_cold_generated_med_shutdown: 520.32</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":54,"branch":1,"machine":566}]'>ts_places_generated_max: 485.58</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":51,"branch":1,"machine":566}]'>ts_places_generated_max_shutdown: 347.42</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":53,"branch":1,"machine":566}]'>ts_places_generated_med: 481.79</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":50,"branch":1,"machine":566}]'>ts_places_generated_med_shutdown: 352.68</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":52,"branch":1,"machine":566}]'>ts_places_generated_min: 481.89</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":49,"branch":1,"machine":566}]'>ts_places_generated_min_shutdown: 359.58</a><br>


And this "cold" log reports:
TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":16,"branch":1,"machine":687}]'>ts: 470.32</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":36,"branch":1,"machine":687}]'>ts_shutdown: 405.74</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":55,"branch":1,"machine":687}]'>ts_cold: 7406.21</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[{"test":56,"branch":1,"machine":687}]'>ts_cold_shutdown: 446.84</a><br>

Comment 2

[alice nodelman [:alice] [:anode]]

Looks like you get Ts twice (once as part of the cold set, once as part of the dirty set).  I think that I included it in each set to make a quick baseline before running the more exotic ts tests.  We can easily remove it from one or the other.

Comment 3

[Chris AtLee [:catlee]]

(In reply to comment #2)
> Looks like you get Ts twice (once as part of the cold set, once as part of the
> dirty set).  I think that I included it in each set to make a quick baseline
> before running the more exotic ts tests.  We can easily remove it from one or
> the other.

Does Ts in the 'dirty' suite run with all the dirty profiles loaded?

Comment 4

[alice nodelman [:alice] [:anode]]

Ts in the dirty suite is still just Ts.  If has a different name (like ts_generated_min) if is using a dirty profile.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Tweaking summary, and pushing over to Testing after discussions above.

Comment 6

[alice nodelman [:alice] [:anode]]

I don't know what the bug is here - they are both running just straight ts before starting the more complicated versions.  There are not two ts tests that mean different things, if that is what is being implied.

Comment 7

[Shawn Wilsher :sdwilsh]

(In reply to comment #6)
> I don't know what the bug is here - they are both running just straight ts
> before starting the more complicated versions.  There are not two ts tests that
> mean different things, if that is what is being implied.
I suppose the bug is that more work is being done than needed, right?

Comment 8

[alice nodelman [:alice] [:anode]]

Attachment: run ts once per-changeset

This patch removes ts from the cold + dirty suites.  Ts will only run as part of the standard chrome tests.

Comment 9

[Chris AtLee [:catlee]]

Comment on attachment 469623 [details] [diff] [review]
run ts once per-changeset

>diff -r 7bd7b20145ed mozilla-tests/config.py
>--- a/mozilla-tests/config.py	Thu Aug 26 16:17:07 2010 -0400
>+++ b/mozilla-tests/config.py	Thu Aug 26 14:42:53 2010 -0700
>@@ -17,11 +17,11 @@
> TALOS_ADDON_OPTS = {'addonTester' : True, 'plugins': 'zips/plugins.zip', 'pageset': 'zips/pagesets.zip'}
> 
> SUITES = {
>-    'chrome': GRAPH_CONFIG + ['--activeTests', 'tdhtml:twinopen:tsspider:tgfx'],
>+    'chrome': GRAPH_CONFIG + ['--activeTests', 'ts:tdhtml:twinopen:tsspider:tgfx'],

Did you mean to remove it from here too?

Comment 10

[Chris AtLee [:catlee]]

Comment on attachment 469623 [details] [diff] [review]
run ts once per-changeset

Checked in as changeset:   2933:bfbffec2105e, and masters reconfigured.

Comment 11

[alice nodelman [:alice] [:anode]]

This is done.