Closed Bug 1078391 Opened 10 years ago Closed 10 years ago

ensure that talos e10s posts to different graph server / tree herder locations than non e10s

Categories

(Testing :: Talos, defect)

defect
Not set
normal

Tracking

(e10s+)

RESOLVED FIXED
Tracking Status
e10s + ---

People

(Reporter: jmaher, Assigned: jmaher)

References

Details

Attachments

(1 file)

e10s talos is posting to the same graphs as non e10s talos.  We need to adjust the test names to be unique for e10s.  

In the rare case that we plan to just switch regular off and only run e10s, then this shouldn't be a problem.
The results are stored by branch anyway, so the holly branch results will not intermix with m-c/i results anyway.

And once e10s is turned on by default, we WILL want to be able to compare pre/post e10-switch on the same graph because.. it's the same branch which changed.

So I don't quite get why we'll need different kind of reporting, other than the separation which happens automatically from using a different branch...
I don't know if we are switching over 100%.  if we are, lets close this as resolved wontfix; if we will run in parallel for atime, then lets not.  We do need to think about downstream branches as well.
I may be missing something, but as far as I understand, we have to consider before and after inbound switches to e10s.

1. Before (now), the holly branch results and m-c/m-i results don't mix due to the different branch.

2. After the switch to e10s, everything switches to e10s, and we will want to be able to compare, on the same branch and graphs (m-c/m-i), how the values changed from the switch to e10s, so we will want the (e10s-ified) m-c to report to the same place as the older results.

If the above covers the cases we're concerned about, then I don't see the point of this bug still.

So what am I missing?
I need confirmation that we will switch 100% from regular to e10s and I would be happy to close this.  If there is a chance that we will be running in parallel on a integration and mainline branch (inbound, fxteam, central, etc.), then we need to adjust the test names to support both.
Agreed that if we're going to test both e10s and non-e10s of the same branch, then some differentiation is required, and naming variant could address this.

However, how would that be different than us running the holly branch now in parallel to m-i/m-c? once we switch, if we still want to test the non-e10s variant, couldn't we just create an "unholy" branch to test the non-e10s variant?
so holly branch numbers are useless look at while we run in parallel.  I really don't know the goal here mostly because we are running mochitest in parallel on inbound and talos in parallel on holly.
How hard is this to do? I could easily imagine us wanting to run talos in both e10s and non-e10s modes on a single branch (which, as Joel points out, we are doing right now on holly). We might be able to get away without it, but it would be nicer to have it.
(In reply to Bill McCloskey (:billm) from comment #7)
> How hard is this to do?

Should be quite easy I'd imagine.

However, the main issue I'm trying to avoid is that once we switch, we won't be able to see the before/after numbers at the same graph. In other words, I want us to be able to compere before/after e10s switch numbers on the same graph - which truly reflects the evolution of the branch.

This is why I suggested that we maintain a branch with the opposite e10s config to m-c. Such that right now the holly branch tests only e10s (and its results on graphserver would appear under the holly branch, separate from m-c), and after m-c switches to e10s, either reuse the holly branch to test the non e10s config, or create a new branch for it.

> I could easily imagine us wanting to run talos in
> both e10s and non-e10s modes on a single branch (which, as Joel points out,
> we are doing right now on holly).

I must have missed the point where we already run both variants on the holly branch. Isn't the holly branch meant for testing the e10s config? and if yes, why do we run also non-e10s tests on it - which should be the same tests/results which we'd get on m-i/m-c anyway?
wlach, do you see anything wrong with disabling the regular talos runs on holly?  Then we could compare numbers more realistically between holly/inbound.
Flags: needinfo?(wlachance)
(In reply to Joel Maher (:jmaher) from comment #9)
> wlach, do you see anything wrong with disabling the regular talos runs on
> holly?  Then we could compare numbers more realistically between
> holly/inbound.

I don't see any problem. Go for it!
Flags: needinfo?(wlachance)
tracking-e10s: --- → +
Depends on: 1092194
simple hack here, also disabling a few tests so we can get to green- we still need to fix tests, but this gets us running!
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #8518148 - Flags: review?(wlachance)
Comment on attachment 8518148 [details] [diff] [review]
hack machine/platform names for e10s (1.0)

LGTM!
Attachment #8518148 - Flags: review?(wlachance) → review+
Merged: https://github.com/mozilla/treeherder-service/commit/8fae856593e630ec5de4acf9747383c19e152c20
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
(In reply to William Lachance (:wlach) from comment #13)
> Merged:
> https://github.com/mozilla/treeherder-service/commit/
> 8fae856593e630ec5de4acf9747383c19e152c20

Oops, that was for the wrong bug. Ignore.
Status: RESOLVED → UNCONFIRMED
Ever confirmed: false
Resolution: FIXED → ---
Blocks: 1094961
this is ready for deployment, will be updated this week on inbound.
Status: UNCONFIRMED → RESOLVED
Closed: 10 years ago10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: