All users were logged out of Bugzilla on October 13th, 2018

Create treeherder help docs for services (e2e) tests

RESOLVED DUPLICATE of bug 1213542

Status

RESOLVED DUPLICATE of bug 1213542
4 years ago
2 years ago

People

(Reporter: rpapa, Assigned: rpapa)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Per discussion between A-Team, Platform & Services QA, we'd like to begin the process of enabling reports from external test environments within treeherder.  This bug is intended to track creating documentation for that process.

NOTE:
We'll start with Loop and document the process so we have a guideline we can provide as we bring other services tests for Find-my-device, Firefox Accounts, etc. online.
Assignee: nobody → rpappalardo
Jonathan - 
Can you please provide links to some existing documentation we can use as a guideline - or that we would likely be adding to as we go through this process? If there is any documentation around sheriffing policy as well, that would be great.  thanks!
Flags: needinfo?(jgriffin)

Comment 2

4 years ago
(In reply to Richard Pappalardo [:rpapa][:rpappalardo] from comment #1)
> If there is any documentation around sheriffing policy as well,
> that would be great.  thanks!

https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy
I think we should document this at something like https://wiki.mozilla.org/Auto-tools/Adding_a_new_test_harness (Ed may have a better naming suggestion).  We currently don't have any specific docs related to this (other than the existing visibility guidelines above), so we'll have to flesh this out as we go.

Step 1 seems to be to get these tests reporting results to Treeherder, and from there we can figure out how Treeherder should display them (if different from current Tier 1 tests), what the sheriffing rules are, and what additional tooling (if any) we need to support that.

Cam and/or Ed, where would you recommend that someone starts when they start the process of adapting their tools to report test results to Treeherder?
Flags: needinfo?(jgriffin)
Flags: needinfo?(emorley)
Flags: needinfo?(cdawson)
thClient is the normal mechanism for submitting test results to Treeherder.  I think the docs for it could likely use some polish, however:

https://github.com/mozilla/treeherder-client

But that's the right place to start.  Initially, we like to test submission of data to our staging service (treeherder.allizom.org) and then once those work well, point them a production.
Flags: needinfo?(cdawson)

Comment 5

4 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> I think we should document this at something like
> https://wiki.mozilla.org/Auto-tools/Adding_a_new_test_harness (Ed may have a
> better naming suggestion).  We currently don't have any specific docs
> related to this (other than the existing visibility guidelines above), so
> we'll have to flesh this out as we go.

That wiki page name/location makes sense to me :-)

> Cam and/or Ed, where would you recommend that someone starts when they start
> the process of adapting their tools to report test results to Treeherder?

I'd probably speak to :lightsofapollo and :bc since they've been through this process already.
Flags: needinfo?(emorley)
Richard, please speak to :bc (Bob Clary) who has been through this process with Autophone, about where to start.  Please needinfo me on this bug if you need more information, Treeherder feature enhancements, etc.

Comment 7

4 years ago
The first attempt at documenting using Treeherder client using Autophone as an example is now at https://wiki.mozilla.org/Auto-tools/Reporting_Test_Results_to_Treeherder_for_non-buildbot_Test_Frameworks#Sheriffing

Comment 8

4 years ago
(In reply to Bob Clary [:bc:] from comment #7)
> The first attempt at documenting using Treeherder client using Autophone as
> an example is now at
> https://wiki.mozilla.org/Auto-tools/
> Reporting_Test_Results_to_Treeherder_for_non-
> buildbot_Test_Frameworks#Sheriffing

Do we think some of that content should be moved to the visibility policy page instead? It's going to very quickly get out of date otherwise :-)

Comment 9

4 years ago
(In reply to Ed Morley [:edmorley] from comment #8)

> 
> Do we think some of that content should be moved to the visibility policy
> page instead? It's going to very quickly get out of date otherwise :-)

Actually, I didn't mean to link directly to the Sheriffing section but to the whole document. The Sheriffing section doesn't contain anything that is not already in the Policy document except for the Tier distinction perhaps. It wasn't meant to be a replacement for the Policy document in any way but to serve as glue for the narrative. If you think it is better to remove the Tier 1/Tier 2 stuff and put it in the Policy document that would be fine with me.
Ah I see. My only concern is duplicating content, though I know it's not always possible to avoid it. The policy page could do with some cleanup (or more, changes now that we have tier 2 support in the treeherder UI), at which point directly linking to it may make more sense. Anyhow, happy to leave remaining feedback to the sheriffs to make :-)
See yesterday's auto-tools/sheriffs thread. Updating the policy is on my radar.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1213542
You need to log in before you can comment on or make changes to this bug.