1108259 - Create treeherder help docs for services (e2e) tests

Status: RESOLVED DUPLICATE of bug 1213542

(Testing :: General, defect)

Description

[Richard Pappalardo [:rpapa][:rpappalardo]]

Per discussion between A-Team, Platform & Services QA, we'd like to begin the process of enabling reports from external test environments within treeherder.  This bug is intended to track creating documentation for that process.

NOTE:
We'll start with Loop and document the process so we have a guideline we can provide as we bring other services tests for Find-my-device, Firefox Accounts, etc. online.

Comment 1

[Richard Pappalardo [:rpapa][:rpappalardo]]

Jonathan - 
Can you please provide links to some existing documentation we can use as a guideline - or that we would likely be adding to as we go through this process? If there is any documentation around sheriffing policy as well, that would be great.  thanks!

Comment 2

[Ed Morley [:emorley]]

(In reply to Richard Pappalardo [:rpapa][:rpappalardo] from comment #1)
> If there is any documentation around sheriffing policy as well,
> that would be great.  thanks!

https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy

Comment 3

[Jonathan Griffin (:jgriffin)]

I think we should document this at something like https://wiki.mozilla.org/Auto-tools/Adding_a_new_test_harness (Ed may have a better naming suggestion).  We currently don't have any specific docs related to this (other than the existing visibility guidelines above), so we'll have to flesh this out as we go.

Step 1 seems to be to get these tests reporting results to Treeherder, and from there we can figure out how Treeherder should display them (if different from current Tier 1 tests), what the sheriffing rules are, and what additional tooling (if any) we need to support that.

Cam and/or Ed, where would you recommend that someone starts when they start the process of adapting their tools to report test results to Treeherder?

Comment 4

[Cameron Dawson [:camd]]

thClient is the normal mechanism for submitting test results to Treeherder.  I think the docs for it could likely use some polish, however:

https://github.com/mozilla/treeherder-client

But that's the right place to start.  Initially, we like to test submission of data to our staging service (treeherder.allizom.org) and then once those work well, point them a production.

Comment 5

[Ed Morley [:emorley]]

(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> I think we should document this at something like
> https://wiki.mozilla.org/Auto-tools/Adding_a_new_test_harness (Ed may have a
> better naming suggestion).  We currently don't have any specific docs
> related to this (other than the existing visibility guidelines above), so
> we'll have to flesh this out as we go.

That wiki page name/location makes sense to me :-)

> Cam and/or Ed, where would you recommend that someone starts when they start
> the process of adapting their tools to report test results to Treeherder?

I'd probably speak to :lightsofapollo and :bc since they've been through this process already.

Comment 6

[Jonathan Griffin (:jgriffin)]

Richard, please speak to :bc (Bob Clary) who has been through this process with Autophone, about where to start.  Please needinfo me on this bug if you need more information, Treeherder feature enhancements, etc.

Comment 7

[Bob Clary [:bc] (inactive)]

The first attempt at documenting using Treeherder client using Autophone as an example is now at https://wiki.mozilla.org/Auto-tools/Reporting_Test_Results_to_Treeherder_for_non-buildbot_Test_Frameworks#Sheriffing

Comment 8

[Ed Morley [:emorley]]

(In reply to Bob Clary [:bc:] from comment #7)
> The first attempt at documenting using Treeherder client using Autophone as
> an example is now at
> https://wiki.mozilla.org/Auto-tools/
> Reporting_Test_Results_to_Treeherder_for_non-
> buildbot_Test_Frameworks#Sheriffing

Do we think some of that content should be moved to the visibility policy page instead? It's going to very quickly get out of date otherwise :-)

Comment 9

[Bob Clary [:bc] (inactive)]

(In reply to Ed Morley [:edmorley] from comment #8)

> 
> Do we think some of that content should be moved to the visibility policy
> page instead? It's going to very quickly get out of date otherwise :-)

Actually, I didn't mean to link directly to the Sheriffing section but to the whole document. The Sheriffing section doesn't contain anything that is not already in the Policy document except for the Tier distinction perhaps. It wasn't meant to be a replacement for the Policy document in any way but to serve as glue for the narrative. If you think it is better to remove the Tier 1/Tier 2 stuff and put it in the Policy document that would be fine with me.

Comment 10

[Ed Morley [:emorley]]

Ah I see. My only concern is duplicating content, though I know it's not always possible to avoid it. The policy page could do with some cleanup (or more, changes now that we have tier 2 support in the treeherder UI), at which point directly linking to it may make more sense. Anyhow, happy to leave remaining feedback to the sheriffs to make :-)

Comment 11

[Ryan VanderMeulen [:RyanVM]]

See yesterday's auto-tools/sheriffs thread. Updating the policy is on my radar.

Comment 12

[Ed Morley [:emorley]]

Yeah I saw that, I meant specifally the duplicated content at https://wiki.mozilla.org/Auto-tools/Reporting_Test_Results_to_Treeherder_for_non-buildbot_Test_Frameworks#Sheriffing