1200501 - Move reftest analyzer from hg.mozilla.org to Treeherder

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder, defect)

Description

[Gregory Szorc [:gps]]

We want to disable serving of appropriate Content-Type headers from hg.mozilla.org. Apparently reftest analyzer may be relying on this feature to run directly from hg.mozilla.org. If so, we need to find new hosting for it.

First thing is first: does the reftest analyzer run from hg.mozilla.org? If so, what's the scope of this change? If not, this bug can likely be marked INVALID.

Comment 1

[Gregory Szorc [:gps]]

botond was kind enough to link me to https://treeherder.mozilla.org/logviewer.html#?job_id=1094928&repo=mozilla-aurora which has an "open analyzer" link that sure enough opens up to a raw-file link on hg.mozilla.org. This will fail to work in the future.

I'll just reassign this to Treeherder. Hopefully enough reftest people saw these comments and will know to chime in, if necessary.

Comment 2

[Cameron McCormack (:heycam)]

There's nothing that requires the reftest analyzer to be run from hg.mozilla.org.  As you say Botond said, that's just where we currently link to it (and presumably we do it this way so we can always link to the latest version).  The reftest analzyer doesn't have any particular dependency on running on hg.mozilla.org; it's purely client side, apart from an XHR to ftp.mozilla.org (which sends Access-Control-Allow-Origin: *).

Comment 3

[Ed Morley [:emorley]]

Treeherder links to the reftest analyser on hg.m.o in two places - the logviewer and the job details panel in the main UI:
https://github.com/mozilla/treeherder/search?q=reftest-analyzer.xhtml

Fixing this is easy - find somewhere else to host the reftest analyzer and update those two links.

Harder will be deciding where to host it. Ideally we'd just move the source to say the Treeherder repo and host directly, however I'm guessing some people use the tool locally from their mozilla-central checkout. This means we would have to copy rather than move it, then have to decide which location is the canonical one, and hope people remember to sync between the two. 

David, as the person who I believe owns the reftest analyser (or at least has expressed interest in it's development in the past) - what would your preferred way forwards be here?

Comment 4

[Ed Morley [:emorley]]

Also as-is, I'm reluctant to host it on treeherder.m.o, looking at the way it does an unconditional GET on whatever is passed in as logurl. I have little faith that the various tools that have whitelisted treeherder.m.o (eg buildapi) will honour the fact it's a GET and not a POST/PUT.

http://hg.mozilla.org/mozilla-central/file/cafb1c90f794/layout/tools/reftest/reftest-analyzer.xhtml#l118

Comment 5

[Ed Morley [:emorley]]

Particularly when it's a feature:
https://secure.pub.build.mozilla.org/buildapi/self-serve
"PUT, POST, and DELETE requests (which can be faked by setting a '_method' field to 'PUT' or 'DELETE' in a regular POST request if your client doesn't support PUT/DELETE easiliy) represent requests to change buildbot state."

eg:
https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-inbound/request/79292619?_method=DELETE

So if the reftest analyser was hosted on treeherder.m.o, this would work:
treeherder.m.o/reftest-analyzer.xhtml#logurl=https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-inbound/request/79292619?_method=DELETE

Comment 6

[Ed Morley [:emorley]]

Moving this back to the reftest component, to: decide on the plan and improve the reftest analyser to avoid the issue in comment 5 (plus check for any other potential XSS issues). Once that's done please file a Treeherder bug depending on this one, that either just updates the links in Treeherder if it's being hosted somewhere else, or will also copy it into the Treeherder repo. Thanks :-)

Comment 7

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to Ed Morley [:emorley] from comment #3)
> David, as the person who I believe owns the reftest analyser (or at least
> has expressed interest in it's development in the past) - what would your
> preferred way forwards be here?

I would prefer to keep it in somewhere such that changes to it immediately deploy.  It sounds like that might be hard to do if it's in hg.mozilla.org now -- or is it?

Maybe the next-best thing is to put it in a github repository (on a gh-pages branch) and access it off of mozilla.github.io or dbaron.github.io ?  (Do we use mozilla.github.io?)

Comment 8

[Ed Morley [:emorley]]

Github pages sounds great to me. If the repo were created under the Mozilla org, with a default branch of gh-pages, it would auto-publish on push to the repo, to:
//mozilla.github.io/name-of-repo/

Comment 9

[Gregory Szorc [:gps]]

All you need is a HTTP server that proxies https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml and adds the appropriate Content-Type header. This could be written in nearly any language. You could probably host it on paas.allizom.org.

Comment 10

[GitHub Autolander Bot]

Attachment: [treeherder] indygreg:proxy-reftest-analyzer > mozilla:master

Comment 11

[Gregory Szorc [:gps]]

New plan: have Treeherder proxy the reftest analyzer code from hg.mozilla.org. The reftest analyzer will exist at a well-known url (like https://treeherder.mozilla.org/reftest-analyzer).

Comment 12

[Ed Morley [:emorley]]

Comments left on the PR. Once addressed, Travis run passing & rebased on master, please request review using the Bugzilla flags to ensure it doesn't get missed :-)

Comment 13

[Ed Morley [:emorley]]

From the PR...

jgraham wrote:
> Why don't we just put the reftest analyzer in treeherder directly?
> The last commit was July 2015, so it's not like moving it would have
> an undue burden on developers (also I have a forked version that works
> with structured logs — although so far not from traditional reftest
> structured logs — that I plan to incorporate into the structured test
> viewer).

Comment 14

[Ed Morley [:emorley]]

dbaron, ping?

Comment 15

[Gregory Szorc [:gps]]

I'd support putting the reftest analyzer in treeherder. That removes a potential security issue from the perspective of treeherder.

Comment 16

[Ed Morley [:emorley]]

:jst/:jgriffin, a needinfo has been open on :dbaron for 5 months and 1 week - could you see if there's a better way to try and get an answer from him?

That said, I'm not sure we should support non-standard workflows, IMO needinfos are a really essential part of working with Bugzilla, and if people have explicitly disabled the automated daily reminders, I'm at a loss as to how to improve efficiency for inter-team communication.

Many thanks!

Comment 17

[Jonathan Griffin (:jgriffin)]

Will take this offline.

Comment 18

[Jonathan Griffin (:jgriffin)]

We've been asked to improve our compliance with Mozilla's website security guidelines, so our plan of action is to integrate the reftest-analyzer with Treeherder. The only open question is whether we should then remove it from hg. I know some developers use the reftest-analyzer locally, so I'm guessing we should leave a copy in hg but with clear instructions on how to get the Treeherder copy updated.

Please post any objections to this plan by Dec 12.

Comment 20

[William Lachance (:wlach)]

I opened a pull request to use a mirrored copy inside treeherder here: https://github.com/mozilla/treeherder/pull/2107

Just my opinion: I don't personally think anything more is required here. I suppose we could dump a README file inside mozilla-central if we liked.

Comment 21

[Ed Morley [:emorley]]

Due to the risk of XSS (particularly now Taskcluster tokens are stored in localstorage), before hosting the reftest analyser on treeherder.m.o, we'd need to either:
* get a security review
* add a CSP header for just that page

Alternatively it may just be easier to host the reftest analyser on GitHub pages or similar in the meantime.

Comment 22

[GitHub Autolander Bot]

Attachment: [treeherder] wlach:1333928 > mozilla:master

Comment 23

[Ed Morley [:emorley]]

Longer term I think it makes sense for us to decide a strategy for tools that tie into test harness output: should they reside with the harness code or with tools like Treeherder? (Particularly now we're moving in the direction of splitting these tools out, eg the unified log viewer)

However until that happens and a new reftest analyser (that avoids the issues in comment 21) is written, I don't believe Treeherder is the correct place for the legacy reftest analyser, nor is this something the Treeherder team should own.

As such short term the options are:
1) Move the reftest analyser to it's own hosting somewhere (eg github pages, heroku app etc)
2) Just agree to leave the current reftest analon hg.mozilla.org for now

If someone wants to do (2), then please file a new bug in Testing::Reftest for setting that up, and when the new hosting is ready, file a new Treeherder bug for updating the URL Treeherder uses.

Comment 24

[Ed Morley [:emorley]]

Eugh copy-paste-rearranging typos. Should have read:

As such short term the options are:
1) Move the reftest analyser to it's own hosting somewhere (eg github pages, heroku app etc)
2) Just agree to leave the current reftest analyser on hg.mozilla.org for now

If someone wants to do (1), then please file a new bug in Testing::Reftest for setting that up, and when the new hosting is ready, file a new Treeherder bug for updating the URL Treeherder uses.