685903 - remove firebug tests + harness from mozilla-central

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Jeff Hammel]

http://mxr.mozilla.org/mozilla-central/source/testing/firebug/

This version is obselete and, as I understand it, is not utilized at all and is instead taking up time + bandwidth being shipped over with tests.zip.

They should probably be removed, for now

Comment 1

[cmtalbert]

So, whether we remove, or update depends on how likely we are to get movement on the apache changes required to run the firebug tests on the build slaves.  That work stalled out in bug 635019.  

Is this something that releng wants to do, or do we want to formalize the ad-hoc testing that we've had ongoing with firebug this entire time.

If we decide to do this then here are the next steps:
* Add PHP and mod_rewrite support to all test slaves
* Update the firebug code to use mozbase
* Update the firebug code in-tree (or pull it in separately from a zip file aka talos, however)
* Write the buildbot steps to integrate firebug into the test system (partially done in a bug somewhere)

If we decide that this isn't something releng can support then we:
* remove firebug from m-c
* formalize the ad-hoc automated system we have been running these last few months.

What do y'all want to do?

Comment 2

[Andrew Halberstadt [:ahal]]

Personally, even though I've spent a fair amount of time working on this, I'd be in favour of removing firebug from m-c. Reasons as follow..

1) This is a project that was thought up and created right in the middle of all the craziness leading up to the fx4 launch. Bottom line was that we didn't want to ship with a broken Firebug. This goal is very reasonable and arguably even more important now with rapid release cycles. However there were a few things that the architects of this project weren't aware of. Namely...

2) The tests are fundamentally different from any of the current automated tests. They require their own webserver and different configurations on all the build machines. Furthermore...

3) We don't really have control of the tests. The tests are written and maintained by the Firebug team (which now only consists of Jan Odvarko). It is possible to pipe these through our own mercurial repo, and possibly even add our own tests, but it is an extra layer of problems waiting to happen.

4) Work on built-in Firefox dev tools is ramping up while work on Firebug is winding down. Obviously Firebug still is and will remain a very important extension for the foresee-able future, but I can only see the usefulness of this project diminishing over time.

I know I'm only listing the cons here, and that there are many benefits to finishing this project, but I think that maybe time would be better spent formalizing the system already in place.

Things we might do to our existing system...

A) Create our own results dashboard tailored to our needs (if the one at www.getfirebug.com/testresults is unsuitable)

B) Run the tests on a larger variety of builds and platforms

C) Automatic regression finding/notification

Comment 3

[Andrew Halberstadt [:ahal]]

I want to point out that maybe some of the cons I listed aren't as big of a deal as I think they are and should we go forward I'd be fine with helping out anyway I can.

Comment 4

[Jeff Hammel]

For the record, I disagree with point 3 in comment 2 regarding the extra layer of problems (this is a standard practice ... mirroring subsets of tests between locations ... and I'd rather attack a problem like this than saying its hard and we can't do it).  I also think point 4 is a bit overstated.  Firebug will be a go to tool for quite some time and if we make a decision along these lines it should be driven by usage statistics and consensus.

Comment 5

[cmtalbert]

If I have a finite number of buildbot requests (which I believe I do) then I'd rather focus on getting user responsiveness into the buildbot automation rather than firebug.  The firebug tests are running, have been running, and we can keep them running for as long as we want, but having them in m-c hasn't been an issue.  I'm going to side with Ahal and decide that we should remove them from m-c and throw the buildbot support out for them at this time.

We can always revisit later, but right now our buildbot resources should concentrate on other things like user responsiveness testing rather than this.

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Patch 1.0 - Remove firebug

Wow, just noticed that this is still open. This patch has DONTBUILD in the commit message, though I pushed a -p linux -u none job to try just to be safe: https://tbpl.mozilla.org/?tree=Try&rev=d3ed4cb3c014

Comment 7

[Andrew Halberstadt [:ahal]]

So there was a build exception. I'm 95% sure it's unrelated, but pushed to try again with all platforms this time anyway:

https://tbpl.mozilla.org/?tree=Try&rev=68697bfcf6e2

Comment 8

[Andrew Halberstadt [:ahal]]

Hmm, same infra exception. I'll ask around, though searching the log for 'firebug' yields no results.

Comment 9

[[e-mail] (:bhearsum)]

That exception is definitely not related to your patch:
https://bugzilla.mozilla.org/show_bug.cgi?id=793675
https://bugzilla.mozilla.org/show_bug.cgi?id=793016

Comment 10

[Jeff Hammel]

Comment on attachment 664976 [details] [diff] [review]
Patch 1.0 - Remove firebug

remove it with a vengeance!

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6a5b311a12e6

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/6a5b311a12e6