554197 - [httpd] Use the XPCOM component of the HTTPd server instead of the loadable script

Status: RESOLVED WONTFIX

(Testing Graveyard :: Mozmill, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Right now we are loading the HTTPd server via the script:

var httpd = {};   Components.utils.import('resource://mozmill/stdlib/httpd.js', httpd);

We should avoid that because it requires updates to the file after we pulled it from hg. At least we have to add an export for getServer. Instead of those modifications we can place that file under the components folder of the extension and make it accessible as XPCOM component even for release builds. It's platform independent so I feel that should be the best way to go.

Any objections?

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As Clint pointed out on IRC we have to create the .xpt file first. That would give us two options:

1. If you want to update httpd we would have to check if the xpidl tool is available.

2. Someone from the core team has to update the xpt file when necessary.

By all means we shouldn't install the whole build environment for each user. Or is there an easier way to build this tool?

Comment 2

[Mikeal Rogers [:mikeal] (mrogers@)]

Is there something extra that you get out of this?

There are a lot of modifications that I made to httpd, it does really excessive logging by default that, last time i checked, you couldn't turn off.

The httpd.js in mozmill does everything we need it to, is there anything that we need that we get out of doing it this way instead?

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The version we are using is damn old and a couple of new features went in, which can be really helpful. We will have to extend the way how tests can interact with the server. Right now you can only set a folder and nothing more. For upcoming tests we need e.g. our own handlers and using the server as a component would make things easier. The only downside here is the creation of the .xpt file.

I just want to minimize all the additional stuff which has been added to that file too. Mikeal, can you remember which version of the server you have used? Or do you still know which modifications you did, except the export statement and the getServer function.

I cc Jeff to that bug so we can clarify some open questions.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

With the script on bug 554199 we have now the possibility to always have an updated version available. 

The problem we still have is the creation of the .xpt file.

Comment 5

[cmtalbert]

We should have some way to do httpd.js stuff in Mozmill.  Not sure if we should just reuse what mochitest does or if we can use this bundled ourselves.

Comment 6

[Jeff Hammel]

There has been somewhat overlapping discussion between this bug and bug 563748 . I will try to consolidate here, as this is the blocking bug. But please see related discussion there for parallel suggestions.

As far as how mochitest does things, a cursory survey tells me a few things:

 1. mochitest makes use of automation.py(.in) to read the server locations from server-locations.txt and sets the proxy on the user test using the network.proxy.autoconfig_url in the profile prefs:

http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py.in#423
->
http://mxr.mozilla.org/mozilla-central/source/build/automation.py.in#363

 2. http.js is then invoked via xpcshell:

http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py.in#306

I am not sure how this would work in general without using the packaged tests which is suggested as not a possibility in https://bugzilla.mozilla.org/show_bug.cgi?id=563748#c7

For completeness, I will note memtest.py also and independently does proxying, albeit in a much less thorough way than automation.py(.in):

http://mxr.mozilla.org/mozilla-central/source/testing/tests/memtest.py#185

http://mxr.mozilla.org/mozilla-central/source/testing/tests/memtest.py#197

also, and this purely speaks to my naivete, not sure how any of this relates to .xpt files.  Can anyone qualify here?

Also, is any of this "how things work" worth moving to a wiki page? (and if so, which?)

Comment 7

[cmtalbert]

We need a better packaging mechanism for httpd.js and we need to solve that outside of mozmill.  We'll dupe this bug to that bug once it is filed.  Either way, it's not a mozmill 2.0 blocker. Minusing.

Comment 8

[Mikeal Rogers [:mikeal] (mrogers@)]

Unless it has changed in the last year httpd.js has modifications to it for mozmill and isn't exactly the same as the one used by other packages.

IMHO, mozmill exposes an API on top of httpd and test authors don't touch it directly so most improvements aren't really applicable to mozmill anyway so keeping an outdated copy shouldn't be a problem. 

In fact taking a new version for features you don't use is just going to increase the risk that there was a bug introduced. It's a pretty simple server, i doubt there are bugs fixed that often, most of the continued development is adding application layer features for mochitest that mozmill won't use.

Comment 9

[Jeff Hammel]

Currently we have special scripts to download httpd.js and sync this to mozmill.  We shouldn't do this.  A stupid solution would be to fork httpd.js and maintain it completely independently.  A much better solution would be to not have mozmill depend on this or any other specific http server.  As clint points out, this is a larger issue than 2.0 or rather we just don't have time to figure this out right now. But no, mozmill should not be intertwined with httpd.js.  This should go in other code.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Well, the modification is AFAIR necessary because we don't use httpd as XPCOM object. If we would do that, we only have to stick the xpt file into the extension and update some of our code which handles httpd instances. Lets see if I find a bit of time over a weekend to play with it.

Comment 11

[Jeff Hammel]

Though again I'm not sure if we should bundle httpd.js at all.  I think doing server and proxying is a separate test package.  Why should mozmill be a web server?

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

How is it handled in the testing package for Mochitests?

Comment 13

[Jeff Hammel]

Badly! The short answer is it runs via a Makefile calling xpcshell to instantiate httpd.js from a known location within the mozilla-central tree patching PYTHONPATH in the process.  Mozmill doesn't (natively) run via a Makefile, doesn't have xpcshell embedded, and isn't canonically part of Mozilla central.

We can make all of these things happen, but I think its usually better to ask ourselves what we actually want to do versus copying an existing methodology and calling it correct except in the interest of convergence.  Why should Mozmill serve web?  Shouldn't a webserver serve web? Should mozmill care which webserver it is?  I'm going to say "It shouldn't; yes, and no" in that order.

Again, I don't think this is 2.0 worthy, as its not something to be entered into lightly and there are several related issues attached (which I won't enumerate here).  But I don't think copying other harnesses' methodologies blindly is a good idea unless they do it right or there is an intent and plan to converge.

Comment 14

[cmtalbert]

(In reply to comment #13)
> Badly! The short answer is it runs via a Makefile calling xpcshell to
> instantiate httpd.js from a known location within the mozilla-central tree
> patching PYTHONPATH in the process.  Mozmill doesn't (natively) run via a
> Makefile, doesn't have xpcshell embedded, and isn't canonically part of Mozilla
> central.
In the python runner for mochitest you can fully specify all the paths and PYTHONPATH is not patched.  This is the standard way of running our mochitests.

> 
> We can make all of these things happen, but I think its usually better to ask
> ourselves what we actually want to do versus copying an existing methodology
> and calling it correct except in the interest of convergence.  Why should
> Mozmill serve web?  Shouldn't a webserver serve web? Should mozmill care which
> webserver it is?  I'm going to say "It shouldn't; yes, and no" in that order.
Yes, but there are advantages to using our test webserver.  It can do things at run time that most other servers cannot.  I think having hooks to help tests interact with the httpd.js webserver is a win.  Those hooks would be no-ops against a non-httpd.js webserver that isn't as flexible.

We need a way to package it, we need a way to make it easier to start/stop/interact with.  This is outside of the scope of mozmill.  Once we have that, we can look at test hooks for it.

The current packaging mechanism for httpd.js assumes that you have several files from the bin/ directory of the test packages, several shared libraries on your path or LD_LIBRARY_PATH to link to from a compatible build of the platform, and (if you actually want to serve things the way mochitest does) a couple of other js files that must be included in your directory and several bits of javascript code that must be specified on the command line for xpcshell when httpd.js is invoked.

Now you begin to see why I say we need a packaging solution, yes?

Comment 15

[Jeff Hammel]

(In reply to comment #14)
> (In reply to comment #13)
> > Badly! The short answer is it runs via a Makefile calling xpcshell to
> > instantiate httpd.js from a known location within the mozilla-central tree
> > patching PYTHONPATH in the process.  Mozmill doesn't (natively) run via a
> > Makefile, doesn't have xpcshell embedded, and isn't canonically part of Mozilla
> > central.
> In the python runner for mochitest you can fully specify all the paths and
> PYTHONPATH is not patched.  This is the standard way of running our mochitests.

Sure, I was just pointing out that we don't do this for Mozmill.  We *could* of course, python doesn't care, but we don't and its not a tiny switch (I also tend to believe its a bad idea).  

> 
> > 
> > We can make all of these things happen, but I think its usually better to ask
> > ourselves what we actually want to do versus copying an existing methodology
> > and calling it correct except in the interest of convergence.  Why should
> > Mozmill serve web?  Shouldn't a webserver serve web? Should mozmill care which
> > webserver it is?  I'm going to say "It shouldn't; yes, and no" in that order.
> Yes, but there are advantages to using our test webserver.  It can do things at
> run time that most other servers cannot.  I think having hooks to help tests
> interact with the httpd.js webserver is a win.  Those hooks would be no-ops
> against a non-httpd.js webserver that isn't as flexible.
> 
> We need a way to package it, we need a way to make it easier to
> start/stop/interact with.  This is outside of the scope of mozmill.  Once we
> have that, we can look at test hooks for it.
> 
> The current packaging mechanism for httpd.js assumes that you have several
> files from the bin/ directory of the test packages, several shared libraries on
> your path or LD_LIBRARY_PATH to link to from a compatible build of the
> platform, and (if you actually want to serve things the way mochitest does) a
> couple of other js files that must be included in your directory and several
> bits of javascript code that must be specified on the command line for xpcshell
> when httpd.js is invoked.
> 
> Now you begin to see why I say we need a packaging solution, yes?

Yes.  I think the three takeaways are:
- we *will* use httpd.js for several reasons;  however, any reliance on a particular web server should be test-specific and not Mozmill-specific
- we need a way to package this
- ...but this is outside of the scope of Mozmill.  it may live in the same repo, but it doesn't really have anything to do with the test harness (nor with any of the existing packages in the repo)

Comment 16

[Jeff Hammel]

Not sure why this was assigned to me.  I'm not working on it actively

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

A fix for this bug is not necessary anymore. httpd.js has been updated a while back so it exports all the symbols which should be visible to the outside. That means we no longer have to do any modifications on that file.