564648 - Mozmill crowd v0.1 tracking bug

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Crowd Extension, defect)

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This bug tracks the work which is necessary to create the Mozmill crowd testing extension. Project details can be found here:

https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Cloud_Testing

QA will start to implement this extension soon. Therefore I have to put together all the specs in the next days. While talking with Clint last week, he mentioned some security concerns we should avoid, which I wasn't aware of.

Jorge, I hope you can help us to make sure that we deliver a secure and trustful extension to run the mentioned tasks.

As far as what Clint told, we are not allowed to ship any external binary components with our extension. Due to outdated versions, e.g. for Python, we could open security holes. That spoken we would have to think about alternatives which can be used to create a temporary environment for running Mozmill tests. 

1. Let the user manually download and unzip a simplified version of MozillaBuild on Windows, which will offer us a Python interpreter, easy_install/pip, mercurial, and the git client preinstalled. The folder of the unzipped package would have to be specified in the extensions preference pane. On Linux and OS X we will use the default Python version. Once this has been done we can do the next step and prepare the local, sandboxed environment for the test-run inside the current profile folder automatically:

* Download the latest version of virtualenv
* (OS X/Linux) install/update easy_install or pip
* (OS X/Linux) install/update mercurial (mozmill tests on hg.mozilla.org)
* (OS X/Linux) install/update git (test-run scripts on github)
* install/update mozmill, mozrunner, and jsbridge 
* clone/pull the script repository from Github

Nothing more should be necessary now.

2. We offer a separate application which completely bundles all the necessary software (see proposal 1) within a package for download.


Personally I'm leaning towards proposal 2 because it is inflexible and forces us to update everything once a new version has been released. Also creating a complete new application is probably outside of my knowledge. Having an extension we can really let it install with a single click and after the preparation steps on Windows it will be functional.

Jorge, when you read the project description and the proposals above, what is your opinion about what's the safest and best way for our users - from a security perspective. I'm looking forward to your expertise. Thanks!

Comment 2

[Jorge Villalobos [:jorgev] (he/him)]

From a security perspective I think either approach is good. As long as the add-on doesn't ship with the binaries and there's a way to keep them up to date, the user should be safe.

I also think proposal #2 is the simplest and easier to manage. The add-on could check regularly if there are new versions of the bundle, and handle the download and maybe the installation. HTTPS and checksums should be used to verify the origin and integrity of the installer. The trickier part is making sure the installer works well with different combinations of installed packages in the host system.

It also looks like you're downloading test code when running tests. It would also be a very good idea to use HTTPS for that.

For user privacy I also thinks it's a good idea to give users the choice of sending test data or not, and to keep it if the want to. You can look into the Test Pilot add-on for an example of this.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks for your feedback Jorge! I will incorporate everything into a spec. Would be nice if you could check it again, once it has been finished. I will add another comment to this bug by that time.

(In reply to comment #2)
> I also think proposal #2 is the simplest and easier to manage. The add-on could
> check regularly if there are new versions of the bundle, and handle the
> download and maybe the installation. HTTPS and checksums should be used to
> verify the origin and integrity of the installer. The trickier part is making
> sure the installer works well with different combinations of installed packages
> in the host system.

In case of proposal #2 we would have more work over time because we would have to supply new binary updates every time a software gets updated. I will investigate how difficult this solution is against proposal #1. Perhaps we can have an automated way to create those binaries and upload them to a accessible location for download.

> It also looks like you're downloading test code when running tests. It would
> also be a very good idea to use HTTPS for that.

Will do that. Seems to work fine with https://hg.mozilla.org/qa/mozmill-tests/

> For user privacy I also thinks it's a good idea to give users the choice of
> sending test data or not, and to keep it if the want to. You can look into the
> Test Pilot add-on for an example of this.

This is something which is already on my list. We need a privacy agreement which has to be accepted by the user before it will be possible to send results. I'll have a look at the Test Pilot extension. Also Grafx Bot from jgriffin is a good example.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I have created the specs for the cloud testing extension:
https://docs.google.com/Doc?id=d9cz3wk_17hmkx6p2w

I would really appreciate it if everyone can have a look at it by today and give feedback on this bug. Thanks.

Comment 5

[cmtalbert]

(In reply to comment #4)
> I have created the specs for the cloud testing extension:
> https://docs.google.com/Doc?id=d9cz3wk_17hmkx6p2w
> 
> I would really appreciate it if everyone can have a look at it by today and
> give feedback on this bug. Thanks.

So I think this is a good proposal all in all.  I do have a couple of technical questions about it, though.

The first question is very much an overall focus question, and I ask it not to put stop energy into this (because I think the idea of making it easier for people to run our automated tests is a great one) but because by not asking it, I think we make some implicit assumptions here that are going to unnecessarily complicate solution we build.

The question: What problem are we trying to solve?  

If we're really trying to run these tests in Firefox without closing people's browser or using another profile, then we're putting ourselves into a very narrow technical box.  Inside of those constraints, it's going to be very hard to build this.

If we're just trying to make it easier for people to run our automated tests, then we are a little freer to consider other solutions like stand-alone xulrunner applications and we can get away from the difficulties encountered by forcing this application to live inside a Firefox user's profile.

So, I have some issues I'd like to see addressed in this specification and I also have some ideas for possible recommendations. My recommendations may or may not be any good because I'm really not clear on what constraints you believe the cloud testing extension *must* solve and which constraints can be bent, reinvented or broken.  So my recommendations may be out of left field.  Let's get started.

= Issue # 1 =
How are you going to ensure the safety of the user's profile during and after test runs?  If you are going to run from inside their profile without restarting, without using a different profile, what steps will you take to back up their profile and ensure no destructive tests are performed?
* Note that adding things can be just as destructive as removing things - for example, if you add 100 test bookmarks and fail to remove them, you'd really make people mad.  Or if you change the ordering of folders in the bookmark library.
* One easy answer to say here is that you'll enforce this by making all tests behave, but then I'd argue that you'd be shooting yourself in the foot w.r.t. your testing.  You WANT destructive tests or else you're not really exercising corner cases and recovery tests.  I think at the very least you're going to have to fork what we run internally and what is run by this extension.  Perhaps what the extension runs is a subset of what we run internally, but it will never be all the tests.
* This brings up another question - how will you manage this second set of tests?  How will a test be ensured to be safe, how will it be tracked, how will the tool find it?

= Recommendation for Issue 1 =
I'd like to see how you're going to handle this mentioned in the proposal because I think it's a design feature you'll need to factor in from the start, especially if you plan to run this on people's day-to-day profiles.

= Issue # 2 =
* Why do you need mercurial?
* There are a number of reasons - if you need to manage two sets of tests, perhaps you just want to check out a specific branch of "known safe" tests.  Perhaps you want to report against a known "revision" of tests?  I'm just not clear on why you have that dependency, and it feels to me like it is unnecessary.

= Recommendation for Issue 2 =
* Why not host a set of datetime stamped tests on some public server (like brasstacks) and have the tool download it?  It can report the datetime stamp of the test package that it uses when it uploads results.  And this frees you from having to carry around a mercurial dependency, which is really nice.  This also means that if in the future you decide to change where or how you store the tests in the repository you don't break every person that installs this application.  It's easy to keep serving something from the same URL, and you can use a redirect on that URL if things change.  It's less easy to future proof an Hg repository checkout.

= Issue # 3 =
* Why do you need msys on Windows?

= Recommendation for Issue 3 =
* Drop it.  I'm really not clear on why you need a unix shell on windows for what you're doing.

= Issue # 4 =
* What's your delivery mechanism to get all this installed on people's machines?

= Recommendation # 4 =
* The current way of performing extension installs gives you very little control over recovery and error handling. You're installing quite a bit of stuff that all has to work together.  In the old days, you could actually do some amount of recovery and error handling when installing an extension.  These days you can't.  I'm not sure if that's good or bad from how you see it.

= Issue # 5 =
* You're talking about running different versions of the browser from inside a running instance of the browser.  This is fraught with problems:
** Most users do not have multiple versions of Firefox on their machine.  Are you planning to install these for them?
** It is actually impossible to have two versions of Firefox running against the same profile at the same time (and if you found a way to do it, you'd almost certainly corrupt the profile).
** Most people do not have more than 1 profile.  Will you automatically create other profiles for these extra spawned Firefoxes?
** There is no way for spawned Firefox to use the Mozmill extension contained inside of its parent Firefox process.  Firefoxes can't cross profile boundaries at all, so you'd have to insert mozmill into the spawned Firefox and restart the browser.

= Recommendation # 5 =
* This is why I think you should really reconsider what problem you're trying to solve.  You're butting up against all of Gecko's security constraints and a bunch of Gecko design assumptions.  This makes what you're trying to do very very hard.
* Perhaps what you really want here is a stand-alone application and not an extension at all.  If you start thinking bigger about a stand-alone application you can start thinking about all kinds of exciting possibilities - installing versions of firefox for the user, managing their profiles for them, enabling them to run either automated or manual tests, and automatically updating all those versions of firefox yourself without them needing to be concerned and without affecting their standard install.

= Issue # 6 =
* Related to issue # 5, there is no generic "spawn" utility inside the Gecko platform.
** This means you can't execute "python myprogram.py foo" from inside the browser.  Nor can you call out to the system, nor can you call out to mozmill and point it at a different instance of Firefox.

= Recommendation # 6 =
* Really reconsider this approach.  We are building this ability for the profile manager (which is a standalone application) and can share the code with you, but I wouldn't allow this code to be in an extension -- it's a security hole waiting to happen.
* The only way we have to spawn a process is to create a heavily controlled subprocess (like for OOPP) but that's not what you want here.

= Issue # 7 =
* Why do we expose the "trust unsecure addons" preference on the preference screen of this addon?

= Recommendation # 7 =
* This is not a preference we really expose, nor is it something that we really recommend that people set.  We'd prefer to handle this by a whitelist of trusted sites so that when some random site "NoImReallyNotEvil.com" tries to install the "SpamZapperAddon" you get an alert.  If you set this pref then (I think) you no longer see these alerts for any site.  Having it exposed in an otherwise innocuous looking preference pane is setting up users for disaster.  Do you have a good reason for doing this?

I hope this isn't too negative.  I do like the idea, and Henrik has put a lot of thought into making this paper.  I think it's a good idea, and well-executed it could be a great boon to the community.  However, I do think we really need to fully understand what we want and how we can make that happen without pitting ourselves head-to-head against our platform's design decisions and while ensuring we do not put our volunteer's data or their security at risk.

Comment 6

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Looking over the spec itself, everything seems like good things to want. 

I do agree with ctalbert that if significant problems are solved/avoided by going standalone, we consider that.  I think it's an acceptable speedbump, especially if it potentially reduces some other speedbumps or gotchas.  

Once we have things running solid in a standalone context, we could review whether it'd make sense or is technically possible to move some subset of tests into an extension context (possibly into the MozMill extension itself?).  In the meantime, standalone may get us up and running robustly in the fastest manner, and thus have the most short/medium-term value.

Comment 7

[Jeff Hammel]

(In reply to comment #5)
> (In reply to comment #4)
<snip/>
> If we're really trying to run these tests in Firefox without closing people's
> browser or using another profile, then we're putting ourselves into a very
> narrow technical box.  Inside of those constraints, it's going to be very hard
> to build this.

I agree with Clint's sentiments here for a few reasons:

 - intent:  having the testing live in the browser just so that it can spawn another browser is a bit, at best, a seeming conflation just so "things can look familiar"; unless I'm also missing some of the purpose here

 - overlap:  In working on the new ProfileManager, I've encountered several of these identical problems.  Regardless of how Cloud Testing goes, these will still be problems that I will have to deal with independently.  The ideal solution would be to work in such a way that as much code+intent isn't duplicated between the too.  If that's not possible or practical (which is my guess), then I am much more confident that doing two standalone applications can lead to something unified (with elements living in m-c) than a standalone application and a firefox extension

 - ease of implementation:  as much trouble as I've had with the build system this week, I would be surprised if a Firefox extension that performs tasks largely dependent on a large external installation (of python, virtualenv, etc) vs something that is encapsulated and encapsulable with its own build scripts that can do whatever they want vs trying to do magic within the extension context.  This said, my knowledge of extensions is not great, but I have a very good idea what can be done from a real command line.

> If we're just trying to make it easier for people to run our automated tests,
> then we are a little freer to consider other solutions like stand-alone
> xulrunner applications and we can get away from the difficulties encountered by
> forcing this application to live inside a Firefox user's profile.

Again, +1

 
> = Issue # 1 =
> How are you going to ensure the safety of the user's profile during and after
> test runs?  If you are going to run from inside their profile without
> restarting, without using a different profile, what steps will you take to back
> up their profile and ensure no destructive tests are performed?
> * Note that adding things can be just as destructive as removing things - for
> example, if you add 100 test bookmarks and fail to remove them, you'd really
> make people mad.  Or if you change the ordering of folders in the bookmark
> library.
> * One easy answer to say here is that you'll enforce this by making all tests
> behave, but then I'd argue that you'd be shooting yourself in the foot w.r.t.
> your testing.  You WANT destructive tests or else you're not really exercising
> corner cases and recovery tests.  I think at the very least you're going to
> have to fork what we run internally and what is run by this extension.  Perhaps
> what the extension runs is a subset of what we run internally, but it will
> never be all the tests.

Again, +1.  It is my opinion that tests should never leak data into user land and neither should test harnesses.  mozmill has already had leaks.  What if something awful happens?  It is my opinion that if we're going to try to get a strong cloud, we must do our damndest to not mess up our testors' computers or profiles.  If I were a volunteer tester and my profile was borked or bzip2.tar.gz appeared in my home directory, I would probably discontinue volunteering.

> * This brings up another question - how will you manage this second set of
> tests?  How will a test be ensured to be safe, how will it be tracked, how will
> the tool find it?
> 
> = Recommendation for Issue 1 =
> I'd like to see how you're going to handle this mentioned in the proposal
> because I think it's a design feature you'll need to factor in from the start,
> especially if you plan to run this on people's day-to-day profiles.
> 

> = Issue # 3 =
> * Why do you need msys on Windows?
> 
> = Recommendation for Issue 3 =
> * Drop it.  I'm really not clear on why you need a unix shell on windows for
> what you're doing.
> 
> = Issue # 4 =
> * What's your delivery mechanism to get all this installed on people's
> machines?
> 
> = Recommendation # 4 =
> * The current way of performing extension installs gives you very little
> control over recovery and error handling. You're installing quite a bit of
> stuff that all has to work together.  In the old days, you could actually do
> some amount of recovery and error handling when installing an extension.  These
> days you can't.  I'm not sure if that's good or bad from how you see it.

This again points to a standalone program in which you have complete freedom to do whatever you want.
 
> = Issue # 5 =
> * You're talking about running different versions of the browser from inside a
> running instance of the browser.  This is fraught with problems:
> ** Most users do not have multiple versions of Firefox on their machine.  Are
> you planning to install these for them?
> ** It is actually impossible to have two versions of Firefox running against
> the same profile at the same time (and if you found a way to do it, you'd
> almost certainly corrupt the profile).
> ** Most people do not have more than 1 profile.  Will you automatically create
> other profiles for these extra spawned Firefoxes?
> ** There is no way for spawned Firefox to use the Mozmill extension contained
> inside of its parent Firefox process.  Firefoxes can't cross profile boundaries
> at all, so you'd have to insert mozmill into the spawned Firefox and restart
> the browser.
> 
> = Recommendation # 5 =
> * This is why I think you should really reconsider what problem you're trying
> to solve.  You're butting up against all of Gecko's security constraints and a
> bunch of Gecko design assumptions.  This makes what you're trying to do very
> very hard.
> * Perhaps what you really want here is a stand-alone application and not an
> extension at all.  If you start thinking bigger about a stand-alone application
> you can start thinking about all kinds of exciting possibilities - installing
> versions of firefox for the user, managing their profiles for them, enabling
> them to run either automated or manual tests, and automatically updating all
> those versions of firefox yourself without them needing to be concerned and
> without affecting their standard install.
> 
> = Issue # 6 =
> * Related to issue # 5, there is no generic "spawn" utility inside the Gecko
> platform.
> ** This means you can't execute "python myprogram.py foo" from inside the
> browser.  Nor can you call out to the system, nor can you call out to mozmill
> and point it at a different instance of Firefox.

This is only partially true.  There is nsProcess, which will let you run other programs.  It, however, is like subprocess and not like exec.  Also, there is no way (currently) to get output from said programs (bug 68702).  In general, I am very skeptical of going down this route with FF for several reasons (wide disparities on how things are installed, security concerns).  Whereas, in a standalone app, you are free to ensure an installation as you like.

> = Recommendation # 6 =
> * Really reconsider this approach.  We are building this ability for the
> profile manager (which is a standalone application) and can share the code with
> you, but I wouldn't allow this code to be in an extension -- it's a security
> hole waiting to happen.
> * The only way we have to spawn a process is to create a heavily controlled
> subprocess (like for OOPP) but that's not what you want here.

Another recommendation:  There is strong overlap in purpose between your proposal here and profile manager.  As a strong advocate of '-> unity', I would recommend trying to figure out what we can share here.  This would be my opinion regardless of whether or not it was my project (which, of course, it is).

> = Issue # 7 =
> * Why do we expose the "trust unsecure addons" preference on the preference
> screen of this addon?
> 

> I hope this isn't too negative.  I do like the idea, and Henrik has put a lot
> of thought into making this paper.  I think it's a good idea, and well-executed
> it could be a great boon to the community.  However, I do think we really need
> to fully understand what we want and how we can make that happen without
> pitting ourselves head-to-head against our platform's design decisions and
> while ensuring we do not put our volunteer's data or their security at risk.

I also like the idea, and hope that my suggestions are helpful.  I am concerned about security, having a program that will not disrupt testers' user experiences (well, anymore than testing anything in mozmill already demands; I want to retain as many dedicated testers as possible), and doing something "now" that ends up costing more time/effort in maintainence vs getting it right.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #5)
> The question: What problem are we trying to solve?  

Meanwhile we already talked about it on IRC and I have updated the spec. But to repeat it here: Right now we do not have that much and in-detail test results for all of our BFT and FFT tests for localized builds. Those suites are quite too heavy that people don't run all the tests. With Mozmill we can empower people to run those tests in a split of the time which would be needed for manual tests. Further we can let people run Mozmill tests against Add-ons.

> If we're really trying to run these tests in Firefox without closing people's
> browser or using another profile, then we're putting ourselves into a very
> narrow technical box.  Inside of those constraints, it's going to be very hard
> to build this.

I don't want to do that. Clicking on start will use the configured test environment in the users profile to execute Mozmill and bring up another process with a fresh profile. The users folder will never be touched, except the test environment files.

> If we're just trying to make it easier for people to run our automated tests,
> then we are a little freer to consider other solutions like stand-alone
> xulrunner applications and we can get away from the difficulties encountered by
> forcing this application to live inside a Firefox user's profile.

Installing another application is another barrier we wanted to avoid. Installing an extension is the simplest part. And having it as a Jetpack available in the future would even be more streamlined.

> = Issue # 1 =
> How are you going to ensure the safety of the user's profile during and after
> test runs?  If you are going to run from inside their profile without
> restarting, without using a different profile, what steps will you take to back
> up their profile and ensure no destructive tests are performed?
> * Note that adding things can be just as destructive as removing things - for
> example, if you add 100 test bookmarks and fail to remove them, you'd really
> make people mad.  Or if you change the ordering of folders in the bookmark
> library.
> * One easy answer to say here is that you'll enforce this by making all tests
> behave, but then I'd argue that you'd be shooting yourself in the foot w.r.t.
> your testing.  You WANT destructive tests or else you're not really exercising
> corner cases and recovery tests.  I think at the very least you're going to
> have to fork what we run internally and what is run by this extension.  Perhaps
> what the extension runs is a subset of what we run internally, but it will
> never be all the tests.
> * This brings up another question - how will you manage this second set of
> tests?  How will a test be ensured to be safe, how will it be tracked, how will
> the tool find it?

All those questions have been solved by my above answer. We have another Firefox instance running with a fresh profile. We will run the same set of tests as what we are doing now from the command line. There is no difference and nothing will be damaged / deleted in the users profile.

> = Issue # 2 =
> * Why do you need mercurial?
> * There are a number of reasons - if you need to manage two sets of tests,
> perhaps you just want to check out a specific branch of "known safe" tests. 
> Perhaps you want to report against a known "revision" of tests?  I'm just not
> clear on why you have that dependency, and it feels to me like it is
> unnecessary.

Downloading the tests from hg.mozilla.org doesn't allow you to switch between named branches. Or I haven't found that yet. But that's something we have to do, when the user can select between the different supported branches of Firefox.

> = Recommendation for Issue 2 =
> * Why not host a set of datetime stamped tests on some public server (like
> brasstacks) and have the tool download it?  It can report the datetime stamp of
> the test package that it uses when it uploads results.  And this frees you from
> having to carry around a mercurial dependency, which is really nice.  This also
> means that if in the future you decide to change where or how you store the
> tests in the repository you don't break every person that installs this
> application.  It's easy to keep serving something from the same URL, and you
> can use a redirect on that URL if things change.  It's less easy to future
> proof an Hg repository checkout.

If the location of the mozmill-automation repository changes we can offer an update of the extension which will then fix the url. Further we do not want to have a static set of tests on a file server, which we will have to update at least once a day. Pulling the latest changes will give us better and more in-time results on brasstacks. That means fixed tests are immediately detected for the next test-run.

> = Issue # 3 =
> * Why do you need msys on Windows?

It's something I wasn't sure yet. But as source for the environment I want to use MozillaBuild. We have to strip away all the unnecessary tools and only take the ones I have mentioned above. We are using msys to call the Mozmill command line client. If I can do it without msys, I would be even happier.

> = Issue # 4 =
> * What's your delivery mechanism to get all this installed on people's
> machines?

Nothing will be installed. We only extract the test environments and download necessary modules via setuptools or pip.

> ** Most users do not have multiple versions of Firefox on their machine.  Are
> you planning to install these for them?

No. This is not part of the extension.

> ** Most people do not have more than 1 profile.  Will you automatically create
> other profiles for these extra spawned Firefoxes?

We use the default mechanism of Mozmill which creates fresh profiles for each test-run or restart test.

> ** There is no way for spawned Firefox to use the Mozmill extension contained
> inside of its parent Firefox process.  Firefoxes can't cross profile boundaries
> at all, so you'd have to insert mozmill into the spawned Firefox and restart
> the browser.

Mozmill is used as a command line client from within the Msys or Python process. It will install automatically inside the spawned Firefox process.

> = Issue # 6 =
> * Related to issue # 5, there is no generic "spawn" utility inside the Gecko
> platform.
> ** This means you can't execute "python myprogram.py foo" from inside the
> browser.  Nor can you call out to the system, nor can you call out to mozmill
> and point it at a different instance of Firefox.

Enigmail is using a component which is able to spawn new processes on all platforms. I will have to check its licensing but I plan to use the same or a similar way.

> * Really reconsider this approach.  We are building this ability for the
> profile manager (which is a standalone application) and can share the code with
> you, but I wouldn't allow this code to be in an extension -- it's a security
> hole waiting to happen.

Well, it's something I will have to consider. But as said above Enigmail is using this method already for a couple of years. I'm sure that Patrick has found a way to make it secure.

> * Why do we expose the "trust unsecure addons" preference on the preference
> screen of this addon?

It's not for this add-on but for the add-ons test-run. That means add-ons which are not hosted on AMO will not be executed if this preference has not been enabled. It's disabled by default and we will display a warning once it gets enabled.

> * This is not a preference we really expose, nor is it something that we really
> recommend that people set.  We'd prefer to handle this by a whitelist of
> trusted sites so that when some random site "NoImReallyNotEvil.com" tries to
> install the "SpamZapperAddon" you get an alert.  If you set this pref then (I
> think) you no longer see these alerts for any site.  Having it exposed in an
> otherwise innocuous looking preference pane is setting up users for disaster. 
> Do you have a good reason for doing this?

That's another good point. But once we start the test for a specific add-on from the command line, we do not know about any white listed website. That's why we only allow extensions from AMO per default.

> I hope this isn't too negative.  I do like the idea, and Henrik has put a lot
> of thought into making this paper.  I think it's a good idea, and well-executed

Thanks Clint for this fantastic feedback (you can compete with Jeff). I really can see that the spec puts questions up. I will have to fix most of it but it's probably something for next month. As I know from our talk all those questions have been solved. This is just a write up, to give a better explanation.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Also thanks to Geo and Jeff! Has my last comment also solved your questions?

Comment 10

[Jeff Hammel]

>> ** Most users do not have multiple versions of Firefox on their machine.  Are
>> you planning to install these for them?

> No. This is not part of the extension.

So where does this list come from?  I ask because, again, it would be a nice thing to have for ProfileManager

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #10)
> So where does this list come from?  I ask because, again, it would be a nice
> thing to have for ProfileManager

Those are recently used entries, the user has been selected via the system file open dialog. It's will be limited to 5 entries which should be enough.

If you have questions for the profile manager lets discuss it on IRC or on another bug.

Comment 12

[Jeff Hammel]

Relatedly to all of this meta-stuff and my profilemanager references, see

http://k0s.org/mozilla/hg/ProfileManagerXUL/file/327c61c2ffe1/README.txt#l136

I'll paste here as this repository is likely to move to hg.m.o/automation:

There are some issues here that point to a much broader sweeping unity
amongst XUL applications.  These issues are beyond the scope of the
current effort (a standalone application to manage profiles and launch
e.g. Firefox), but point to something more elegant in the unforseeable future:

 - currently, profiles exist on a per-app basis (see above).  This
   makes a lot of reinvention necessary when dealing with an
   application that wants access to another application with the same
   profile information.  While ProfileManager is an extreme case here,
   in the general trend toward software unification, it is likely to
   become preferable and desirable, for both technical as well as
   user-facing reasons, to have a unity of profile information.  For
   instance, 

 - changing and manipulating profiles within an application:
   currently, once you start (e.g.) Firefox with a profile, that is
   the profile that that instance will use until shutdown.  Ideally,
   an app should be able to switch profiles while running and
   manipulate the active and alternate profiles programmatically from
   within the application. (See also:
   https://wiki.mozilla.org/Auto-tools/Projects/CrossWeave )

 - profiles are not structured or currently defined as a contract: as
   stated above, the current state of a profile is defined only by a
   few attributes, whereas the attributes inherent in a profile (SQL
   databases, preferences, etc) are not collectively manipulable.  The
   question asked is: "Does this profile have XYZ?" when in fact the
   desired question is "What does this profile have?".  Additionally,
   things like prefs.js are not intelligibly manipulable vs. a
   "standard" format such as an .ini file. In short, profiles should
   have a defined contract that gives:
   - what is in the profile
   - methods (objects, really) for manipulating each part of the
   profile
   Add-ons should be exempt from this manifest nature (though they
   *could* support if as desired by the author), but by the same
   token, they should only be allowed to manipulate the profile using
   this unified API.

 - multiple versions of (Firefox, Thunderbird, etc): ideally,
   ProfileManager would allow you to start a profile with a drop-down
   list of whatever versions of the apps you have "installed" (or
   have run in the past for a close second-place).  However, this
   information is not globally stored.  Ideally, the available
   versions should be kept in a central registry.  This is to say,
   profiles of each application and version should be kept.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Wow, that should be "Crowd" and not "Cloud" :) Fixing all traces.

Comment 14

[Aaron Train [:aaronmt]]

I might have missed somewhere else but how will you engage the problem with the requirement of some tests requiring a focused testing browser window? What happens if a user wishes to keep the testing window in the background or minimized?

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #14)
> I might have missed somewhere else but how will you engage the problem with the
> requirement of some tests requiring a focused testing browser window? What
> happens if a user wishes to keep the testing window in the background or
> minimized?

Once we have native event support in Mozmill that will not be a problem anymore. Until then tests will have to be run in the foreground. We should explicitly state that.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Lets use this bug as tracker for version 0.1.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I had to move the work on the non-blocking nsIProcess call out to v0.2. It's more work involved as I had expected. If you are interested in v0.2 please follow bug 623204.

I will upload the final patch for v0.1 in a couple of minutes.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Final patch v0.1

This patch updates the version number, some data in the install.rdf file, and adds the missing license blocks to some files. I have also disabled sending of reports to brasstacks per default, because with the blocking call we could get a lot of wrong results due to bug 604648.

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Landed as:
https://github.com/whimboo/mozmill-crowd/commit/77f925d39b9cd954a6feb98ff9a466711d54da74

I will now create an AMO page we can use to distribute the extension. Nothing more to tell, this bug is fixed.

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Version 0.1 has been tagged:
https://github.com/whimboo/mozmill-crowd/tree/v0.1