568943 - Add possibility to load account information from local files for restricted tests

Status: RESOLVED WONTFIX

(Testing Graveyard :: Mozmill, enhancement)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

In the future we will have Mozmill tests which will test websites which restrict the usage to members. For those it would be great when we have the possibility to load account information (user/password) from a local file which is not located inside the repository.

Clint, I would say we should do that as a shared module instead of implementing it into Mozmill itself. Would you agree?

Comment 1

[Jeff Hammel]

can't this be done by creating + using a profile with the appropriate data?

Comment 2

[Marc-Aurèle DARCHE]

(In reply to comment #1)
> can't this be done by creating + using a profile with the appropriate data?

Using separate profiles is one possibility but I would prefer having a local file because:

* a local file can be tracked in a versioning system
* code would be clearer to understand if it refers to a file (explicit)
* (it seems to me) that code would be harder to understand if the account information is located in a profile (implicit)

And finally, is it possible to track a Profile in a versioning system?

But in the short term, yes, putting account information in the preferences is a very pragmatic solution that works. But it seems to me that it's more of a manual setting that of an industrial process that could be used for a test farm.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It's also more a privacy issue. I wouldn't start such a test-run which gets the data from a profile you don't know in the first step where it is located. Having a file stored inside of the tests folder would make more sense. Not sure if this is even a Mozmill issue. We should be able to completely integrate it via a shared module and offer that function for any kind of Mozmill tests.

What's your opinion?

Comment 4

[Jeff Hammel]

This is an enhancement proposal; marking as such

Comment 5

[Andrew Halberstadt [:ahal]]

If we do do this maybe it should be more generalized to read in any data from a config file?

For example giving the user the ability to create python ConfigParser objects should be *relatively* simple.

Comment 6

[Jeff Hammel]

This seems like something that should be specced out.  I'd like to see the ability to load prefs from a file, which is actually a separate to-be-filed bug.  In general what is required, as best I can tell, is the ability to create a profile from e.g. an .ini file.  While I think this is a valuable effort, it is both a separate package and something that we should really think out, IMHO, vs just satisfying a specific use case.  I'm also not sure if this is a priority.  I'd like what we want to be specified.

Comment 7

[cmtalbert]

Ok, pulling together the threads I'm seeing here:
This is for a local file that can be specified at mozmill runtime on the command line.  This local file has the ability to:

* Set preferences for the generated mozmill profile
* Contain test specific data that a test can ask mozmill for at run time.
** i.e. there will be a test call like: mozmill.getTestConfigData() which returns a bunch of key value pairs from the file.

= Questions =
My proposal if we take this for 2.0 would be:
* When using this config file to configure a profile, I'm thinking you are allowed to only specify preferences.  You cannot seed a profile with anything else at the moment.
* The config file has a section that allows you to specify test-specifc data.  This data will be key value pairs.  And will be provided to the test in a dictionary style object.

I'd like to start with this feature simply, and we can add on from there.  Does this satisfy the basic use-case?

Comment 8

[cmtalbert]

Henrik, M.A. Darche, Ahal, Jhammel?  WHat is the use case here? (see comment 7).

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We don't have a usecase for our basic Firefox tests at the moment and probably not in the near future. Not sure about add-ons tests. Could happen earlier.

Comment 10

[cmtalbert]

Without a use case, we're going to wait and revisit this for 2.1.  Also there is other code outside of mozmill that is working on creating a profile, so perhaps once that work is formalized and completed we may be able to take that module and use it in mozmill.

Comment 11

[Marc-Aurèle DARCHE]

The use case is described in bug 671256.(In reply to comment #10)
> Without a use case, we're going to wait and revisit this for 2.1.  Also
> there is other code outside of mozmill that is working on creating a
> profile, so perhaps once that work is formalized and completed we may be
> able to take that module and use it in mozmill.

Yes, as Jeff Hammel said, most of what's described here can be done with the "--profile" option now available in Mozmill.

Only passwords should not be stored and handled as preferences. This is discouraged as this is an unsecure practice.

The use case is described in bug 671256 which focuses only on the password setting now that profiles loading is available.

So I request that this ticket is closed. Thank you.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

There is no need to create a new bug. Everything is still the same as described here in comment 0.

All passwords are stored in a sqlite database so it will not be that easy to prepare such a profile. It would depend on the version of Firefox. Instead we should probably have restart tests for those scenarios. The first test simply sets the password, and the following can make use of it. All other ways are too failure-prone and should probably not be done.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hm, a restart test will not solve that. IMO we should create sqlite files with the lowest version of the supported schema. So it will be usable by all versions of Firefox. If it's too old Firefox will automatically upgrade it. Those files should be put into a newly created profile or an already existent one, if specified. We could use that to pre-populate profiles. There should be another bug for the basic support of it around. We should add the dependency.

Comment 15

[cmtalbert]

(In reply to comment #14)
> Hm, a restart test will not solve that. IMO we should create sqlite files
> with the lowest version of the supported schema. So it will be usable by all
> versions of Firefox. If it's too old Firefox will automatically upgrade it.
> Those files should be put into a newly created profile or an already
> existent one, if specified. We could use that to pre-populate profiles.
> There should be another bug for the basic support of it around. We should
> add the dependency.

For this explicit use case, I don't understand why you can't simply write a non-restart test that stores the password by writing to the password manager directly before loading the site.  It sounds to me like you're not really wanting to test the "save password" work flow here.  What you want to test is the content on the password protected site, so why not simply store your login, browse to the site, and go on with your test?

(Remember to flush the password cache after the test finishes).

You can use the nsILoginManager interfaces from your test to achieve this.[1]

[1]: http://mxr.mozilla.org/mozilla-central/source/toolkit/components/passwordmgr/nsILoginManager.idl

Comment 16

[Marc-Aurèle DARCHE]

(In reply to comment #15)
> (In reply to comment #14)
> > Hm, a restart test will not solve that. IMO we should create sqlite files
> > with the lowest version of the supported schema. So it will be usable by all
> > versions of Firefox. If it's too old Firefox will automatically upgrade it.
> > Those files should be put into a newly created profile or an already
> > existent one, if specified. We could use that to pre-populate profiles.
> > There should be another bug for the basic support of it around. We should
> > add the dependency.
> 

I can't imagine a better solution. I'm not saying that it shouldn't be done otherwise. I'm just saying that it seems practical and resilient enough.

> For this explicit use case, I don't understand why you can't simply write a
> non-restart test that stores the password by writing to the password manager
> directly before loading the site.
>

The Fidesfit-client usecase is not about using a login + password for filling in forms on web sites.

In the Fidesfit-client, the login + password are used to create XMLHttpRequests (with always the same
login + password) that are then later used to query a singe online services provider.

Concretely for this usecase, the "hostname" in 
http://mxr.mozilla.org/mozilla-central/source/toolkit/components/passwordmgr/nsILoginManager.idl
calls, is "chrome://fidesfit-client" and not a web site URL, just as explained in https://developer.mozilla.org/en/XPCOM_Interface_Reference/nsILoginManager/Using_nsILoginManager#Creating_a_local_extension_login

> It sounds to me like you're not really
> wanting to test the "save password" work flow here.  What you want to test
> is the content on the password protected site, so why not simply store your
> login, browse to the site, and go on with your test?
> 

What the Fidesfit company has asked me to do is to add a mandatory check at the extension starts (which is delayed until the browser window loads) that the user id, the password and the provider URL are set. And if they are not set a configuration window pops up and makes it mandatory to fill in the fields.

If that was not possible to preset the password with Mozmill I would have to put code related to testing in the production code, which is allegedly a bad practice.

Is the usecase clearer with those details?

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Marc, for now I would propose you are using a separate profile on your side to run those tests, which already has the password set. I know that those tests could not be run with Mozmill Crowd or our automation scripts. From the Mozmill side I would expect we need bug 642843 resolved first.

From the automation side we can try to find a temporary solution for your tests. I would suggest you file a bug in the Mozmill Tests component and make it blocking bug 568958.

Comment 18

[Marc-Aurèle DARCHE]

(In reply to comment #17)
> Marc, for now I would propose you are using a separate profile on your side
> to run those tests, which already has the password set.
>

I have done exactly that for all the Fidesfit-client Mozmill tests at the end of last week. And this works well :-) Thanks again. This way of doing just seems not really safe for the long-term future.

So, as for myself, *this present ticket doesn't block anything anymore*, both for the initial landing of the Fidesfit-client tests and the new versions on which I'm still very active at the moment.

Please note that this change of setting a password in a profile has been coded only on the latest Fidesfit-client versions, not the version in the process of being reviewed in bug 570493.

> I know that those tests could not be run with Mozmill Crowd or 
> our automation scripts.
>

:'(

> From the Mozmill side I would expect we need bug 642843 resolved first.
> 

Yes, bug 642843 would be a must have! At present I have added many dedicated macros in my test Makefile to clean the profile after each test run.

> From the automation side we can try to find a temporary solution for your
> tests. I would suggest you file a bug in the Mozmill Tests component and
> make it blocking bug 568958.

I don't really get it, what would be the subject of this new ticket you are suggesting?

Thanks

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #18)
> I don't really get it, what would be the subject of this new ticket you are
> suggesting?

I.e. the chance to select a subfolder in the add-ons test directory which contains a pre-configured profile with sample credentials our automation script can use to somehow inject it into the newly created profile. That way we wouldn't be dependent on the Mozmill integration.

Comment 20

[cmtalbert]

(In reply to comment #19)
> (In reply to comment #18)
> > I don't really get it, what would be the subject of this new ticket you are
> > suggesting?
> 
> I.e. the chance to select a subfolder in the add-ons test directory which
> contains a pre-configured profile with sample credentials our automation
> script can use to somehow inject it into the newly created profile. That way
> we wouldn't be dependent on the Mozmill integration.

I don't understand why this is useful.  It feels like a complicated workaround to a simple problem (simple problem being the rather complex issue around cloning an existing profile).

Comment 21

[cmtalbert]

(In reply to comment #16)
> (In reply to comment #15)
> > (In reply to comment #14)
> 
> > For this explicit use case, I don't understand why you can't simply write a
> > non-restart test that stores the password by writing to the password manager
> > directly before loading the site.
> >
> 
> The Fidesfit-client usecase is not about using a login + password for
> filling in forms on web sites.
> 
> In the Fidesfit-client, the login + password are used to create
> XMLHttpRequests (with always the same
> login + password) that are then later used to query a singe online services
> provider.
> 
> Concretely for this usecase, the "hostname" in 
> http://mxr.mozilla.org/mozilla-central/source/toolkit/components/passwordmgr/
> nsILoginManager.idl
> calls, is "chrome://fidesfit-client" and not a web site URL, just as
> explained in
> https://developer.mozilla.org/en/XPCOM_Interface_Reference/nsILoginManager/
> Using_nsILoginManager#Creating_a_local_extension_login
> 
> > It sounds to me like you're not really
> > wanting to test the "save password" work flow here.  What you want to test
> > is the content on the password protected site, so why not simply store your
> > login, browse to the site, and go on with your test?
> > 
> 
> What the Fidesfit company has asked me to do is to add a mandatory check at
> the extension starts (which is delayed until the browser window loads) that
> the user id, the password and the provider URL are set. And if they are not
> set a configuration window pops up and makes it mandatory to fill in the
> fields.
> 
> If that was not possible to preset the password with Mozmill I would have to
> put code related to testing in the production code, which is allegedly a bad
> practice.
> 
> Is the usecase clearer with those details?
So, I think I understand.  Let me rephrase to see if I really understand.  The issue here is that the login stuff is going to get queried when the browser window loads.  Therefore, if you put the code to set the login in your mozmill test, then you're racing with the extension's load process.  

Ideally you need the extension's credentials saved in the profile so that they can be reliably used onload, and so that there is no chance of a race condition.

Is that a good statement of the problem?

Comment 22

[Marc-Aurèle DARCHE]

(In reply to comment #21)
> Let me rephrase to see if I really understand. 
> The issue here is that the login stuff is going to get queried when the
> browser window loads.  Therefore, if you put the code to set the login in
> your mozmill test, then you're racing with the extension's load process.  
> 

Exactly.

> Ideally you need the extension's credentials saved in the profile so that
> they can be reliably used onload, and so that there is no chance of a race
> condition.
> 

This is a solution for sure. There may be others we haven't come up with yet.

> Is that a good statement of the problem?

This really is.

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mozmill will reach its end of life soon. We are currently working on getting all the tests for Firefox ported to Marionette. For status updates please see bug 1080766.