944471 - Introduce a Mutt testcase class

Status: RESOLVED WONTFIX

(Testing Graveyard :: Mozmill, enhancement)

Description

[Corentin Smith (:cosmith)]

As discussed in https://bugzilla.mozilla.org/show_bug.cgi?id=842717 the make_test methods are being replaced by a common do_test one, and the JS tests have moved to their own files.

It would probably be a good idea to introduce a MuttTestCase class subclassing unittest.TestCase to avoid the duplication of the code in all these do_test methods.

Here is a simple example of what this could look like, mostly taken from test_addons.py:


import unittest
import os
import mozmill
import manifestparser


class MuttTestCase(unittest.TestCase):
    """Mutt test case"""

    def do_test(self, relative_test_path=None, manifest_path=None,
                persisted=None, passes=0, fails=0, skips=0):

        abspath = os.path.dirname(os.path.abspath(__file__))

        if manifest_path:
            manifestpath = os.path.join(abspath, manifest_path)
            manifest = manifestparser.TestManifest(manifests=[manifestpath], strict=False)
            tests = manifest.active_tests()
        elif relative_test_path:
            testpath = os.path.join(abspath, relative_test_path)
            tests = [{'path': testpath}]

        m = mozmill.MozMill.create()
        if persisted:
            m.persisted.update(persisted)
        m.run(tests)
        results = m.finish(())

        self.assertEqual(len(results.passes), passes, 'Passes should match')
        self.assertEqual(len(results.fails), fails, 'Fails should match')
        self.assertEqual(len(results.skipped), skips, 'Skips should match')

        return (results, m.persisted)



I'm waiting for your thoughts and suggestions!

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I can't add more details. You already said everything. All sounds fine. We only would have to investigate what the minimum requirements would be for such a class and the do_test() method (or however it will be called finally).

Comment 2

[Dharmit Shah]

Hi Henrik,

I would like to work on this bug. It's my first time trying to contribute to an opensource project and I am interested in this bug as it's Python. Automation interests me as well.

Could you please guide me further on this? Should I start with setting up a dev environment first as mentioned on https://github.com/mozilla/mozmill? And set myself under Assigned To?

Thanks!
Dharmit.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hi Dharmit, thank you for your interest in working on this bug. But before we can get you started I have to ask back to Corentin if he still wants to work on his own filed bug. If it is the case, we can find something else for you. Otherwise you could take this. If you directly want to get started we can certainly find something else challenging for you! Just let me know.

Corentin, can you please tell us about your plans on this bug? Thanks.

Comment 4

[Corentin Smith (:cosmith)]

Hey, I don't have a lot of free time at the moment so you can work on this bug. I'll look for another one to work on when I have more time.

Comment 5

[Dharmit Shah]

Hi Henrik,

Since Corentin is busy, I will take this up and try to learn the stuff.

Could you suggest what should be my initial steps to get started? I am a beginner in Python and chose this bug as it's tagged as "Good First Bug". :)

Thank you!

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This bug hasn't been set as good first bug by myself. Actually I wouldn't consider it that easy. But if you are eager to learn and enjoy challenges I would be happy to mentor you on this one. Otherwise we can work on to find something else for you.

Comment 7

[Dharmit Shah]

Hi Henrik,

Okay. I would like to work on this one.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Dharmit, that sounds great! Please see the added URL to our Mozmill repo and how to set it up. Once you got the first mutt tests running, check comment 0 for proposals how to get this feature implemented. Let me know if you have questions.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Dharmit, do you have any questions in regards of getting started? Would be nice if you could give a feedback.

Comment 10

[Dharmit Shah]

Hi Henrik, sorry for the prolonged inactivity on this bug. I see you're away till 08/01. Could we talk about this post your return?

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Sure. Please let me know when you have time and we can meet on IRC, or put all the questions here in the bug. Then I can answer them if we cannot find a time.

Comment 12

[Dharmit Shah]

Henrik,

As per our conversation, I did "mutt testall -b /usr/bin/firefox" but the tests keep failing with the error:

    Exception: Cannot connect to jsbridge extension, port XXXXX


Where XXXXX is some port number. 

I did a quick grep over the codebase and it seems like the error is arising from the function 'wait_and_create_network()' in the file 'jsbridge/jsbridge/__init__.py'. For some reason the value of "connected" in this function stays "False" because of which we keep getting the exception "Cannot connect to jsbridge extension".

I tried with both "master" and "hotfix-2.0" branches. Platform is Fedora 20.

How do I move further on this? I am planning to setup a clean virtual machine (F20) and retry this.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Looks like that the jsbridge extension inside of Firefox doesn't work correctly. First please have a look into the Add-on Manager, and check if jsbridge and mozmill are enabled under extensions. If that is the case it would be helpful if you could run a test directly with mozmill and additionally specify the `--debug` argument. This will give a lot of output, which would help me to figure out what's wrong. Lets try to get this solved via IRC if possible.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

No activity happened on this bug for the last half year. So I'm un-assigning you.

Also Mozmill will reach its end of life soon. We are currently working on getting all the tests for Firefox ported to Marionette. For status updates please see bug 1080766.