1159421 - Release marionette-driver 0.5

Status: RESOLVED FIXED

(Testing :: Marionette Client and Harness, defect)

Comment 1

[David Burns :automatedtester]

dburns in ~/development/mozilla/mozilla-inbound/testing/marionette/driver λ python setup.py sdist upload
running sdist
running egg_info
creating marionette_driver.egg-info
writing requirements to marionette_driver.egg-info/requires.txt
writing marionette_driver.egg-info/PKG-INFO
writing top-level names to marionette_driver.egg-info/top_level.txt
writing dependency_links to marionette_driver.egg-info/dependency_links.txt
writing manifest file 'marionette_driver.egg-info/SOURCES.txt'
reading manifest file 'marionette_driver.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'marionette_driver.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt

running check
creating marionette_driver-0.5
creating marionette_driver-0.5/marionette_driver
creating marionette_driver-0.5/marionette_driver.egg-info
making hard links in marionette_driver-0.5...
hard linking requirements.txt -> marionette_driver-0.5
hard linking setup.py -> marionette_driver-0.5
hard linking marionette_driver/__init__.py -> marionette_driver-0.5/marionette_driver
hard linking marionette_driver/application_cache.py -> marionette_driver-0.5/marionette_driver
hard linking marionette_driver/by.py -> marionette_driver-0.5/marionette_driver
hard linking marionette_driver/date_time_value.py -> marionette_driver-0.5/marionette_driver
hard linking marionette_driver/decorators.py -> marionette_driver-0.5/marionette_driver
hard linking marionette_driver/errors.py -> marionette_driver-0.5/marionette_driver
hard linking marionette_driver/expected.py -> marionette_driver-0.5/marionette_driver
hard linking marionette_driver/geckoinstance.py -> marionette_driver-0.5/marionette_driver
hard linking marionette_driver/gestures.py -> marionette_driver-0.5/marionette_driver
hard linking marionette_driver/keys.py -> marionette_driver-0.5/marionette_driver
hard linking marionette_driver/marionette.py -> marionette_driver-0.5/marionette_driver
hard linking marionette_driver/selection.py -> marionette_driver-0.5/marionette_driver
hard linking marionette_driver/wait.py -> marionette_driver-0.5/marionette_driver
hard linking marionette_driver.egg-info/PKG-INFO -> marionette_driver-0.5/marionette_driver.egg-info
hard linking marionette_driver.egg-info/SOURCES.txt -> marionette_driver-0.5/marionette_driver.egg-info
hard linking marionette_driver.egg-info/dependency_links.txt -> marionette_driver-0.5/marionette_driver.egg-info
hard linking marionette_driver.egg-info/not-zip-safe -> marionette_driver-0.5/marionette_driver.egg-info
hard linking marionette_driver.egg-info/requires.txt -> marionette_driver-0.5/marionette_driver.egg-info
hard linking marionette_driver.egg-info/top_level.txt -> marionette_driver-0.5/marionette_driver.egg-info
Writing marionette_driver-0.5/setup.cfg
creating dist
Creating tar archive
removing 'marionette_driver-0.5' (and everything under it)
running upload
Submitting dist/marionette_driver-0.5.tar.gz to https://pypi.python.org/pypi
Server response (200): OK

Comment 2

[David Burns :automatedtester]

Changelog

Bug 1154691: Align Marionette with WebDriver errors
Bug 1083131: Always remove a profile created by marionette when the runner shuts down.
Bug 1154681: Use static lookups in errors.py
Bug 1154525: Make HTMLElement#location and #size use #rect internally
Bug 1157665: Add mozrunner dependency to marionette_driver
Bug 1158219: Don't set `id' field if undefined when switching frame

Comment 3

[David Burns :automatedtester]

changelog generated using hg log -M -r cc8eb386f147:: --template "{desc|firstline}\n" .

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

At least one of the changes (bug 1154691) was also to marionette-client, and we need to bump/release that package to restore compatibility. Also, this new package was automatically picked up by gaiatest (via marionette-client) so for now all on device Gaia tests are failing.

Comment 5

[Armen [:armenzg]]

@davehunt: how can we pin those gaiatest tests to specific versions of the packages?

FYI AutomatedTester uploaded version 0.12.

And how can we prevent them from grabbing the package from pypi.python.org?
We can upload them to http://pypi.pub.build.mozilla.org
We have had issues in the past where python.org would not be available in the past and all jobs would start failing, hence, we created pypi.p.b.m.o

Comment 6

[Armen [:armenzg]]

also, where can we see those jobs? Has the issue cleared? are they using version 0.5 of marionette driver? (even though the package as unpublished, we have found out that it was still reachable; in other words, unpublishing did nothing IIUC).

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Armen: gaiatest is already pinned to a specific version of marionette-client, however marionette-client is not pinned to a specific version of marionette-driver. We could add this pin, however it would need to be kept in sync to prevent buildbot failures as that uses whatever versions are in tree.

For PyPI, we've not seen any availability issues for our on device tests, but if we needed to switch to our internal PyPI then it should just be a case of setting the PIP index via environment variable. We should however still ensure that the versions on PyPI are compatible and working.

The jobs can be seen here (requires VPN): http://jenkins1.qa.scl3.mozilla.com:8080/view/UI/ and they are now operational again. We also publish results to Treeherder but they're hidden by default: https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&exclusion_profile=false&filter-job_type_symbol=Gip

This particular issue would have caused no results to be published to Treeherder, rather than a failed report to appear.

If we land a change in marionette-client that is necessary due to a marionette-driver change then we should version bump and release the client promptly so that we can update any downstream consumers before the marionette-driver is released. Alternatively, we need to pin marionette-driver. I suspect we could also do this in gaiatest. In many ways it was simpler when we only had one package. :)

Comment 8

[Armen [:armenzg]]

Thanks Dave for the explanation!

For tracking purposes, can I attach one of the logs? I don't know if there is a reason for the log not to be public.
I'm pasting below any relevant code which I considered OK to show that happened and is still OK to paste public [1].

We should indeed upload packages simultaneously. We need to figure out the right way to coordinate this.
Who are the people involved with marionette client/driver that could have helped coordinate this?

Besides the disjoint release, I highly encourage looking into preventing yesterday's issue on the harness. We have been there with mozharness and I remember how bad it was at some point for everyone involved. We worked on making sure that "uploading a package should not affect any of our jobs".

In mozharness we pin all versions on the harness and we block by doing this that the python packages would install whatever is latest.

I assume that before we uploaded marionette-driver 0.5, the version 0.4 was being used, however, as soon as we uploaded the latest the gaiatest started using it. What I'm trying to get to is that we would not have this issue if they had been pinned inside of the gaiatest.

In mozharness a year or two ago, we pinned all versions for all jobs and we switched to using the packages inside of the test.zip (instead of what mozilla's pypi). We did this because uploading a package to pypi was like a russian roulette.
The upload could affect any tree or any job and we had no way to control it.

In order not to make gaia-ui-tests requirements to strict you can perhaps try this trick I came up with.
On the harness that runs the firefox-ui-tests I created a requirements.txt that has all the packages needed by firefox-ui-tests and I pinned it [2][3].
So I take two steps, one to set up the environment with all dependencies (-r requirements.txt) and then pip install the main package I want.
In fact, the second step should not installing anything else besides itself.
Perhaps this can be used for gaiatest:
https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/tbpl_requirements.txt

# firefox-ui-tests has few requirements, however, its requirements do not have
# hard pinned versions. If a new version of those dependencies would be uploaded
# to the internal pypi, we would start using it unless we hard pinned them here
pip install $pip_options -r $DIR/firefox_ui_updates_requirements.txt  || exit
# We install firefox-ui-tests separetedly to cleary distinct the main tool from its dependencies
pip install $pip_options firefox-ui-tests==0.2  || exit


On the other side, gaia-test is not my harness so I can't say what you decide taking on or not, however, as much as uploading the package took of your time and other people, it also took a lot of my time trying to figure out how we could recover as soon as possible. In other words, if we could fix this we could prevent a lot of hours being saved. I hope this can be of any help and let me know if I can be of any assitance.

Side note, the others reasons for the mozilla pypi is not only availabilty but control and security of the environment.

[1]
..
Traceback (most recent call last):
  File "/var/jenkins/2/workspace/flame-kk-319.b2g-inbound.tinderbox.ui.functional.sanity/.env/bin/gaiatest", line 9, in <module>
    load_entry_point('gaiatest==0.33', 'console_scripts', 'gaiatest')()
  File "/var/jenkins/2/workspace/flame-kk-319.b2g-inbound.tinderbox.ui.functional.sanity/.env/local/lib/python2.7/site-packages/pkg_resources.py", line 356, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/var/jenkins/2/workspace/flame-kk-319.b2g-inbound.tinderbox.ui.functional.sanity/.env/local/lib/python2.7/site-packages/pkg_resources.py", line 2431, in load_entry_point
    return ep.load()
  File "/var/jenkins/2/workspace/flame-kk-319.b2g-inbound.tinderbox.ui.functional.sanity/.env/local/lib/python2.7/site-packages/pkg_resources.py", line 2147, in load
    ['__name__'])
  File "/var/jenkins/2/workspace/flame-kk-319.b2g-inbound.tinderbox.ui.functional.sanity/tests/python/gaia-ui-tests/gaiatest/__init__.py", line 5, in <module>
    from gaia_test import *
  File "/var/jenkins/2/workspace/flame-kk-319.b2g-inbound.tinderbox.ui.functional.sanity/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 12, in <module>
    from marionette import MarionetteTestCase, B2GTestCaseMixin
  File "/var/jenkins/2/workspace/flame-kk-319.b2g-inbound.tinderbox.ui.functional.sanity/.env/local/lib/python2.7/site-packages/marionette_client-0.11-py2.7.egg/marionette/__init__.py", line 5, in <module>
    from marionette_test import MarionetteTestCase, MarionetteJSTestCase, CommonTestCase, expectedFailure, skip, SkipTest
  File "/var/jenkins/2/workspace/flame-kk-319.b2g-inbound.tinderbox.ui.functional.sanity/.env/local/lib/python2.7/site-packages/marionette_client-0.11-py2.7.egg/marionette/marionette_test.py", line 18, in <module>
    from marionette_driver.errors import (
ImportError: cannot import name InvalidResponseException

[2]
https://github.com/armenzg/build-tools/blob/update_testing/release/common/setup_firefox_ui_updates.sh#L80
[3]
https://github.com/armenzg/build-tools/blob/update_testing/release/common/firefox_ui_updates_requirements.txt

Comment 9

[David Burns :automatedtester]

I would rather have downstream users of marionette being explicit about their requirements like in Firefox-ui-tests [1]. This would mean that downstream consumers would always have a reproduceable environments. We split out driver and client(yes there was a change here that required both releasing) so they can have their own release cycles. Having to bump the client requirements and release with each means that we have wasted effort in the past and will significantly reduce our development speed on these projects


https://github.com/armenzg/build-tools/blob/update_testing/release/common/firefox_ui_updates_requirements.txt

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚BST]

To close this off, marionette-driver will be pinned in gaiatest once bug 1162131 is resolved.

Comment 11

[Armen [:armenzg]]

(In reply to Dave Hunt (:davehunt) from comment #10)
> To close this off, marionette-driver will be pinned in gaiatest once bug
> 1162131 is resolved.

Excellent! Thanks Dave!

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Moving bugs for Marionette client due to component changes.