838374 - release mozprofile, mozprocess, mozdevice (and other?) and mirror to m-c

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Jeff Hammel]

We'll need to have a new version of mozprofile (and potentially other)
on m-c for bug 790765 upon the landing of bug 837897.  There may be
other packages we need to bump and mirror as well.

Comment 1

[Jeff Hammel]

ah, yes, mozprocess: bug 838075

Comment 2

[Andrew Halberstadt [:ahal]]

I'll have a patch up soon for bug 837719 if you want to wait for that. Not sure how much of a rush you are in though.

Comment 3

[Jeff Hammel]

(In reply to Andrew Halberstadt [:ahal] from comment #2)
> I'll have a patch up soon for bug 837719 if you want to wait for that. Not
> sure how much of a rush you are in though.

I'm not in a huge hurry.  While bug 838075 looks straightforward, I suspect there will be difficulties encountered in the mozprocess tests on try. In any case, if we have to bump a few times, I'm not too worried. I'm aiming for as few bumps as possible in the immediate future, but that said I'm not going to waste worry over it.

Comment 4

[Jeff Hammel]

also probably mozfile: https://bugzilla.mozilla.org/show_bug.cgi?id=838390 ; will change summary when things become clearer

Comment 5

[Jeff Hammel]

< wlach> jhammel: syncing mozdevice to m-c as well would be useful

Comment 6

[Jeff Hammel]

It has been decided to block on bug 837719

Comment 7

[Jeff Hammel]

Looking closely, ABICT the following changes are needed and the packages to be mirrored:

- mozdevice 0.19 -> 0.20
- mozprocess 0.8 -> 0.9
- mozprofile 0.4 -> 0.5
- mozrunner 5.14 -> 5.15

Should we change mozrunner to specify {mozprofile,mozprocess} >= (version)
instead of ==? (Similarly, mozdevice?)

Comment 8

[Andrew Halberstadt [:ahal]]

(In reply to Jeff Hammel [:jhammel] from comment #7)
> - mozdevice 0.19 -> 0.20
> - mozprocess 0.8 -> 0.9
> - mozprofile 0.4 -> 0.5
> - mozrunner 5.14 -> 5.15

Makes sense to me. Everything I personally need is in these packages.

> Should we change mozrunner to specify {mozprofile,mozprocess} >= (version)
> instead of ==? (Similarly, mozdevice?)

I would be a fan of doing this. Theoretically we shouldn't run into any problems as long as we remember to bump the required dependencies anytime we break compatibility between two mozbase packages.

Comment 9

[Jeff Hammel]

(In reply to Andrew Halberstadt [:ahal] from comment #8)
> (In reply to Jeff Hammel [:jhammel] from comment #7)
> > - mozdevice 0.19 -> 0.20
> > - mozprocess 0.8 -> 0.9
> > - mozprofile 0.4 -> 0.5
> > - mozrunner 5.14 -> 5.15
> 
> Makes sense to me. Everything I personally need is in these packages.

Cool, thanks.

> > Should we change mozrunner to specify {mozprofile,mozprocess} >= (version)
> > instead of ==? (Similarly, mozdevice?)
> 
> I would be a fan of doing this. Theoretically we shouldn't run into any
> problems as long as we remember to bump the required dependencies anytime we
> break compatibility between two mozbase packages.

Cool.  I think I will take that as part of this bug (vs separately) because A. this needs to be done when versions are bumped (basically); and B. all the stakeholders are here.

BTW, as explained here http://k0s.org/mozilla/blog/20121113102023 in conjunction with here https://plus.google.com/+IanBicking/posts/M8Wpsfsyykc really all approaches are wrong.  I would *highly* recommend reading Ian's post on the subject if you're at all interested in versioning

Comment 10

[Jeff Hammel]

<ot:meta>
Also, BTW, not that a bug is the right platform for this, but let's not block in the future on this so much.  I want to see more releases vs large releases going forward.  If releasing is hard, we can streamline that, but blocking and doing a big-hunka stuff doesn't make things any easier.

I'll try to bring this up at tomorrow's meeting, if i remember
</ot:meta>

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

we need to add in mozcrash as well.

Comment 12

[Jeff Hammel]

So we'll need mozfile too:

│./generate_diff.py --develop mozdevice mozprocess mozprofile mozrunner mozcrashCloning into 'mozbase'...
remote: Counting objects: 4265, done.
remote: Compressing objects: 100% (1957/1957), done.
remote: Total 4265 (delta 2424), reused 4095 (delta 2274)
Receiving objects: 100% (4265/4265), 1.24 MiB | 451 KiB/s, done.
Resolving deltas: 100% (2424/2424), done.
Traceback (most recent call last):
  File "./generate_diff.py", line 379, in <module>
    main()
  File "./generate_diff.py", line 342, in main
    check_consistency(*current_package_info.values())
  File "./generate_diff.py", line 208, in check_consistency
    raise Exception('\n'.join(errors))
Exception: Dependency for package 'mozcrash' failed: mozfile-0.2 not >= 0.3


We need better functions where appropriate to introspect this sort of thing.  Yes, you can manually look at the setup.pys, but you can also manually use an abacus to do arithmetic.

Comment 13

[Jeff Hammel]

Filed bug 850285 for this.  In some perfect world there would be a Venn-ish sort of way to compare what is on m-c vs what is in github.

Comment 14

[Jeff Hammel]

Attachment: b57f3d45844fc57b04700f993f2198dd224060d2.diff

./generate_diff.py --develop mozdevice mozprocess mozprofile mozrunner mozcrash mozfile=0.3

Comment 15

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #14)
> Created attachment 724033 [details] [diff] [review]
> b57f3d45844fc57b04700f993f2198dd224060d2.diff
> 
> ./generate_diff.py --develop mozdevice mozprocess mozprofile mozrunner
> mozcrash mozfile=0.3

So something is obviously wrong:

│lsdiff b57f3d45844fc57b04700f993f2198dd224060d2.diff 
a/testing/mozbase/mozcrash/mozcrash/mozcrash.py
a/testing/mozbase/mozcrash/setup.py
a/testing/mozbase/mozfile/README.md
a/testing/mozbase/mozfile/mozfile/mozfile.py
a/testing/mozbase/mozfile/setup.py
b/testing/mozbase/mozfile/tests/is_url.py
a/testing/mozbase/mozfile/tests/manifest.ini
a/testing/mozbase/mozprocess/README.md
a/testing/mozbase/mozprocess/setup.py
a/testing/mozbase/mozprofile/README.md
a/testing/mozbase/mozprofile/mozprofile/__init__.py
a/testing/mozbase/mozprofile/mozprofile/addons.py
a/testing/mozbase/mozprofile/mozprofile/cli.py
a/testing/mozbase/mozprofile/mozprofile/permissions.py
a/testing/mozbase/mozprofile/mozprofile/prefs.py
a/testing/mozbase/mozprofile/mozprofile/profile.py
a/testing/mozbase/mozprofile/setup.py
a/testing/mozbase/mozrunner/README.md
a/testing/mozbase/mozrunner/setup.py

The patch to fix the mozprocess tests , bug 838075, is not reflected here :(

Comment 16

[Jeff Hammel]

Oh, this is because git is crazy.  Who would have known?
. 
git checkout tag results in a detached HEAD state.  Then checkout out HEAD will give you the same damn thing you already have.

Comment 17

[Jeff Hammel]

Attachment: bre-bumped diff

the lsdiff should basically be the same for this (for posting to try) as following bumping:

│lsdiff /home/jhammel/mozilla/src/mozilla-central/testing/mozbase/ac6e95d5a975b12f71f2c2d470cbf2f10f5944f7.diff
a/testing/mozbase/mozcrash/mozcrash/mozcrash.py
a/testing/mozbase/mozcrash/setup.py
a/testing/mozbase/mozdevice/README.md
a/testing/mozbase/mozdevice/mozdevice/__init__.py
a/testing/mozbase/mozdevice/mozdevice/b2gemulator.py
a/testing/mozbase/mozdevice/mozdevice/devicemanager.py
a/testing/mozbase/mozdevice/mozdevice/devicemanagerADB.py
a/testing/mozbase/mozdevice/mozdevice/devicemanagerSUT.py
a/testing/mozbase/mozdevice/mozdevice/dmcli.py
a/testing/mozbase/mozdevice/mozdevice/droid.py
a/testing/mozbase/mozdevice/mozdevice/emulator.py
a/testing/mozbase/mozdevice/mozdevice/emulator_battery.py
b/testing/mozbase/mozdevice/mozdevice/sutini.py
a/testing/mozbase/mozdevice/setup.py
a/testing/mozbase/mozdevice/sut_tests/dmunit.py
a/testing/mozbase/mozdevice/sut_tests/runtests.py
a/testing/mozbase/mozdevice/sut_tests/test_cat2.py
a/testing/mozbase/mozdevice/sut_tests/test_datachannel.py
a/testing/mozbase/mozdevice/sut_tests/test_exec.py
a/testing/mozbase/mozdevice/sut_tests/test_exec_env.py
a/testing/mozbase/mozdevice/sut_tests/test_getdir.py
a/testing/mozbase/mozdevice/sut_tests/test_info.py
a/testing/mozbase/mozdevice/sut_tests/test_isdir.py
a/testing/mozbase/mozdevice/sut_tests/test_prompt.py
a/testing/mozbase/mozdevice/sut_tests/test_ps.py
a/testing/mozbase/mozdevice/sut_tests/test_pull.py
a/testing/mozbase/mozdevice/sut_tests/test_push1.py
a/testing/mozbase/mozdevice/sut_tests/test_push2.py
a/testing/mozbase/mozdevice/sut_tests/test_pushbinary.py
a/testing/mozbase/mozdevice/sut_tests/test_pushsmalltext.py
a/testing/mozbase/mozdevice/tests/sut_mkdir.py
a/testing/mozbase/mozfile/README.md
a/testing/mozbase/mozfile/mozfile/mozfile.py
a/testing/mozbase/mozfile/setup.py
b/testing/mozbase/mozfile/tests/is_url.py
a/testing/mozbase/mozfile/tests/manifest.ini
a/testing/mozbase/mozprocess/README.md
a/testing/mozbase/mozprocess/setup.py
a/testing/mozbase/mozprocess/tests/manifest.ini
a/testing/mozbase/mozprocess/tests/mozprocess1.py
a/testing/mozbase/mozprocess/tests/mozprocess2.py
b/testing/mozbase/mozprocess/tests/test_mozprocess.py
a/testing/mozbase/mozprofile/README.md
a/testing/mozbase/mozprofile/mozprofile/__init__.py
a/testing/mozbase/mozprofile/mozprofile/addons.py
a/testing/mozbase/mozprofile/mozprofile/cli.py
a/testing/mozbase/mozprofile/mozprofile/permissions.py
a/testing/mozbase/mozprofile/mozprofile/prefs.py
a/testing/mozbase/mozprofile/mozprofile/profile.py
b/testing/mozbase/mozprofile/mozprofile/webapps.py
a/testing/mozbase/mozprofile/setup.py
b/testing/mozbase/mozprofile/tests/bug785146.py
b/testing/mozbase/mozprofile/tests/files/prefs_with_comments.js
b/testing/mozbase/mozprofile/tests/files/webapps1.json
b/testing/mozbase/mozprofile/tests/files/webapps2.json
a/testing/mozbase/mozprofile/tests/manifest.ini
b/testing/mozbase/mozprofile/tests/test_clone_cleanup.py
a/testing/mozbase/mozprofile/tests/test_preferences.py
b/testing/mozbase/mozprofile/tests/test_webapps.py
a/testing/mozbase/mozrunner/README.md
a/testing/mozbase/mozrunner/setup.py

Comment 18

[Jeff Hammel]

pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=6694d0dfc90e

build-only; not sure if this suffices

Comment 19

[William Lachance (:wlach)]

Just released mozdevice 0.20 with all the latest stuff for another purpose, so no need to version bump that (unless something changes before this bug is completed)

Comment 20

[Jeff Hammel]

Going to go ahead and bump.  Worst comes to worst, we have bumps that are pointless.

Mozdevice will have to be rebumped. https://github.com/mozilla/mozbase/blob/master/mozdevice/setup.py#L9 . Since we're bumping mozprocess and require strict versions, we must bump again.

To reiterate:

> > - mozprocess 0.8 -> 0.9
> > - mozprofile 0.4 -> 0.5
> > - mozrunner 5.14 -> 5.15
> > - mozdevice 0.20 -> 0.21

I will also make mozrunner and mozdevice loosely (>=) require versions vs strictly (==). 

Note that we are also mirroring mozfile, but the last released version, 0.3, seems to suffice.

Comment 21

[Jeff Hammel]

> I will also make mozrunner and mozdevice loosely (>=) require versions vs strictly (==). 

Since this requires a separate commit, filed blocker bug 850869 for it

Comment 22

[Jeff Hammel]

The versions are bumped and the packages uploaded to pypi:

(mozbase)│./versionbump.py --strict -m 'Bug 838374 - release mozprofile, mozprocess, mozdevice (and other?) and mirror to m-c' mozprocess=0.9 mozprofile=0.5 mozrunner=5.15 mozdevice=0.21
Checking for pypirc: /home/jhammel/.pypirc
Pulling from git://github.com/mozilla/mozbase.git master
Running ['git', 'pull', 'git://github.com/mozilla/mozbase.git', 'master'], {'cwd': '/home/jhammel/mozbase/src/mozbase', 'stderr': None, 'stdout': None}
From git://github.com/mozilla/mozbase
 * branch            master     -> FETCH_HEAD
Already up-to-date.
Bumping mozrunner == 5.14 => 5.15
Bumping mozprocess == 0.8 => 0.9
Bumping mozprofile == 0.4 => 0.5
Bumping mozdevice == 0.20 => 0.21
Commit changes to [e-mail]:mozilla/mozbase.git: Bug 838374 - release mozprofile, mozprocess, mozdevice (and other?) and mirror to m-c
Running ['git', 'commit', '-a', '-m', 'Bug 838374 - release mozprofile, mozprocess, mozdevice (and other?) and mirror to m-c'], {'cwd': '/home/jhammel/mozbase/src/mozbase'}
Running ['git', 'push', '[e-mail]:mozilla/mozbase.git', 'master'], {'cwd': '/home/jhammel/mozbase/src/mozbase', 'stderr': None, 'stdout': None}
Enter passphrase for key '/home/jhammel/.ssh/id_rsa': 
Counting objects: 19, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (10/10), done.
Writing objects: 100% (10/10), 1012 bytes, done.
Total 10 (delta 5), reused 0 (delta 0)
To [e-mail]:mozilla/mozbase.git
   0d8ad89..54d57da  master -> master
Updating tags for [e-mail]:mozilla/mozbase.git: mozrunner-5.15, mozprocess-0.9, mozprofile-0.5, mozdevice-0.21
Running ['git', 'pull', '--tags', '[e-mail]:mozilla/mozbase.git', 'master'], {'cwd': '/home/jhammel/mozbase/src/mozbase', 'stderr': None, 'stdout': None}
Enter passphrase for key '/home/jhammel/.ssh/id_rsa': 
remote: Counting objects: 2, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 2 (delta 0), reused 2 (delta 0)
Unpacking objects: 100% (2/2), done.
From github.com:mozilla/mozbase
 * branch            master     -> FETCH_HEAD
 * [new tag]         mozb2g-0.2 -> mozb2g-0.2
 * [new tag]         mozcrash-0.2 -> mozcrash-0.2
 * [new tag]         mozcrash-0.2.1 -> mozcrash-0.2.1
 * [new tag]         mozcrash-0.3 -> mozcrash-0.3
 * [new tag]         mozcrash-0.3.1 -> mozcrash-0.3.1
 * [new tag]         mozcrash-0.4 -> mozcrash-0.4
 * [new tag]         mozcrash-0.5 -> mozcrash-0.5
 * [new tag]         mozdevice-0.12 -> mozdevice-0.12
 * [new tag]         mozdevice-0.13 -> mozdevice-0.13
 * [new tag]         mozdevice-0.14 -> mozdevice-0.14
 * [new tag]         mozdevice-0.15 -> mozdevice-0.15
 * [new tag]         mozdevice-0.16 -> mozdevice-0.16
 * [new tag]         mozdevice-0.17 -> mozdevice-0.17
 * [new tag]         mozdevice-0.18 -> mozdevice-0.18
 * [new tag]         mozdevice-0.19 -> mozdevice-0.19
 * [new tag]         mozdevice-0.20 -> mozdevice-0.20
 * [new tag]         mozfile-0.1 -> mozfile-0.1
 * [new tag]         mozfile-0.2 -> mozfile-0.2
 * [new tag]         mozfile-0.3 -> mozfile-0.3
 * [new tag]         mozhttpd-0.5 -> mozhttpd-0.5
 - [tag update]      mozilla-central -> mozilla-central
 * [new tag]         mozinstall-1.5 -> mozinstall-1.5
 * [new tag]         moznetwork-0.2 -> moznetwork-0.2
 * [new tag]         moznetwork-0.21 -> moznetwork-0.21
 * [new tag]         mozprocess-0.8 -> mozprocess-0.8
 * [new tag]         mozrunner-5.14 -> mozrunner-5.14
Already up-to-date.
Running ['git', 'tag', 'mozrunner-5.15'], {'cwd': '/home/jhammel/mozbase/src/mozbase'}
Running ['git', 'tag', 'mozprocess-0.9'], {'cwd': '/home/jhammel/mozbase/src/mozbase'}
Running ['git', 'tag', 'mozprofile-0.5'], {'cwd': '/home/jhammel/mozbase/src/mozbase'}
Running ['git', 'tag', 'mozdevice-0.21'], {'cwd': '/home/jhammel/mozbase/src/mozbase'}
Running ['git', 'push', '--tags', '[e-mail]:mozilla/mozbase.git', 'master'], {'cwd': '/home/jhammel/mozbase/src/mozbase', 'stderr': None, 'stdout': None}
Enter passphrase for key '/home/jhammel/.ssh/id_rsa': 
Total 0 (delta 0), reused 0 (delta 0)
To [e-mail]:mozilla/mozbase.git
 * [new tag]         mozdevice-0.21 -> mozdevice-0.21
 * [new tag]         mozprocess-0.9 -> mozprocess-0.9
 * [new tag]         mozprofile-0.5 -> mozprofile-0.5
 * [new tag]         mozrunner-5.15 -> mozrunner-5.15
Uploading to pypi: mozrunner-5.15, mozprocess-0.9, mozdevice-0.21, mozprofile-0.5
Running ['/home/jhammel/mozbase/bin/python', 'setup.py', 'egg_info', '-RDb', '', 'sdist', 'upload'], {'cwd': '/home/jhammel/mozbase/src/mozbase/mozrunner'}
Running ['/home/jhammel/mozbase/bin/python', 'setup.py', 'egg_info', '-RDb', '', 'sdist', 'upload'], {'cwd': '/home/jhammel/mozbase/src/mozbase/mozprocess'}
Running ['/home/jhammel/mozbase/bin/python', 'setup.py', 'egg_info', '-RDb', '', 'sdist', 'upload'], {'cwd': '/home/jhammel/mozbase/src/mozbase/mozdevice'}
Running ['/home/jhammel/mozbase/bin/python', 'setup.py', 'egg_info', '-RDb', '', 'sdist', 'upload'], {'cwd': '/home/jhammel/mozbase/src/mozbase/mozprofile'}

Comment 23

[Jeff Hammel]

Attachment: post-bump diff

Comment 24

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #23)
> Created attachment 724849 [details] [diff] [review]
> post-bump diff

pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=8fe7bcac5b74

I only used, again build and xpcshell test as, AFAIK, these are the only things that can hit mozbase.  Am I wrong?  We should probably document this somewhere...

Comment 25

[Andrew Halberstadt [:ahal]]

Comment on attachment 724849 [details] [diff] [review]
post-bump diff

Review of attachment 724849 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good. Re: mozbase in m-c, the b2g tests currently copy over devicemanagerADB.py and devicemanager.py to tests.zip, but I checked and don't think there are any changes there that would warrant a new try push. Also, this behaviour is about to change very soon.

Comment 26

[Jeff Hammel]

Thanks, Andrew

Try run looks green. The tree is closed currently but this should be pushed when reopened

Comment 27

[Jeff Hammel]

pushed to inbound: http://hg.mozilla.org/integration/mozilla-inbound/rev/7d2ea731f5a5

Comment 28

[Jonathan Griffin (:jgriffin)]

Marionette uses mozprocess and mozdevice, and on desktop runs, mozprofile.  You can add marionette tests to a try run for desktop by adding 'marionette' to the -u field, you can add them to B2G by adding 'marionette-webapi'.

Comment 29

[Jeff Hammel]

filed bug 851286 about mozbase on try

Comment 30

[Ryan VanderMeulen [:RyanVM]]

Backed out for B2G test bustage.
https://hg.mozilla.org/integration/mozilla-inbound/rev/b3c09fc390c5

https://tbpl.mozilla.org/php/getParsedLog.php?id=20658521&tree=Mozilla-Inbound

14:36:28     INFO -  waiting for system-message-listener-ready...
14:36:28     INFO -  done
14:36:28     INFO -  will use zip to push directories
14:36:28     INFO -  args: ['/home/cltbld/talos-slave/test/build/xre/bin/xpcshell', '-g', '/home/cltbld/talos-slave/test/build/xre/bin', '-v', '170', '-f', '/home/cltbld/talos-slave/test/build/tests/reftest/reftest/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpD89Fzw';const _SERVER_PORT = '8888'; const _SERVER_ADDR ='10.0.2.2';", '-f', '/home/cltbld/talos-slave/test/build/tests/reftest/server.js']
14:36:28     INFO -  INFO | remotereftests.py | Server pid: 2604
14:36:35     INFO -  Traceback (most recent call last):
14:36:35     INFO -    File "runreftestb2g.py", line 568, in main
14:36:35     INFO -      retVal = reftest.runTests(manifest, options, cmdlineArgs)
14:36:35     INFO -    File "/home/cltbld/talos-slave/test/build/tests/reftest/runreftest.py", line 125, in runTests
14:36:35     INFO -      self.createReftestProfile(options, profileDir, reftestlist)
14:36:35     INFO -    File "runreftestb2g.py", line 454, in createReftestProfile
14:36:35     INFO -      if self._devicemanager._useDDCopy:
14:36:35     INFO -  AttributeError: 'DeviceManagerADB' object has no attribute '_useDDCopy'
14:36:35     INFO -  zip/unzip failure: falling back to normal push
14:36:35     INFO -  profileDir: /tmp/tmppvo42P
14:36:35     INFO -  Automation Error: Exception caught while running tests
14:36:36    ERROR - Return code: 1

Comment 31

[Jonathan Griffin (:jgriffin)]

So this error is caused by the fact that mozharness uses a now-removed function in mozdevice, catFile.  To fix this, we'll need to update mozharness.  See bug 848843.

This probably underscores the point that we can't categorically know right now who uses a particular API, and so we should be extremely cautious about removing/changing them.

Comment 32

[Jeff Hammel]

Oh and puns for "underscores" if intended.  This also underscores (teehee) the fact that underscored prefixes as non-public attributes as python convention dictates ARE COMPLETELY IGNORED by consumers.  Let's *not* do that in the future.  I'll curtail my rant for now, but TL;DR - underscoring attributes/method by itself does not constitute an API strategy.

Comment 33

[Jonathan Griffin (:jgriffin)]

Whoops, I think my comment was for a different mozbase breakage in gaia-ui-tests, not this patch.  *This* breakage is caused by https://github.com/mozilla/mozbase/commit/66bd6b7207bc3a359e831f9671983ba33b99d25c.  The solution here is to update the testrunner(s) not to use _useDDCopy.

Comment 34

[Andrew Halberstadt [:ahal]]

(In reply to Jeff Hammel [:jhammel] from comment #32)
> Oh and puns for "underscores" if intended.  This also underscores (teehee)
> the fact that underscored prefixes as non-public attributes as python
> convention dictates ARE COMPLETELY IGNORED by consumers.  Let's *not* do
> that in the future.  I'll curtail my rant for now, but TL;DR - underscoring
> attributes/method by itself does not constitute an API strategy.

We'll have to agree to disagree :)

a) underscores at the module level *do* have an affect (they won't be imported)
b) underscores at the class level at least signify to consumers that it is intended to be private. e.g "You can use this if you need to, but don't come running to us if we break it"

In this particular case, the bug happened because the harnesses were written before we started using underscores. We were in a situation where it was really difficult to tell which methods/attributes were meant to be private and which ones weren't (and of course used many of the properties that weren't). Then when we updated the API with underscores, we only updated the consumers to match instead of updating them to use the API properly (this is where we made the mistake). We never would have hit this if we had used underscores in mozdevice from the outset.

Comment 35

[Andrew Halberstadt [:ahal]]

I pushed this to try with the changes from bug 851374

https://tbpl.mozilla.org/?tree=Try&rev=4cfc772a4b82

Comment 36

[William Lachance (:wlach)]

(In reply to Jonathan Griffin (:jgriffin) from comment #31)
> So this error is caused by the fact that mozharness uses a now-removed
> function in mozdevice, catFile.  To fix this, we'll need to update
> mozharness.  See bug 848843.
> 
> This probably underscores the point that we can't categorically know right
> now who uses a particular API, and so we should be extremely cautious about
> removing/changing them.

That isn't the only incorrect usage of mozdevice in mozharness. :(

Just searching through device.py, I see the following api usage which tries to check return values which no longer exist (since we throw exceptions now, see bug 795456):

https://github.com/escapewindow/mozharness/blob/master/mozharness/mozilla/testing/device.py#L499
https://github.com/escapewindow/mozharness/blob/master/mozharness/mozilla/testing/device.py#L572

Last I remember hearing from aki (a few months ago at least), mozharness was set up to use an older version of mozdevice so as not to run into these problems. Anyway, if we want to fix these things, I *suspect* we can just delete the return code checking stuff. The commands that we're trying there aren't supposed to fail under normal circumstances. I'm not sure if there aren't any more subtle incompatbilities/breakages in mozdevice usage here lurking under the covers though.

Comment 37

[Jeff Hammel]

pushed to try, again: https://tbpl.mozilla.org/?tree=Try&rev=9417d760d0b3

This time with all unit tests

Comment 38

[Jeff Hammel]

Try was green except for a few known intermittents when last it loaded.  However, "OPEN. Try was reset 1730 PDT March 21. Older pushes are no longer available, use https://secure.pub.build.mozilla.org/buildapi". ABICT, things are good.  Going to be bold and push to inbound.  If something doesn't work, please feel free to backout.  ...ya try to cover all your bases and...

Comment 39

[Jeff Hammel]

pushed to inbound: https://hg.mozilla.org/integration/mozilla-inbound/rev/27fb990d7fc7

Comment 40

[Phil Ringnalda (:philor)]

The androids say:

Traceback (most recent call last):
  File "mochitest/runtestsremote.py", line 16, in <module>
    from automation import Automation
  File "/builds/tegra-249/test/build/tests/mochitest/automation.py", line 35, in <module>
    import mozcrash
  File "/builds/tegra-249/test/build/tests/mozbase/mozcrash/mozcrash/__init__.py", line 8, in <module>
    from mozcrash import *
  File "/builds/tegra-249/test/build/tests/mozbase/mozcrash/mozcrash/mozcrash.py", line 18, in <module>
    from mozfile import extract_zip
ImportError: No module named mozfile

Backed out in https://hg.mozilla.org/integration/mozilla-inbound/rev/10080f945066.

Comment 41

[Phil Ringnalda (:philor)]

(b2g, on the other hand, was quite happy - other than not having MINIDUMP_STACKWALK set, which I think is supposed to be set to the path to the binary, they were actually properly catching their constant crashes)

Comment 42

[Jonathan Griffin (:jgriffin)]

Looks like we probably need to add mozfile here: 
http://mxr.mozilla.org/mozilla-central/source/build/automation.py.in#36

and here:
http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/runxpcshelltests.py#30

Comment 43

[Jonathan Griffin (:jgriffin)]

I added these two things and pushed to try again:

https://tbpl.mozilla.org/?tree=Try&rev=68c78e862932

Comment 44

[Jonathan Griffin (:jgriffin)]

try run looked good, so: https://hg.mozilla.org/integration/mozilla-inbound/rev/78733111adb6

Comment 45

[Jeff Hammel]

Thank you, Jonathan!

Comment 46

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/78733111adb6