1440437 - tp6 failing on windows in production

Status: RESOLVED DUPLICATE of bug 1280365

(Testing :: Talos, defect)

Description

[Robert Wood [:rwood]]

Talos tp6 test is failing in production on the new tc win hw. This same test is green on try on the new hardware (why??). This is the failure on the new win machines in production [1]:

19:15:24     INFO -  Starting mitmproxy playback using command: C:\Users\task_1519244165\py3venv\Scripts\mitmdump -k -s ""C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\alternate-server-replay.py C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-google.mp C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-youtube.mp C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-amazon.mp C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-facebook.mp""
19:15:24     INFO -  C:\Users\task_1519244165\py3venv\Scripts\mitmdump: No such script: "C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\alternate-server-replay.py C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-google.mp C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-youtube.mp C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-amazon.mp C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-facebook.mp"
19:15:34     INFO -  Aborting: mitmproxy playback process failed to start, poll returned: 1

[1] https://treeherder.mozilla.org/logviewer.html#?job_id=163763253&repo=autoland&lineNumber=1290

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

try run (green):
21:58:39     INFO -  Mitmproxy version: 2.0.2 (release version)
21:58:39     INFO -  Python version: 3.6.1
21:58:39     INFO -  Platform: Windows-10-10.0.15063-SP0
21:58:39     INFO -  SSL version: OpenSSL 1.1.0f  25 May 2017
21:58:39     INFO -  Windows version: 10 10.0.15063 SP0 Multiprocessor Free
21:58:39     INFO - Return code: 0
21:58:39     INFO - Downloading the mitmproxy recording set using tooltool


autoland (failed):
9:15:22     INFO -  Mitmproxy: 3.0.0
19:15:22     INFO -  Python:    3.6.1
19:15:22     INFO -  OpenSSL:   OpenSSL 1.1.0g  2 Nov 2017
19:15:22     INFO -  Platform:  Windows-10-10.0.15063-SP0
19:15:23     INFO - Return code: 0
19:15:23     INFO - Downloading the mitmproxy recording set using tooltool

Comment 2

[Robert Wood [:rwood]]

FYI: Moving tp6 win to tier3 in Bug 1440444

Comment 3

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7f7cce66a2ff65baaec00dd7d46af0fa81f5f0b3

Comment 4

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f7bd52e11b4e740e83c129a6bc89d671c745f7d7

Comment 5

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1265e6e5f15634a4363e051f91249eb3fc0e5ed2

Comment 6

[Robert Wood [:rwood]]

Note: When this is fixed and we re-enable tp6 on windows, also enable h2

Comment 7

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=139580eef09112c17f9c9f16804c4270a2a5fe5a

Comment 8

[Robert Wood [:rwood]]

I *think* I figured it out:

mitmproxy_requirements specifies 'mitmproxy==2.0.2' however on the new hardware, installing those requirements fails:

19:31:42     INFO - Running command: ['C:\\Users\\task_1519242428\\py3venv\\Scripts\\pip', 'install', '--timeout', '120', '--no-index', '--find-links', 'http://pypi.pvt.build.mozilla.org/pub', '--find-links', 'http://pypi.pub.build.mozilla.org/pub', '--trusted-host', 'pypi.pub.build.mozilla.org', '--trusted-host', 'pypi.pvt.build.mozilla.org', '-r', 'C:\\Users\\task_1519242428\\build\\tests\\talos\\talos\\mitmproxy\\mitmproxy_requirements.txt']
...
19:31:59     INFO -  Collecting cryptography==1.8.2 (from -r C:\Users\task_1519242428\build\tests\talos\talos\mitmproxy\mitmproxy_requirements.txt (line 9))
19:32:00     INFO -    Could not find a version that satisfies the requirement cryptography==1.8.2 (from -r C:\Users\task_1519242428\build\tests\talos\talos\mitmproxy\mitmproxy_requirements.txt (line 9)) (from versions: 0.5.2, 1.5.2, 1.8.1)
19:32:00     INFO -  No matching distribution found for cryptography==1.8.2 (from -r C:\Users\task_1519242428\build\tests\talos\talos\mitmproxy\mitmproxy_requirements.txt (line 9))
19:32:00    ERROR - Return code: 1
19:32:00     INFO - Installing mitmproxy

Later on, inside the py3 virtualenv, there's another 'pip install mitmproxy' - however normally it would find it local already from the requirements install above, so it would use that version in the py3 venv; but in this case the install failed above, so looks like in the py3 venv install, it is going external and getting the new mitmproxy 3, that doesn't work.

Comment 9

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cee652fb7b8d7cb009e18f075bf609f8f35e30d2

Comment 10

[Robert Wood [:rwood]]

It works when running tp6 on windows locally because in talos mach [1] it sets:

'pypi_url': 'http://pypi.python.org/simple'

and is able to find crypto 1.8.2 there.

[1] https://searchfox.org/mozilla-central/rev/bd05e3853c6e982e2a35c1cc404b987b2bc914d6/testing/talos/mach_commands.py#66

Comment 11

[Robert Wood [:rwood]]

We were just lucky on production in the past, after failing to install the mitmproxy requirements, when it did the mimproxy install in the py3 venv, it went to the external pypi it would get mitmproxy 2.0.2 anyway. Now it gets the new 3.0 which doesn't work.

I *believe* the solution is to get 'cryptography==1.8.2' added to our internal pypi at:

'--find-links', 'http://pypi.pvt.build.mozilla.org/pub', '--find-links', 'http://pypi.pub.build.mozilla.org/pub', '--trusted-host', 'pypi.pub.build.mozilla.org',

Comment 12

[Robert Wood [:rwood]]

:jlund, can you (or someone on your team) confirm that 'cryptography' 1.8.2 is not available on our internal pypi servers, and if not - how do we go about getting that package added? Thanks! :)

Comment 13

[Robert Wood [:rwood]]

Looking on http://pypi.pub.build.mozilla.org/pub/

Confirmed there is no 1.8.2 (latest is cryptography-1.8.1.tar.gz).

Package is here:

https://pypi.python.org/pypi/cryptography/1.8.2

Comment 14

[Robert Wood [:rwood]]

3:03 PM <jmaher> probably https://pypi.python.org/packages/67/2b/f0bbd0329e10ba3f33588bb85b14cb9305564c17e49716f61f3b36972194/cryptography-1.8.2-cp36-cp36m-win_amd64.whl#md5=26e474df395a5ae42378619220ec6809

Comment 15

[Intermittent Failures Robot]

54 failures in 675 pushes (0.08 failures/push) were associated with this bug in the last 7 days. 

This is the #19 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 54

Platform breakdown:
* windows7-32: 27
* windows10-64: 27

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1440437&startday=2018-02-19&endday=2018-02-25&tree=all

Comment 16

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b779f0504ce45bd280e066731c3dfda0a609e2c2

Comment 17

[Robert Wood [:rwood]]

Attachment: Bug 1440437 - temporary fix/workaround for tp6 win production,

Review commit: https://reviewboard.mozilla.org/r/223198/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/223198/

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8954044 [details]
Bug 1440437 - temporary fix/workaround for tp6 win production,

https://reviewboard.mozilla.org/r/223198/#review229138

this gets us going for now and we can unblock other work until we solve this the right way.

Comment 19

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1d5575f5908862f99770968d4a4012bd2d6d085a

Comment 20

[Robert Wood [:rwood]]

Comment on attachment 8954044 [details]
Bug 1440437 - temporary fix/workaround for tp6 win production,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/223198/diff/1-2/

Comment 21

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bea663d56ca2f21951447b395aebb9aec21cccea

Comment 22

[Jordan Lund (:jlund)]

Sorry, I missed this.

@rwood, if you would like a python package to be uploaded to our pypi servers, please file a buildduty bug stating so with a link to the package/zip and expected shasum.

Comment 23

[Robert Wood [:rwood]]

Thanks Jordan, I filed Bug 1442240

Comment 24

[Robert Wood [:rwood]]

Awesome, tp6 should work in production again now with Bug 1442240 being resolved

Comment 25

[Robert Wood [:rwood]]

Comment on attachment 8954044 [details]
Bug 1440437 - temporary fix/workaround for tp6 win production,

Work-around no longer required

Comment 26

[Robert Wood [:rwood]]

As :jmaher suggested, I found a previous try run where tp6 was failing (on new moonshot h/w) because cryptography 1.8.2 wasn't available - and I re-triggered tp6 [1].

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=b9de33e3b54bd0aef8cc5644dd6caa3e0ab611af

Comment 27

[Joel Maher ( :jmaher ) (UTC -8)]

:rwood- can we turn these back on now? I believe our dependency on the crypto package has been updated

Comment 28

[Robert Wood [:rwood]]

The re-triggers on the new Win h/w did not work (comment 26). Do we want this turned on existing hw? I'm not even sure why it isn't running on the old h/w now via buildbot...

Comment 29

[Joel Maher ( :jmaher ) (UTC -8)]

hmm, it sounds like we will be turning it right off again as soon as we update- until we can get some machines to test on via try to get this fixed- :markco last said EOD Thursday for another test cycle, lets see if that works and we can test tp6 as well.

Comment 30

[Joel Maher ( :jmaher ) (UTC -8)]

the patch in bug 1280365 will solve our problems here.