Closed Bug 1440437 Opened 3 years ago Closed 3 years ago

tp6 failing on windows in production

Categories

(Testing :: Talos, defect)

Version 3
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1280365

People

(Reporter: rwood, Assigned: rwood)

References

Details

(Whiteboard: [PI:March][stockwell disabled])

Attachments

(1 obsolete file)

Talos tp6 test is failing in production on the new tc win hw. This same test is green on try on the new hardware (why??). This is the failure on the new win machines in production [1]:

19:15:24     INFO -  Starting mitmproxy playback using command: C:\Users\task_1519244165\py3venv\Scripts\mitmdump -k -s ""C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\alternate-server-replay.py C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-google.mp C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-youtube.mp C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-amazon.mp C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-facebook.mp""
19:15:24     INFO -  C:\Users\task_1519244165\py3venv\Scripts\mitmdump: No such script: "C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\alternate-server-replay.py C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-google.mp C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-youtube.mp C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-amazon.mp C:\\\Users\\\task_1519244165\\\build\\\tests\\\talos\\\talos\\\mitmproxy\\\mitmproxy-recording-facebook.mp"
19:15:34     INFO -  Aborting: mitmproxy playback process failed to start, poll returned: 1

[1] https://treeherder.mozilla.org/logviewer.html#?job_id=163763253&repo=autoland&lineNumber=1290
try run (green):
21:58:39     INFO -  Mitmproxy version: 2.0.2 (release version)
21:58:39     INFO -  Python version: 3.6.1
21:58:39     INFO -  Platform: Windows-10-10.0.15063-SP0
21:58:39     INFO -  SSL version: OpenSSL 1.1.0f  25 May 2017
21:58:39     INFO -  Windows version: 10 10.0.15063 SP0 Multiprocessor Free
21:58:39     INFO - Return code: 0
21:58:39     INFO - Downloading the mitmproxy recording set using tooltool


autoland (failed):
9:15:22     INFO -  Mitmproxy: 3.0.0
19:15:22     INFO -  Python:    3.6.1
19:15:22     INFO -  OpenSSL:   OpenSSL 1.1.0g  2 Nov 2017
19:15:22     INFO -  Platform:  Windows-10-10.0.15063-SP0
19:15:23     INFO - Return code: 0
19:15:23     INFO - Downloading the mitmproxy recording set using tooltool
FYI: Moving tp6 win to tier3 in Bug 1440444
Whiteboard: [PI:February]
Note: When this is fixed and we re-enable tp6 on windows, also enable h2
Blocks: 1434056
Assignee: nobody → rwood
Status: NEW → ASSIGNED
I *think* I figured it out:

mitmproxy_requirements specifies 'mitmproxy==2.0.2' however on the new hardware, installing those requirements fails:

19:31:42     INFO - Running command: ['C:\\Users\\task_1519242428\\py3venv\\Scripts\\pip', 'install', '--timeout', '120', '--no-index', '--find-links', 'http://pypi.pvt.build.mozilla.org/pub', '--find-links', 'http://pypi.pub.build.mozilla.org/pub', '--trusted-host', 'pypi.pub.build.mozilla.org', '--trusted-host', 'pypi.pvt.build.mozilla.org', '-r', 'C:\\Users\\task_1519242428\\build\\tests\\talos\\talos\\mitmproxy\\mitmproxy_requirements.txt']
...
19:31:59     INFO -  Collecting cryptography==1.8.2 (from -r C:\Users\task_1519242428\build\tests\talos\talos\mitmproxy\mitmproxy_requirements.txt (line 9))
19:32:00     INFO -    Could not find a version that satisfies the requirement cryptography==1.8.2 (from -r C:\Users\task_1519242428\build\tests\talos\talos\mitmproxy\mitmproxy_requirements.txt (line 9)) (from versions: 0.5.2, 1.5.2, 1.8.1)
19:32:00     INFO -  No matching distribution found for cryptography==1.8.2 (from -r C:\Users\task_1519242428\build\tests\talos\talos\mitmproxy\mitmproxy_requirements.txt (line 9))
19:32:00    ERROR - Return code: 1
19:32:00     INFO - Installing mitmproxy

Later on, inside the py3 virtualenv, there's another 'pip install mitmproxy' - however normally it would find it local already from the requirements install above, so it would use that version in the py3 venv; but in this case the install failed above, so looks like in the py3 venv install, it is going external and getting the new mitmproxy 3, that doesn't work.
It works when running tp6 on windows locally because in talos mach [1] it sets:

'pypi_url': 'http://pypi.python.org/simple'

and is able to find crypto 1.8.2 there.

[1] https://searchfox.org/mozilla-central/rev/bd05e3853c6e982e2a35c1cc404b987b2bc914d6/testing/talos/mach_commands.py#66
We were just lucky on production in the past, after failing to install the mitmproxy requirements, when it did the mimproxy install in the py3 venv, it went to the external pypi it would get mitmproxy 2.0.2 anyway. Now it gets the new 3.0 which doesn't work.

I *believe* the solution is to get 'cryptography==1.8.2' added to our internal pypi at:

'--find-links', 'http://pypi.pvt.build.mozilla.org/pub', '--find-links', 'http://pypi.pub.build.mozilla.org/pub', '--trusted-host', 'pypi.pub.build.mozilla.org',
:jlund, can you (or someone on your team) confirm that 'cryptography' 1.8.2 is not available on our internal pypi servers, and if not - how do we go about getting that package added? Thanks! :)
Flags: needinfo?(jlund)
Looking on http://pypi.pub.build.mozilla.org/pub/

Confirmed there is no 1.8.2 (latest is cryptography-1.8.1.tar.gz).

Package is here:

https://pypi.python.org/pypi/cryptography/1.8.2
Comment on attachment 8954044 [details]
Bug 1440437 - temporary fix/workaround for tp6 win production,

https://reviewboard.mozilla.org/r/223198/#review229138

this gets us going for now and we can unblock other work until we solve this the right way.
Attachment #8954044 - Flags: review?(jmaher) → review+
Summary: tp6 failing on new win h/w in production → tp6 failing on windows in production
Sorry, I missed this.

@rwood, if you would like a python package to be uploaded to our pypi servers, please file a buildduty bug stating so with a link to the package/zip and expected shasum.
Flags: needinfo?(jlund) → needinfo?(rwood)
Thanks Jordan, I filed Bug 1442240
Flags: needinfo?(rwood)
Awesome, tp6 should work in production again now with Bug 1442240 being resolved
Depends on: 1442240
Comment on attachment 8954044 [details]
Bug 1440437 - temporary fix/workaround for tp6 win production,

Work-around no longer required
Attachment #8954044 - Attachment is obsolete: true
As :jmaher suggested, I found a previous try run where tp6 was failing (on new moonshot h/w) because cryptography 1.8.2 wasn't available - and I re-triggered tp6 [1].

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=b9de33e3b54bd0aef8cc5644dd6caa3e0ab611af
Whiteboard: [PI:February] → [PI:March]
Whiteboard: [PI:March] → [PI:March][stockwell disabled]
:rwood- can we turn these back on now? I believe our dependency on the crypto package has been updated
Flags: needinfo?(rwood)
The re-triggers on the new Win h/w did not work (comment 26). Do we want this turned on existing hw? I'm not even sure why it isn't running on the old h/w now via buildbot...
Flags: needinfo?(rwood)
hmm, it sounds like we will be turning it right off again as soon as we update- until we can get some machines to test on via try to get this fixed- :markco last said EOD Thursday for another test cycle, lets see if that works and we can test tp6 as well.
the patch in bug 1280365 will solve our problems here.
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1280365
You need to log in before you can comment on or make changes to this bug.