I have checked a couple of our Firefox UI jobs on Treeherder and noticed that the execution times for our tests on Linux take twice the time as on OS X and Windows. This is by using mozharness as runner. A first look showed me that we have huge wait times while installing Python packages in the create_virtualenv step: 07:21:36 INFO - Downloading/unpacking psutil>=0.7.1 07:21:36 INFO - http://pypi.pub.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org has it available 07:21:36 INFO - http://pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com has it available 07:21:36 INFO - http://pypi.pub.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org has it available 07:23:37 INFO - Running setup.py (path:/home/mozauto/jenkins/workspace/mozilla-central_update/build/venv/build/psutil/setup.py) egg_info for package psutil Not sure why the proxxy1 index is queried for given that this should not happen in developer mode.
The problem is clearly located in the create_virtualenv step. Here what happens on those Linux boxes: 07:12:36 INFO - MultiFileLogger online at 20160207 07:12:36 in /home/mozauto/jenkins/workspace/mozilla-central_functional [..] 07:21:01 INFO - ##### Running install step. Compared to OS X: 05:25:18 INFO - MultiFileLogger online at 20160207 05:25:18 in /Users/mozauto/jenkins/workspace/mozilla-central_functional [..] 05:26:11 INFO - ##### Running install step. As you can see we have 8.5 minutes compared to 53s.
This goes back to release but keep in mind that we always make use of the mozharness version from mozilla-central for now. This might change so that we use the version of the branch /revision we are testing. That means we should take care of it down to beta (which becomes the next ESR) but not release.
Interestingly this is only a problem on production but not for staging Linux nodes. Maybe bug 1241811 plays into account here and it's something related to the pip v8 warning. I will see this later.
Looks like that this is really the case here. Having mm-ub-1404-1 updated I executed an update job and now the times are fine with 3 minutes again: https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=f19a31a82ae3&filter-searchStr=firefox%20ui%20update&filter-tier=1&filter-tier=2&filter-tier=3&selectedJob=1890599 Marking the dependency.
Fixed on all branches with bug 1241811 done.