867005 - Mac OSX 10.6 nodes failing on release1.qa.mtv1.mozilla.com

Status: VERIFIED FIXED

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[u279076]

I was trying to run the ondemand_update automation for Firefox 21.0b5 updates on beta channel today but the OSX 10.6 nodes kept failing. I asked Dave Hunt to look into it but it was too late in his day to make any progress toward a solution.

Here's the output of one of the failed testruns:
> Started by user anonymous
> Building remotely on mm-osx-106-1 in workspace jenkins/workspace/ondemand_update
>
> Deleting project workspace... done
> 
> No emails were triggered.
> Copied 569 artifacts from "get_mozmill-environments » mac,master" build number 2
> Copied 1,200 artifacts from "get_mozmill-tests" build number 2
> Copied 76 artifacts from "get_mozmill-automation" build number 2
> [ondemand_update] $ mozmill-env/run mozdownload --type=$BUILD_TYPE --platform=$PLATFORM --version=$VERSION --locale=$LOCALE --build-number=$BUILD_NUMBER --directory=builds
> Traceback (most recent call last):
>  File "/Users/mozauto/jenkins/workspace/ondemand_update/mozmill-env/bin/mozdownload", line 5, in <module>
>    from pkg_resources import load_entry_point
>  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.py", line 2560, in <module>
>    parse_requirements(__requires__), Environment()
>  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.py", line 518, in resolve
>    raise DistributionNotFound(req)  # XXX put more info here pkg_resources.DistributionNotFound: mozdownload==1.7
> Build step 'Invoke XShell command' marked build as failure
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
> Sending email to: [e-mail]
> Finished: FAILURE

You can see more examples of this failure here:
http://release1.qa.mtv1.mozilla.com:8080/view/@ondemand/job/ondemand_update/65
http://release1.qa.mtv1.mozilla.com:8080/view/@ondemand/job/ondemand_update/64
http://release1.qa.mtv1.mozilla.com:8080/view/@ondemand/job/ondemand_update/63

It seems like this is only happening on the Mac OSX 10.6 nodes, 10.7 and 10.8 testruns worked fine.

Note that this impacts Firefox 21.0b5 sign-off today but I will cover it by manually spotchecking Mac OSX 10.6 updates.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I will check that. Not sure what's the cause of this problem actually. :/ The Mac Minis are the same and nothing has changed in their configuration.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So what I have seen is that in the environment we had both 1.6 and 1.7 egg information present. I filed https://github.com/whimboo/mozmill-environment/issues/39 to get this fixed.

That at least fixed the current failure but a new one popped up for the progressbar implementation where 'total_seconds' is not an attributre of datetime.timedelta. I will investigate further soon.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Henrik Skupin (:whimboo) from comment #2)
> That at least fixed the current failure but a new one popped up for the
> progressbar implementation where 'total_seconds' is not an attributre of
> datetime.timedelta. I will investigate further soon.

This is actually a regression in our code from mozdownload 1.6 which means we lost backward compatibility with Python 2.6. We have to get this fixed and I will temporarily upload a new environment for mac which includes 1.6.

I filed the issue as:
https://github.com/mozilla/mozdownload/issues/73

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Both pull requests are awaiting review from Dave now. As mentioned before I reverted mozdownload to version 1.6 for Mac and this solved the problem. Jobs can successfully grab builds again.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We have released mozdownload 1.7.1 with the fix in place. I have uploaded new environments and updated both CI to make use of it.

Comment 6

[u279076]

Thanks for the quick turnaround. I will verify this with the upcoming Firefox 21.0b6 release.

Comment 7

[u279076]

I did not encounter this issue yesterday with the ondemand update tests for Firefox 21.0b6 on betatest and releasetest channel.