Today we had this in the agent log on bm36: 2012-05-21 12:19:06,770 Loading masters from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json 2012-05-21 12:19:06,976 Loading branches from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json then nothing. There was a related issue with machines cloning try, and two established connections to hg.m.o. I think we should give up if we're not successful in 10 seconds or so. We use urllib.urlopen() at the moment, which doesn't take a timeout. Switching to urllib2.urlopen might be the way to go.
Probably won't be fixing this because buildapi is going to be deprecated after all scheduling moves to Taskcluster.