969052 - Dashboard outage at iriscouch on Feb 6th

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[u279076]

I've just run Firefox 28.0b1 update tests on betatest and in spite of reports being pushed correctly to mozmill-release they are not showing up on the dashboard. I see no errors in Mozmill-CI so it's completely baffling to me how this happened. 

For example:
http://mozauto.iriscouch.com/mozmill-release/817d21b2201918e7c56aeecf0e150433

The above report shows up nowhere on:
http://mozmill-release.blargon7.com/#/update/reports?branch=28.0&platform=All&from=2014-02-03&to=2014-02-06

None of the nodes reported any test failures so I'm not blocking Firefox 28.0b1 sign-off on this problem but it is still very severe, and somewhat exacerbated by the fact that no one was around to help be debug this issue.

Comment 1

[u279076]

The same goes for releasetest:
http://mozmill-release.blargon7.com/#/update/overview?branch=28.0&channel=releasetest&from=2014-02-06&to=2014-02-07

No reports in the dashboard despite successful testruns being reported by Mozmill-CI.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

iriscouch is down so that is the reason why you cannot access the dashboards. A simple test to do is to load https://mozauto.iriscouch.com/_utils/index.html. If that doesn't work the whole system is f&2§ed up. I will get in touch with iriscouch support so that they can bring it back.

Also I filed bug 969261 to bring the couch instance back in-house.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This has been fixed by nodejitsu. All is working fine again.

Comment 4

[u279076]

Thanks Henrik.