1147853 - Widespread "InternalError: Starting video failed" failures across all trees on AWS-based test instances

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

This is happening across all trees, including on pushes that had previously-green runs. AMI change or something maybe?

https://treeherder.mozilla.org/logviewer.html#?job_id=1224907&repo=mozilla-central

 1835 INFO TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/identity/test_peerConnection_peerIdentity.html | Error in test execution: InternalError: Starting video failed - expected PASS 

etc

All trees closed.

Comment 1

[Chris Cooper [:coop] (he/him)]

As mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1143018#c11, we normalized the basedir for slaves yesterday, both in the slavealloc db and in runslave.py. Assuming AMI generation happened properly last night, that change would have hit AWS today.

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #1)
> As mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1143018#c11, we
> normalized the basedir for slaves yesterday, both in the slavealloc db and
> in runslave.py. Assuming AMI generation happened properly last night, that
> change would have hit AWS today.

*However* we didn't remove the old dir or any references that pointed to it, so the resources should still be there. We might be relying on a relative path though in the tests.

Comment 3

[Chris Cooper [:coop] (he/him)]

Bug 1143018 is implicated in that it moved where the various buildbot state files (twistd.pid, logs) are created from /builds/slave/talos-slave to /builds/slave.

Morgan is looking into whether runner is failing because of this.

Comment 4

[Chris Cooper [:coop] (he/him)]

In the absence of an obvious cause, we're rolling back the AMIs to yesterday's versions while we continue to investigate. Existing instances should self-terminate and restart shortly.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Rolling back the AMI seems to have worked. Trees reopened.

Comment 6

[Jordan Lund (:jlund)]

resolved by https://bugzil.la/1149580