1739397 - Hardware for fxrecord startup performance harness in Toronto Office is unavailable

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

According to bug 1738369, no workers are available for gecko-t-fxrecorder, causing tasks to fail with deadline exceeded.

This Jira ticket indicates the following hardware is set up in the Toronto server room:

• fxrecorder01.corp.tor1.mozilla.com (10.242.24.97) - asset tag - 09390
• fxrunner01.corp.tor1.mozilla.com (10.242.24.96) - asset tag - 35944

Could someone please investigate the status of these boxes?

Comment 1

[:dhouse]

:markco do you know about these machines for fxrecord startup? re: https://bugzilla.mozilla.org/show_bug.cgi?id=1554314#c8 :fubar had mentioned you, but then it looks like :barret stood up the machines

Comment 2

[Mike Conley (:mconley) (:⚙️)]

I think jlin has also had a hand in setting these up in the Toronto office.

Comment 3

[Jonathan Lin - Not watching NI/offline [:jlin] [[e-mail]]]

Yes I set these up over in the toronto server room as indicated in the jira ticket.

There is an upcoming power maintenance this weekend at the Toronto office, and I will be on site on Monday Nov 8 to do the powerup - I can check these hosts to make sure they are back online on Monday Nov 8 after the office is back up, probably around end of day on Nov 8.

Comment 4

[Mark Cornmesser [:markco] OOO 2024/04/15]

(In reply to Dave House [:dhouse] from comment #1)

:markco do you know about these machines for fxrecord startup? re: https://bugzilla.mozilla.org/show_bug.cgi?id=1554314#c8 :fubar had mentioned you, but then it looks like :barret stood up the machines

There was discussion on us taking over management of this, but we never did.

Comment 5

[Jonathan Lin - Not watching NI/offline [:jlin] [[e-mail]]]

I powered on both fxrecorder01 as well as fxrunner01, both seem to be doing some reboots (probably queued up updates, processes, or tasks?) I don't know if it's working correctly, but usually I will see the led on the HD60S capture device light up when it's doing a record - so far it hasn't lit up yet, I'll come back to them in 30 min- 1 hour to see if they are done with their reboot cycles

Comment 6

[:dhouse]

It looks like it is working.
I retriggered this task (from https://bugzilla.mozilla.org/show_bug.cgi?id=1738369#c0) and it is running now: https://firefox-ci-tc.services.mozilla.com/tasks/fzahkkV1TQijYvaPrN-Flw

Comment 7

[Jonathan Lin - Not watching NI/offline [:jlin] [[e-mail]]]

excellent sounds like it's working - I didn't check in on it since my last comment 5 but if it's giving you successful runs that should be good.

Comment 8

[Cosmin Sabou [:CosminS]]

Hi, looks like that machine is not working. Jonathan, any chance you could check it out again?
Job in TH that's with deadline exceeded is here. Taskcluster link.

Comment 9

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The machine will be down until air conditioning in the room got fixed.

Comment 10

[Barret Rennie [:barret] (they/them)]

Bug 1743015 was filed for taking the tests offline during the outage, but it didn't hit m-c quick enough to stop a task from getting queued. Bug 1743026 will bring the tests back online once the Toronto office outage is over.

Comment 11

[Barret Rennie [:barret] (they/them)]

Re-resolving as FIXED because the tasks using these machines are not running so it shouldn't be causing further problems.

Comment 12

[Barret Rennie [:barret] (they/them)]

A few jobs timed out since the machines were turned back on. I just restarted the worker daemon on the fxrecorder01 machine and it seems to be accepting jobs again.