Closed Bug 1053977 Opened 10 years ago Closed 10 years ago

Request EDID for all QA maintained Mac minis in qa.scl3.mozilla.com

Categories

(Infrastructure & Operations :: DCOps, task)

Product:
Infrastructure & Operations
Component:
DCOps
Platform:
All
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: whimboo, Assigned: vinh)

References

Details

(Whiteboard: [qa-automation-blocked]) 

QA has a couple of macs running in SCL3 for testing with Mozmill and TPS. All those behave very slow when being connected via VNC, and the screenresolution can't be set to a higher value. All this should be related to the missing connected screen. So in case of RelEng all machines got those dongles or resistors. I also would like to request those for all of our machines.

Here the list of all the mac minis we have in SCL3:

qa-deploystudio1.qa.scl3.mozilla.com 10.22.73.46

mm-osx-106.qa.scl3.mozilla.com 10.22.73.76
mm-osx-107.qa.scl3.mozilla.com 10.22.73.78
mm-osx-108.qa.scl3.mozilla.com 10.22.73.79
mm-osx-109.qa.scl3.mozilla.com 10.22.73.77

mm-osx-106-1.qa.scl3.mozilla.com 10.22.73.50
mm-osx-106-2.qa.scl3.mozilla.com 10.22.73.51
mm-osx-106-3.qa.scl3.mozilla.com 10.22.73.52
mm-osx-106-4.qa.scl3.mozilla.com 10.22.73.135
mm-osx-107-1.qa.scl3.mozilla.com 10.22.73.53
mm-osx-107-2.qa.scl3.mozilla.com 10.22.73.54
mm-osx-107-3.qa.scl3.mozilla.com 10.22.73.55
mm-osx-107-4.qa.scl3.mozilla.com 10.22.73.136
mm-osx-108-1.qa.scl3.mozilla.com 10.22.73.56
mm-osx-108-2.qa.scl3.mozilla.com 10.22.73.57
mm-osx-108-3.qa.scl3.mozilla.com 10.22.73.58
mm-osx-108-4.qa.scl3.mozilla.com 10.22.73.137
mm-osx-109-1.qa.scl3.mozilla.com 10.22.73.131
mm-osx-109-2.qa.scl3.mozilla.com 10.22.73.132
mm-osx-109-3.qa.scl3.mozilla.com 10.22.73.133
mm-osx-109-4.qa.scl3.mozilla.com 10.22.73.134

tps-osx-109-1.qa.scl3.mozilla.com 10.22.73.164
tps-osx-109-2.qa.scl3.mozilla.com 10.22.73.165

Dustin, is that request doable for your team? If not, can you please move it to the appropriate component?
dcops should be able to help you with this.
Assignee: relops → server-ops-dcops
Component: RelOps → Server Operations: DCOps
Product: Infrastructure & Operations → mozilla.org
QA Contact: arich → dmoore
Thanks Amy! Please be informed that I will be away next week. If you want to test it first with a single dongle / resistor please talk to Andrei or Andreea. We can then test the speed improvements by comparing minis with and without those hw extension.
:vinh, did we keep the dongles from scl1?
colo-trip: --- → scl3
The VGA dongles were handed out along with each donated r3 minis.  We have only a handful left but they do not have the resistors on them.
releng had a contractor make these dongles and additional hardware will need to be purchased. the information/mana page isn't complete though. https://mana.mozilla.org/wiki/display/SYSADMIN/Mac+Mini+Video+Dongles?src=search

we'll test an EDID as a work-around the next time we're at the data center.
(In reply to Van Le [:van] from comment #5)
> we'll test an EDID as a work-around the next time we're at the data center.

Is that the kind of dongle all releng machines are using right now? If that is the case it would be what we want. Dustin may be able to give a better answer. I'm not familiar with all that.
Yes, we're using EDID simulators, not dongles.  Dongles don't work with r4 and higher minis.
:whimboo - Many of these mac minis will require power down so that I can best fit the EDIDs.  Would like to go down the list of minis that you provided.
(In reply to Vinh Hua [:vinh] from comment #8)
> :whimboo - Many of these mac minis will require power down so that I can
> best fit the EDIDs.  Would like to go down the list of minis that you
> provided.

Henrik is on PTO this week.
As he mentioned in comment 2 we should probably start by installing the EDID in one machine, and test if it really makes a difference.

All OSX machines are currently in use. Once they finish their workload I will take one machine offline and will let you know which one we target.
Flags: needinfo?(andrei.eftimie)
I took mm-osx-108-4 offline from our CI infrastructure.

This is: mm-osx-108-4.qa.scl3.mozilla.com 10.22.73.137

Vinh, please use this machine to install the EDID simulator.
You can safely power it down if needed.
Let me know once its done so we'll test it and report back.

Thanks
Flags: needinfo?(andrei.eftimie) → needinfo?(vhua)
Andrei - I've attached the EDID to mm-osx-108-4.qa.scl3.mozilla.com.  Native resolution should be at 1600x1200.
Flags: needinfo?(vhua)
Thanks Vinh, and sorry that no-one replied yet for this update. So I finally checked the box which you have updated, and I can successfully connect via VNC. When I open the display preferences I see all the available screen resolutions now! Also switching to a different one doesn't end-up with a black screen until the next boot anymore. So here it has made a good impact.

So what I would propose is that we update next our staging nodes with the EDID, so we can also check the differences for 10.6, 10.7, and 10.9. Vinh, do you have 4 more EDID for the following boxes?

mm-osx-106.qa.scl3.mozilla.com 10.22.73.76
mm-osx-107.qa.scl3.mozilla.com 10.22.73.78
mm-osx-108.qa.scl3.mozilla.com 10.22.73.79
mm-osx-109.qa.scl3.mozilla.com 10.22.73.77
Summary: Request for dongles / resistors for QA maintained minis in qa.scl3.mozilla.com → Request EDID for all QA maintained Mac minis in qa.scl3.mozilla.com
Blocks: 996629
> mm-osx-106.qa.scl3.mozilla.com 10.22.73.76
> mm-osx-107.qa.scl3.mozilla.com 10.22.73.78
> mm-osx-108.qa.scl3.mozilla.com 10.22.73.79
> mm-osx-109.qa.scl3.mozilla.com 10.22.73.77

:whimboo- These four minis are sitting on one rack tray that does not have enough space to place the EDIDs.  Will need to split the minis onto different trays, which requires some downtime.
That sounds fine. Given that you are in PDT timezone we could do this later today? It's one of the days when I work in the evening, so we should be able to cover it. Best you tell me a time, and I will be around.
I have to mark this as qa-automation-blocked now, given that without the EDID connected, puppetagain will fail on those boxes for the screenresolution module.
Assignee: server-ops-dcops → vhua
Status: NEW → ASSIGNED
Whiteboard: [qa-automation-wanted] → [qa-automation-blocked]
If you can comment when you take them down, I can then start working on the move.  Should only take about 30 mins.
Would it possible for you to join IRC for coordination? Not sure if your last comment means that you can do it right now or not.
Oh sorry Henrik I meant later.  It's currently 1:48am over here.
> mm-osx-106.qa.scl3.mozilla.com 10.22.73.76
> mm-osx-107.qa.scl3.mozilla.com 10.22.73.78
> mm-osx-108.qa.scl3.mozilla.com 10.22.73.79
> mm-osx-109.qa.scl3.mozilla.com 10.22.73.77

has EDIDs attached.
Thanks Vinh! I just tested the 10.7 box in regards of puppetagain and its screenresolution module. Now with the EDID installed it no longer raises errors, but succeeds successfully:

Aug 27 13:34:02 mm-osx-107 screenresolution[924]: starting screenresolution argv=/usr/local/bin/screenresolution get 
Aug 27 13:34:02 mm-osx-107 screenresolution[924]: Display 0: 1600x1200x32
Aug 27 13:34:02 mm-osx-107 screenresolution[926]: starting screenresolution argv=/usr/local/bin/screenresolution set 1024x768x32 
Aug 27 13:34:02 mm-osx-107 screenresolution[926]: set mode on display 0 to 1024x768x32
Aug 27 13:34:03 mm-osx-107 puppet-agent[173]: (/Stage[main]/Screenresolution/Exec[set-resolution]/returns) executed successfully

I will bring back up all the staging nodes now, and we check those over the next day. Once we are confident that all is working fine we could continue here.

Vinh, are there enough EDIDs left for all the other minis? Additionally to the minis listed above, I missed two more, which are used for TPS testing and which also need an EDID.
Another positive observation is that now all Java related windows are shown without any artifacts (partly updated). So far I really like that change. Lets see how tomorrows testruns will work.
Everything looks great on all staging machines which got an EDID installed.
They feel snappier, and I don't see any graphical artifacts we used to have (slow to none loading of certain windows, especially Java).

All tests on these machines appear to have run well. I don't see any negative side-effect.
That's great to hear! Thanks Andrei.

Vinh, in regards of all the other machines left, would they also have to be turned off? Or can the EDID plugged in while they are running? If they have to be turned off, I would propose that we continue work on Wednesday next week when all the Firefox releases are out.
Flags: needinfo?(vhua)
:whimboo - Yup will need to be powered down, mainly because they require some shuffling around for space.
Flags: needinfo?(vhua)
Ok, so lets pass the releases, and then lets continue on Wednesday next week.
Vinh, can we get the EDID installed on the qa-deploystudio1.qa.scl3.mozilla.com as soon as possible. It seems to block us a bit in correctly working via VNC on that machine.
EDID installed on qa-deploystudio1.qa.scl3.mozilla.com
(In reply to Vinh Hua [:vinh] from comment #27)
> EDID installed on qa-deploystudio1.qa.scl3.mozilla.com

Damn. I made a mistake the actual box to install the EDID should have been deploystudio1.qa.scl3.mozilla.com. Dustin and myself already had a discussion this afternoon about the confusing naming. Sorry. If it doesnt hurt you can leave the EDIT on qa-deploystudio1.
EDIDs attached:
mm-osx-106-1.qa.scl3.mozilla.com 10.22.73.50
mm-osx-106-2.qa.scl3.mozilla.com 10.22.73.51
mm-osx-106-3.qa.scl3.mozilla.com 10.22.73.52
mm-osx-107-1.qa.scl3.mozilla.com 10.22.73.53
mm-osx-107-2.qa.scl3.mozilla.com 10.22.73.54
mm-osx-107-3.qa.scl3.mozilla.com 10.22.73.55
mm-osx-108-1.qa.scl3.mozilla.com 10.22.73.56
mm-osx-108-2.qa.scl3.mozilla.com 10.22.73.57
mm-osx-108-3.qa.scl3.mozilla.com 10.22.73.58
mm-osx-109-2.qa.scl3.mozilla.com 10.22.73.132
mm-osx-109-3.qa.scl3.mozilla.com 10.22.73.133
mm-osx-109-4.qa.scl3.mozilla.com 10.22.73.134


Still needs EDID:
mm-osx-106-4.qa.scl3.mozilla.com 10.22.73.135	
mm-osx-107-4.qa.scl3.mozilla.com 10.22.73.136
mm-osx-108-4.qa.scl3.mozilla.com 10.22.73.137	
mm-osx-109-1.qa.scl3.mozilla.com 10.22.73.131	
tps-osx-109-1.qa.scl3.mozilla.com 10.22.73.164
tps-osx-109-2.qa.scl3.mozilla.com 10.22.73.165
(In reply to Vinh Hua [:vinh] from comment #29)
> EDIDs attached:
> mm-osx-106-1.qa.scl3.mozilla.com 10.22.73.50
> mm-osx-106-2.qa.scl3.mozilla.com 10.22.73.51
> mm-osx-106-3.qa.scl3.mozilla.com 10.22.73.52
> mm-osx-107-1.qa.scl3.mozilla.com 10.22.73.53
> mm-osx-107-2.qa.scl3.mozilla.com 10.22.73.54
> mm-osx-107-3.qa.scl3.mozilla.com 10.22.73.55
> mm-osx-108-1.qa.scl3.mozilla.com 10.22.73.56
> mm-osx-108-2.qa.scl3.mozilla.com 10.22.73.57
> mm-osx-108-3.qa.scl3.mozilla.com 10.22.73.58
> mm-osx-109-2.qa.scl3.mozilla.com 10.22.73.132
> mm-osx-109-3.qa.scl3.mozilla.com 10.22.73.133
> mm-osx-109-4.qa.scl3.mozilla.com 10.22.73.134

Great to hear that this was possible without having to shutdown those boxes! That saved us a lot of time. For all the above I have set the resolution to 1024x768 now.

Vinh, do you know when the remaining ones can be done?
We need to procure additional EDIDs.  This may be 2 weeks lead time.
All minis now have EDIDs attached.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
(In reply to Vinh Hua [:vinh] from comment #29)
> mm-osx-106-4.qa.scl3.mozilla.com 10.22.73.135	
> mm-osx-107-4.qa.scl3.mozilla.com 10.22.73.136
> mm-osx-108-4.qa.scl3.mozilla.com 10.22.73.137	
> mm-osx-109-1.qa.scl3.mozilla.com 10.22.73.131	
> tps-osx-109-1.qa.scl3.mozilla.com 10.22.73.164
> tps-osx-109-2.qa.scl3.mozilla.com 10.22.73.165

All machines have been updated for our wanted screen resolution. Thanks Vinh for installing all those EDIDs!!
Status: RESOLVED → VERIFIED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.