1805214 - MacOS workers of the NSS continuous integration are failing

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, defect)

Description

[Benjamin Beurdouche [:beurdouche]]

The MacOS workers of the NSS continuous integration are failing apparently from the following push to nss-try on November 4th.
https://treeherder.mozilla.org/jobs?repo=nss-try&revision=07171669dbc765a77fa7bfa9cef7e8faefb9980d

The error message is abort: error: EOF occurred in violation of protocol (_ssl.c:661) which seems to be an infrastructure related issue.
https://treeherder.mozilla.org/logviewer?job_id=395529314&repo=nss-try&lineNumber=24

Comment 1

[:dhouse]

:beurdouche we've managed the hosting for these machines, but not the machines themselves. Do you have login information? I'll write out how to fix this (following comment), or I can get access and work on it if you do not have a login.

Comment 2

[:dhouse]

I thought this was a case needing filesystem cleanup for generic-worker, but it looks like something different:
This specific task failure appears to be that a hg clone failed/interrupted (and the error "file-missing-on-worker: Could not read directory '/Users/administrator/worker8/tasks/task_1667567316/public'" usually happens when a task doesn't complete and the artifact is not created).

Have you see many failures with hg clones and does there appear to be any pattern to it? We can ask macstadium to investigate or troubleshoot the network with us and ideally we can have specific times it has happened in the last few days or can reproduce the failure.

Comment 3

[Benjamin Beurdouche [:beurdouche]]

Unfortunately, I don't have credentials for these machines. We never had this pattern of failure before (at least not for the 3 years I have been here). I don't have access to MacStadium ressources (which would actually be good for the team) so asking them to look at the network connectivity might be good.

Comment 4

[Michelle Goossens [:masterwayz]]

I made a nss-try push and according to https://treeherder.mozilla.org/logviewer?job_id=405478210&repo=nss-try&lineNumber=4057 it fails but with a make error and no more SSL errors.

Comment 5

[Benjamin Beurdouche [:beurdouche]]

Kai, any intuition on why this is failing ?
Is this supposed to be a real failure ? https://treeherder.mozilla.org/logviewer?job_id=405478210&repo=nss-try&lineNumber=2781

Comment 6

[Michelle Goossens [:masterwayz]]

It appears to be passing on https://treeherder.mozilla.org/jobs?repo=nss-try&searchStr=NSPR&selectedTaskRun=i70iPb00RLacD8CIAwxCLw.0 so I'm not sure what's going on here.
When I pushed, I was on nss tip if that helps.

Comment 7

[Benjamin Beurdouche [:beurdouche]]

Hmm, actually the MacOS builds seem to work on latest nss-try pushes, did you do anything for it to work @Michelle ?

Comment 8

[Michelle Goossens [:masterwayz]]

I did not. Perhaps there were some network hiccups. I did do some troubleshooting but that also came to the conclusion that everything just worked.
Had been wondering for a while since when I grabbed this ticket I looked on try and already saw green builds. I guess we can say that this is fixed now?

Comment 9

[Benjamin Beurdouche [:beurdouche]]

Something likely changed on the infrastructure side somehow... Let's call this fixed for now, we can reopen if needed.
Thanks Michelle !! : )

Comment 10

[Christoph Kerschbaumer [:ckerschb]]

An it seems the problem is coming back :-(
https://treeherder.mozilla.org/jobs?repo=nss-try

Michelle, can you help us out again?

Comment 11

[Michelle Goossens [:masterwayz]]

https://treeherder.mozilla.org/jobs?repo=nss-try&revision=3cfcaf2e60bff5504bb8fa5d48ffa60c5befaae4 looks good again
I did some networking adjustments

Comment 12

[Michelle Goossens [:masterwayz]]

Still looks good, so I think we are done?
If the network issue happens again I'll have to get in touch with MacStadium

Comment 13

[Christoph Kerschbaumer [:ckerschb]]

Yes, it seems that caused the problem to go away. Let's just hope for the best, I guess. I'll close the bug for now, worst case we re-open it.

Thank you!