1248213 - Push to try-comm-central fails too often.

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[ISHIKAWA, Chiaki]

Sorry, I am not sure what component to use.
I simply followed the embedded link
https://bugzilla.mozilla.org/enter_bug.cgi?comment=please%20schedule%20a%20reset%20of%20the%20try%20repository%20ASAP%20with%20sheriffs%20%26%20releng.&component=WebOps%3A%20Source%20Control&op_sys=All&product=Infrastructure%20%26%20Operations&rep_platform=All&short_desc=Push%20to%20try%20taking%20XXX%20minutes
in
https://wiki.mozilla.org/ReleaseEngineering/TryServer
and modfied the description.)

Since the end of January, I have noticed that try-comm-central jobs do not build simply because of the following errors many times.

CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/thunderbird/bundles/try-comm-central.hg']' returned non-zero exit status 255

Sometimes all the platforms I try (linux, windows, OSX) fail due basically the same issues.
Sometimes only a single platform, say, windows is affected.

A quote from: Bug 1241762 - Build infrastructure should NOT use https://ftp-ssl.mozilla.org/ to obtain bundle. Use S3 or the CloudFront CDN. 

Gregory Szorc's comment:
--- begin quote

I consider it a bug that we're obtaining bundles from https://ftp-ssl.mozilla.org/. All automation should be using the clone bundles / bundleclone facilities in Mercurial for cloning repos from S3 or the CloudFront CDN. See https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmo/bundleclone.html.

--- end quote

I hope something can be done about it.
(If this is a matter of replacing a script to fetch bundle from a different place, it may not be too time-consuming.)

TIA


PS; I have noticed a new error message.
abort: certificate for s3-us-west-2.amazonaws.com has unexpected fingerprint 44:91:09:38:91:64:f7:a1:4a:12:dc:d4:dd:0b:53:f2:bb:b0:36:3c
in
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&selectedJob=16133
(See linux64 opt build.)
The build seems to have been restarted automatically, though.

This is the first time I saw this, I think.

Comment 1

[Peter Radcliffe [:pir]]

The only bug link I can see in https://wiki.mozilla.org/ReleaseEngineering/TryServer is to mozilla.org -> Release Engineering, which doesn't seem to exist any more, so that needs updating.

Comment 2

[Wayne Mery (:wsmwk)]

still seeing this? 

> CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/thunderbird/bundles/try-comm-central.hg']' returned non-zero exit status 255

Comment 3

[ISHIKAWA, Chiaki]

Maybe it is only me: but
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=0cdc7f3e413d0e9d3a5b8dccab4bea5b7c4d4508

This was July 1st.

Please note the first line in the error log.

CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/thunderbird/bundles/try-comm-central.hg']' returned non-zero exit status 255
/builds/slave/tb-try-c-cen-l64-d-00000000000/build/mail/app/nsMailApp.cpp:365:29: error: 'EARLY_GLUESTARTUP_HARD_FAULTS' is not a member of 'mozilla::Telemetry'
/builds/slave/tb-try-c-cen-l64-d-00000000000/build/mail/app/nsMailApp.cpp:369:31: error: 'GLUESTARTUP_HARD_FAULTS' is not a member of 'mozilla::Telemetry'
make[5]: *** [nsMailApp.o] Error 1make[5]: Leaving directory `/builds/slave/tb-try-c-cen-l64-d-00000000000/build/objdir-tb/mail/app' 

Any errors afterwards is a suspect: I have to ignore it more or less thinking that the initial file fetch failed.

TIA

Comment 4

[ISHIKAWA, Chiaki]

On the previous day, mkmelin's submission also failed due to the same error.
(June 30)
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=d95e90e5513088854a12d86cfc569b545eaa2556

Still another day earlier, my job submission failed due to the same error.
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=8da0a26cf6e518d1797f1179e1e32178b52663be

I think there *IS* something wrong in the cloud infrastructure.

TIA

Comment 5

[Aki Sasaki (not active)]

A lot has changed over the past 6 years; marking incomplete.