917692 - Aurora desktop B2G localizer builds failing with "Can't checkout https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo!"

Status: RESOLVED FIXED

(Firefox OS Graveyard :: GonkIntegration, defect)

Description

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/?tree=Mozilla-Aurora&jobname=localizer

eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=28024944&tree=Mozilla-Aurora#error16

{
03:49:57     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'clone', u'https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo', u'/builds/hg-shared/releases/gaia-l10n/v1_2/eo']
03:49:57     INFO - Copy/paste: hg --config ui.merge=internal:merge clone https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo /builds/hg-shared/releases/gaia-l10n/v1_2/eo
03:49:58     INFO -  warning: hg.mozilla.org certificate with fingerprint 10:78:e8:57:2d:95:de:7c:de:90:bd:22:e1:38:17:67:c5:a7:9c:14 not verified (check hostfingerprints or web.cacerts config setting)
03:49:58    ERROR -  abort: HTTP Error 404: Not Found
03:49:58    ERROR -  Automation Error: hg not responding
03:49:58    ERROR - Return code: 255
03:49:58     INFO - rmtree: eo
03:49:58     INFO - retry: Failed, sleeping 60 seconds before retrying
03:50:58     INFO - retry: Calling <bound method B2gMultilocale._get_revision of <__main__.B2gMultilocale object at 0x1782250>> with args: (<mozharness.base.vcs.mercurial.MercurialVCS object at 0x1782c50>, u'eo'), kwargs: {}, attempt #2
03:50:58     INFO - Setting /builds/slave/m-aurora-l64_g_lz-ntly-0000000/build-gaia-l10n/eo to https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo using shared directory /builds/hg-shared.
03:50:58     INFO - Updating shared repo
03:50:58     INFO - Cloning https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo to /builds/hg-shared/releases/gaia-l10n/v1_2/eo.
03:50:58     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'clone', u'https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo', u'/builds/hg-shared/releases/gaia-l10n/v1_2/eo']
03:50:58     INFO - Copy/paste: hg --config ui.merge=internal:merge clone https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo /builds/hg-shared/releases/gaia-l10n/v1_2/eo
03:50:58     INFO -  warning: hg.mozilla.org certificate with fingerprint 10:78:e8:57:2d:95:de:7c:de:90:bd:22:e1:38:17:67:c5:a7:9c:14 not verified (check hostfingerprints or web.cacerts config setting)
03:50:58    ERROR -  abort: HTTP Error 404: Not Found
03:50:58    ERROR -  Automation Error: hg not responding
03:50:58    ERROR - Return code: 255
03:50:58     INFO - rmtree: eo
03:50:58     INFO - retry: Failed, sleeping 120 seconds before retrying
03:52:58     INFO - retry: Calling <bound method B2gMultilocale._get_revision of <__main__.B2gMultilocale object at 0x1782250>> with args: (<mozharness.base.vcs.mercurial.MercurialVCS object at 0x1782c50>, u'eo'), kwargs: {}, attempt #3
03:52:58     INFO - Setting /builds/slave/m-aurora-l64_g_lz-ntly-0000000/build-gaia-l10n/eo to https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo using shared directory /builds/hg-shared.
03:52:58     INFO - Updating shared repo
03:52:58     INFO - Cloning https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo to /builds/hg-shared/releases/gaia-l10n/v1_2/eo.
03:52:58     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'clone', u'https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo', u'/builds/hg-shared/releases/gaia-l10n/v1_2/eo']
03:52:58     INFO - Copy/paste: hg --config ui.merge=internal:merge clone https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo /builds/hg-shared/releases/gaia-l10n/v1_2/eo
03:52:59     INFO -  warning: hg.mozilla.org certificate with fingerprint 10:78:e8:57:2d:95:de:7c:de:90:bd:22:e1:38:17:67:c5:a7:9c:14 not verified (check hostfingerprints or web.cacerts config setting)
03:52:59    ERROR -  abort: HTTP Error 404: Not Found
03:52:59    ERROR -  Automation Error: hg not responding
03:52:59    ERROR - Return code: 255
03:52:59     INFO - rmtree: eo
03:52:59     INFO - retry: Failed, sleeping 240 seconds before retrying
03:56:59     INFO - retry: Calling <bound method B2gMultilocale._get_revision of <__main__.B2gMultilocale object at 0x1782250>> with args: (<mozharness.base.vcs.mercurial.MercurialVCS object at 0x1782c50>, u'eo'), kwargs: {}, attempt #4
03:56:59     INFO - Setting /builds/slave/m-aurora-l64_g_lz-ntly-0000000/build-gaia-l10n/eo to https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo using shared directory /builds/hg-shared.
03:56:59     INFO - Updating shared repo
03:56:59     INFO - Cloning https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo to /builds/hg-shared/releases/gaia-l10n/v1_2/eo.
03:56:59     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'clone', u'https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo', u'/builds/hg-shared/releases/gaia-l10n/v1_2/eo']
03:56:59     INFO - Copy/paste: hg --config ui.merge=internal:merge clone https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo /builds/hg-shared/releases/gaia-l10n/v1_2/eo
03:56:59     INFO -  warning: hg.mozilla.org certificate with fingerprint 10:78:e8:57:2d:95:de:7c:de:90:bd:22:e1:38:17:67:c5:a7:9c:14 not verified (check hostfingerprints or web.cacerts config setting)
03:56:59    ERROR -  abort: HTTP Error 404: Not Found
03:56:59    ERROR -  Automation Error: hg not responding
03:56:59    ERROR - Return code: 255
03:56:59     INFO - rmtree: eo
03:56:59     INFO - retry: Failed, sleeping 300 seconds before retrying
04:01:59     INFO - retry: Calling <bound method B2gMultilocale._get_revision of <__main__.B2gMultilocale object at 0x1782250>> with args: (<mozharness.base.vcs.mercurial.MercurialVCS object at 0x1782c50>, u'eo'), kwargs: {}, attempt #5
04:01:59     INFO - Setting /builds/slave/m-aurora-l64_g_lz-ntly-0000000/build-gaia-l10n/eo to https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo using shared directory /builds/hg-shared.
04:01:59     INFO - Updating shared repo
04:01:59     INFO - Cloning https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo to /builds/hg-shared/releases/gaia-l10n/v1_2/eo.
04:01:59     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'clone', u'https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo', u'/builds/hg-shared/releases/gaia-l10n/v1_2/eo']
04:01:59     INFO - Copy/paste: hg --config ui.merge=internal:merge clone https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo /builds/hg-shared/releases/gaia-l10n/v1_2/eo
04:02:00     INFO -  warning: hg.mozilla.org certificate with fingerprint 10:78:e8:57:2d:95:de:7c:de:90:bd:22:e1:38:17:67:c5:a7:9c:14 not verified (check hostfingerprints or web.cacerts config setting)
04:02:00    ERROR -  abort: HTTP Error 404: Not Found
04:02:00    ERROR -  Automation Error: hg not responding
04:02:00    ERROR - Return code: 255
04:02:00     INFO - rmtree: eo
04:02:00    FATAL - Can't checkout https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo!
04:02:00    FATAL - Caught exception: Unable to clone https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo to /builds/hg-shared/releases/gaia-l10n/v1_2/eo!
}

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Bug 915249 never asked for eo to be mirrored, so either either an oversight or the locale manifest is incorrect for aurora.

Comment 2

[Axel Hecht [:Pike]]

Yeah, we should strip down gaia's languages_all.json to the contents of languages_dev.json for 1.2.

Moving over to b2g.

Comment 3

[Axel Hecht [:Pike]]

Attachment: sync languages_all.json with _dev.json

Kaze, can you review this PR, and merge if it's cool?

This is explicitly for 1.2 only, and shouldn't land on master.

Comment 4

[Fabien Cazenave [:kaze]]

Comment on attachment 806571 [details]
sync languages_all.json with _dev.json

A bit afraid that this might cause confusion (languages_dev ≠ languages_all) but as this is limited to the 1.2 branch, I guess it’s OK.

Merged on 1.2: https://github.com/mozilla-b2g/gaia/commit/f70292e01d42ee8874e52a1047ecfee82c5e9c19

Comment 5

[Ed Morley [:emorley]]

Jobs retriggered on aurora.
Thank you :-)

Comment 6

[Ed Morley [:emorley]]

Still failing:

https://tbpl.mozilla.org/php/getParsedLog.php?id=28033066&tree=Mozilla-Aurora#error15

{
08:20:51     INFO - Cloning https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo to /builds/hg-shared/releases/gaia-l10n/v1_2/eo.
08:20:51     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'clone', u'https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo', u'/builds/hg-shared/releases/gaia-l10n/v1_2/eo']
08:20:51     INFO - Copy/paste: hg --config ui.merge=internal:merge clone https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo /builds/hg-shared/releases/gaia-l10n/v1_2/eo
08:20:52     INFO -  warning: hg.mozilla.org certificate with fingerprint 10:78:e8:57:2d:95:de:7c:de:90:bd:22:e1:38:17:67:c5:a7:9c:14 not verified (check hostfingerprints or web.cacerts config setting)
08:20:52    ERROR -  abort: HTTP Error 404: Not Found
08:20:52    ERROR -  Automation Error: hg not responding
08:20:52    ERROR - Return code: 255
08:20:52     INFO - rmtree: eo
08:20:52    FATAL - Can't checkout https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo!
08:20:52    FATAL - Caught exception: Unable to clone https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo to /builds/hg-shared/releases/gaia-l10n/v1_2/eo!
08:20:52    FATAL - Caught exception: Unable to clone https://hg.mozilla.org/releases/gaia-l10n/v1_2/eo to /builds/hg-shared/releases/gaia-l10n/v1_2/eo!
}

However, higher up in the log is our answer:

{
    "revision": "eb59735156986ce7a697063021ad74f1fca0bc12", 
    "repo_path": "/integration/gaia-central"
}
...
gaiaRepoPath: '/integration/gaia-central'
gaiaRevision: 'eb59735156986ce7a697063021ad74f1fca0bc12'

We're still pulling the mirror of gaia master as far as I can see (the in-tree gaia.json manifest needs updating to point at the gaia 1.2 mirror, and the new commit that landed in comment 4).

Comment 7

[Ed Morley [:emorley]]

(Looks like bug 916911 might need some tweaks?)

Comment 8

[Rail Aliiev [:rail]]

"eo" wasn't a part of the initial request for v1.2 (see https://bugzilla.mozilla.org/show_bug.cgi?id=915249#c0)

Axel, should we add eo to v1.2 or remove this locale from the automation?

Comment 9

[Ed Morley [:emorley]]

This locale has already been removed (comment 4), the problem is that these builds aren't using the updated 1.2 branch of gaia (last sentence of comment 6).

Comment 10

[Aki Sasaki (not active)]

Hm, I failed to merge bug 915596 into production like I thought. Done.

Comment 11

[Aki Sasaki (not active)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/3c6244bf7686

Comment 12

[Aki Sasaki (not active)]

If 1.2 localizer builds are the same locale list as non-localizer builds, should we turn them off?

Comment 13

[Ed Morley [:emorley]]

Thank you, they're working now:
https://tbpl.mozilla.org/?tree=Mozilla-Aurora&jobname=localizer 

:-)

Comment 14

[Ed Morley [:emorley]]

(In reply to Aki Sasaki [:aki] from comment #12)
> If 1.2 localizer builds are the same locale list as non-localizer builds,
> should we turn them off?

Comment 15

[Axel Hecht [:Pike]]

Are there any updates or the like for those? If not, I guess we can save those cycles, yeah.

Comment 16

[Aki Sasaki (not active)]

I was tempted to reuse this bug, but filed https://bugzilla.mozilla.org/show_bug.cgi?id=918524 with patch.

Comment 17

[Aki Sasaki (not active)]

... and didn't mean to reopen.