929335 - staging gecko-git hg corruption

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Aki Sasaki (not active)]

Error log is non-zero!

00:34:11    ERROR -  abort: data/b2g/config/gaia.json.i@327bccdb70e8: no node!
00:34:11    ERROR -  Automation Error: hg not responding
00:34:11    ERROR - Return code: 255
00:34:11    ERROR -  abort: unknown revision '177bf37a49f5'!
00:34:11    ERROR -  Automation Error: hg not responding
00:34:11    ERROR - Return code: 255

On gd2.  I'd like to be able to debug this and hopefully get it to auto-fix.

Comment 1

[Aki Sasaki (not active)]

Attachment: hg_corruption

I wasn't able to fix this in place, but once I moved the stage_source/mozilla-central away, that made the script re-clone and we were in business.

This patch is an attempt to do so in an automated fashion.

Comment 2

[(no longer active)]

This is bug 737865.  I've been hitting this bug on a regular basis, and to deal with that I ended up writing a fixup script which detects the corrupted hg repo, reclones, puts hg-git stuff in place again, and retries.

Comment 3

[John Schoenick [:johns]]

My git mirror hits this often as well. Doing:
> hg strip --no-backup ${lastsuccessfulexport_revision}:

After a failed gexport solves the situation 99% of the time and is much quicker than a re-clone (though still needs the full re-clone fallback if it fails twice)

Comment 4

[Hal Wine [:hwine] use NI!]

Comment on attachment 820591 [details] [diff] [review]
hg_corruption

Review of attachment 820591 [details] [diff] [review]:
-----------------------------------------------------------------

looks good -- good use of disk to avoid the need to save meta data

Comment 5

[Aki Sasaki (not active)]

(In reply to John Schoenick [:johns] from comment #3)
> My git mirror hits this often as well. Doing:
> > hg strip --no-backup ${lastsuccessfulexport_revision}:
> 
> After a failed gexport solves the situation 99% of the time and is much
> quicker than a re-clone (though still needs the full re-clone fallback if it
> fails twice)

Good to know, I'll look into this later.

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 820591 [details] [diff] [review]
hg_corruption

https://hg.mozilla.org/build/mozharness/rev/7c89adb82021
transplanted to production.

I think we're good here, though I'll look into the hg strip later.

Comment 7

[Aki Sasaki (not active)]

Filed bug 934879 for the hg strip.
Resolving this one, to clarify the dependency list for bug 929334.