1220117 - b2g-inbound and gaia closed for bumper bot failures

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Carsten Book [:Tomcat]]

seems since about 11 hours was no push from bumper bot to b2g-inbound and there are pushes to gaia. So seems there is a problem it seems 

closed b2g-inbound and gaia

Comment 1

[Carsten Book [:Tomcat]]

could someone from releng please update this bug ? Its a deadline day for b2g folks and they get upset with this gaia closure

Comment 2

[Carsten Book [:Tomcat]]

from irc:

07:58 < catlee> rail: do you know what's going on with gaia 2.5. vcssync?
07:58 < Tomcat|sheriffduty> catlee: yeah that rev question is a good one, i don't know, seems this problem started around 3pm pacific yesterday
07:58 < catlee> all those commits look super old
07:58 < Tomcat|sheriffduty> catlee: wondered why i'm the top pusher to b2g-i today ;) 
08:00 < rail> catlee: last h.wine checked yesterday it was still syncing 
08:00 < catlee> ah
08:00 < rail> Need to check with him 
08:00 < catlee> Tomcat|sheriffduty:  I think that's the real problem
08:01 < catlee> nothing's happened on the hg repo since yesterday afternoon

hwine, fubar could you take a look into this ?

Comment 3

[Kendall Libby [:fubar] (he/him)]

2015-10-30T14:31:50+0000: starting: hg --cwd /home/vcs2vcs/repos/integration-gaia-2_5 push --force hg.m.o
pushing to ssh://hg.m.o/integration/gaia-2_5
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 41105 changesets with 114465 changes to 22879 files (+15 heads)
remote: *** pushing unrelated repository ***
remote: 
remote: Changeset 346536343139 introduces a new root changeset into this repository. This
remote: almost certainly means you accidentally force pushed to the wrong
remote: repository and/or URL.
remote: 
remote: Your push is being rejected because this is almost certainly not what you
remote: intended.
remote: transaction abort!
remote: rollback completed
remote: abort: pretxnchangegroup.single_root hook failed
2015-10-30T14:41:24+0000:  exit 1: hg --cwd /home/vcs2vcs/repos/integration-gaia-2_5 push --force hg.m.o

Comment 4

[Hal Wine [:hwine] use NI!]

Attachment: mapfile.patch

Comment 5

[Hal Wine [:hwine] use NI!]

Comment on attachment 8681351 [details] [diff] [review]
mapfile.patch

Adds mapfile (reason bumper was bad -- it didn't have any data to work with)

Fixes inclusion of tags (not wanted - diagnosed last night, but that doesn't impact data, just size of conversion repo on vcs-sync machine)

Fixes file mode on "config" - no idea how that got set wonky.

Comment 6

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8681351 [details] [diff] [review]
mapfile.patch

Review of attachment 8681351 [details] [diff] [review]:
-----------------------------------------------------------------

shipit

Comment 7

[Hal Wine [:hwine] use NI!]

Actually, this is a problem with vcs-sync, moving to that product/component

Comment 8

[Hal Wine [:hwine] use NI!]

status update: vcs-sync conversion repo corrupted, regenerating. This will take several hours. I'll update with an eta on this stage.

Comment 9

[Hal Wine [:hwine] use NI!]

status update: had to go back to clean slate and rebuild. Conversion takes ~430m, and is done. Preparing fresh hg repo.

Comment 10

[Hal Wine [:hwine] use NI!]

NOTE: part of the clean slate process included the deletion and recreation of the integration/gaia-2_5 mercurial repository. This is the first time we've done this step, and I don't believe there will be any repercussions. If any glitch occurs, it would be on slaves which had already cached the prior version of the repo.

Comment 11

[Hal Wine [:hwine] use NI!]

fixed! will write up detailed notes later, but wanted to get the word out.

Comment 12

[Hal Wine [:hwine] use NI!]

long story short, initial conversion attempt was an optimization (using pre converted data from another gaia branch). When that failed to push, the assumption was that the optimization had failed, so the conversion was retried (but again with an optimization). When that failed to push, a "from scratch" conversion was started, which takes ~450m to run.

Only when the fresh conversion also failed did attention turn elsewhere. The hook issuing the error in comment 3 (single_root hook) was only deployed around 2015-05-27, well after the previous b2g branch activity.

Disabling the single root hook on hg.mozilla.org enabled the push to be successful. Subsequent updates also pushed successfully.

Bug 1220393 has been opened to address the problems with the single root hook.