596327 - Linux mozilla-central build failures with "abort: data/mozilla2/mobile/android/mobile-browser/nightly/mozconfig.i@1e3cd603e8ab: no match found!"

Status: RESOLVED INCOMPLETE

(mozilla.org Graveyard :: Server Operations, task)

Description

[Daniel Holbert [:dholbert]]

We just got two linux opt builds in a row with this failure:
{
hg clone http://hg.mozilla.org/build/buildbot-configs configs
 in dir /builds/slave/mozilla-central-linux64/build (timeout 1200 secs)
 watching logfiles {}
 argv: ['hg', 'clone', 'http://hg.mozilla.org/build/buildbot-configs', 'configs']
 environment:
[SNIP]
requesting all changes
adding changesets
adding manifests
adding file changes
added 2963 changesets with 7091 changes to 1403 files
updating working directory
abort: data/mozilla2/mobile/android/mobile-browser/nightly/mozconfig.i@1e3cd603e8ab: no match found!
}

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1284489438.1284489853.31317.gz
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1284489142.1284489767.30898.gz

Filing this as an ASAP high-priority issue, on the assumption that this is something that'll continue to be broken until it's fixed.

Comment 1

[Daniel Holbert [:dholbert]]

This seems to have started failing on aki's push to this repo:  http://hg.mozilla.org/build/buildbot-configsThis push updated the file mentioned in the error message.  (aki says, & I agree, that this is some sort of hg issue -- his push wasn't at fault)

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

I wonder if this is an issue with caching and/or locking of the repository.


Example repository:

[cltbld@moz2-linux-slave30 configs]$ hg verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
 mozilla2/mobile/android/mobile-browser/nightly/mozconfig@2962: 1e3cd603e8ab in manifests not found
 mozilla2/mobile/android/mobile-browser/release/mozconfig@2962: 44c57a77e711 in manifests not found
1403 files, 2963 changesets, 7091 total revisions
2 integrity errors encountered!
(first damaged changeset appears to be 2962)
[cltbld@moz2-linux-slave30 configs]$ hg ident
000000000000
[cltbld@moz2-linux-slave30 configs]$ hg tip
changeset:   2962:38eda43c57e6
tag:         tip
user:        Aki Sasaki <[e-mail]>
date:        Tue Sep 14 11:41:15 2010 -0700
summary:     android updates! r=alexp

Comment 4

[[e-mail] (:bhearsum)]

For what it's worth, I've had multiple jobs clone buildbot-configs without issue, and I've cloned it manually without issue. For example: http://production-master.build.mozilla.org:8011/builders/macosx_build/builds/0

Comment 5

[Daniel Holbert [:dholbert]]

Hm - so a respin from one of the purple builds just completed successfullly:
   http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1284489978.1284491243.6708.gz
and a Linux debug build completed successfully too:
   http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1284489407.1284491385.7407.gz

Not sure if that's due to luck or to someone fixing this.  Reducing severity, in any case, since this doesn't block work if it's not stopping *all* linux builds.

Comment 6

[Justin Dow [:jabba]]

I spoke to bear on IRC about this and he said it is probably best for us to wait for aravind to get back to take a look at this, since he is most familiar with the setup. If things get worse before he gets back, push this bug back to server-ops and it will page oncall again.

Comment 7

[[e-mail] (:bhearsum)]

Has there actually been *any* failures since the first batch? I can't find any, which makes me think this is a one-off issue.

Comment 8

[Daniel Holbert [:dholbert]]

I haven't seen any since those first two mentioned in comment 0.

Comment 9

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Symptoms are different, but unclear if this is caused by same root cause as intermittent hg errors in bug#588711.

Lets keep this open (and critical) until we know whats going on.

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Moving to correct component.

Comment 11

[[e-mail] (:bhearsum)]

I hit this on a different file:
requesting all changes
adding changesets
adding manifests
adding file changes
added 2966 changesets with 7095 changes to 1403 files
updating working directory
abort: data/thunderbird/release_config.py.i@2b7a7725d65f: no match found!

Comment 12

[Aravind Gottipati [:aravind]]

Is this still consistently happening?  Is there a reproducible test case I can use?

Comment 13

[Daniel Holbert [:dholbert]]

(In reply to comment #12)
> Is this still consistently happening?

I don't think so. The command in comment 0...
 hg clone http://hg.mozilla.org/build/buildbot-configs
... completes just fine on my own Linux machine now, and "hg verify" in the fresh checkout completes successfully too.

I'm not sure what caused this or why it became magically fixed.

Comment 14

[Chris AtLee [:catlee]]

This is an ongoing intermittent issue, it's especially bad with the buildbot-configs repository for some reason.  I wouldn't call this "fixed", it's just not happening right now.

Comment 15

[Chris AtLee [:catlee]]

see also bug 574348

Comment 16

[Aravind Gottipati [:aravind]]

Have we seen this in recent times?

Comment 17

[matthew zeier [:mrz]]

Let's close this until we see it again.  Having trouble duplicating and tracking down.