905197 - please check hg repository builds/buildbot-configs for integrity

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[hwine]

We've had reports of the same exact file being reported as corrupted in clones on 4 separate boxes.

See bug 900693 comment 0 for details.

We also see this occasionally in the vsc-sync system.

In general, this might be due to running the webheads on NFS, it's somewhat concerning that it occurs on the same file across multiple machines.

If the master copy verifies okay, would it also make sense to verify from each of the webheads?

Comment 1

[Aki Sasaki (not active)]

I think this is a known problem where the webheads don't deal well with repo integrity while an active push happens.

Comment 2

[Ben Kero [:bkero]]

[[e-mail]3 buildbot-configs]# hg verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
2606 files, 8938 changesets, 18418 total revisions

[[e-mail]3 buildbot-configs]# echo $?
0


I don't believe running this on webheads will make any difference. The problem is that the webheads have daemons running that might not pick up on the file alterations until they are restarted, which happens after they serve ~500 requests.

Comment 3

[hwine]

(In reply to Aki Sasaki [:aki] from comment #1)
> I think this is a known problem where the webheads don't deal well with repo
> integrity while an active push happens.

Possibly, but I would expect the impacted file to be relatively random for that cause (depending on the file block being written during the contention). 

It's the report from :rail that we've seen the identical file impacted on 4 separate hosts that concerns me.

Comment 4

[hwine]

(In reply to Ben Kero [:bkero] from comment #2)
> I don't believe running this on webheads will make any difference. The
> problem is that the webheads have daemons running that might not pick up on
> the file alterations until they are restarted, which happens after they
> serve ~500 requests.

As I understand NFS, any changes should be picked up as soon as the file is reopened (re-stat'd actually). So, unless all 4 hosts happened to clone from the same webhead quickly enough that the file stayed open in a filesystem cache, AND the NFS client daemon that doesn't notice it was updated on the server (which would invalidate the cache), I'd expect different files to be identified as the source of corruption.

That said, I'll defer to the experts -- I have no proof, but remain troubled by the odds as I perceive them.

Thanks for checking this out.