1296872 - Investigate cause of Ceph inconsistency during BMO DR

Status: RESOLVED WONTFIX

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[Dylan Hardison [:dylan] (he/him)]

Attachment: GOOD-a9c76a2405d11dde53d5e18165188aef.js

During the DR there was a consistency inconsistency between multiple javascript asset files. Some webheads had the files begin with about 8kb of ascii NUL.
All webheads saw the same file sizes.

Comment 1

[Dylan Hardison [:dylan] (he/him)]

Attachment: BAD-a9c76a2405d11dde53d5e18165188aef.js

Comment 2

[Dylan Hardison [:dylan] (he/him)]

Possibly look at what File::Slurp does when it writes out a file.

Comment 3

[Philippe M. Chiasson (:gozer)]

No actual information yet, but now that you mentioned File::Slurp, I wonder, do you use it with its atomic option or not?

I'd be curious to have a peek if you could point me at the bz code that deals with writing these assets.

Comment 4

[:glob ✱]

gozer: _concatenate_js and _concatenate_css in https://github.com/mozilla-bteam/bmo/blob/master/Bugzilla/Template.pm
they aren't using the atomic switch.

this looks like http://tracker.ceph.com/issues/11749
> The first one is fixed by Yehuda on https://github.com/ceph/ceph/pull/4661
> in such condition, the first stripe of a part will be lost.

Comment 5

[Amy Rich [:arr] [:arich]]

:gozer: :dlan: are we just going to resolve this in preference to removing the network filesystem?

Comment 6

[Mark Côté [:mcote]]

I talked to dylan about this (and cshields).  I don't think we'd be able to remove dependencies on the filesystem anytime soon.  I think it makes sense for the nubis team to continue to investigate efs.

Comment 7

[Dylan Hardison [:dylan] (he/him)]

I think it's safe to say this is WONTFIX.