Mozillians.org prod Chief: .git/index.lock error

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: Other
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: sancus, Assigned: cturra)

Tracking

Details

(Reporter)

Description

5 years ago
[localhost] err: [2013-04-15 13:31:21] [localhost] failed: cd /data/genericrhel6/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['mozillians.org']' (1.289s)
[localhost] err: [localhost] err: fatal: Unable to create '/data/genericrhel6/www/.git/index.lock': File exists.

Happened when I tried to push mozillians.org prod just now. Not too sure why, the last push was successful....
(Assignee)

Comment 1

5 years ago
this might intermittently happen since there are other deploys using the same git repository. the git repo in question is an internal one which is used to push/pull code from the admin node in the web cluster to each of the web heads. if you see this error, another chief push should resolve this.
(Reporter)

Comment 2

5 years ago
Hmm, I tried to push once more after this failure... but I can try again.
(Reporter)

Comment 3

5 years ago
Something is definitely wrong, it's continuously failing and 

https://mozillians.org/media/revision.txt reports that the revision is 2d974799f9d2c51414ae18e2f45251c5c0e98aff however that can't be the case because the admin panel is missing the feature that I pushed for, whereas staging has it.

So it's both not updated properly and reporting that it is via revision.txt...

[localhost] err: [2013-04-15 13:46:54] [localhost] failed: cd /data/genericrhel6/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['mozillians.org']' (6.075s)
[localhost] err: [localhost] out: # On branch master
[localhost] err: [localhost] out: nothing to commit (working directory clean)

It seems like it's possibly failing because it thinks there's nothing new to deploy, but the old deploy wasn't fully successful?
(Assignee)

Comment 4

5 years ago
i believe i know what is going on here. there was a successful `git commit` to the internal git repo at some point, which is why you're seeing this "err" for the deploy. and since you're seeing that, the deploy script is not moving onto the step where it `touch(es)` the wsgi file to reload the code. i have manually done this from the generic admin host for you -- can you please test again?
Hardware: x86 → All
(Reporter)

Comment 5

5 years ago
Yep, I can confirm that the site properly reloaded the code now.

Thanks!
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Assignee)

Comment 6

5 years ago
it might make sense to check for the .lock file before attempting the `git commit` as part of the deploy script. i will poke around at that to see what our options are.
Assignee: server-ops-webops → cturra
OS: Mac OS X → All
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.