Closed Bug 766810 Opened 9 years ago Closed 8 years ago

Pushing to mozilla-inbound fails with abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks

Categories

(Developer Services :: General, task)

x86
macOS
task
Not set
normal

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: Margaret, Assigned: fox2mike)

References

Details

I'm getting this error:

pushing to ssh://hg.mozilla.org/integration/mozilla-inbound
searching for changes
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string
Drew says he's seeing this too
Severity: normal → major
Assignee: server-ops → server-ops-devservices
Component: Server Operations → Server Operations: Developer Services
QA Contact: phong → shyam
Others are able to push - http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=d047caf12f04&tochange=6d5a6c8120c0 - so this might be intermittent. How many nodes do we have behind ssh traffic to hg.m.o ?
(In reply to Nick Thomas [:nthomas] from comment #2)
> Others are able to push -
> http://hg.mozilla.org/integration/mozilla-inbound/
> pushloghtml?fromchange=d047caf12f04&tochange=6d5a6c8120c0 - so this might be
> intermittent. How many nodes do we have behind ssh traffic to hg.m.o ?

Just one active atm. (same way it's always been).
Assignee: server-ops-devservices → shyam
Wondering if this is somehow related to bug 766533
I don't see this file in there :
(quoting to prevent wrapping)

> [root@hgssh1.dmz.scl3 mozilla-inbound]# ls -l .hg
> total 15060
> -rw-rw-r--   1 root                       scm_level_3       0 Jun 20 07:02 bookmarks
> -rw-rw-r--   1 hg                         scm_level_3       8 Jun  6  2011 branch
> -rw-rw-r--   1 lrocha@mozilla.com         scm_level_3    2725 Jan 13 08:21 branchheads.cache
> drwxrwsr-x   2 bobbyholley@gmail.com      scm_level_3    4096 Jun 20 20:24 cache
> -rw-rw-r--   1 hg                         scm_level_3 3124085 Jun  6  2011 dirstate
> -rw-rw-r--   1 hg                         scm_level_3     282 Oct 29  2011 hgrc
> -rw-rw-r--   1 pweitershausen@mozilla.com scm_level_3 4542464 Jun 20 20:24 pushlog2.db
> -rw-r--r--   1 root                       scm_level_3 4529152 Jun 20 06:32 pushlog2.db.20120620-cshields
> -rw-rw-r-- 162 hg                         scm_level_2      15 May  9  2007 requires
> drwxrwsr-x   3 hg                         scm_level_3    4096 Jun 20 20:24 store
> drwxr-sr-x   2 root                       scm_level_3    4096 Jun 20 07:01 strip-backup
> -rw-rw-r--   1 mak77@bonardo.net          scm_level_3   12130 Jan 13 00:54 tags.cache
> -rw-rw-r--   1 dwillcoxon@mozilla.com     scm_level_3       0 Jun 20 20:24 undo.bookmarks
> -rw-rw-r--   1 dwillcoxon@mozilla.com     scm_level_3       7 Jun 20 20:24 undo.branch
> -rw-rw-r--   1 dwillcoxon@mozilla.com     scm_level_3      38 Jun 20 20:24 undo.desc
> -rw-rw-r--   1 dwillcoxon@mozilla.com     scm_level_3 3124085 Jun 20 20:24 undo.dirstate
Group: mozilla-corporation-confidential
Meh, marking confidential for the emails above.
What are the permissions on .hg itself ?
(In reply to Mike Hommey [:glandium] from comment #7)
> What are the permissions on .hg itself ?

drwxrwsr-x  5 hg scm_level_3   4096 Jun 21 01:03 .hg
Well, it worked for me now. Maybe just a temporary problem yesterday?
Yup, going to call it that for now.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME
(In reply to Shyam Mani [:fox2mike] from comment #6)
> Meh, marking confidential for the emails above.

those emails are in pushlog, no need to hide this for that
Group: mozilla-corporation-confidential
I just hit this three time in a row.  I haven't managed to land my patch, yet.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
I tried pushing my patches to a different local repo and then pushing to inbound from there, and that failed too, so it's not something to do with my local repos.
Summary: Pushing to mozilla-inbound fails with abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks → Pushing to mozilla-inbound intermittently fails with abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
This is also currently affecting me when I try to push to inbound.
This also appears to have merged my patch with another one!

  [1 patch in queue series]
  qfin -a
  hg push    // get error string like everyone else in this bug report
  hg out     // still shows my patch
  hg qimport -r tip  // imports changeset with my changeset message, but there's other code in it now too: some "foo.orig' files in the /dom directory. Yay!
I'm using mercurial 2.0-2, BTW, on ubuntu.  And this keeps failing no matter how many times I try (FYI: no one else has landed anything in the interim, in case that matters.  Perhaps I need someone else to land to reset file permissions somehow?

 http://selenic.com/pipermail/mercurial/2011-November/040629.html
Happening here too.
And here.  (Is this really intermittent, or is it biting everyone now?)
Because everybody loves getting uninformed wild guesses, umask?

(In reply to Shyam Mani [:fox2mike] from comment #5)
> I don't see this file in there :

You shouldn't - near as I can tell, it should create it (and several others, which oddly should fail first, but apparently don't) as part of the push, copying in the contents of the 0 byte bookmarks file, to use to rollback if the transaction gets interrupted, and then it should get renamed to undo.bookmarks for use in revert. So I'd expect the only way to see the file would be to look during a push.

Since it is hitting everybody (at least, everybody who has tried to push for the last five hours), this has effectively closed mozilla-inbound.
Severity: major → blocker
Summary: Pushing to mozilla-inbound intermittently fails with abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks → Pushing to mozilla-inbound fails with abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
Right. It seems like someone's push failed and then everything stacked up after it.

> [root@hgssh1.dmz.scl3 .hg]# ls -l
> total 18252
> -rw-rw-r--   1 root                       scm_level_3       0 Jun 20 07:02 bookmarks
> -rw-rw-r--   1 hg                         scm_level_3       8 Jun  6  2011 branch
> -rw-rw-r--   1 lrocha@mozilla.com         scm_level_3    2725 Jan 13 08:21 branchheads.cache
> drwxrwsr-x   2 bobbyholley@gmail.com      scm_level_3    4096 Jun 28 17:03 cache
> -rw-rw-r--   1 hg                         scm_level_3 3124085 Jun  6  2011 dirstate
> -rw-rw-r--   1 hg                         scm_level_3     282 Oct 29  2011 hgrc
> -rw-rw-r--   1 wjohnston@mozilla.com      scm_level_3       0 Jun 28 23:55 journal.bookmarks
> -rw-rw-r--   1 felipc@gmail.com           scm_level_3       7 Jun 28 23:55 journal.branch
> -rw-rw-r--   1 felipc@gmail.com           scm_level_3      38 Jun 28 23:55 journal.desc
> -rw-rw-r--   1 felipc@gmail.com           scm_level_3 3124085 Jun 28 23:55 journal.dirstate
> -rw-rw-r--   1 pweitershausen@mozilla.com scm_level_3 4670464 Jun 28 16:37 pushlog2.db
> -rw-r--r--   1 root                       scm_level_3 4529152 Jun 20 06:32 > pushlog2.db.20120620-cshields
> -rw-rw-r-- 165 hg                         scm_level_1      15 May  9  2007 requires
> drwxrwsr-x   3 hg                         scm_level_3    4096 Jun 28 23:55 store
> drwxr-sr-x   2 root                       scm_level_3    4096 Jun 20 07:01 strip-backup
> -rw-rw-r--   1 mak77@bonardo.net          scm_level_3   12130 Jan 13 00:54 tags.cache
> -rw-rw-r--   1 jduell@mozilla.com         scm_level_3       0 Jun 28 16:37 undo.bookmarks
> -rw-rw-r--   1 jduell@mozilla.com         scm_level_3       7 Jun 28 16:37 undo.branch
> -rw-rw-r--   1 jduell@mozilla.com         scm_level_3      38 Jun 28 16:37 undo.desc
> -rw-rw-r--   1 jduell@mozilla.com         scm_level_3 3124085 Jun 28 16:37 undo.dirstate

This shouldn't be happening. And wasn't. I'm not sure what's changed to cause this.
I've blown away the journal.* files, but I'll keep an eye on this. Bug is still open, but mozilla-inbound should be okay to push to.
Did Wesley try to push something that didn't go well?
This is the concerning bit : 

> -rw-rw-r--   1 wjohnston@mozilla.com      scm_level_3       0 Jun 28 23:55 journal.bookmarks
> -rw-rw-r--   1 felipc@gmail.com           scm_level_3       7 Jun 28 23:55 journal.branch
> -rw-rw-r--   1 felipc@gmail.com           scm_level_3      38 Jun 28 23:55 journal.desc
> -rw-rw-r--   1 felipc@gmail.com           scm_level_3 3124085 Jun 28 23:55 journal.dirstate

It seems like wesley's push failed or didn't clean up after itself and when felipe tried, it failed. The timestamps don't match the first report of failure from Nicholas, so right now this is all pure speculation.
Hmm... I tried to push something yesterday and had hg reject it because I had qfolded a patch with try chooser stuff in the commit message. That then turned into a mess in my tree that I still haven't cleaned up, but could that have caused this?
Severity: blocker → major
We're now getting this on inbound consistently, as a result, nobody can push to inbound.
Severity: major → critical
This is blocking people (including me) from pushing to inbound.  I have a lot of important backouts on central which needs to go on inbound asap.
Severity: critical → blocker
Unassigning in the hopes that this gets someone's attention.
Assignee: shyam → nobody
(In reply to Ehsan Akhgari [:ehsan] from comment #27)
> Unassigning in the hopes that this gets someone's attention.

Wrong assignee, please reset to default or this won't page. 

> -rw-rw-r--   1 jblandy@mozilla.com        scm_level_3       0 Jul  4 17:54 journal.bookmarks
> -rw-rw-r--   1 hfiguiere@mozilla.com      scm_level_3       7 Jul  4 17:54 journal.branch
> -rw-rw-r--   1 hfiguiere@mozilla.com      scm_level_3      38 Jul  4 17:54 journal.desc
> -rw-rw-r--   1 hfiguiere@mozilla.com      scm_level_3 3124085 Jul  4 17:54 journal.dirstate

Either Jim or Hub had a push fail on them? This is pretty hard to debug coming in hours after it's happened :|
Assignee: nobody → shyam
Alright, please poke the oncall sysadmin (in #it) to page either me or bkero when this happens next. We need to catch this as it happens (first dev that triggers this) to see why it's happening.
Whiteboard: poke bkero/fox2mike at the first failure
 
> > -rw-rw-r--   1 jblandy@mozilla.com        scm_level_3       0 Jul  4 17:54 journal.bookmarks
> > -rw-rw-r--   1 hfiguiere@mozilla.com      scm_level_3       7 Jul  4 17:54 journal.branch
> > -rw-rw-r--   1 hfiguiere@mozilla.com      scm_level_3      38 Jul  4 17:54 journal.desc
> > -rw-rw-r--   1 hfiguiere@mozilla.com      scm_level_3 3124085 Jul  4 17:54 journal.dirstate
> 
> Either Jim or Hub had a push fail on them? This is pretty hard to debug
> coming in hours after it's happened :|

I had it fail on the message mentioned in the bug title.
Just had this occur for me pushing to inbound at ~15:05 UTC+1 2012-07-12

pushing to ssh://hg.mozilla.org/integration/mozilla-inbound/
searching for changes
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string

bkero/fox2mike pinging in #it
Paged bkero. In the meantime, adding some debug info:

> [root@hgssh1.dmz.scl3 .hg]# ls -l journal*
> -rw-rw-r-- 1 cjones@mozilla.com  scm_level_3       0 Jul 12 07:05 journal.bookmarks
> -rw-rw-r-- 1 emorley@mozilla.com scm_level_3       7 Jul 12 07:05 journal.branch
> -rw-rw-r-- 1 emorley@mozilla.com scm_level_3      38 Jul 12 07:05 journal.desc
> -rw-rw-r-- 1 emorley@mozilla.com scm_level_3 3124085 Jul 12 07:05 journal.dirstate
> [root@hgssh1.dmz.scl3 .hg]# stat journal.*
>   File: `journal.bookmarks'
>   Size: 0               Blocks: 0          IO Block: 65536  regular empty file
> Device: 13h/19d Inode: 17124171    Links: 1
> Access: (0664/-rw-rw-r--)  Uid: ( 1129/cjones@mozilla.com)   Gid: (  679/scm_level_3)
> Access: 2012-07-12 05:52:06.491817000 -0700
> Modify: 2012-07-12 07:05:52.817403000 -0700
> Change: 2012-07-12 07:05:52.817403000 -0700
>   File: `journal.branch'
>   Size: 7               Blocks: 0          IO Block: 65536  regular file
> Device: 13h/19d Inode: 17124169    Links: 1
> Access: (0664/-rw-rw-r--)  Uid: ( 1943/emorley@mozilla.com)   Gid: (  679/scm_level_3)
> Access: 2012-07-12 07:05:52.814445000 -0700
> Modify: 2012-07-12 07:05:52.815347000 -0700
> Change: 2012-07-12 07:05:52.815347000 -0700
>   File: `journal.desc'
>   Size: 38              Blocks: 0          IO Block: 65536  regular file
> Device: 13h/19d Inode: 17124170    Links: 1
> Access: (0664/-rw-rw-r--)  Uid: ( 1943/emorley@mozilla.com)   Gid: (  679/scm_level_3)
> Access: 2012-07-12 07:05:52.816317000 -0700
> Modify: 2012-07-12 07:05:52.816360000 -0700
> Change: 2012-07-12 07:05:52.816360000 -0700
>   File: `journal.dirstate'
>   Size: 3124085         Blocks: 6128       IO Block: 65536  regular file
> Device: 13h/19d Inode: 17124168    Links: 1
> Access: (0664/-rw-rw-r--)  Uid: ( 1943/emorley@mozilla.com)   Gid: (  679/scm_level_3)
> Access: 2012-07-12 07:05:52.782370000 -0700
> Modify: 2012-07-12 07:05:52.813335000 -0700
> Change: 2012-07-12 07:05:52.813335000 -0700
Just remembered, cjones said this at ~1353 UTC+1 in #developers:
"inbound says open, hg tells me otherwise"

So presume he tried to push and hit the closure hook. (Inbound was closed at the time, cjones just hadn't refreshed the tbpl page).
FWIW, just hit this; pinged bkero/fox2make in #it for bonus points.  Happy to provide more information.  Seems to happen repeatedly for me.
[root@hgssh1.dmz.scl3 mozilla-inbound]# hg verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
101525 files, 99059 changesets, 502734 total revisions
I moved the journal.* and undo.* files to a backup directory and functionality was restored.

Please ignore the latest committed changeset, that's how I verified this working.

[root@hgssh1.dmz.scl3 .hg]# hg log -l 4
changeset:   99059:ed4888197cb5
tag:         tip
user:        Malini Das <mdas@mozilla.com>
date:        Thu Jul 12 11:01:20 2012 -0400
summary:     Bug 771224 - use chrome:// instead of loading remote xul. r=jgriffin

changeset:   99058:f0be4b70b814
user:        Ed Morley <emorley@mozilla.com>
date:        Thu Jul 12 13:04:51 2012 +0100
summary:     Backout 6bbf3f22bb5d (bug 753158), 38a703b244c2 (bug 753145), c9a5dfa1b07d (bug 767750), cd782fd66995 & 6cf7aa93994c (bug 765956), 0253f34f6bc2 & 41d5c8529748 (bug 771039),94f6bf99a4aa (bug 766447),fad7d06d7dd5 (bug 772303) for winxp pgo-only jsreftest failures (caused by fad7d06d7dd5) and the rest for conflicts, on a CLOSED TREE

changeset:   99057:64cff7aafcc4
user:        Anant Narayanan <anant@kix.in>
date:        Thu Jul 12 04:53:48 2012 -0700
summary:     Bug 771833: Assign an nsIPrincipal to media streams returned by getUserMedia; r=roc

changeset:   99056:1d3ef213c75b
user:        Anant Narayanan <anant@kix.in>
date:        Thu Jul 12 04:53:08 2012 -0700
summary:     Bug 691234: Part 3/3: Add DOM binding for getUserMedia on Desktop; r=jst
Next time this happens we should investigate using hg rollback (maybe with -n to dry-run) to roll the repository back to the previous state and see if that yields any clues as to the reason for these problems.
I just got this again when pushing:

ehsanakhgari:~/moz/inbound [02:19:01]$ hg push
pushing to ssh://hg.mozilla.org/integration/mozilla-inbound/
searching for changes
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string
Me too!

07-13 11:28 > hg push
pushing to ssh://myk%40mozilla.com@hg.mozilla.org/integration/mozilla-inbound/
searching for changes
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string
A subsequent attempt succeeded after a rebase:

07-13 11:28 > hg push
pushing to ssh://myk%40mozilla.com@hg.mozilla.org/integration/mozilla-inbound/
searching for changes
abort: push creates new remote head f44a483b2f92!
(you should pull and merge or use push -f to force)
07-13 12:02 > hg pull --rebase
pulling from ssh://myk%40mozilla.com@hg.mozilla.org/integration/mozilla-inbound/
searching for changes
adding changesets
adding manifests
adding file changes
added 7 changesets with 25 changes to 20 files (+1 heads)
saved backup bundle to c:\Users\myk\Mozilla\inbound\.hg\strip-backup\f44a483b2f92-backup.hg
07-13 12:03 > hg outgoing
comparing with ssh://myk%40mozilla.com@hg.mozilla.org/integration/mozilla-inbound/
searching for changes
changeset:   99213:06756c4481d5
tag:         tip
user:        Marco Castelluccio <mar.castelluccio@studenti.unina.it>
date:        Fri Jul 13 11:27:49 2012 -0700
summary:     Bug 768768 - Launch app from shell, close, then launch it from terminal results in no icon showing up in task bar in ubuntu; r=karlt

07-13 12:03 > hg push
pushing to ssh://myk%40mozilla.com@hg.mozilla.org/integration/mozilla-inbound/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.
I was able to investigate a bit today and this seems to be related to pushes while the tree is closed.

Earlier today (around 9AM) the trees closed and gavin (who wasn't sure if the trees were closed or not) was able to push to mozilla-inbound.  Afterwards, journal.* and undo.* files were created and persisted past the successful commit.

Armed with this new information, I'm going to read through the treeclosed script and as a last measure I could add part of the hook to be deleting the journal.* and undo.* files if no other solution becomes evident.
(In reply to Ben Kero [:bkero] from comment #41)
>
> [...]
> 
> Armed with this new information, I'm going to read through the treeclosed
> script and as a last measure I could add part of the hook to be deleting the
> journal.* and undo.* files if no other solution becomes evident.

There are two related bugs in mercurial: one for journal.bookmarks[1] and one for
journal.phaseroots[2].

Mercurial 2.1.2 and 2.2 contains these fixes, but I hope they could be ported back easily.

[1] - http://bz.selenic.com/show_bug.cgi?id=3318
[2] - http://bz.selenic.com/show_bug.cgi?id=3376

Sorry if this was already listed.
I already have a bug open for testing new version of mercurial (bug #741353 )

Additionally today I cloned mozilla-inbound as a bare repository, in the hopes that the storefiles changed in 2.0.2 since the last mozilla-inbound clone was on 1.x.
(In reply to Ben Kero [:bkero] from comment #43)
> Additionally today I cloned mozilla-inbound as a bare repository, in the
> hopes that the storefiles changed in 2.0.2 since the last mozilla-inbound
> clone was on 1.x.

I just got:
remote: abort: could not lock repository /repo/hg/mozilla/integration/mozilla-inbound: Permission denied                                              
abort: unexpected response: empty string

when trying to push to inbound. Related?
pushing to ssh://hg.mozilla.org/integration/mozilla-inbound
searching for changes
remote: abort: could not lock repository /repo/hg/mozilla/integration/mozilla-inbound: Permission denied
abort: unexpected response: empty string

I'm getting the same thing.  I was able to push earlier today - this just started happening now.
Spoke to bkero on IRC, it was indeed fallout from his earlier change - he fixed it yesterday shortly after my comment.
Has this happened recently?
I was able to push a couple of hours after Comment 45.  Haven't pushed to m-i since though so I don't know if it showed up again.
We've had this open a month without any reports...and now that I've jinxed it, let's see if we see it again.
Reopen if we run into this again, please.
Status: REOPENED → RESOLVED
Closed: 9 years ago8 years ago
Resolution: --- → FIXED
Your wish is my command!

$ hg push
pushing to ssh://hg.mozilla.org/integration/mozilla-inbound
searching for changes
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Caused by bug 797910?
This is now confirmed. If we strip, the repo goes mad. 

I'm not sure how we can fix this, tbh. 

Ehsan?
(In reply to Ryan VanderMeulen from comment #51)
> Your wish is my command!
> 
> $ hg push
> pushing to ssh://hg.mozilla.org/integration/mozilla-inbound
> searching for changes
> remote: abort: Operation not permitted:
> /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
> abort: unexpected response: empty string

Don't see those files now, so you shouldn't be blocked on this anymore.

It seems like hg strip (which is NOT recommended, maybe for reasons like this) causes later side effects like leaving behind files it shouldn't. The same thing happened the last time we stripped inbound.
Did we ever understand why this happens exactly?
bz and I are both hitting this tonight trying to push to inbound
(In reply to Ehsan Akhgari [:ehsan] from comment #55)
> Did we ever understand why this happens exactly?

No. But we know it happens after we strip.
(In reply to Ryan VanderMeulen from comment #56)
> bz and I are both hitting this tonight trying to push to inbound

Cleaned up :

-rw-rw-r-- 1 kvijayan@mozilla.com scm_level_3       0 Oct  5 22:34 journal.bookmarks
-rw-rw-r-- 1 mrbkap@mozilla.com   scm_level_3       7 Oct  5 22:34 journal.branch
-rw-rw-r-- 1 mrbkap@mozilla.com   scm_level_3      39 Oct  5 22:34 journal.desc
-rw-rw-r-- 1 mrbkap@mozilla.com   scm_level_3       0 Oct  5 22:34 journal.dirstate

Should be good now.
Whiteboard: poke bkero/fox2mike at the first failure
Severity: blocker → normal
pushing to ssh://hg.mozilla.org/integration/mozilla-inbound/
searching for changes
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string

:-(
Neither myself, gcp, cpearce can push to inbound.

I need to backout bustage in order to sheriff the tree, but cannot do so.
Severity: normal → blocker
Depends on: 800853
Pinged by Callek about this happening:

[root@hgssh1.dmz.scl3 ~]# cd /repo/hg/mozilla/integration/mozilla-inbound/.hg/
[root@hgssh1.dmz.scl3 .hg]# ls -aFl journal.bookmarks journal.branch journal.desc journal.dirstate
-rw-rw-r-- 1 bnicholson@mozilla.com scm_level_3  0 Oct 12 03:05 journal.bookmarks
-rw-rw-r-- 1 kwierso@gmail.com      scm_level_3  7 Oct 12 03:05 journal.branch
-rw-rw-r-- 1 kwierso@gmail.com      scm_level_3 39 Oct 12 03:05 journal.desc
-rw-rw-r-- 1 kwierso@gmail.com      scm_level_3  0 Oct 12 03:05 journal.dirstate
[root@hgssh1.dmz.scl3 .hg]# date
Fri Oct 12 03:18:10 PDT 2012
[root@hgssh1.dmz.scl3 .hg]# mkdir ~/2012-10-12-03:18
[root@hgssh1.dmz.scl3 .hg]# mv journal.bookmarks journal.branch journal.desc journal.dirstate ~/2012-10-12-03:18
I'm currently running into this again.
Me too.
looking...
Offending files moved to ~/2012-10-13-05\:42/

Shyam, do we have an hg bug opened for this?  if not, please do ASAP and link here.
A note for oncall:

The quick-fix for this is to move the journal files out of place.  Keep them just in case it does not work and we need to move them back.  

(on hgssh1)
mkdir ~/foo
cd /repo/hg/mozilla/integration/mozilla-inbound/.hg  (for mozilla-inbound problems only!)
mv journal.bookmarks journal.branch journal.desc journal.dirstate ~/foo

Then verify with someone in #developers
(In reply to Ryan VanderMeulen from comment #62)
> I'm currently running into this again.

... per IRC with Corey...

When this hits in the future, its best to create a *new* blocker bug, so that it pages on-call. And just tie deps of it to here. Said bug can be resolved when that instance of us hitting the problem is resolved, this bug can track the overall problem + its recurrence.
No longer blocks: 799849
Blocks: 802467
(In reply to Corey Shields [:cshields] from comment #67)
> ...
> 
> Shyam, do we have an hg bug opened for this?  if not, please do ASAP and
> link here.

IMHO the talk in comment #42 and comment #43 is related, isn't it?
Depends on: 741353
(In reply to Corey Shields [:cshields] from comment #67)
> Offending files moved to ~/2012-10-13-05\:42/
> 
> Shyam, do we have an hg bug opened for this?  if not, please do ASAP and
> link here.

See https://bugzilla.mozilla.org/show_bug.cgi?id=766810#c42 :) So if we upgrade mercurial, this should go away.
Hitting this currently.
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string
Applied the "fix" from comment 68.
I just hit this again.
Fixed it for Matt.
hitting this right now


pushing to ssh://hg.mozilla.org/integration/mozilla-inbound
searching for changes
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string
The last few times this happened (including comment 74, comment 76, and others) were all on right after closing inbound.  So that supports the theory from comment 41 that this is triggered by the closure hook.  Maybe when it rejects an attempted push, it leaves the journal in an inconsistent state?
Getting this as well :(.
Don't see any issues with it right now.

(In reply to Matt Brubeck (:mbrubeck) from comment #77)
> The last few times this happened (including comment 74, comment 76, and
> others) were all on right after closing inbound.  So that supports the
> theory from comment 41 that this is triggered by the closure hook.  Maybe
> when it rejects an attempted push, it leaves the journal in an inconsistent
> state?

Ted, any ideas here?
Of course, no more issues here after a while. Closing INCOMPLETE, this will only be fixed (hopefully) by the Hg upgrade.
Status: REOPENED → RESOLVED
Closed: 8 years ago8 years ago
Resolution: --- → INCOMPLETE
Another report on this for the try repo:

[root@hgssh1.dmz.scl3 .hg]# ls -aFl
total 19368
drwxrwsr-x   4 hg                    scm_level_1     4096 Apr  1 09:42 ./
drwxrwsr-x   3 hg                    scm_level_1     4096 Mar 21 17:25 ../
-rw-rw-r--   1 matekm@gmail.com      scm_level_1       94 Mar 28 09:02 bookmarks
drwxr-sr-x   2 hg                    scm_level_1     4096 Apr  1 08:50 cache/
-rw-rw-r--   1 hg                    scm_level_1      334 Mar 22 01:16 hgrc
-rw-rw-r--   1 masayuki@d-toybox.com scm_level_1       94 Apr  1 09:42 journal.bookmarks
-rw-rw-r--   1 gpascutto@mozilla.com scm_level_1        7 Apr  1 09:42 journal.branch
-rw-rw-r--   1 gpascutto@mozilla.com scm_level_1       39 Apr  1 09:42 journal.desc
-rw-rw-r--   1 gpascutto@mozilla.com scm_level_1        0 Apr  1 09:42 journal.dirstate
-rw-rw-r--   1 hg                    scm_level_1 19718144 Apr  1 08:50 pushlog2.db
-rwxrwxr-x 193 hg                    scm_level_2       15 May  9  2007 requires*
drwxrwsr-x   3 hg                    scm_level_1     4096 Apr  1 09:42 store/
-rw-rw-r--   1 bzbarsky@mozilla.com  scm_level_1       94 Apr  1 08:50 undo.bookmarks
-rw-rw-r--   1 bzbarsky@mozilla.com  scm_level_1        7 Apr  1 08:50 undo.branch
-rw-rw-r--   1 bzbarsky@mozilla.com  scm_level_1       39 Apr  1 08:50 undo.desc
-rw-rw-r--   1 bzbarsky@mozilla.com  scm_level_1        0 Apr  1 08:50 undo.dirstate
Status: RESOLVED → UNCONFIRMED
Ever confirmed: false
Resolution: INCOMPLETE → ---
[root@hgssh1.dmz.scl3 .hg]# mkdir ~/try-fix-2013-04-01
[root@hgssh1.dmz.scl3 .hg]# mv journal.* ~/try-fix-2013-04-01
Peter,

Welcome to report further occurrences here, but there's nothing we do but manually fix until releng gives us the okay to upgrade Hg (as per comment #71).
Status: UNCONFIRMED → RESOLVED
Closed: 8 years ago8 years ago
Resolution: --- → INCOMPLETE
Severity: blocker → normal
Component: Server Operations: Developer Services → General
Product: mozilla.org → Developer Services
You need to log in before you can comment on or make changes to this bug.