766810 - Pushing to mozilla-inbound fails with abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks

Status: RESOLVED INCOMPLETE

(Developer Services :: General, task)

Description

[:Margaret Leibovic]

I'm getting this error:

pushing to ssh://hg.mozilla.org/integration/mozilla-inbound
searching for changes
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string

Comment 1

[:Gavin Sharp [email: [e-mail]]]

Drew says he's seeing this too

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Others are able to push - http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=d047caf12f04&tochange=6d5a6c8120c0 - so this might be intermittent. How many nodes do we have behind ssh traffic to hg.m.o ?

Comment 3

[Shyam Mani [:fox2mike]]

(In reply to Nick Thomas [:nthomas] from comment #2)
> Others are able to push -
> http://hg.mozilla.org/integration/mozilla-inbound/
> pushloghtml?fromchange=d047caf12f04&tochange=6d5a6c8120c0 - so this might be
> intermittent. How many nodes do we have behind ssh traffic to hg.m.o ?

Just one active atm. (same way it's always been).

Comment 4

[Shyam Mani [:fox2mike]]

Wondering if this is somehow related to bug 766533

Comment 5

[Shyam Mani [:fox2mike]]

I don't see this file in there :
(quoting to prevent wrapping)

> [[e-mail]3 mozilla-inbound]# ls -l .hg
> total 15060
> -rw-rw-r--   1 root                       scm_level_3       0 Jun 20 07:02 bookmarks
> -rw-rw-r--   1 hg                         scm_level_3       8 Jun  6  2011 branch
> -rw-rw-r--   1 [e-mail]         scm_level_3    2725 Jan 13 08:21 branchheads.cache
> drwxrwsr-x   2 [e-mail]      scm_level_3    4096 Jun 20 20:24 cache
> -rw-rw-r--   1 hg                         scm_level_3 3124085 Jun  6  2011 dirstate
> -rw-rw-r--   1 hg                         scm_level_3     282 Oct 29  2011 hgrc
> -rw-rw-r--   1 [e-mail] scm_level_3 4542464 Jun 20 20:24 pushlog2.db
> -rw-r--r--   1 root                       scm_level_3 4529152 Jun 20 06:32 pushlog2.db.20120620-cshields
> -rw-rw-r-- 162 hg                         scm_level_2      15 May  9  2007 requires
> drwxrwsr-x   3 hg                         scm_level_3    4096 Jun 20 20:24 store
> drwxr-sr-x   2 root                       scm_level_3    4096 Jun 20 07:01 strip-backup
> -rw-rw-r--   1 [e-mail]          scm_level_3   12130 Jan 13 00:54 tags.cache
> -rw-rw-r--   1 [e-mail]     scm_level_3       0 Jun 20 20:24 undo.bookmarks
> -rw-rw-r--   1 [e-mail]     scm_level_3       7 Jun 20 20:24 undo.branch
> -rw-rw-r--   1 [e-mail]     scm_level_3      38 Jun 20 20:24 undo.desc
> -rw-rw-r--   1 [e-mail]     scm_level_3 3124085 Jun 20 20:24 undo.dirstate

Comment 6

[Shyam Mani [:fox2mike]]

Meh, marking confidential for the emails above.

Comment 7

[Mike Hommey [:glandium]]

What are the permissions on .hg itself ?

Comment 8

[Shyam Mani [:fox2mike]]

(In reply to Mike Hommey [:glandium] from comment #7)
> What are the permissions on .hg itself ?

drwxrwsr-x  5 hg scm_level_3   4096 Jun 21 01:03 .hg

Comment 9

[:Margaret Leibovic]

Well, it worked for me now. Maybe just a temporary problem yesterday?

Comment 10

[Shyam Mani [:fox2mike]]

Yup, going to call it that for now.

Comment 11

[:Gavin Sharp [email: [e-mail]]]

(In reply to Shyam Mani [:fox2mike] from comment #6)
> Meh, marking confidential for the emails above.

those emails are in pushlog, no need to hide this for that

Comment 12

[Nicholas Nethercote [inactive]]

I just hit this three time in a row.  I haven't managed to land my patch, yet.

Comment 13

[Nicholas Nethercote [inactive]]

I tried pushing my patches to a different local repo and then pushing to inbound from there, and that failed too, so it's not something to do with my local repos.

Comment 14

[:Gavin Sharp [email: [e-mail]]]

This is also currently affecting me when I try to push to inbound.

Comment 15

[Jason Duell]

This also appears to have merged my patch with another one!

  [1 patch in queue series]
  qfin -a
  hg push    // get error string like everyone else in this bug report
  hg out     // still shows my patch
  hg qimport -r tip  // imports changeset with my changeset message, but there's other code in it now too: some "foo.orig' files in the /dom directory. Yay!

Comment 16

[Jason Duell]

I'm using mercurial 2.0-2, BTW, on ubuntu.  And this keeps failing no matter how many times I try (FYI: no one else has landed anything in the interim, in case that matters.  Perhaps I need someone else to land to reset file permissions somehow?

 http://selenic.com/pipermail/mercurial/2011-November/040629.html

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Happening here too.

Comment 18

[Jeff Walden [:Waldo]]

And here.  (Is this really intermittent, or is it biting everyone now?)

Comment 19

[Phil Ringnalda (:philor)]

Because everybody loves getting uninformed wild guesses, umask?

(In reply to Shyam Mani [:fox2mike] from comment #5)
> I don't see this file in there :

You shouldn't - near as I can tell, it should create it (and several others, which oddly should fail first, but apparently don't) as part of the push, copying in the contents of the 0 byte bookmarks file, to use to rollback if the transaction gets interrupted, and then it should get renamed to undo.bookmarks for use in revert. So I'd expect the only way to see the file would be to look during a push.

Since it is hitting everybody (at least, everybody who has tried to push for the last five hours), this has effectively closed mozilla-inbound.

Comment 20

[Shyam Mani [:fox2mike]]

Right. It seems like someone's push failed and then everything stacked up after it.

> [[e-mail]3 .hg]# ls -l
> total 18252
> -rw-rw-r--   1 root                       scm_level_3       0 Jun 20 07:02 bookmarks
> -rw-rw-r--   1 hg                         scm_level_3       8 Jun  6  2011 branch
> -rw-rw-r--   1 [e-mail]         scm_level_3    2725 Jan 13 08:21 branchheads.cache
> drwxrwsr-x   2 [e-mail]      scm_level_3    4096 Jun 28 17:03 cache
> -rw-rw-r--   1 hg                         scm_level_3 3124085 Jun  6  2011 dirstate
> -rw-rw-r--   1 hg                         scm_level_3     282 Oct 29  2011 hgrc
> -rw-rw-r--   1 [e-mail]      scm_level_3       0 Jun 28 23:55 journal.bookmarks
> -rw-rw-r--   1 [e-mail]           scm_level_3       7 Jun 28 23:55 journal.branch
> -rw-rw-r--   1 [e-mail]           scm_level_3      38 Jun 28 23:55 journal.desc
> -rw-rw-r--   1 [e-mail]           scm_level_3 3124085 Jun 28 23:55 journal.dirstate
> -rw-rw-r--   1 [e-mail] scm_level_3 4670464 Jun 28 16:37 pushlog2.db
> -rw-r--r--   1 root                       scm_level_3 4529152 Jun 20 06:32 > pushlog2.db.20120620-cshields
> -rw-rw-r-- 165 hg                         scm_level_1      15 May  9  2007 requires
> drwxrwsr-x   3 hg                         scm_level_3    4096 Jun 28 23:55 store
> drwxr-sr-x   2 root                       scm_level_3    4096 Jun 20 07:01 strip-backup
> -rw-rw-r--   1 [e-mail]          scm_level_3   12130 Jan 13 00:54 tags.cache
> -rw-rw-r--   1 [e-mail]         scm_level_3       0 Jun 28 16:37 undo.bookmarks
> -rw-rw-r--   1 [e-mail]         scm_level_3       7 Jun 28 16:37 undo.branch
> -rw-rw-r--   1 [e-mail]         scm_level_3      38 Jun 28 16:37 undo.desc
> -rw-rw-r--   1 [e-mail]         scm_level_3 3124085 Jun 28 16:37 undo.dirstate

This shouldn't be happening. And wasn't. I'm not sure what's changed to cause this.

Comment 21

[Shyam Mani [:fox2mike]]

I've blown away the journal.* files, but I'll keep an eye on this. Bug is still open, but mozilla-inbound should be okay to push to.

Comment 22

[Shyam Mani [:fox2mike]]

Did Wesley try to push something that didn't go well?

Comment 23

[Shyam Mani [:fox2mike]]

This is the concerning bit : 

> -rw-rw-r--   1 [e-mail]      scm_level_3       0 Jun 28 23:55 journal.bookmarks
> -rw-rw-r--   1 [e-mail]           scm_level_3       7 Jun 28 23:55 journal.branch
> -rw-rw-r--   1 [e-mail]           scm_level_3      38 Jun 28 23:55 journal.desc
> -rw-rw-r--   1 [e-mail]           scm_level_3 3124085 Jun 28 23:55 journal.dirstate

It seems like wesley's push failed or didn't clean up after itself and when felipe tried, it failed. The timestamps don't match the first report of failure from Nicholas, so right now this is all pure speculation.

Comment 24

[Wesley Johnston (:wesj)]

Hmm... I tried to push something yesterday and had hg reject it because I had qfolded a patch with try chooser stuff in the commit message. That then turned into a mess in my tree that I still haven't cleaned up, but could that have caused this?

Comment 25

[(no longer active)]

We're now getting this on inbound consistently, as a result, nobody can push to inbound.

Comment 26

[(no longer active)]

This is blocking people (including me) from pushing to inbound.  I have a lot of important backouts on central which needs to go on inbound asap.

Comment 27

[(no longer active)]

Unassigning in the hopes that this gets someone's attention.

Comment 28

[Shyam Mani [:fox2mike]]

(In reply to Ehsan Akhgari [:ehsan] from comment #27)
> Unassigning in the hopes that this gets someone's attention.

Wrong assignee, please reset to default or this won't page. 

> -rw-rw-r--   1 [e-mail]        scm_level_3       0 Jul  4 17:54 journal.bookmarks
> -rw-rw-r--   1 [e-mail]      scm_level_3       7 Jul  4 17:54 journal.branch
> -rw-rw-r--   1 [e-mail]      scm_level_3      38 Jul  4 17:54 journal.desc
> -rw-rw-r--   1 [e-mail]      scm_level_3 3124085 Jul  4 17:54 journal.dirstate

Either Jim or Hub had a push fail on them? This is pretty hard to debug coming in hours after it's happened :|

Comment 29

[Shyam Mani [:fox2mike]]

Alright, please poke the oncall sysadmin (in #it) to page either me or bkero when this happens next. We need to catch this as it happens (first dev that triggers this) to see why it's happening.

Comment 30

[Hubert Figuiere [:hub]]

 
> > -rw-rw-r--   1 [e-mail]        scm_level_3       0 Jul  4 17:54 journal.bookmarks
> > -rw-rw-r--   1 [e-mail]      scm_level_3       7 Jul  4 17:54 journal.branch
> > -rw-rw-r--   1 [e-mail]      scm_level_3      38 Jul  4 17:54 journal.desc
> > -rw-rw-r--   1 [e-mail]      scm_level_3 3124085 Jul  4 17:54 journal.dirstate
> 
> Either Jim or Hub had a push fail on them? This is pretty hard to debug
> coming in hours after it's happened :|

I had it fail on the message mentioned in the bug title.

Comment 31

[Ed Morley [:emorley]]

Just had this occur for me pushing to inbound at ~15:05 UTC+1 2012-07-12

pushing to ssh://hg.mozilla.org/integration/mozilla-inbound/
searching for changes
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string

bkero/fox2mike pinging in #it

Comment 32

[Ashish Vijayaram [:ashish]]

Paged bkero. In the meantime, adding some debug info:

> [[e-mail]3 .hg]# ls -l journal*
> -rw-rw-r-- 1 [e-mail]  scm_level_3       0 Jul 12 07:05 journal.bookmarks
> -rw-rw-r-- 1 [e-mail] scm_level_3       7 Jul 12 07:05 journal.branch
> -rw-rw-r-- 1 [e-mail] scm_level_3      38 Jul 12 07:05 journal.desc
> -rw-rw-r-- 1 [e-mail] scm_level_3 3124085 Jul 12 07:05 journal.dirstate
> [[e-mail]3 .hg]# stat journal.*
>   File: `journal.bookmarks'
>   Size: 0               Blocks: 0          IO Block: 65536  regular empty file
> Device: 13h/19d Inode: 17124171    Links: 1
> Access: (0664/-rw-rw-r--)  Uid: ( 1129/[e-mail])   Gid: (  679/scm_level_3)
> Access: 2012-07-12 05:52:06.491817000 -0700
> Modify: 2012-07-12 07:05:52.817403000 -0700
> Change: 2012-07-12 07:05:52.817403000 -0700
>   File: `journal.branch'
>   Size: 7               Blocks: 0          IO Block: 65536  regular file
> Device: 13h/19d Inode: 17124169    Links: 1
> Access: (0664/-rw-rw-r--)  Uid: ( 1943/[e-mail])   Gid: (  679/scm_level_3)
> Access: 2012-07-12 07:05:52.814445000 -0700
> Modify: 2012-07-12 07:05:52.815347000 -0700
> Change: 2012-07-12 07:05:52.815347000 -0700
>   File: `journal.desc'
>   Size: 38              Blocks: 0          IO Block: 65536  regular file
> Device: 13h/19d Inode: 17124170    Links: 1
> Access: (0664/-rw-rw-r--)  Uid: ( 1943/[e-mail])   Gid: (  679/scm_level_3)
> Access: 2012-07-12 07:05:52.816317000 -0700
> Modify: 2012-07-12 07:05:52.816360000 -0700
> Change: 2012-07-12 07:05:52.816360000 -0700
>   File: `journal.dirstate'
>   Size: 3124085         Blocks: 6128       IO Block: 65536  regular file
> Device: 13h/19d Inode: 17124168    Links: 1
> Access: (0664/-rw-rw-r--)  Uid: ( 1943/[e-mail])   Gid: (  679/scm_level_3)
> Access: 2012-07-12 07:05:52.782370000 -0700
> Modify: 2012-07-12 07:05:52.813335000 -0700
> Change: 2012-07-12 07:05:52.813335000 -0700

Comment 33

[Ed Morley [:emorley]]

Just remembered, cjones said this at ~1353 UTC+1 in #developers:
"inbound says open, hg tells me otherwise"

So presume he tried to push and hit the closure hook. (Inbound was closed at the time, cjones just hadn't refreshed the tbpl page).

Comment 34

[Nathan Froyd [:froydnj]]

FWIW, just hit this; pinged bkero/fox2make in #it for bonus points.  Happy to provide more information.  Seems to happen repeatedly for me.

Comment 35

[Ben Kero [:bkero]]

[[e-mail]3 mozilla-inbound]# hg verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
101525 files, 99059 changesets, 502734 total revisions

Comment 36

[Ben Kero [:bkero]]

I moved the journal.* and undo.* files to a backup directory and functionality was restored.

Please ignore the latest committed changeset, that's how I verified this working.

[[e-mail]3 .hg]# hg log -l 4
changeset:   99059:ed4888197cb5
tag:         tip
user:        Malini Das <[e-mail]>
date:        Thu Jul 12 11:01:20 2012 -0400
summary:     Bug 771224 - use chrome:// instead of loading remote xul. r=jgriffin

changeset:   99058:f0be4b70b814
user:        Ed Morley <[e-mail]>
date:        Thu Jul 12 13:04:51 2012 +0100
summary:     Backout 6bbf3f22bb5d (bug 753158), 38a703b244c2 (bug 753145), c9a5dfa1b07d (bug 767750), cd782fd66995 & 6cf7aa93994c (bug 765956), 0253f34f6bc2 & 41d5c8529748 (bug 771039),94f6bf99a4aa (bug 766447),fad7d06d7dd5 (bug 772303) for winxp pgo-only jsreftest failures (caused by fad7d06d7dd5) and the rest for conflicts, on a CLOSED TREE

changeset:   99057:64cff7aafcc4
user:        Anant Narayanan <[e-mail]>
date:        Thu Jul 12 04:53:48 2012 -0700
summary:     Bug 771833: Assign an nsIPrincipal to media streams returned by getUserMedia; r=roc

changeset:   99056:1d3ef213c75b
user:        Anant Narayanan <[e-mail]>
date:        Thu Jul 12 04:53:08 2012 -0700
summary:     Bug 691234: Part 3/3: Add DOM binding for getUserMedia on Desktop; r=jst

Comment 37

[Ben Kero [:bkero]]

Next time this happens we should investigate using hg rollback (maybe with -n to dry-run) to roll the repository back to the previous state and see if that yields any clues as to the reason for these problems.

Comment 38

[(no longer active)]

I just got this again when pushing:

ehsanakhgari:~/moz/inbound [02:19:01]$ hg push
pushing to ssh://hg.mozilla.org/integration/mozilla-inbound/
searching for changes
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string

Comment 39

[Myk Melez [:myk] [@mykmelez]]

Me too!

07-13 11:28 > hg push
pushing to ssh://[e-mail]/integration/mozilla-inbound/
searching for changes
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string

Comment 40

[Myk Melez [:myk] [@mykmelez]]

A subsequent attempt succeeded after a rebase:

07-13 11:28 > hg push
pushing to ssh://[e-mail]/integration/mozilla-inbound/
searching for changes
abort: push creates new remote head f44a483b2f92!
(you should pull and merge or use push -f to force)
07-13 12:02 > hg pull --rebase
pulling from ssh://[e-mail]/integration/mozilla-inbound/
searching for changes
adding changesets
adding manifests
adding file changes
added 7 changesets with 25 changes to 20 files (+1 heads)
saved backup bundle to c:\Users\myk\Mozilla\inbound\.hg\strip-backup\f44a483b2f92-backup.hg
07-13 12:03 > hg outgoing
comparing with ssh://[e-mail]/integration/mozilla-inbound/
searching for changes
changeset:   99213:06756c4481d5
tag:         tip
user:        Marco Castelluccio <[e-mail]>
date:        Fri Jul 13 11:27:49 2012 -0700
summary:     Bug 768768 - Launch app from shell, close, then launch it from terminal results in no icon showing up in task bar in ubuntu; r=karlt

07-13 12:03 > hg push
pushing to ssh://[e-mail]/integration/mozilla-inbound/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.

Comment 41

[Ben Kero [:bkero]]

I was able to investigate a bit today and this seems to be related to pushes while the tree is closed.

Earlier today (around 9AM) the trees closed and gavin (who wasn't sure if the trees were closed or not) was able to push to mozilla-inbound.  Afterwards, journal.* and undo.* files were created and persisted past the successful commit.

Armed with this new information, I'm going to read through the treeclosed script and as a last measure I could add part of the hook to be deleting the journal.* and undo.* files if no other solution becomes evident.

Comment 42

[Szabolcs Hubai (:xabolcs)]

(In reply to Ben Kero [:bkero] from comment #41)
>
> [...]
> 
> Armed with this new information, I'm going to read through the treeclosed
> script and as a last measure I could add part of the hook to be deleting the
> journal.* and undo.* files if no other solution becomes evident.

There are two related bugs in mercurial: one for journal.bookmarks[1] and one for
journal.phaseroots[2].

Mercurial 2.1.2 and 2.2 contains these fixes, but I hope they could be ported back easily.

[1] - http://bz.selenic.com/show_bug.cgi?id=3318
[2] - http://bz.selenic.com/show_bug.cgi?id=3376

Sorry if this was already listed.

Comment 43

[Ben Kero [:bkero]]

I already have a bug open for testing new version of mercurial (bug #741353 )

Additionally today I cloned mozilla-inbound as a bare repository, in the hopes that the storefiles changed in 2.0.2 since the last mozilla-inbound clone was on 1.x.

Comment 44

[:Gavin Sharp [email: [e-mail]]]

(In reply to Ben Kero [:bkero] from comment #43)
> Additionally today I cloned mozilla-inbound as a bare repository, in the
> hopes that the storefiles changed in 2.0.2 since the last mozilla-inbound
> clone was on 1.x.

I just got:
remote: abort: could not lock repository /repo/hg/mozilla/integration/mozilla-inbound: Permission denied                                              
abort: unexpected response: empty string

when trying to push to inbound. Related?

Comment 45

[Kannan Vijayan [:djvj]]

pushing to ssh://hg.mozilla.org/integration/mozilla-inbound
searching for changes
remote: abort: could not lock repository /repo/hg/mozilla/integration/mozilla-inbound: Permission denied
abort: unexpected response: empty string

I'm getting the same thing.  I was able to push earlier today - this just started happening now.

Comment 46

[:Gavin Sharp [email: [e-mail]]]

Spoke to bkero on IRC, it was indeed fallout from his earlier change - he fixed it yesterday shortly after my comment.

Comment 47

[Shyam Mani [:fox2mike]]

Has this happened recently?

Comment 48

[Kannan Vijayan [:djvj]]

I was able to push a couple of hours after Comment 45.  Haven't pushed to m-i since though so I don't know if it showed up again.

Comment 49

[Shyam Mani [:fox2mike]]

We've had this open a month without any reports...and now that I've jinxed it, let's see if we see it again.

Comment 50

[Shyam Mani [:fox2mike]]

Reopen if we run into this again, please.

Comment 51

[Ryan VanderMeulen [:RyanVM]]

Your wish is my command!

$ hg push
pushing to ssh://hg.mozilla.org/integration/mozilla-inbound
searching for changes
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string

Comment 52

[Ed Morley [:emorley]]

Caused by bug 797910?

Comment 53

[Shyam Mani [:fox2mike]]

This is now confirmed. If we strip, the repo goes mad. 

I'm not sure how we can fix this, tbh. 

Ehsan?

Comment 54

[Shyam Mani [:fox2mike]]

(In reply to Ryan VanderMeulen from comment #51)
> Your wish is my command!
> 
> $ hg push
> pushing to ssh://hg.mozilla.org/integration/mozilla-inbound
> searching for changes
> remote: abort: Operation not permitted:
> /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
> abort: unexpected response: empty string

Don't see those files now, so you shouldn't be blocked on this anymore.

It seems like hg strip (which is NOT recommended, maybe for reasons like this) causes later side effects like leaving behind files it shouldn't. The same thing happened the last time we stripped inbound.

Comment 55

[(no longer active)]

Did we ever understand why this happens exactly?

Comment 56

[Ryan VanderMeulen [:RyanVM]]

bz and I are both hitting this tonight trying to push to inbound

Comment 57

[Shyam Mani [:fox2mike]]

(In reply to Ehsan Akhgari [:ehsan] from comment #55)
> Did we ever understand why this happens exactly?

No. But we know it happens after we strip.

Comment 58

[Shyam Mani [:fox2mike]]

(In reply to Ryan VanderMeulen from comment #56)
> bz and I are both hitting this tonight trying to push to inbound

Cleaned up :

-rw-rw-r-- 1 [e-mail] scm_level_3       0 Oct  5 22:34 journal.bookmarks
-rw-rw-r-- 1 [e-mail]   scm_level_3       7 Oct  5 22:34 journal.branch
-rw-rw-r-- 1 [e-mail]   scm_level_3      39 Oct  5 22:34 journal.desc
-rw-rw-r-- 1 [e-mail]   scm_level_3       0 Oct  5 22:34 journal.dirstate

Should be good now.

Comment 59

[Ed Morley [:emorley]]

pushing to ssh://hg.mozilla.org/integration/mozilla-inbound/
searching for changes
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string

:-(

Comment 60

[Ed Morley [:emorley]]

Neither myself, gcp, cpearce can push to inbound.

I need to backout bustage in order to sheriff the tree, but cannot do so.

Comment 61

[Peter Radcliffe [:pir]]

Pinged by Callek about this happening:

[[e-mail]3 ~]# cd /repo/hg/mozilla/integration/mozilla-inbound/.hg/
[[e-mail]3 .hg]# ls -aFl journal.bookmarks journal.branch journal.desc journal.dirstate
-rw-rw-r-- 1 [e-mail] scm_level_3  0 Oct 12 03:05 journal.bookmarks
-rw-rw-r-- 1 [e-mail]      scm_level_3  7 Oct 12 03:05 journal.branch
-rw-rw-r-- 1 [e-mail]      scm_level_3 39 Oct 12 03:05 journal.desc
-rw-rw-r-- 1 [e-mail]      scm_level_3  0 Oct 12 03:05 journal.dirstate
[[e-mail]3 .hg]# date
Fri Oct 12 03:18:10 PDT 2012
[[e-mail]3 .hg]# mkdir ~/2012-10-12-03:18
[[e-mail]3 .hg]# mv journal.bookmarks journal.branch journal.desc journal.dirstate ~/2012-10-12-03:18

Comment 62

[Ryan VanderMeulen [:RyanVM]]

I'm currently running into this again.

Comment 63

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Same here.

Comment 64

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I'm hitting this too.

Comment 65

[(no longer active)]

Me too.

Comment 66

[Corey Shields [:cshields]]

looking...

Comment 67

[Corey Shields [:cshields]]

Offending files moved to ~/2012-10-13-05\:42/

Shyam, do we have an hg bug opened for this?  if not, please do ASAP and link here.

Comment 68

[Corey Shields [:cshields]]

A note for oncall:

The quick-fix for this is to move the journal files out of place.  Keep them just in case it does not work and we need to move them back.  

(on hgssh1)
mkdir ~/foo
cd /repo/hg/mozilla/integration/mozilla-inbound/.hg  (for mozilla-inbound problems only!)
mv journal.bookmarks journal.branch journal.desc journal.dirstate ~/foo

Then verify with someone in #developers

Comment 69

[Justin Wood (:Callek)]

(In reply to Ryan VanderMeulen from comment #62)
> I'm currently running into this again.

... per IRC with Corey...

When this hits in the future, its best to create a *new* blocker bug, so that it pages on-call. And just tie deps of it to here. Said bug can be resolved when that instance of us hitting the problem is resolved, this bug can track the overall problem + its recurrence.

Comment 70

[Szabolcs Hubai (:xabolcs)]

(In reply to Corey Shields [:cshields] from comment #67)
> ...
> 
> Shyam, do we have an hg bug opened for this?  if not, please do ASAP and
> link here.

IMHO the talk in comment #42 and comment #43 is related, isn't it?

Comment 71

[Shyam Mani [:fox2mike]]

(In reply to Corey Shields [:cshields] from comment #67)
> Offending files moved to ~/2012-10-13-05\:42/
> 
> Shyam, do we have an hg bug opened for this?  if not, please do ASAP and
> link here.

See https://bugzilla.mozilla.org/show_bug.cgi?id=766810#c42 :) So if we upgrade mercurial, this should go away.

Comment 72

[Daniel Holbert [:dholbert]]

Hitting this currently.
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string

Comment 73

[Dumitru Gherman [:dumitru]]

Applied the "fix" from comment 68.

Comment 74

[Matt Brubeck (:mbrubeck)]

I just hit this again.

Comment 75

[Shyam Mani [:fox2mike]]

Fixed it for Matt.

Comment 76

[Patrick McManus [:mcmanus]]

hitting this right now


pushing to ssh://hg.mozilla.org/integration/mozilla-inbound
searching for changes
remote: abort: Operation not permitted: /repo/hg/mozilla/integration/mozilla-inbound/.hg/journal.bookmarks
abort: unexpected response: empty string

Comment 77

[Matt Brubeck (:mbrubeck)]

The last few times this happened (including comment 74, comment 76, and others) were all on right after closing inbound.  So that supports the theory from comment 41 that this is triggered by the closure hook.  Maybe when it rejects an attempted push, it leaves the journal in an inconsistent state?

Comment 78

[Benoit Girard (:BenWa)]

Getting this as well :(.

Comment 79

[Shyam Mani [:fox2mike]]

Don't see any issues with it right now.

(In reply to Matt Brubeck (:mbrubeck) from comment #77)
> The last few times this happened (including comment 74, comment 76, and
> others) were all on right after closing inbound.  So that supports the
> theory from comment 41 that this is triggered by the closure hook.  Maybe
> when it rejects an attempted push, it leaves the journal in an inconsistent
> state?

Ted, any ideas here?

Comment 80

[Shyam Mani [:fox2mike]]

Of course, no more issues here after a while. Closing INCOMPLETE, this will only be fixed (hopefully) by the Hg upgrade.

Comment 81

[Peter Radcliffe [:pir]]

Another report on this for the try repo:

[[e-mail]3 .hg]# ls -aFl
total 19368
drwxrwsr-x   4 hg                    scm_level_1     4096 Apr  1 09:42 ./
drwxrwsr-x   3 hg                    scm_level_1     4096 Mar 21 17:25 ../
-rw-rw-r--   1 [e-mail]      scm_level_1       94 Mar 28 09:02 bookmarks
drwxr-sr-x   2 hg                    scm_level_1     4096 Apr  1 08:50 cache/
-rw-rw-r--   1 hg                    scm_level_1      334 Mar 22 01:16 hgrc
-rw-rw-r--   1 [e-mail] scm_level_1       94 Apr  1 09:42 journal.bookmarks
-rw-rw-r--   1 [e-mail] scm_level_1        7 Apr  1 09:42 journal.branch
-rw-rw-r--   1 [e-mail] scm_level_1       39 Apr  1 09:42 journal.desc
-rw-rw-r--   1 [e-mail] scm_level_1        0 Apr  1 09:42 journal.dirstate
-rw-rw-r--   1 hg                    scm_level_1 19718144 Apr  1 08:50 pushlog2.db
-rwxrwxr-x 193 hg                    scm_level_2       15 May  9  2007 requires*
drwxrwsr-x   3 hg                    scm_level_1     4096 Apr  1 09:42 store/
-rw-rw-r--   1 [e-mail]  scm_level_1       94 Apr  1 08:50 undo.bookmarks
-rw-rw-r--   1 [e-mail]  scm_level_1        7 Apr  1 08:50 undo.branch
-rw-rw-r--   1 [e-mail]  scm_level_1       39 Apr  1 08:50 undo.desc
-rw-rw-r--   1 [e-mail]  scm_level_1        0 Apr  1 08:50 undo.dirstate

Comment 82

[Peter Radcliffe [:pir]]

[[e-mail]3 .hg]# mkdir ~/try-fix-2013-04-01
[[e-mail]3 .hg]# mv journal.* ~/try-fix-2013-04-01

Comment 83

[Shyam Mani [:fox2mike]]

Peter,

Welcome to report further occurrences here, but there's nothing we do but manually fix until releng gives us the okay to upgrade Hg (as per comment #71).