847864 - Remote error pushing to hg.mozilla.org/releases/l10n/mozilla-aurora/my

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Axel Hecht [:Pike]]

+++ This bug was initially created as a clone of Bug #842536 +++

lisa-2:my milos$ hg push
pushing to ssh://hg.mozilla.org/releases/l10n/mozilla-aurora/my
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 2 changes to 2 files
remote: error: changegroup.z_loghistory hook raised an exception: attempt to write a readonly database

Push went through, but pushlog didn't seem to catch it(see URL in this bug).

I'm getting the same error on beta/ms:

Übertrage nach ssh://hg.mozilla.org/releases/l10n/mozilla-beta/ms/
Suche nach Änderungen
Entfernt: Füge Änderungssätze hinzu
Entfernt: Füge Manifeste hinzu
Entfernt: Füge Dateiänderungen hinzu
Entfernt: Fügte 1 Änderungssätze mit 1 Änderungen an 1 Dateien hinzu
Entfernt: Fehler: Hook changegroup.z_loghistory löste eine Ausnahme aus: attempt to write a readonly database

http://hg.mozilla.org/releases/l10n/mozilla-beta/ms/rev/e919fc057eb4 has an empty pushlog.

Comment 1

[Axel Hecht [:Pike]]

Any update on this?

Comment 2

[Ben Kero [:bkero]]

[[e-mail]3 .hg]# ls -hl pushlog2.db
-rw-r--r-- 1 hg                      scm_l10n    0 Jul 30  2012 pushlog2.db

Not sure how this was created without g+w. Discussing setting an explicit umask for all connections into hg. I've also fixed the permissions on the pushlog file belonging to mozilla-beta/ms

I've run 'find . -not -perm -g=w' in the l10n/ repositories and have fixed all the problems. Please let me know if you run into anything else.

Comment 3

[Axel Hecht [:Pike]]

Sorry, just tried Occitan on beta, and got the same error again :-/

wokbok:oc axelhecht$ hg push
Übertrage nach ssh://hg.mozilla.org/releases/l10n/mozilla-beta/oc/
Suche nach Änderungen
Entfernt: Füge Änderungssätze hinzu
Entfernt: Füge Manifeste hinzu
Entfernt: Füge Dateiänderungen hinzu
Entfernt: Fügte 1 Änderungssätze mit 1 Änderungen an 1 Dateien hinzu
Entfernt: Fehler: Hook changegroup.z_loghistory löste eine Ausnahme aus: [Errno 1] Operation not permitted: '/repo/hg/mozilla/releases/l10n/mozilla-beta/oc/.hg/pushlog2.db'

Comment 4

[arky [:arky]]

Ben, Can you please take one more shot at this.

Comment 5

[arky [:arky]]

Ben, Can you please take one more shot at this.

Comment 6

[arky [:arky]]

bkero, Can you please look at this bug again.

Comment 7

[arky [:arky]]

bkero, Can you please look at this bug again.

Comment 8

[Ben Kero [:bkero]]

Which repositories is this an issue for? I checked mozilla-beta/oc and mozilla-aurora/my and both seem to match other repositories in terms of ownership and permissions.

[[e-mail]3 .hg]# pwd
/repo/hg/mozilla/releases/l10n/mozilla-beta/oc/.hg

[[e-mail]3 .hg]# ls -hl pushlog2.db 
-rw-rw-r-- 1 hg scm_l10n 9.0K Mar 21 16:17 pushlog2.db


[[e-mail]3 .hg]# pwd
/repo/hg/mozilla/releases/l10n/mozilla-aurora/my/.hg

[[e-mail]3 .hg]# ls -hl pushlog2.db 
-rw-rw-r-- 1 hg scm_l10n 9.0K Mar 23 15:31 pushlog2.db

Could you please tell me which username/ssh key you're attempting to make these changes with? Perhaps it doesn't have scm_l10n group permissions.

Comment 9

[arky [:arky]]

Axel, Any info?

Comment 10

[arky [:arky]]

bkero, fox2mike 

This problem needs more in-depth debugging. Permissions might be the culprit. Please kindly resolve this issue at the earliest. 

Thanks

PS:This is a goal blocker for Q2 for me.

Comment 11

[Axel Hecht [:Pike]]

I'm investigating as we speak. The repos that didn't work back when seem to have worked in the meantime, it's unclear to me why. I'm now trying to compare the pushlog data with the landed changesets to see if there's mismatch in any of the repos as far as I can see.

Comment 12

[Axel Hecht [:Pike]]

Got hit on different repos now, and I'm managing to push, but the pushlog db fails to update. The aurora repos are fine, I'm just hitting this on beta. l10n-central passes, but that may just be because we're only pushing to active repos there. We might want to occasionally push aurora states over there, not sure what'd happen.

pushing to ssh://hg.mozilla.org/releases/l10n/mozilla-beta/xh/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 7 changesets with 629 changes to 335 files
remote: error: changegroup.z_loghistory hook raised an exception: [Errno 1] Operation not permitted: '/repo/hg/mozilla/releases/l10n/mozilla-beta/xh/.hg/pushlog2.db'


(not testing x-testing for now)

kochbuch:ve ahecht$ hg push
pushing to ssh://hg.mozilla.org/releases/l10n/mozilla-beta/ve/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 0 changes to 0 files
remote: error: changegroup.z_loghistory hook raised an exception: [Errno 1] Operation not permitted: '/repo/hg/mozilla/releases/l10n/mozilla-beta/ve/.hg/pushlog2.db'

I've stopped testing now, as this is apparently happening still across various repos.

I'll attach my test script in a minute, which uses the requests library. You'd run it with 

python validate-pushlog.py beta

This has two failure modes: The repo never got pushed to, or the push of the current head of 'default' didn't get entered into the pushlog db. The repos I tested above are actually repos that didn't get pushed to since we created the beta repositories back in the days, bhearsum's tag is the last changeset prior ot me trying to push. 'xh' is actually now in my review queue, so at least some are blocking current work.

My current output is, and all that are 'fail' need investigation, it seems.

http://hg.mozilla.org/releases/l10n/mozilla-beta/ach/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/af/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ak/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/an/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ar/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/as/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ast/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/az/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/be/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/bg/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/bn-BD/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/bn-IN/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/br/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/bs/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ca/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/cs/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/csb/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/cy/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/da/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/de/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/el/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/en-GB/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/en-ZA/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/eo/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/es-AR/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/es-CL/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/es-ES/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/es-MX/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/et/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/eu/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/fa/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ff/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/fi/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/fr/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/fy-NL/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ga-IE/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/gd/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/gl/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/gu-IN/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/he/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/hi-IN/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/hr/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/hu/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/hy-AM/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/id/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/is/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/it/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ja/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ja-JP-mac/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ka/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/kk/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/km/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/kn/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ko/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ku/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/lg/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/lij/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/lt/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/lv/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/mai/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/mk/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ml/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/mn/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/mr/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ms/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/my/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/nb-NO/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ne-NP/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/nl/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/nn-NO/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/nr/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/nso/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/oc/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/or/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/pa-IN/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/pl/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/pt-BR/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/pt-PT/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/rm/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ro/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ru/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/rw/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/sah/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/si/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/sk/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/sl/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/son/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/sq/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/sr/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ss/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/st/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/sv-SE/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/sw/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ta/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ta-LK/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/te/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/th/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/tn/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/tr/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ts/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/uk/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ur/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ve/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/vi/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/wo/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/wo-backup/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/x-testing/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/xh/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/zh-CN/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/zh-TW/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/zu/: pass

Comment 13

[Axel Hecht [:Pike]]

Attachment: validate-pushlog.py, check json-info against json-pushes for 'default'

This is the script I use to compare what the web server knows about the 'default' revision. I compare json-info, which has the hex hash, if there's a changeset, with json-pushes, which has the changesets for the 'default' changeset, if it got that.

Note, I'm not detecting repos that are completely empty, I just realize. Those probably need some extra care, too.

Comment 14

[Shyam Mani [:fox2mike]]

Axel,

Can you test with empty repos too? 

Ben,

http://hg.mozilla.org/releases/l10n/mozilla-beta/ka/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/ms/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/ne-NP/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/nr/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/rw/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/ss/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/st/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/tn/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/ts/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/ve/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/x-testing/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/xh/: fail

Can we fix up perms on these if needed? I have a feeling we had a bad batch of repos created somewhere, never used for ages and we're now hitting previous bugs because of that (bugs that caused this have since been resolved).

Comment 15

[Axel Hecht [:Pike]]

Attachment: helper script, now calling out empty repos explicitly

I've updated the script to call out empty repos explicitly, because they might fail still. Reran it, and the following are empty:

http://hg.mozilla.org/releases/l10n/mozilla-beta/az/
http://hg.mozilla.org/releases/l10n/mozilla-beta/my/
http://hg.mozilla.org/releases/l10n/mozilla-beta/sah/

Comment 16

[arky [:arky]]

Thanks fox2mike, Pike for looking into this. This bug blocks me from landing new locales into Aurora or taking locales into beta channel.

Comment 17

[Ben Kero [:bkero]]

These repositories have had commits landed in them without having entries created in pushlog. IT hasn't solidified a way to synthesize a pushlog.

The easiest course of action (for the failing repositories) would be to:

1) Clone the repositories
2) Destroy the repositories on hg.mozilla.org
3) 'hg init' new repositories on hg.mozilla.org
4) 'hg push' the repositories back to create proper pushlog entries

What does everybody think of this plan?

Comment 18

[Axel Hecht [:Pike]]

The problem is not that we have changesets without pushes, the problem is that new pushes don't get recorded.

We need to be sure that the pushlogs get created in the future, right now it's like throwing dices.

I really don't like the idea that we throw away and create new repositories every time something goes wrong. The root of the problems needs to be figured out.

Comment 19

[Shyam Mani [:fox2mike]]

(In reply to Axel Hecht [:Pike] from comment #18)
> The problem is not that we have changesets without pushes, the problem is
> that new pushes don't get recorded.
> We need to be sure that the pushlogs get created in the future, right now
> it's like throwing dices.

We had a problem with this around the time these repos (and a bunch of other repos for you guys) were created. That has since been fixed.

> I really don't like the idea that we throw away and create new repositories
> every time something goes wrong. The root of the problems needs to be
> figured out.

I'm with you, but no one else has had any issues with any of the other repos we've created. I think starting clean is the right way here, if we see the problem after starting clean, I'm willing to spend more time and resources. Right now, it's a wild goose chase because this issue is a one off (in the sense that the repos were created when we had a bug which has been fixed for a while, but the repos are buggy).

Comment 20

[Axel Hecht [:Pike]]

tbh, that's not inspiring confidence for future repositories.

Anyway, let's move this forward. I veto getting yet more backup repositories in our production hg repo sets, though. I.e., no more http://hg.mozilla.org/releases/l10n/mozilla-beta/wo-backup/

Comment 21

[arky [:arky]]

Alright, let us try this and see if it fixes this long standing problem.

Comment 22

[arky [:arky]]

@bkero Did you get a chance to look at this issue?

Comment 23

[Ben Kero [:bkero]]

I've run your script (with s/ri.json()/ri.json to fix non-callable-dict errors) and found the broken repositories. With that list I

1) cloned them
2) deleted the upstream repository
3) created the upstream repository
4) pushed

After running your script again, none of the repositories fail your script anymore. Can you ensure they're all entered correctly?

Comment 24

[Axel Hecht [:Pike]]

Merge day:

pushing to ssh://hg.mozilla.org/releases/l10n/mozilla-beta/my/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 20 changesets with 833 changes to 345 files
remote: error: changegroup.z_loghistory hook raised an exception: [Errno 1] Operation not permitted: '/repo/hg/mozilla/releases/l10n/mozilla-beta/my/.hg/pushlog2.db'

Comment 25

[Shyam Mani [:fox2mike]]

Kendall,

please take a stab at this and see what's going on?

Comment 26

[Kendall Libby [:fubar] (he/him)]

Not a permissions issue, this time:

[[e-mail]3 .hg]# ls -ald /repo/hg/mozilla/releases/l10n/mozilla-beta/my/.hg{,/pushlog2.db}
drwxrwsr-x 4 hg scm_l10n 4096 Jun 24 13:22 /repo/hg/mozilla/releases/l10n/mozilla-beta/my/.hg
-rw-rw-r-- 1 hg scm_l10n 9216 Jun 24 13:19 /repo/hg/mozilla/releases/l10n/mozilla-beta/my/.hg/pushlog2.db

Into the depths of pushlog I go...

Comment 27

[Kendall Libby [:fubar] (he/him)]

Axel, can you try the push again? 

The file permissions are correct all the way down, and I'm unable to duplicate the error using a tar'ed copy of the repo. On top of that, what's happening in pushlog is pretty limited:

def schemaexists(conn):
    return 1 == conn.execute("SELECT COUNT(*) FROM SQLITE_MASTER WHERE name='pushlog'").fetchone()[0]

def log(ui, repo, node, **kwargs):
    pushdb = os.path.join(repo.path, 'pushlog2.db')
    createdb = False
    if not os.path.exists(pushdb):
        createdb = True
    conn = sqlite.connect(pushdb)
    if not createdb and not schemaexists(conn):
        createdb = True
    if createdb:
        createpushdb(conn)
        st = os.stat(pushdb)
        os.chmod(pushdb, st.st_mode | stat.S_IWGRP)
    t = int(time.time())
    retval = 1
    print "Trying to insert into pushlog."
    print "Please do not interrupt..."

Either the OS is failing to report that the file exists and it's trying to create a new pushlog, or sqlite.connect is failing. And again, I'm unable to duplicate that; I can see the file, and 'sqlite pushlog2.db' works and I can see the schema/tables/db, though there are no table contents. 

If you push and it fails, I'll try and write something to simulate a push and see if I can get it to break, but I want to make sure that it's NOT a transient (helloooo NFS) error.

Comment 28

[Axel Hecht [:Pike]]

Seems to work now. Odd, and I wish the push would actually fail if it fails.

Comment 29

[Kendall Libby [:fubar] (he/him)]

Yeah, I was afraid that it would work again, because that means it's related to the fabulous and long standing pushlog/nfs issues. 

As a recap, for clarity's sake...

The error "changegroup.z_loghistory hook raised an exception: attempt to write a readonly database" is due to file permission issues on pushlog2.db, which we can fix easily enough.

The error "changegroup.z_loghistory hook raised an exception: [Errno 1] Operation not permitted: <path>" appears to be a transient error, likely due to hg/pushlog on NFS. Try pushing again after a short wait; if it fails consistently please do let us know so we can investigate further!