Remote error pushing to hg.mozilla.org/releases/l10n/mozilla-aurora/my

RESOLVED FIXED

Status

RESOLVED FIXED
6 years ago
4 years ago

People

(Reporter: Pike, Assigned: fubar)

Tracking

Details

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

6 years ago
+++ This bug was initially created as a clone of Bug #842536 +++

lisa-2:my milos$ hg push
pushing to ssh://hg.mozilla.org/releases/l10n/mozilla-aurora/my
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 2 changes to 2 files
remote: error: changegroup.z_loghistory hook raised an exception: attempt to write a readonly database

Push went through, but pushlog didn't seem to catch it(see URL in this bug).

I'm getting the same error on beta/ms:

Übertrage nach ssh://hg.mozilla.org/releases/l10n/mozilla-beta/ms/
Suche nach Änderungen
Entfernt: Füge Änderungssätze hinzu
Entfernt: Füge Manifeste hinzu
Entfernt: Füge Dateiänderungen hinzu
Entfernt: Fügte 1 Änderungssätze mit 1 Änderungen an 1 Dateien hinzu
Entfernt: Fehler: Hook changegroup.z_loghistory löste eine Ausnahme aus: attempt to write a readonly database

http://hg.mozilla.org/releases/l10n/mozilla-beta/ms/rev/e919fc057eb4 has an empty pushlog.
(Reporter)

Comment 1

6 years ago
Any update on this?

Updated

6 years ago
Assignee: server-ops-devservices → bkero
(Reporter)

Updated

6 years ago
Blocks: 737970, 723958

Comment 2

6 years ago
[root@hgssh1.dmz.scl3 .hg]# ls -hl pushlog2.db
-rw-r--r-- 1 hg                      scm_l10n    0 Jul 30  2012 pushlog2.db

Not sure how this was created without g+w. Discussing setting an explicit umask for all connections into hg. I've also fixed the permissions on the pushlog file belonging to mozilla-beta/ms

I've run 'find . -not -perm -g=w' in the l10n/ repositories and have fixed all the problems. Please let me know if you run into anything else.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Reporter)

Comment 3

6 years ago
Sorry, just tried Occitan on beta, and got the same error again :-/

wokbok:oc axelhecht$ hg push
Übertrage nach ssh://hg.mozilla.org/releases/l10n/mozilla-beta/oc/
Suche nach Änderungen
Entfernt: Füge Änderungssätze hinzu
Entfernt: Füge Manifeste hinzu
Entfernt: Füge Dateiänderungen hinzu
Entfernt: Fügte 1 Änderungssätze mit 1 Änderungen an 1 Dateien hinzu
Entfernt: Fehler: Hook changegroup.z_loghistory löste eine Ausnahme aus: [Errno 1] Operation not permitted: '/repo/hg/mozilla/releases/l10n/mozilla-beta/oc/.hg/pushlog2.db'
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment 4

6 years ago
Ben, Can you please take one more shot at this.

Comment 5

6 years ago
Ben, Can you please take one more shot at this.

Comment 6

6 years ago
bkero, Can you please look at this bug again.

Comment 7

6 years ago
bkero, Can you please look at this bug again.

Comment 8

6 years ago
Which repositories is this an issue for? I checked mozilla-beta/oc and mozilla-aurora/my and both seem to match other repositories in terms of ownership and permissions.

[root@hgssh1.dmz.scl3 .hg]# pwd
/repo/hg/mozilla/releases/l10n/mozilla-beta/oc/.hg

[root@hgssh1.dmz.scl3 .hg]# ls -hl pushlog2.db 
-rw-rw-r-- 1 hg scm_l10n 9.0K Mar 21 16:17 pushlog2.db


[root@hgssh1.dmz.scl3 .hg]# pwd
/repo/hg/mozilla/releases/l10n/mozilla-aurora/my/.hg

[root@hgssh1.dmz.scl3 .hg]# ls -hl pushlog2.db 
-rw-rw-r-- 1 hg scm_l10n 9.0K Mar 23 15:31 pushlog2.db

Could you please tell me which username/ssh key you're attempting to make these changes with? Perhaps it doesn't have scm_l10n group permissions.

Comment 9

6 years ago
Axel, Any info?
Flags: needinfo?(l10n)

Comment 10

6 years ago
bkero, fox2mike 

This problem needs more in-depth debugging. Permissions might be the culprit. Please kindly resolve this issue at the earliest. 

Thanks

PS:This is a goal blocker for Q2 for me.
Flags: needinfo?(l10n) → needinfo?(bkero)
(Reporter)

Comment 11

6 years ago
I'm investigating as we speak. The repos that didn't work back when seem to have worked in the meantime, it's unclear to me why. I'm now trying to compare the pushlog data with the landed changesets to see if there's mismatch in any of the repos as far as I can see.
Flags: needinfo?(bkero)
(Reporter)

Comment 12

6 years ago
Got hit on different repos now, and I'm managing to push, but the pushlog db fails to update. The aurora repos are fine, I'm just hitting this on beta. l10n-central passes, but that may just be because we're only pushing to active repos there. We might want to occasionally push aurora states over there, not sure what'd happen.

pushing to ssh://hg.mozilla.org/releases/l10n/mozilla-beta/xh/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 7 changesets with 629 changes to 335 files
remote: error: changegroup.z_loghistory hook raised an exception: [Errno 1] Operation not permitted: '/repo/hg/mozilla/releases/l10n/mozilla-beta/xh/.hg/pushlog2.db'


(not testing x-testing for now)

kochbuch:ve ahecht$ hg push
pushing to ssh://hg.mozilla.org/releases/l10n/mozilla-beta/ve/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 0 changes to 0 files
remote: error: changegroup.z_loghistory hook raised an exception: [Errno 1] Operation not permitted: '/repo/hg/mozilla/releases/l10n/mozilla-beta/ve/.hg/pushlog2.db'

I've stopped testing now, as this is apparently happening still across various repos.

I'll attach my test script in a minute, which uses the requests library. You'd run it with 

python validate-pushlog.py beta

This has two failure modes: The repo never got pushed to, or the push of the current head of 'default' didn't get entered into the pushlog db. The repos I tested above are actually repos that didn't get pushed to since we created the beta repositories back in the days, bhearsum's tag is the last changeset prior ot me trying to push. 'xh' is actually now in my review queue, so at least some are blocking current work.

My current output is, and all that are 'fail' need investigation, it seems.

http://hg.mozilla.org/releases/l10n/mozilla-beta/ach/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/af/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ak/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/an/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ar/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/as/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ast/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/az/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/be/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/bg/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/bn-BD/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/bn-IN/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/br/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/bs/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ca/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/cs/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/csb/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/cy/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/da/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/de/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/el/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/en-GB/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/en-ZA/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/eo/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/es-AR/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/es-CL/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/es-ES/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/es-MX/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/et/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/eu/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/fa/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ff/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/fi/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/fr/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/fy-NL/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ga-IE/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/gd/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/gl/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/gu-IN/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/he/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/hi-IN/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/hr/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/hu/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/hy-AM/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/id/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/is/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/it/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ja/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ja-JP-mac/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ka/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/kk/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/km/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/kn/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ko/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ku/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/lg/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/lij/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/lt/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/lv/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/mai/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/mk/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ml/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/mn/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/mr/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ms/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/my/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/nb-NO/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ne-NP/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/nl/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/nn-NO/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/nr/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/nso/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/oc/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/or/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/pa-IN/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/pl/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/pt-BR/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/pt-PT/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/rm/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ro/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ru/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/rw/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/sah/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/si/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/sk/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/sl/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/son/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/sq/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/sr/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ss/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/st/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/sv-SE/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/sw/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ta/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ta-LK/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/te/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/th/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/tn/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/tr/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ts/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/uk/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ur/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/ve/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/vi/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/wo/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/wo-backup/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/x-testing/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/xh/: fail
http://hg.mozilla.org/releases/l10n/mozilla-beta/zh-CN/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/zh-TW/: pass
http://hg.mozilla.org/releases/l10n/mozilla-beta/zu/: pass
(Reporter)

Comment 13

6 years ago
Created attachment 741837 [details]
validate-pushlog.py, check json-info against json-pushes for 'default'

This is the script I use to compare what the web server knows about the 'default' revision. I compare json-info, which has the hex hash, if there's a changeset, with json-pushes, which has the changesets for the 'default' changeset, if it got that.

Note, I'm not detecting repos that are completely empty, I just realize. Those probably need some extra care, too.
(Reporter)

Updated

6 years ago
Attachment #741837 - Attachment mime type: text/x-python-script → text/plain
(Reporter)

Comment 15

6 years ago
Created attachment 742938 [details]
helper script, now calling out empty repos explicitly

I've updated the script to call out empty repos explicitly, because they might fail still. Reran it, and the following are empty:

http://hg.mozilla.org/releases/l10n/mozilla-beta/az/
http://hg.mozilla.org/releases/l10n/mozilla-beta/my/
http://hg.mozilla.org/releases/l10n/mozilla-beta/sah/
Attachment #741837 - Attachment is obsolete: true

Comment 16

6 years ago
Thanks fox2mike, Pike for looking into this. This bug blocks me from landing new locales into Aurora or taking locales into beta channel.

Comment 17

5 years ago
These repositories have had commits landed in them without having entries created in pushlog. IT hasn't solidified a way to synthesize a pushlog.

The easiest course of action (for the failing repositories) would be to:

1) Clone the repositories
2) Destroy the repositories on hg.mozilla.org
3) 'hg init' new repositories on hg.mozilla.org
4) 'hg push' the repositories back to create proper pushlog entries

What does everybody think of this plan?
Flags: needinfo?
(Reporter)

Comment 18

5 years ago
The problem is not that we have changesets without pushes, the problem is that new pushes don't get recorded.

We need to be sure that the pushlogs get created in the future, right now it's like throwing dices.

I really don't like the idea that we throw away and create new repositories every time something goes wrong. The root of the problems needs to be figured out.
Flags: needinfo?
(In reply to Axel Hecht [:Pike] from comment #18)
> The problem is not that we have changesets without pushes, the problem is
> that new pushes don't get recorded.
> We need to be sure that the pushlogs get created in the future, right now
> it's like throwing dices.

We had a problem with this around the time these repos (and a bunch of other repos for you guys) were created. That has since been fixed.

> I really don't like the idea that we throw away and create new repositories
> every time something goes wrong. The root of the problems needs to be
> figured out.

I'm with you, but no one else has had any issues with any of the other repos we've created. I think starting clean is the right way here, if we see the problem after starting clean, I'm willing to spend more time and resources. Right now, it's a wild goose chase because this issue is a one off (in the sense that the repos were created when we had a bug which has been fixed for a while, but the repos are buggy).
(Reporter)

Comment 20

5 years ago
tbh, that's not inspiring confidence for future repositories.

Anyway, let's move this forward. I veto getting yet more backup repositories in our production hg repo sets, though. I.e., no more http://hg.mozilla.org/releases/l10n/mozilla-beta/wo-backup/

Comment 21

5 years ago
Alright, let us try this and see if it fixes this long standing problem.

Comment 22

5 years ago
@bkero Did you get a chance to look at this issue?

Comment 23

5 years ago
I've run your script (with s/ri.json()/ri.json to fix non-callable-dict errors) and found the broken repositories. With that list I

1) cloned them
2) deleted the upstream repository
3) created the upstream repository
4) pushed

After running your script again, none of the repositories fail your script anymore. Can you ensure they're all entered correctly?
(Reporter)

Comment 24

5 years ago
Merge day:

pushing to ssh://hg.mozilla.org/releases/l10n/mozilla-beta/my/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 20 changesets with 833 changes to 345 files
remote: error: changegroup.z_loghistory hook raised an exception: [Errno 1] Operation not permitted: '/repo/hg/mozilla/releases/l10n/mozilla-beta/my/.hg/pushlog2.db'
Kendall,

please take a stab at this and see what's going on?
Assignee: bkero → klibby
(Assignee)

Comment 26

5 years ago
Not a permissions issue, this time:

[root@hgssh1.dmz.scl3 .hg]# ls -ald /repo/hg/mozilla/releases/l10n/mozilla-beta/my/.hg{,/pushlog2.db}
drwxrwsr-x 4 hg scm_l10n 4096 Jun 24 13:22 /repo/hg/mozilla/releases/l10n/mozilla-beta/my/.hg
-rw-rw-r-- 1 hg scm_l10n 9216 Jun 24 13:19 /repo/hg/mozilla/releases/l10n/mozilla-beta/my/.hg/pushlog2.db

Into the depths of pushlog I go...
(Assignee)

Comment 27

5 years ago
Axel, can you try the push again? 

The file permissions are correct all the way down, and I'm unable to duplicate the error using a tar'ed copy of the repo. On top of that, what's happening in pushlog is pretty limited:

def schemaexists(conn):
    return 1 == conn.execute("SELECT COUNT(*) FROM SQLITE_MASTER WHERE name='pushlog'").fetchone()[0]

def log(ui, repo, node, **kwargs):
    pushdb = os.path.join(repo.path, 'pushlog2.db')
    createdb = False
    if not os.path.exists(pushdb):
        createdb = True
    conn = sqlite.connect(pushdb)
    if not createdb and not schemaexists(conn):
        createdb = True
    if createdb:
        createpushdb(conn)
        st = os.stat(pushdb)
        os.chmod(pushdb, st.st_mode | stat.S_IWGRP)
    t = int(time.time())
    retval = 1
    print "Trying to insert into pushlog."
    print "Please do not interrupt..."

Either the OS is failing to report that the file exists and it's trying to create a new pushlog, or sqlite.connect is failing. And again, I'm unable to duplicate that; I can see the file, and 'sqlite pushlog2.db' works and I can see the schema/tables/db, though there are no table contents. 

If you push and it fails, I'll try and write something to simulate a push and see if I can get it to break, but I want to make sure that it's NOT a transient (helloooo NFS) error.
(Reporter)

Comment 28

5 years ago
Seems to work now. Odd, and I wish the push would actually fail if it fails.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago5 years ago
Resolution: --- → FIXED
(Assignee)

Comment 29

5 years ago
Yeah, I was afraid that it would work again, because that means it's related to the fabulous and long standing pushlog/nfs issues. 

As a recap, for clarity's sake...

The error "changegroup.z_loghistory hook raised an exception: attempt to write a readonly database" is due to file permission issues on pushlog2.db, which we can fix easily enough.

The error "changegroup.z_loghistory hook raised an exception: [Errno 1] Operation not permitted: <path>" appears to be a transient error, likely due to hg/pushlog on NFS. Try pushing again after a short wait; if it fails consistently please do let us know so we can investigate further!
Component: Server Operations: Developer Services → General
Product: mozilla.org → Developer Services
You need to log in before you can comment on or make changes to this bug.