Before updating, please indicate the current hg changeset of the live MXR. It should be: 13b77a7fa477 mxr.mozilla.org update warning: The original "fix" for bug 571341 was broken and it was hacked around by someone on mxr.mozilla.org. If you do nothing before updating to tip, you will get a useless conflict when you do the pull on mxr.mozilla.org. You won't on mxr-stage as no one hacked around it. Before updating mxr.mozilla.org, you could revert that file using: hg revert -r . services-central to avoid the conflict. thanks. I think I'd like to have a week or something of the changes running on mxr-stage before we pull this set to production. Please indicate which hg changeset you get when you update mxr. Bug 608196 has introduced a feature which could make the addons repository update much less painful. It's an experimental feature and requires a certain configuration to work, but it could be very beneficial. to work, you need to use: ./update-xref.pl -cron --by-unit $TREE addons old sources, old index addons-stage new sources The code will automatically find the addons tree by chopping off '-stage' from 'addons-strage', and then compare the files top-level-directory at a time in addons to addons-stage, if the directories (these correspond to individual addons) match, then instead of indexing a directory, it will copy the existing index. For cases where the directories do not match, the directory will be indexed. After all top-level-directories are indexed, a global merge is performed. A nice advantage of this is that the only part of the process that requires massive amounts of memory is this last stage, so the general runtime behavior should be nicer. This of course should also be tested (and monitored) on stage before it's deployed. It probably makes sense to do this after we've established that the general update is happy. I don't know of any problems with any of the code, and I'm not particularly worried, but... Of note: I will be in @w3 TPAC next week. I should be able to respond to things as usual if necessary.
Stage, before : [root@dm-webtools04 mxr-stage]# hg parents changeset: 349:13b77a7fa477 tag: tip user: email@example.com date: Mon Oct 18 21:13:13 2010 +0200 summary: Fix bug 596131 Stage, after : [root@dm-webtools04 mxr-stage]# hg pull -u pulling from http://hg.mozilla.org/webtools/mxr/ searching for changes adding changesets adding manifests adding file changes added 9 changesets with 12 changes to 9 files merging genxref 8 files updated, 1 files merged, 0 files removed, 0 files unresolved [root@dm-webtools04 mxr-stage]# hg parents changeset: 358:dd91a798a1e6 tag: tip user: firstname.lastname@example.org date: Fri Oct 29 08:14:46 2010 +0200 summary: Refactoring HGCOMMAND HGUPDATE as sub hg_update I didn't understand the ./update-xref.pl bits, can you explain it to me again? Should I run that by hand or by cron?
Reopen if this is still needed.
sorry, quirk of my system, i don't see comments on bugs i file until i touch them again :( -- i need to change my process to touch them once. basically, in theory you could improve the performance of the addons xref update if you changed how it worked. try this on mxr-stage (once): ./update-xref.pl --by-unit addons-stage then swap addons-stage to active (addons) then, let a update-src.pl addons-stage run, then: ./update-xref.pl --by-unit addons-stage in theory you should get a much faster update-xref pass for this second run (and any subsequent), at which point it'd be good to switch the cron job for mxr-stage for addons-stage to include --by-unit, Before --by-unit could be introduced to mxr (instead of stage), people would have to compare the mxr-stage ident results for addons with those from mxr.
(In reply to comment #4) > sorry, quirk of my system, i don't see comments on bugs i file until i touch > them again :( -- i need to change my process to touch them once. > > basically, in theory you could improve the performance of the addons xref > update if you changed how it worked. > > try this on mxr-stage (once): > > ./update-xref.pl --by-unit addons-stage > [root@dm-webtools04 mxr-stage]# ./update-xref.pl --by-unit addons-stage [Mon Jan 10 08:01:04 2011] fatal: Could not find matching sourceroot: for addons-stage at ./update-xref.pl line 119, <CONFIG> line 180.
oops, it's probably: ./update-xref.pl --by-unit addons
Is this superceded by bug 632643?
this has a tangent which i wanted tested as part of the landing for this set of changes. basically comment 1 "fixed" the primary part of this bug. i'd just like someone to try the command from comment 6 on mxr-stage to see how it goes. but it should probably be done after mxr-stage is updated to tip.
from bug 632643 comment 0: Before updating, please indicate the current hg changeset of the live MXR. It should be: dd91a798a1e6 bug 620256 should finally be fixed, although there's still a bandaid which we should eventually remove. comment 1: actually, i'm wrong about the expected version. it's probably abf6a685f1a4 because we were trying to make the bug go away.
phong: please note that resolving bugs as duplicates does not magically carry over dependencies, and those dependencies are important for people trying to understand what's changing and when.
We have no way to test this on "stage" without affecting production because they both use the same data files (not enough storage available to have separate copies). I just updated both to the tip, there's an existing pass of addons in progress right now, I'll give this a test as soon as the one that's running now completes.
justdave: oh, what you could do is: ./add-root.pl addons-test /data/mxr-data/amo/amo addons ./update-xref.pl --by-unit addons-test this way you only pay for the databases (not the undderlying file data) which will be roughly 2x the current database size (one for the file database and one for the set of intermediate databases). that should be a tolerable cost. of course to do that you'd need to have whatever htaccess magic there is which redirects addons to https.
(In reply to comment #6) > ./update-xref.pl --by-unit addons OK, this has been run. It did something. It took several hours to run, and printed what appeared to be several thousand instances of the "time" command, most of which were only a fraction of a second (but a few of them here and there took 15 minutes or 40 minutes).
As stated in previous comments, I ran timeless' experiements and posted the results here. Without guidance of what those results mean or how that affects the deployment there's not really much more than can be done here. Reopen when someone can tell us what needs to be done next.