Closed Bug 787208 (c-c-m-c-merge) Opened 13 years ago Closed 8 years ago

Move comm-central into mozilla-central

Categories

(MailNews Core :: Build Config, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: ehsan.akhgari, Unassigned)

References

()

Details

Attachments

(4 files, 1 obsolete file)

I don't think there's a good reason why these are two separate repositories. Let's see if we can merge them back in.
Pretty epic. Cc'ing appropriate TB and SM folks.
The reason that I think we should do this is to make it easier for people to remember c-c *if* they choose to. Specifically I'm not proposing any changes to our policies against c-c, or what happens if an m-c patch breaks c-c, etc. This is merely about where the code lives. :-)
As someone who would *increadibly* love this, and knowing it should simplify a good chunk of c-c hacks/code everywhere its touched, I should note there is are at least *SOME* very good reasons for the separation at present. * A checkin to anything comm-central specific currently does not trigger Firefox/Mobile/B2G builds, as is appropriate. (And keeps meaningless build churn low) * Hg has real tangible costs as a repo gets larger in size, at present. (We can see these costs in our uplift times, try-push-times, etc.) * We would lose easy access to much of the blame/history of these changesets. The only way around that is if we merge the two "unrelated" repos manually, which will involve basically dropping on the floor the entire configure.in/build-system generic changes, etc. Not an easy merge to perform, and would be a VERY VERY large push/landing. That if we do, we should pre-seed at the least try, inbound, and m-c. So basically IFF we do this, I think we either need a mechanism to traverse multiple locations for blame/history easily --or-- that unrelated-repo merge. Also a VERY VERY good to have, which might alone block this anyway, is a sane means to identify what files a push touches, and only trigger jobs for those related builds. -- This one would be a releng-code change, and I know none of the Releng Team has time to even consider this until Q1-2013 at the --earliest-- --- ALL THAT SAID -- The above is mostly devils advocate, since suite/ and mail/ under m-c has been a desire of mine since we branched to hg. Though it was initially turned down by "the powers that be". The transition would make the build-system merge/update much easier, and allow TB and SeaMonkey to be maintained from the community in a much better cost-vs-reward. As well as allow SeaMonkey and Releng-TB buildbot code to be simplified quite a bit more due to less differences in these types of codepaths.
Product: Thunderbird → MailNews Core
How hard is it for releng to trigger builds on platform XYZ when only if changes are in/not in a specific directory? Additional difficulties is that we'd probably need to change some models on the try server about how to test Thunderbird related patches. Perhaps a trychooser apps switch? :-)
(In reply to Joshua Cranmer [:jcranmer] from comment #4) > How hard is it for releng to trigger builds on platform XYZ when only if > changes are in/not in a specific directory? The problem there is Releng infra only pays attention to the tip cset in a push, and thus we don't have data for the entire push's changed files. Its a non-trivial amount of work to change these assumptions. It is a good to have, just not something anyone is working on [and would help us even IF we don't move comm-central to mozilla-central]
(In reply to comment #3) > As someone who would *increadibly* love this, and knowing it should simplify a > good chunk of c-c hacks/code everywhere its touched, I should note there is are > at least *SOME* very good reasons for the separation at present. > > * A checkin to anything comm-central specific currently does not trigger > Firefox/Mobile/B2G builds, as is appropriate. (And keeps meaningless build > churn low) Our current m-c infrastructure can detect changes to the js/src directory and run custom changes on them. So I assume it's possible to detect changes to the comm/ directory and not do anything. > * Hg has real tangible costs as a repo gets larger in size, at present. (We can > see these costs in our uplift times, try-push-times, etc.) How large is the history depth of c-c? > * We would lose easy access to much of the blame/history of these changesets. > The only way around that is if we merge the two "unrelated" repos manually, > which will involve basically dropping on the floor the entire > configure.in/build-system generic changes, etc. Not an easy merge to perform, > and would be a VERY VERY large push/landing. That if we do, we should pre-seed > at the least try, inbound, and m-c. No history needs to be lost. We can import the full history. We did that when we imported mobile-browser into m-c/mobile/, for example.
(In reply to comment #4) > How hard is it for releng to trigger builds on platform XYZ when only if > changes are in/not in a specific directory? They do it in the case of js/src, so it's _possible_. How hard, I have no idea. > Additional difficulties is that we'd probably need to change some models on the > try server about how to test Thunderbird related patches. Perhaps a trychooser > apps switch? :-) Oh, one implicit benefit of this change would be that you can test m-c and c-c changes on the try server!
(In reply to comment #5) > (In reply to Joshua Cranmer [:jcranmer] from comment #4) > > How hard is it for releng to trigger builds on platform XYZ when only if > > changes are in/not in a specific directory? > > The problem there is Releng infra only pays attention to the tip cset in a > push, and thus we don't have data for the entire push's changed files. Its a > non-trivial amount of work to change these assumptions. > > It is a good to have, just not something anyone is working on [and would help > us even IF we don't move comm-central to mozilla-central] That would be a problem when we merge across branches, at the very least. :(
(In reply to Ehsan Akhgari [:ehsan] from comment #6) > (In reply to comment #3) > > As someone who would *increadibly* love this, and knowing it should simplify a > > good chunk of c-c hacks/code everywhere its touched, I should note there is are > > at least *SOME* very good reasons for the separation at present. > > > > * A checkin to anything comm-central specific currently does not trigger > > Firefox/Mobile/B2G builds, as is appropriate. (And keeps meaningless build > > churn low) > > Our current m-c infrastructure can detect changes to the js/src directory > and run custom changes on them. So I assume it's possible to detect changes > to the comm/ directory and not do anything. well, presumably if we import to m-c we'd flatten as well, so browser/ is next to suite/ is next to mailnews/ is next to mail/ etc. (but thats a tangential conversation from what we are saying here). As to your point about js/src it is broken technically, but since most people who touch js/src don't push patches to say, browser/* in the same push, no-one has really noticed the problem. Try it yourself, push a js/src change next time, with a patch that doesn't touch those as your tip cset, you won't get those extra jobs. > > * Hg has real tangible costs as a repo gets larger in size, at present. (We can > > see these costs in our uplift times, try-push-times, etc.) > > How large is the history depth of c-c? much more than mobile/ was but much less than m-c is. I'm less concerned with the single-hit-now, as I am with the future-hits later in this point. > > * We would lose easy access to much of the blame/history of these changesets. > > The only way around that is if we merge the two "unrelated" repos manually, > > which will involve basically dropping on the floor the entire > > configure.in/build-system generic changes, etc. Not an easy merge to perform, > > and would be a VERY VERY large push/landing. That if we do, we should pre-seed > > at the least try, inbound, and m-c. > > No history needs to be lost. We can import the full history. We did that > when we imported mobile-browser into m-c/mobile/, for example. Sure, but when importing comm-* we would want to flaten (otherwise we have TONS of code changes to make to support the flip in structure) and that would require a lot of manual hand-tweaking of the build system on the merge, since c-c has tried to mimic relevant stuff, and care would NEED to be taken to avoid accidentally breaking one or the other repos. (In reply to Ehsan Akhgari [:ehsan] from comment #7) > (In reply to comment #4) > > How hard is it for releng to trigger builds on platform XYZ when only if > > changes are in/not in a specific directory? > > They do it in the case of js/src, so it's _possible_. How hard, I have no > idea. In software anything is possible. However tradeoffs for what we did for the few js/src tests are much different than we'd be facing here. If I for example, am dealing with checkin-needed for suite and gecko in one go, import a browser patch, then a TB patch, only TB would build, not Firefox if we reused the code-paths we have now in place to manage the js/src stuff. That alone isn't too trivial, but to make it extended to support all changesets in the push is a lot of work, since we hard-code assumptions lots of places that we don't. And we would need to write mitigation code as well, since we initially made it *only* pay attention to one cset due to Buildbot hanging/OOM-ing/whatever when it encountered a large[ish] push/merge/etc. > > Additional difficulties is that we'd probably need to change some models on the > > try server about how to test Thunderbird related patches. Perhaps a trychooser > > apps switch? :-) > > Oh, one implicit benefit of this change would be that you can test m-c and > c-c changes on the try server! You can test m-c and c-c changes on the [comm-]try server today, in one try push! Its just not as easy as it would be if we merged both of these things.
I'd certainly be for this, although it'd need a more public discussion before we would do it. However, I think we should work on the bigger items first, before raising that discussion, as it will fix things in other areas that we should fix anyway, and I think we may still be a couple of months away from being ready to have the discussion. So I'm adding some of the items like reworking the build system (bug 648979) and push-dependent builds (bug 752836) to the deps for tracking purposes.
Depends on: ccrework, 752836
(In reply to Mark Banner (:standard8) from comment #10) > I'd certainly be for this, although it'd need a more public discussion > before we would do it. Looks like the discussion is happening: https://groups.google.com/d/msg/mozilla.dev.platform/pNwnQGgaIFE/37twR8bPBgAJ
Alias: c-c-m-c-merge
Depends on: 1222301
This project seems to be WONTFIX, or going there. AFAICT, the official MoCo position is that "Thunderbird is levering too much tax on Firefox building resources" while the official Thunderbird position is that "it seems that in the medium term it will become necessary to build Thunderbird based on a frozen Gecko clone, no more dependent on the Firefox backends' 'moving target'". The above is my subjective perception and it might be unintentionally caricatural. Which policy the SeaMonkey Council will adopt is not yet clear to me.
(In reply to Tony Mechelynck [:tonymec] from comment #12) > This project seems to be WONTFIX, or going there. > > AFAICT, the official MoCo position is that "Thunderbird is levering too much > tax on Firefox building resources" while the official Thunderbird position > is that "it seems that in the medium term it will become necessary to build > Thunderbird based on a frozen Gecko clone, no more dependent on the Firefox > backends' 'moving target'". > > The above is my subjective perception and it might be unintentionally > caricatural. Which policy the SeaMonkey Council will adopt is not yet clear > to me. IIUC, the project has not really hit an unpassable wall, it has just been reoriented: see https://groups.google.com/d/topic/tb-planning/YTsw5LRfMZc/discussion (where this bug is mentioned) for details. Instead of merging into mozilla-central (which has indeed become "verboten") an experimental merged repository has been set up at http://hg.mozilla.org/projects/cypress/ to test RelEng automation for possible necessary changes. Then (some day, no ETA yet AFAIK, but possibly as soon as Q1 2016) the merged repo will be pushed to comm-central. Joshua, sorry if I broke your news before you were ready, but IMHO anyone following this bug would be interested by your new newspost. Please correct any errors in my attempt at summarizing of your efforts.
(In reply to Tony Mechelynck [:tonymec] from comment #13) > IIUC, the project has not really hit an unpassable wall, it has just been > reoriented: It's the same work to merge mozilla-central into comm-central as it is to do it the other way around; they only difference is developer ergonomics in the opposite direction. > Joshua, sorry if I broke your news before you were ready, but IMHO anyone > following this bug would be interested by your new newspost. Please correct > any errors in my attempt at summarizing of your efforts. It's an apt summary.
OS: Mac OS X → All
Hardware: x86 → All
As promised, this is the "minimum" I think is needed to get us to build Thunderbird on cypress instead of Firefox. This unsets a Gecko Ver hardcode used by Firefox right now, and enables TB stuff. This also points cypress at using itself as the mozharness repo (incase we/you want to turn on any mozharness based stuff as we get closer to success here, for itereation) I did not test this patch in any way (builders added/removed, a full dump_master etc) but I think if it passes travis its a good first step and can/should land. r? to rail for official releng review, r? to fallen incase he notices any gotchya's in terms of config with a TB focus in mind, anf? to jcranmer incase he knows of anything that I missed to support cypress itself.
Attachment #8718317 - Flags: review?(rail)
Attachment #8718317 - Flags: review?(philipp)
Attachment #8718317 - Flags: feedback?(Pidgeot18)
Attachment #8718317 - Flags: review?(rail) → review+
Comment on attachment 8718317 [details] [diff] [review] [configs] tb-on-cypress Review of attachment 8718317 [details] [diff] [review]: ----------------------------------------------------------------- releng is not my specialty here, so I'm mostly going by seeing if I see everything from bug 929288, which appears to be in the affirmative.
Attachment #8718317 - Flags: feedback?(Pidgeot18) → feedback+
Comment on attachment 8718317 [details] [diff] [review] [configs] tb-on-cypress Review of attachment 8718317 [details] [diff] [review]: ----------------------------------------------------------------- looks pretty straightforward, r=philipp
Attachment #8718317 - Flags: review?(philipp) → review+
(In reply to Release Engineering SlaveAPI Service from comment #19) > In production: https://hg.mozilla.org/build/buildbot-configs/rev/8c7677a7cdb6 This was removing one more buildbot use of mozilla directory being specified, but then on a rebuild we still had graph server issues. I applied the following to production graph server (need to do it to staging) mysql> select * from machines where name LIKE "TB%comm-central%";+------+-------+---------------+-----------+--------------------------------------------+-----------+------------+ | id | os_id | is_throttling | cpu_speed | name | is_active | date_added | +------+-------+---------------+-----------+--------------------------------------------+-----------+------------+ | 2402 | 6 | 0 | NULL | TB_Linux_comm-central | 1 | 1335291895 | | 2403 | 6 | 0 | NULL | TB_Linux_comm-central_leak_test | 1 | 1335291895 | | 2408 | 6 | 0 | NULL | TB_Linux_try-comm-central | 1 | 1335291895 | | 2409 | 6 | 0 | NULL | TB_Linux_try-comm-central_leak_test | 1 | 1335291895 | | 2414 | 18 | 0 | NULL | TB_Linux_x86-64_comm-central | 1 | 1335291895 | | 2415 | 18 | 0 | NULL | TB_Linux_x86-64_comm-central_leak_test | 1 | 1335291895 | | 2420 | 18 | 0 | NULL | TB_Linux_x86-64_try-comm-central | 1 | 1335291895 | | 2423 | 7 | 0 | NULL | TB_OS_X_10.5.2_comm-central_leak_test | 1 | 1335291895 | | 2430 | 17 | 0 | NULL | TB_OS_X_10.6.2_comm-central | 1 | 1335291895 | | 2431 | 17 | 0 | NULL | TB_OS_X_10.6.2_comm-central_leak_test | 1 | 1335291895 | | 2446 | 7 | 0 | NULL | TB_OS_X_10.7_32-bit_comm-central_leak_test | 1 | 1336079016 | | 2454 | 17 | 0 | NULL | TB_OS_X_10.7_64-bit_comm-central_leak_test | 1 | 1336079016 | | 2453 | 17 | 0 | NULL | TB_OS_X_10.7_comm-central | 1 | 1336079016 | | 8947 | 8 | 0 | NULL | TB_WINNT_5.2_comm-central | 1 | 1378830465 | | 2438 | 8 | 0 | NULL | TB_WINNT_5.2_comm-central_leak_test | 1 | 1335291895 | | 8951 | 19 | 0 | NULL | TB_WINNT_6.1_comm-central | 1 | 1378830465 | | 8953 | 19 | 0 | NULL | TB_WINNT_6.1_comm-central_leak_test | 1 | 1378830465 | +------+-------+---------------+-----------+--------------------------------------------+-----------+------------+ 17 rows in set (0.00 sec) mysql> SELECT id, os_id, is_throttling, cpu_speed, REPLACE(name, 'comm-central', 'cypress'), is_active, UNIX_TIMESTAMP() FROM machines WHERE name LIKE "TB%comm-central%"; +------+-------+---------------+-----------+------------------------------------------+-----------+------------------+ | id | os_id | is_throttling | cpu_speed | REPLACE(name, 'comm-central', 'cypress') | is_active | UNIX_TIMESTAMP() | +------+-------+---------------+-----------+------------------------------------------+-----------+------------------+ | 2402 | 6 | 0 | NULL | TB_Linux_cypress | 1 | 1455509970 | | 2403 | 6 | 0 | NULL | TB_Linux_cypress_leak_test | 1 | 1455509970 | | 2408 | 6 | 0 | NULL | TB_Linux_try-cypress | 1 | 1455509970 | | 2409 | 6 | 0 | NULL | TB_Linux_try-cypress_leak_test | 1 | 1455509970 | | 2414 | 18 | 0 | NULL | TB_Linux_x86-64_cypress | 1 | 1455509970 | | 2415 | 18 | 0 | NULL | TB_Linux_x86-64_cypress_leak_test | 1 | 1455509970 | | 2420 | 18 | 0 | NULL | TB_Linux_x86-64_try-cypress | 1 | 1455509970 | | 2423 | 7 | 0 | NULL | TB_OS_X_10.5.2_cypress_leak_test | 1 | 1455509970 | | 2430 | 17 | 0 | NULL | TB_OS_X_10.6.2_cypress | 1 | 1455509970 | | 2431 | 17 | 0 | NULL | TB_OS_X_10.6.2_cypress_leak_test | 1 | 1455509970 | | 2446 | 7 | 0 | NULL | TB_OS_X_10.7_32-bit_cypress_leak_test | 1 | 1455509970 | | 2454 | 17 | 0 | NULL | TB_OS_X_10.7_64-bit_cypress_leak_test | 1 | 1455509970 | | 2453 | 17 | 0 | NULL | TB_OS_X_10.7_cypress | 1 | 1455509970 | | 8947 | 8 | 0 | NULL | TB_WINNT_5.2_cypress | 1 | 1455509970 | | 2438 | 8 | 0 | NULL | TB_WINNT_5.2_cypress_leak_test | 1 | 1455509970 | | 8951 | 19 | 0 | NULL | TB_WINNT_6.1_cypress | 1 | 1455509970 | | 8953 | 19 | 0 | NULL | TB_WINNT_6.1_cypress_leak_test | 1 | 1455509970 | +------+-------+---------------+-----------+------------------------------------------+-----------+------------------+ 17 rows in set (0.00 sec) mysql> INSERT INTO machines (id, os_id, is_throttling, cpu_speed, name, is_active, date_added) SELECT id, os_id, is_throttling, cpu_speed, REPLACE(name, 'comm-central', 'cypress'), is_active, NOW() FROM machines WHERE name LIKE "TB%comm-central%"; ERROR 1062 (23000): Duplicate entry '2402' for key 'PRIMARY' mysql> INSERT INTO machines (os_id, is_throttling, cpu_speed, name, is_active, date_added) SELECT os_id, is_throttling, cpu_speed, REPLACE(name, 'comm-central', 'cypress'), is_active, NOW() FROM machines WHERE name LIKE "TB%comm-central%"; Query OK, 17 rows affected, 17 warnings (0.01 sec) Records: 17 Duplicates: 0 Warnings: 17
This may need in-tree fixes for cypress mozharness configs to support this setup, but I'll leave that to jcranmer and related TB teams to figure out. This should make buildbot schedule them all correctly though.
Attachment #8722561 - Flags: review?(rail)
Attachment #8722561 - Flags: review?(rail) → review+
Depends on: 1258866
Depends on: 1259143
Attached patch cypress.tests (obsolete) — Splinter Review
This should fix cypress tests pulling from archiver.
Attachment #8745040 - Flags: review?(jlund)
Comment on attachment 8745040 [details] [diff] [review] cypress.tests Review of attachment 8745040 [details] [diff] [review]: ----------------------------------------------------------------- assuming this is based off default branch, afaict you will have to import loadDefaultValues
Attachment #8745040 - Flags: review?(jlund) → review-
err yea, import it too
Attachment #8745040 - Attachment is obsolete: true
Attachment #8745049 - Flags: review?(jlund)
Comment on attachment 8745049 [details] [diff] [review] [configs] cypress tests take 2 Review of attachment 8745049 [details] [diff] [review]: ----------------------------------------------------------------- one more thing in line before shipping it ::: mozilla-tests/thunderbird_config.py @@ +472,5 @@ > branch['mozharness_archiver_repo_path'] = '%(moz_repo_path)s' > branch['mozharness_archiver_rev'] = 'default' > > +#Setup Cypress based on default values first > +loadDefaultValues(BRANCHES, cypress, BRANCHES[cypress]) s/BRANCHES[cypress]/BRANCHES["cypress"]/
Attachment #8745049 - Flags: review?(jlund)
Today I wanted to create a new buildbot master on dev-master2 and I'm receiving the following error when I'm running "make checkconfig" $ make checkconfig cd master && /builds/buildbot/vlad.ciobancai/test-linux3/bin/buildbot checkconfig Traceback (most recent call last): File "/builds/buildbot/vlad.ciobancai/test-linux3/lib/python2.7/site-packages/buildbot-0.8.2_hg_114a79935b99_production_0.8-py2.7.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig ConfigLoader(configFileName=configFileName) File "/builds/buildbot/vlad.ciobancai/test-linux3/lib/python2.7/site-packages/buildbot-0.8.2_hg_114a79935b99_production_0.8-py2.7.egg/buildbot/scripts/checkconfig.py", line 31, in __init__ self.loadConfig(configFile, check_synchronously_only=True) File "/builds/buildbot/vlad.ciobancai/test-linux3/lib/python2.7/site-packages/buildbot-0.8.2_hg_114a79935b99_production_0.8-py2.7.egg/buildbot/master.py", line 652, in loadConfig exec f in localDict File "/builds/buildbot/vlad.ciobancai/test-linux3/master/master.cfg", line 11, in <module> import thunderbird_config File "/tmp/tmpMBHXXz/thunderbird_config.py", line 476, in <module> NameError: name 'cypress' is not defined make: *** [checkconfig] Error 1 Can you please have a look over this issue ?
Flags: needinfo?(bugspam.Callek)
Comment on attachment 8745049 [details] [diff] [review] [configs] cypress tests take 2 In loadDefaultValues, should the second parameter be 'cypress' and not cypress?
(In reply to Edmund Wong (:ewong) from comment #29) > Comment on attachment 8745049 [details] [diff] [review] > [configs] cypress tests take 2 > > In loadDefaultValues, should the second parameter be 'cypress' and not > cypress? I checked the thunderbird_config from default branch and it is with quota, more details http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla-tests/thunderbird_config.py#l476
Attached is the patch to fix the issue . I tested on dev-master2
Attachment #8745321 - Flags: review?(bugspam.Callek)
Comment on attachment 8745321 [details] [diff] [review] bug787208_thunderbird_config.py.patch Review of attachment 8745321 [details] [diff] [review]: ----------------------------------------------------------------- Whops, I already landed this exact fix before I saw the review request -- thanks for the assist though. https://hg.mozilla.org/build/buildbot-configs/rev/af5f7251deeb
Attachment #8745321 - Flags: review?(bugspam.Callek) → review+
Flags: needinfo?(bugspam.Callek)
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: