688847 - WebRTC repo management and upstream/downstream merge design

Status: RESOLVED INCOMPLETE

(Core :: Audio/Video, defect)

Description

[Randell Jesup [:jesup] (needinfo me)]

+++ This bug was initially created as a clone of Bug #688187 +++

This is to document the plan for WebRTC repo management and how we plan to handle patches for upstream and merging patches from upstream.

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

Comments copied from bug 688187:

Here's a pretty good picture of how to import updates to an external vendor lib in mercurial.  Looks fairly straightforward.  It won't directly track renames, etc, but that's probably ok (that's hard to do across different SVC systems).

http://www.selenic.com/pipermail/mercurial/2009-April/025304.html


Randell Jesup [:jesup] 2011-09-23 12:44:25 PDT

Ok, summarizing from #media:  (And remembering O'Duin's exhortation to document all decisions in writing).  This is the outcome of myself, rillian and dmose chatting.

Tentative plan -

1) Start with webrtc in the tree on our branch repo, but built as a standalone lib for now.

2) To enable easy tracking of upstream, use a structure like this:

a) Changes are checked into our trunk.  If they're not moz-specific (make stuff, etc) we add [upstream-merge-needed] to whiteboard.
b) upstream-merge-needed patches are merged to a separate head that has a clean copy of the library with the changes we want to go back to Google; whiteboard is changed to [upstream-pending]
c) downstream pulls are done via a script; it puts them on a "vendor-branch" head which is then merged to the upstream-pending head.  Conflicts are dealt with manually.  [upstream-pending] patches now in the upstream should be marked [upstreamed].  Then the upstream-pending head is merged to trunk.

Comments?

It should be easy to get lists of pending upstream patches, etc.  Biggest place it can fall down is patches in trunk not getting [upstream-merge-needed]; we could make a tool to catch those.


Comment 9 Randell Jesup [:jesup] 2011-09-23 12:59:13 PDT

Oh, yeah, we'll work out with Google if they want batches of patches for upstreaming or done one at a time as generated.  For us, some level of batching for most patches makes sense.  We might want to add a 4th upstream state: [upstream-waiting] after it's submitted and not yet in a downstream

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

Ralph Giles (:rillian) 2011-09-23 13:07:57 PDT

Certainly we should update the bug here with links to google's rietveld instance when the upstream status for a patch changes. I'd suggest, therefore, that our webrtc patch bugs not be closed until they're accepted or rejected upstream.

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

So, mark as fixed when we check into trunk; but don't have QA/etc close them until upstream accepts them?

Comment 4

[Ralph Giles (:rillian)]

I think we shouldn't even mark them fixed until upstream has also reviewed on their side. But I don't know who QA/etc would be for our branch.

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

Hmmm.  Not sure how leaving them 'open' would work with our process here in bugzilla.  I can check with a few people.  We generally close bugs when fixed in trunk; for example a bug found in Aurora is closed when the fix is in trunk; we use flags to mark the handling for aurora.  (We *could* use flags here, but we wouldn't really want them global, so I think whiteboard tags makes more sense - effectively on-demand private flags.)  I think Matt is our QA contact; chatted briefly at All-Hands.

Per discussion in #media, we'll want to avoid perturbing their internal directory structure if we can - it makes merges in either direction a pain.  For pieces we want to "break out" (AEC, ice, etc) we'll make the build system create a stand-along .a or .so to pull in elsewhere, I think.

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

I'm preparing the initial import onto alder (our project repo), setting up a 'vendor branch' per the discussion above, and building a tool for merging updates.  Before pushing the import, I'm going to test the tool and sequence on a dummy repo using the real webrtc files.

I plan to pull it into media/webrtc, and because it has a bunch of dependencies on living in a directory named "trunk" I'm planning on have all the code live in media/webrtc/trunk.  Also, we'll have other webrtc code which can live in media/webrtc and subdirs of it.

I plan to name the vendor tag "webrtc-import-last", the pending upstream tag "webrtc-pending".  I also plan to push a tag at the import point: "webrtc-import" to make rollbacks and diffs easy.  I plan to exclude all .svn/* files from import.

Here's the .hg/hgrc you'll need:
[paths]
default = http://hg.mozilla.org/projects/alder/
default-push = ssh://hg.mozilla.org/projects/alder/

(Adding the rest of the team for heads-up)

Comment 7

[Timothy B. Terriberry (:derf)]

(In reply to Randell Jesup [:jesup] from comment #6)
> I plan to pull it into media/webrtc, and because it has a bunch of
> dependencies on living in a directory named "trunk" I'm planning on have all

What depends on this?

> the code live in media/webrtc/trunk.  Also, we'll have other webrtc code
> which can live in media/webrtc and subdirs of it.

Traditionally, our 1st party code has lived in content/media.

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

You can't even pull the code with sync if you use the --name option to change the directory name.

Comment 9

[Randell Jesup [:jesup] (needinfo me)]

Ok, after talking with sfink and derf on #developers last night, and looking more at pages about vendor branches in Mercurial, I'm leaning towards either a true branch (though it clutters up the commits a little, it avoids mistakenly leaving the tip on one of my sidelines), or bookmarks, which are like branches but don't clutter metadata.  If we use bookmarks, when we commit to the bookmark head I think we'll have to do a commit to the development head to make it the tip again, which is a pain and prone to race conditions and errors.

I trialed bookmarks on a local dummy repo, and the sequence for merging updates from upstream works - the one caveat is that you must force it to create a new head for the bookmark before trying to merge to it. (I.e. commit a do-nothing commit to upstream-pending before merging to it).

Because of the problems about tip handling (unless 2 branches for handling vendor tracking is ok), it might make sense to handle the vendor branch (webrtc-import-last) and upstream-pending on a separate repo.  The branches might make more sense if they could be shared by *all* the external libraries we import (or at least new ones) - and I suspect they can be shared, but it needs thought.  This will need a little more thought and reading.  I hope I can have it ready by Monday, or ready for approval of the scheme from some the other stakeholders in tree management.

Comment 10

[Randell Jesup [:jesup] (needinfo me)]

Attachment: Sample WIP script using bookmarks

Comment 11

[Randell Jesup [:jesup] (needinfo me)]

Ok - things aren't so simple:

Talked with ted and kyle in #developers.  hg sub-repos may make sense (to a private mirror or local hg repo that implements the scheme above).  Note that per hg docs they recommend not having sub-repos under real ones, since the sub-repo can block attempts to push to the main repo.

Also, it's even bigger than I remember (maybe they added more tests): 

480K	./media/webrtc/trunk/third_party_mods
71036K	./media/webrtc/trunk/test
24964K	./media/webrtc/trunk/src
4580K	./media/webrtc/trunk/tools
408K	./media/webrtc/trunk/build
200K	./media/webrtc/trunk/peerconnection
4K	./media/webrtc/trunk/.svn
4828K	./media/webrtc/trunk/testing
43224K	./media/webrtc/trunk/third_party
149864K	./media/webrtc/trunk

and that's NOT built.  >1/2 of it is the test and testing dirs.  We only have about 550MB of code in mozilla-central today...

So, a little more thought and prep is going to be needed, even if we go with this overall idea.

Factors to consider:
a) ease of taking updates from upstream
b) ease of collecting and submitting updates to upstream
c) affect on mozilla tree and build management
d) risks of fubars
e) affect on development times for developers: tree pulls, build times, etc

Or we deal with it as an pure external lib with no mods.  This has a bunch of downsides from our point-of-view; I think we want it in our tree.  But I'm open to argument.

Comment 12

[Timothy B. Terriberry (:derf)]

(In reply to Randell Jesup [:jesup] from comment #11)
> So, a little more thought and prep is going to be needed, even if we go with
> this overall idea.

I think we should just try something, and work from there. We're on a branch, so we can't really screw anyone else up yet. If it's untenable, we'll notice pretty quickly, and we can figure out what to do based on actual experience.

Comment 13

[Ralph Giles (:rillian)]

+1 for just trying something.

When we land feature branches, do we import the whole development history, or just merge a patches with the working code? If the later we don't have to worry about the code size.

Comment 14

[Randell Jesup [:jesup] (needinfo me)]

Could be either (import history or just code).

After talking with people on #mercurial, I think I'm going to simplify it for now by not making an upstream-pending bookmark.  We can add it later if we want.  I'll have a bookmark that we can use to "trim" down the release for import.  I'll probably import from the 'trim' bookmark, but I may not bother to trim yet.  Later on we can hg remove any files we don't want in our m-c build repo.

What I may do when we have patches to upstream is to hold them in a (revision-controlled, shared) queue (mq) in the merge repo.  "hg init --mq"  -- though for now, the merge repo can probably just be local, as there's minimal important info in it.  So I'll start with it local.  Or we could just leave them in the m-c (alder for now) repo, and use whiteboard to manage.

So this simplifies things.

Comment 15

[Randell Jesup [:jesup] (needinfo me)]

Attachment: Updated update script

Updated update script - most of the "modify the repos" and "push to alder"
commands are currently echoed for the user to run, until I'm comfortable
its working well.

Comment 16

[Randell Jesup [:jesup] (needinfo me)]

My webrtc_import repo is now hosted at hg.mozilla.org/users/rjesup_wgate.com/webrtc_import
(The address should change when my checkin credentials are updated to my current email address)

Sequence to build an import repo:

hg init webrtc_import
mkdir media/webrtc
cd media/webrtc
mkdir webrtc_update
cd webrtc_update
gclient config http://webrtc.googlecode.com/svn/trunk
gclient sync --force
cd trunk
hg tag -l old-tip
hg bookmark webrtc-import-last
rm -rf trunk
mv webrtc_update/trunk trunk
mv webrtc_update/.g* .
rmdir webrtc_update
hg addremove --exclude "**.svn" --similarity 75 trunk
hg add media/webrtc/.g*
hg commit -m 'Webrtc import 2011-09-27 09:00 EDT'
hg bookmark webrtc-pending
hg bookmark webrtc-trim
echo > media/webrtc/dummy foo
hg add media/webrtc/dummy
hg commit -m "Add dummy file to force bookmark to make a head"
hg remove media/webrtc/dummy
hg commit -m "Remove dummy file"

Comment 18

[Randell Jesup [:jesup] (needinfo me)]

Import process here didn't work in the end - too much explosion of history in hg.  We use a manual process with reapplying all patches (ad a single mass) on top of a pull of new code from upstream