Closed Bug 1050477 Opened 5 years ago Closed 5 years ago

Move mcMerge to it's own repo/hosting solution (part 1)

Categories

(Tree Management :: Bugherder, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: KWierso, Assigned: emorley)

References

Details

Attachments

(1 file)

mcMerge's code is currently hosted within TBPL's repository: https://hg.mozilla.org/webtools/tbpl/file/ae716547090e/mcmerge

We should probably make sure it finds a new home when TBPL is decomissioned.
Yeah we'll need to do this - however this doesn't block bug 1030636 since that's not about eol (see bug 1030636 comment 2). Switching off tbpl is several steps after bug 1030636.
OS: Windows 8.1 → All
Priority: -- → P3
Hardware: x86_64 → All
Blocks: tbpl-eol
I'm not sure what we should do with mcMerge - we could either:
* move it into treeherder as-is for now, and work on integrating it better in the future
* move it into version-control-tools and have new hosting set up to point to there
Summary: Make sure mcMerge makes the transition over to Treeherder. → Find a new home for mcMerge
Depends on: 1072923
(In reply to Ed Morley [:edmorley] from comment #2)
> I'm not sure what we should do with mcMerge - we could either:
> * move it into treeherder as-is for now, and work on integrating it better
> in the future
> * move it into version-control-tools and have new hosting set up to point to
> there

Whichever we do, it will simplify the move greatly if we can remove the only server side component to mcMerge (particularly since it's PHP, which may not even be set up on the treeherder boxes), which can only be done if bug 774766 is fixed.
Component: Treeherder → mcMerge
Assignee: nobody → emorley
I would eventually like mcMerge to be driven automatically merely by pushing to a repository. Unless some other arrangement is made, my guess is that replacing mcMerge with something more automated will make its way onto the Developer Services team roadmap for 2015.
I'm not sure if all of the functionality of mcMerge can be replaced with a tool that has no interactive mode - at least not whilst we're clumsily scraping commit messages and relying on a human to check the interpretation & handle things like security bugs, closing bugs, the myriad of B2G status flags etc. (All things we should be fixing, but will take a while). However something that manages to do so would be very welcome :-)

That said, this bug is needed shorter term than a solution to the above would be available, since it would hold up the switching off of tbpl.m.o.

However fortunately this bug will be pretty easy to fix, now that hg.m.o returns the correct access control headers - since I'll just fix bug 1072923 (and so make mcMerge static content only) and sling it up on Github pages to keep us going until another tool replaces it.
Status: NEW → ASSIGNED
Yeah, some manual functionality is still needed.

Once Autoland is in place and driving 99% of landings, Autoland effectively becomes mcMerge.

I agree it is a security nightmare to work out. But we'll get there. We need to: we can't continue having humans doing work machines can do more efficiently.
Agree totally :-)
So couple of decisions to make:
1) Are we definitely putting this on Github (I'm leaning this way) vs say the version control tools repo?
2) Are we going to host it using Github pages or not (they support HTTPS, but forcing it is tricky unless we use Cloudflare or similar). I'd like to if possible, just because it won't require using Chief or managing some box somewhere (for what is a fairly simple client side only app).

Presuming the answer to #1 is yes, #2 for now only really affects the Git branch name (master vs gh-pages).

Either way, these are the steps that will be required to export the mcmerge commits into their own git repo:

1) Setup:
git clone http://repo.or.cz/r/fast-export.git

2) Break out mcmerge commits to their own Hg repo (and rewrite paths so it's at the repo root):
echo include mcmerge > ~filemap
echo rename mcmerge . >> ~filemap
hg convert --filemap ~filemap tbpl mcmerge-hg

3) Checking to see if author mapping required when converting to Git:
cd mcmerge-hg
hg log | grep user: | sort | uniq | sed 's/user: *//' > ../authors
cd ..

-> Inspect output to see if any changes required. 
(None needed in this case, so this step can be omitted).

4) Convert the standalone Hg repo to a Git repo, whose branch is named gh-pages:
git init mcmerge
cd mcmerge
../fast-export/hg-fast-export.sh -r ../mcmerge-hg -M gh-pages
git checkout HEAD

5) <Add a remote and push>
This is now blocked on making a decision on where to host this.

Some considerations:
* As a quick reminder, mcMerge is client-side only, with Bugzilla credentials only ever being posted to Bugzilla's bzapi compatibility layer over HTTPS. The kicker being that if someone MITM'd the mcMerge JS assets they could obviously do whatever they like with the credentials.
* Only a half-dozen people use mcMerge, so the audience with respects to encouraging people to only ever accessing mcMerge via Treeherder's HTTPS URL links/HTTPS bookmarks (option 1) is fairly small.
* We could always consider making mcMerge work only with a bot account, so people get in the habit of using the credentials for that, and not their personal (potentially security bug permission granted) bugzilla account (as they do at present), making option 1 less of a big deal. The added advantage is that it solves some existing problems we have: mcMerge activity cluttering up people's personal Bugzilla activity/profile, plus other devs not realising a bug comment/closure was performed by a script ("Hey you closed the wrong bug, why did you do that!?").

Choices:

Option 1: github pages...
* HTTPS is supported, but we have no way of forcing people visit the HTTPS version over the HTTP one, either than a MITM-able client side redirect.
* That said, Treeherder (where the vast majority of accesses come from) will link to the HTTPS version, and the redirect would at least ensure people bookmark or copy/paste/share the HTTPS URL.
* Motivation: Extremely simple, to set up & deploy. I don't have to file bugs to request someone from IT deploy it - we just push to the gh-pages branch on the new github repo.

Option 2: github pages with cloudflare CDN in front...
* HTTPS can be forced on the cloudflare protected custom domain, however the connection between github and cloudflare is susceptible, for the reasons mentioned at https://me.net.nz/blog/github-pages-secure-with-cloudflare/
* For this reason I think it's inferior to option 1.

Option 3: Hosted by IT/in some AWS instance with an SSL cert
* Ideally we'd have deployment done on a cron, so we can still just check into the github repo and be done with it. (Chief is much better than the previous, but it's still an unnecessary extra step IMO).
* I'm probably just put off this one because I'm over-estimating the hassle based on previous experience. Happy to be convinced otherwise :-)

Thoughts?
Kendall, I don't suppose you could suggest who I should discuss comment 11 with - in particular the ease with which we could prop up Mozilla hosting for this extremely low usage client-side only webapp, that just needs HTTPS for MITM protection? Thanks :-)
Flags: needinfo?(klibby)
I'm still an AWS noob, else I'd stand it up there (I still could, it just wouldn't be fast). WebOps could probably make this happen fairly quickly on the generic cluster, which would alleviate having to spin up a VM somewhere. 

Let me throw jakem under the bus and see what he has to say. :-D
Flags: needinfo?(klibby) → needinfo?(nmaul)
Priority: P3 → P2
As much as I'd love to make mcMerge a service that runs automatically and is derived from events on the repository server, I don't think I'll have time to do that until at least Q2.
Yeah the scope for this bug is purely "move mcMerge to new hosting", even if comment 14 is the desired future end state.
(In reply to Kendall Libby [:fubar] from comment #13)
> I'm still an AWS noob, else I'd stand it up there (I still could, it just
> wouldn't be fast). WebOps could probably make this happen fairly quickly on
> the generic cluster, which would alleviate having to spin up a VM somewhere. 
> 
> Let me throw jakem under the bus and see what he has to say. :-D

Jake, any idea when you might be able to let us know our options/next steps? :-)
Laura, any luck with this? (following on from the email)

Also seems like we can go with a AWS/VM solution and have a pretty simple deploy by pushing to the Git repo, using Git 2.3's new feature:
https://github.com/blog/1957-git-2-3-has-been-released
Flags: needinfo?(laura)
Laura asked me to set this up in AWS, but I forgot to come back and find the bug.
Flags: needinfo?(nmaul)
Flags: needinfo?(laura)
(In reply to Kendall Libby [:fubar] from comment #18)
> Laura asked me to set this up in AWS, but I forgot to come back and find the
> bug.

Would it be helpful for me to file another bug for the ops side of this? If so, in which component? :-)
With TBPL EOL looming, we'll need somewhere to put mcMerge soon.
Flags: needinfo?(klibby)
Summary: Find a new home for mcMerge → Move mcMerge to it's own repo/hosting solution
Depends on: 1140889
I've broken the "we need somewhere to host" part out to webops bug 1140889.
Flags: needinfo?(klibby)
I'm not sure if mcMerge found a new repo home, but I'd be happy to have it in version-control-tools. We already have the test infrastructure in place to provide Docker containers for running Bugzilla and Selenium tests so we can verify end-to-end behavior works as expected.
That's a very kind offer, but I think it might be a bit heavyweight given the inactivity of mcmerge and how much would be involved in getting sensible test data mocked up (well, more than I have time for at the moment anyway). That and I can't get the version-control-tools environment to work properly (bug 1110345), the jenkins run for it takes hours & I'm really starting to like Git.

I previously created https://github.com/mozilla/mcMerge but I'm waiting on the new hosting to be ready before running my conversion script locally, so the old and new repos don't get out of sync.
mcMerge has now been extracted from the TBPL repo, converted to git & pushed to:
https://github.com/mozilla/mcMerge

I won't delete it from the TBPL repo yet - but we should no longer make changes to mcMerge there.

--

[~/src]$ echo include mcmerge > ~filemap
[~/src]$ echo rename mcmerge . >> ~filemap
[~/src]$ hg convert --filemap ~filemap tbpl mcmerge-hg
initializing destination mcmerge-hg repository
scanning source...
sorting...
converting...
1514 Initial import.
<snip>
0 Bug 911273 - Add the submit button to the top of the page as well. r=edmorley

[~/src]$ cd mcmerge-hg
[~/src/mcmerge-hg]$ hg log | grep user: | sort | uniq | sed 's/user: *//' > ../authors
[~/src/mcmerge-hg]$ cd ..

[~/src]$ git init mcMerge
Initialized empty Git repository in C:/Users/Ed/src/mcMerge/.git/
[~/src]$ cd mcMerge/
[~/src/mcMerge]$ ../fast-export/hg-fast-export.sh -r ../mcmerge-hg -M master
master: Exporting full revision 1/171 with 46/0/0 added/changed/removed files
master: Exporting simple delta revision 2/171 with 0/1/0 added/changed/removed files
<snip>
master: Exporting simple delta revision 171/171 with 0/2/0 added/changed/removed files
Issued 171 commands
C:\Program Files (x86)\Git\libexec\git-core\git-fast-import.exe statistics:
---------------------------------------------------------------------
Alloc'd objects:       5000
Total objects:          841 (        35 duplicates                  )
      blobs  :          312 (        25 duplicates        138 deltas of        310 attempts)
      trees  :          358 (        10 duplicates        308 deltas of        331 attempts)
      commits:          171 (         0 duplicates          0 deltas of          0 attempts)
      tags   :            0 (         0 duplicates          0 deltas of          0 attempts)
Total branches:           1 (         1 loads     )
      marks:           1024 (       171 unique    )
      atoms:             70
Memory total:          2294 KiB
       pools:          2098 KiB
     objects:           195 KiB
---------------------------------------------------------------------
pack_report: getpagesize()            =      65536
pack_report: core.packedGitWindowSize =   33554432
pack_report: core.packedGitLimit      =  268435456
pack_report: pack_used_ctr            =        527
pack_report: pack_mmap_calls          =        328
pack_report: pack_open_windows        =          1 /          1
pack_report: pack_mapped              =     990321 /     990321
---------------------------------------------------------------------

[~/src/mcMerge]$ git checkout HEAD
[~/src/mcMerge]$ git remote add origin git@github.com:mozilla/mcMerge.git
[~/src/mcMerge]$ git push -u origin master
Counting objects: 841, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (386/386), done.
Writing objects: 100% (841/841), 966.82 KiB | 0 bytes/s, done.
Total 841 (delta 446), reused 841 (delta 446)
To git@github.com:mozilla/mcMerge.git
 * [new branch]      master -> master
Branch master set up to track remote branch master from origin.
This is pretty much complete now after much blood, sweat, tears & forking. We're having to use my fork of the buildpack for now, until upstream accept my PRs.

I can't quite decide whether using Stackato like this (as an overkill, but unavoidable alternative to force-https github pages) is awful or awesome, but hey - it works haha.

To run the instance follow the Mozilla Stackato setup docs then |stackato push --no-prompt| from the repo root. However once I deploy it live, it should need no further intervention, since the cron will update it.

Temporary dev (allizom) instance for testing:
https://mcmerge.paas.allizom.org/

The http URL 301s to https :-)

Deployed revision info available at:
https://mcmerge.paas.allizom.org/deploy.txt
(updates on 5 min cron).

The allizom paas sites have a wildcard ssl cert; for mozilla.org we'll need to decide re the "shall we rename mcMerge" email first, to avoid churn.
Blocks: 1145713
We want to EOL TBPL at the end of the month, but I'm on PTO from Monday until 1st April (back 2nd).

So we don't have to rush the decision around choosing a new name for mcMerge, and given that we already have the Github repo set up with the current name (since it was created before the name change was raised) - let's just use the allizom instance with the old name for now (https://mcmerge.paas.allizom.org/) - and we can rename (and create a proper X.mozilla.org) later.
Summary: Move mcMerge to it's own repo/hosting solution → Move mcMerge to it's own repo/hosting solution (part 1)
Comment on attachment 8579065 [details] [review]
Add configuration files for Stackato

I've just landed this as-is, since (a) it works, (b) there isn't really anyone who understands this stuff who can review, (c) my PTO ends after TBPL will be EOLed, (d) this stuff is easy to revise later.

https://github.com/mozilla/mcMerge/commit/cfeedd2b7398ecf4a69bc4926c2eb90f5f212208

I'll file other bugs for:
* Moving to X.mozilla.org rather than allizom or X.pass.mozilla.org
* Updating links in Treeherder
* Renaming mcMerge
* Updating the readme with info on the deployment (the comments at the top of stackato.yml are pretty comprehensive and will be fine for now)

For mcMerge, for now use:
https://mcmerge.paas.allizom.org/

To check how the auto-deploy is running, see:
https://mcmerge.paas.allizom.org/deploy.txt
Attachment #8579065 - Flags: checkin+
mcMerge removed from the TBPL repo:
https://hg.mozilla.org/webtools/tbpl/rev/5507d424a283
Blocks: 1145836
Depends on: 1145840
All done here apart from:
1) Deploy treeherder master to production (I've just updated stage, looks fine)
2) Deploy tbpl default to production (to remove mcmerge from there)

Will close this bug once those are done.
I've run merges tonight from https://mcmerge.paas.allizom.org/ and it all appears to be working, for whatever that's worth...
(In reply to Wes Kocher (:KWierso) from comment #29)
> I've run merges tonight from https://mcmerge.paas.allizom.org/ and it all
> appears to be working, for whatever that's worth...

Awesome, thank you :-)
(In reply to Ed Morley (Away 23rd March -> 1st April) [:edmorley] from comment #28)
> All done here apart from:
> 1) Deploy treeherder master to production (I've just updated stage, looks
> fine)

Done a couple of days ago.

> 2) Deploy tbpl default to production (to remove mcmerge from there)

Done now.

Treeherder now links to mcmerge.pass.allizom.o, and anyone using the old links will still get redirected correctly thanks to the rules added in bug 1145713.

Bug 1145836 will handle things like: picking a different name for mcMerge, switching to NAME.mozilla.org rather than something on *.pass.* or anything on allizom etc.
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Switched back to the upstream buildpack, now that the changes in my fork have been accepted/merged:
https://github.com/mozilla/mcMerge/commit/05184f7dd4f0b848c31eb07b76e741bce7f2ecbe
You need to log in before you can comment on or make changes to this bug.