Closed Bug 949600 Opened 6 years ago Closed 6 years ago

Move mozbase from github to m-c

Categories

(Testing :: Mozbase, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla30

People

(Reporter: ahal, Assigned: ahal)

References

Details

Attachments

(2 files, 4 obsolete files)

This bug will track:

1) Preserving history (if possible)
2) Version bumping all mozbase packages
3) Performing one last sync to m-c
4) Deprecating the github repo
(In reply to Andrew Halberstadt [:ahal] from comment #0)
> This bug will track:
> 
> 1) Preserving history (if possible)
> 2) Version bumping all mozbase packages
> 3) Performing one last sync to m-c
> 4) Deprecating the github repo

Don't forget to update the doc autogeneration!
Preliminary try run:
https://tbpl.mozilla.org/?tree=Try&rev=e0ef15568b44

This isn't a cutoff point or anything, I just wanted try results as soon as possible to see if there were issues we'd have to hunt down and fix.
Depends on: 947974
Depends on: 944684
Bug 950227 handles the problems as we can see on Windows. Wrong file permissions for files extracted from a zip archive is causing this problem.
Depends on: 950829
Wlach pointed out that we also need to encourage devs to run the mozbase tests locally before committing to avoid unnecessary backouts. Filed bug 950829 to add a mach target for them.
When we do the merge, we also have to enable mozrunner tests by updating make check, which should pass in the binary. Only then we will be able to run those tests.
Whiteboard: [see comment 6]
Been awhile since the last try run. There are still some known issues that need fixing, but here's a new try run anyway, just to see if the latest changes to mozbase have broken anything else:
https://tbpl.mozilla.org/?tree=Try&rev=d7cdf4cf4d12
You have removed the packages.txt file for mozbase which caused all builds to burn. But even with it included we have a lot of bustage given the pe file inclusion. So I don't think its worth doing a try on latest mozbase from github until we got bug 963176 solved.
Oops. Ok I'll wait for bug 963176 before further testing.
Depends on: 970415
Since the pefile problem was fixed we discovered two new issues related to adding new modules. Here's a new try run with those fixed as well:
https://tbpl.mozilla.org/?tree=Try&rev=842b31a7fa8d
Depends on: 971243
Depends on: 971833
Attached patch Patch 1.0 - the sync (obsolete) — Splinter Review
Fixed the remaining errors in that last try run.. assuming they weren't hiding further problems we should be good to go.

Here's a new try run:
https://tbpl.mozilla.org/?tree=Try&rev=dd631ea627ad
Assignee: nobody → ahalberstadt
Status: NEW → ASSIGNED
Attachment #8375044 - Flags: review?(wlachance)
Attachment #8375045 - Flags: review?(wlachance)
Comment on attachment 8375045 [details] [diff] [review]
Patch 1.2 - update reftest harness for mozdevice changes

Review of attachment 8375045 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm. As you know, _checkCmd is an internal api that we shouldn't use, but I'm fine with fixing that later.
Attachment #8375045 - Flags: review?(wlachance) → review+
Comment on attachment 8375044 [details] [diff] [review]
Patch 1.1 - make mozversion known to m-c

Review of attachment 8375044 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm
Attachment #8375044 - Flags: review?(wlachance) → review+
Depends on: 972059
Blocks: 972518
Depends on: 972966
Bug 966441 tracks pulling the mozbase tests out of make check, so I don't think we should do anything there to run the mozrunner tests. I updated that bug with a comment explaining the mozrunner situation.
Whiteboard: [see comment 6]
Attached patch Patch 1.0 - the sync (obsolete) — Splinter Review
This looks good. Only a handful of oranges, none of which look related to this.
Attachment #8375043 - Attachment is obsolete: true
Attachment #8376839 - Flags: review?(wlachance)
Attached file output from qstatus
Here's the output from qstatus to make the review a little easier.
Comment on attachment 8376839 [details] [diff] [review]
Patch 1.0 - the sync

Review of attachment 8376839 [details] [diff] [review]:
-----------------------------------------------------------------

Great!

::: testing/mozbase/README.md
@@ +1,1 @@
> +# Mozbase [![Build Status](https://travis-ci.org/mozilla/mozbase.png)](https://travis-ci.org/mozilla/mozbase)

Delete this line
Attachment #8376839 - Flags: review?(wlachance) → review+
(by the way I didn't really bother doing a detailed review, just assuming this was a literal copy of the mozbase repo)
https://hg.mozilla.org/integration/mozilla-inbound/rev/a7f8c25c07ac

Yeah, I just did a 'rm -rf testing/mozbase && cp -r ~/git/mozbase testing/'. I needed to be careful about added/removed files, but I stepped through hg status line by line to make sure I didn't leave anything behind.
Whiteboard: [leave-open]
Filed bug 974064 to track reconfiguring readthedocs (and patching where necessary) so we generate stuff from m-c.
See Also: → 974064
Along the way I decided not to do steps 1 and 2 from comment 0. Steps 3 and 4 are already done, and bug 974064 tracks updating our readthedocs instance. So I don't think there is anything left to do. One thing people may want to look into is porting the versionbump.py script over to hg. Personally I don't have much interest in doing this as I always version bumped manually anyway. Just adding it as a footnote.
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Backed out in http://hg.mozilla.org/mozilla-central/rev/8cfebc57002c due to changes that apparently broke CLOBBER:

[15:52]	<philor>	wlach: https://tbpl.mozilla.org/php/getParsedLog.php?id=34869690&tree=Mozilla-Inbound - we touched CLOBBER and now OS X is busted, is it possible I'm right in wanting to blame ahal's mozbase update from this morning?
[15:53]	<wlach>	philor: hmmmmmm....
	<philor>	it did touch mozfile/mozfile/mozfile/mozfile/mozfile/mozfile.py...
	<wlach>	it did
[15:54]	this mozfile stuff makes me grumpy
	let me take a closer look at what it's trying to do
	<KWierso>	wlach, philor: as a side point, I hit the clobberer for linux opt on inbound and retriggered the build, and it seems to be going just fine
[15:55]	<philor>	KWierso: yeah, that would cause the build system's autocobberer to say "nothing to do, no need for me to break"
	* AutomatedTester	mutters about people wanting to centralise a DVCS...
	<philor>	just another reason to clobber early often and constantly
[15:58]	<wlach>	philor: so this is weird
	philor: we're throwing an exception, except that exception is supposed to be caught http://hg.mozilla.org/integration/mozilla-inbound/file/f8cd1f2d686e/testing/mozbase/mozfile/mozfile/mozfile.py#l151
	oh
[15:59]	it's an errno.EPERM, which we don't retry on
[16:00]	philor: I think the problem is the build slave, not mozbase
	why shouldn't we be able to delete that thing if we have permissions?
	especially on mac
	<KWierso>	wlach: it's more than just the one slave: https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=f8cd1f2d686e
[16:02]	<wlach>	KWierso: ok, guess there's some bigger problem
	god there were so many changes here it's difficult to unwind
	I hate mozfile
[16:04]	ok, looking more closely I think it's a good bet that the new update permissions code in mozfile is to blame
[16:05]	this update_permissions stuff (where we're seeing the error) seems to be new
	http://hg.mozilla.org/integration/mozilla-inbound/diff/a7f8c25c07ac/testing/mozbase/mozfile/mozfile/mozfile.py
[16:06]	philor: I think we should backout
	philor: could you please reference my irc comments about the new permissions code in mozfile being to blame
[16:08]	hopefully the fix is not too difficult
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Andrew Halberstadt [:ahal] from comment #25)
> ... One thing people may want to
> look into is porting the versionbump.py script over to hg. Personally I
> don't have much interest in doing this as I always version bumped manually
> anyway. 

I think this would be useful, since whilst much of the old versionbump.py functionality is now redundant (diff to apply to m-c, tagging of mozbase repo etc), the bumping of versions for dependencies & also rolling up of package/upload to pypi is something that would take a bit of figuring out for me & other infrequent contributors each time, having forgotten since the last etc.
Blocks: 974184
Duplicate of this bug: 974225
Duplicate of this bug: 974226
> /builds/slave/m-in-osx64-0000000000000000000/build/obj-firefox/i386/_virtualenv/include/python2.7

So can anyone please tell me what kind of virtual environment this is? For what is it used? Is it still active?
Also which steps I would have to perform to reproduce this on my local system? Is triggering a clobber enough after building Firefox?
(In reply to Ed Morley [:edmorley UTC+0] from comment #27)
> I think this would be useful, since whilst much of the old versionbump.py
> functionality is now redundant (diff to apply to m-c, tagging of mozbase
> repo etc), the bumping of versions for dependencies & also rolling up of
> package/upload to pypi is something that would take a bit of figuring out
> for me & other infrequent contributors each time, having forgotten since the
> last etc.

Yes, that's something I see as must have. There are too many dependencies between the different packages that we would need at least the --info option working. Without that it would be even hard for me to figure out what version to bump, to not break dependencies. We should really include this file.
(In reply to Henrik Skupin (:whimboo) from comment #30)
> > /builds/slave/m-in-osx64-0000000000000000000/build/obj-firefox/i386/_virtualenv/include/python2.7
> 
> So can anyone please tell me what kind of virtual environment this is? For
> what is it used? Is it still active?

So attachment 8377995 [details] gives the answer here. We are trying to modify the permissions of the /usr/include/python2.7/ folder instead for the symlink:

$ ll obj-android/_virtualenv/include/python2.7 
lrwxrwxrwx 1 mcomella users 22 Feb 14 17:53 obj-android/_virtualenv/include/python2.7 -> /usr/include/python2.7/

I will morph bug 974225 to testing/mozbase to get this problem fixed in mozfile.
Blocks: 972872
Attached patch final syncSplinter Review
For simplicity, here's a single patch containing the 3 patches in this bug, plus Henrik's fix in bug 974225. Carrying r+ forward.
Attachment #8375044 - Attachment is obsolete: true
Attachment #8375045 - Attachment is obsolete: true
Attachment #8376839 - Attachment is obsolete: true
Attachment #8378558 - Flags: review+
Let us join hands and pray to the tinderbox gods:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b36a250468e7

This time, forcing a CLOBBER as well:
https://hg.mozilla.org/integration/mozilla-inbound/rev/cc7d6d3ebc7e
Whiteboard: [leave-open]
https://hg.mozilla.org/mozilla-central/rev/b36a250468e7
https://hg.mozilla.org/mozilla-central/rev/cc7d6d3ebc7e
Status: REOPENED → RESOLVED
Closed: 6 years ago6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla30
Blocks: 975085
No longer blocks: 975085
You need to log in before you can comment on or make changes to this bug.