meta bug for a Mozilla sponsored project this summer. place to report progress and link in other pages related to work planned in http://sukhoy.public.iastate.edu/soc2007/proposal-mozilla.txt
/me grumbles about people who don't get bugspam when they are cc:ed, and tracking bugs filed without cc:ing the person being tracked.
Ok, it looks like I cannot assign the bug to myself due to permissions. This probably has to be done...
0th day at work is over. 2all subscribers: I'm going to post regularly here, so... You've been warned ;). Any feedback is welcome at this point.
What's done so far
* [almost] finished setup of the build environment, i.e. incremental builds of all three major branches. I will test on the following platforms: XP Pro (32-bit) [setup is mostly complete], XP Pro (64-bit) [did not start setting yet], Open SUSE 10.2 (with most recent packages) [setup is mostly complete], Fedora Core 6 (with most recent packages) [setup is mostly complete], MacOS X (Intel) [setup is mostly complete]. All of above except MacOS is under VMWare. MacOS test builds run on enhanced (2GB RAM) Intel Mac Mini. Shall I throw in FreeBSD test machine there?. I don't have experience with it except that I'm a MacOS user...
Unfortunately, I do not have access to Vista at this time (this may change though).
* The plan for this week:
* Start with MathML DOM implementation.
* Development of the test suite (I intend to follow they way they do it here: http://www.w3.org/DOM/Test/ ). Create at least the minimal test of the very basic things, like if certain attributes are supported etc. The idea is to develop an outline of the extendable MathML DOM testing framework by the mid-to-end of the week.
* Get the start on a patch to satisfy the test suite, i.e. a pair of draft test suite/simple patch to satisfy it is a strong goal by the week's end.
* look more in-depth: #276267, #379178 (somehow I was not a subscriber there...) provides a number of patches/test cases by Daniel Kraft which I intend to reuse here, some of the code, in particular - generic stylable base class code) seems to be on its way to the tree...
* Finally wrap up with the builds, setup nightly builds, backups of whatever I do in the source code tree.
I don't know whether this is a bug or a feature, but using '=' symbol in the name of the source code directory or objdir breaks assertion in mozila/config/Preprocessor.py in handleCommandLine, function handleD, the code fragment is:
def handleD(option, opt, value, parser):
vals = value.split('=')
assert len(vals) < 3
if len(vals) == 1:
My directory choice was something like $STUFF/objdirs/hacking+mozilla+all+branch=HEAD for the objdir and $STUFF/source/hacking+mozilla+all+branch=HEAD for the source code dir.
Created attachment 264936 [details]
Python wrapper for nmedit to fix macos mozilla build on SMB/CIFS share.
I run into a problem building Trunk source on a Samba share, in particular nmedit tool wants to rename an open file, which is ok in POSIX semantics, but not allowed in SMB/CIFS. So, a solution was to wrap nmedit transparently, copy the files it needs somewhere like /tmp which is likely not a SMB/CIFS share, invoke nmedit over them, copy back.
Obviously, $PATH should be setup so that wrapper scripts takes priority over nmedit. Unfortunately, the build does not allow to configure nmedit location (like, there is no support for $NMEDIT or whatever), it seems.
Status report for the last week is delayed (will be filed on Tuesday), some progress was made :).
Created attachment 265757 [details] [diff] [review]
MathML DOM (MathML 2, 2nd ed) testcase set patch, revision 0.
Here comes the (late) weekly report.
What was done:
* More work on build environment (it looks like being "finished" with it is more of an idea rather than a realistic milestone :)). In particular, designed "tag"-based builds, so that different flavours are designated via the directory name (i.e., if I want to build debug flavour, head branch i say that the objdir will be "mozilla+all+branch-HEAD+debug" and appropriate mozconfig is built on the fly by scripts, as well as the appropriate source code drop is selected automatically). I keep enhancing this system.
* For the builds - I'm not going to test xppro64 builds at this time since it turned out that vmware on xp32 cannot run 64-bit OSes even though the processor can do it. I should've figured :(.
* Backups of the build support code and scripts is done via private svn repository.
* Got a start on the MathML DOM test set. Reviewed http://www.w3.org/DOM/Test/, rejected the idea of using exactly the same technology, reasons:
2. Due to 1, they need yet another build step and use ant, this seems too much of unneeded burden in my situation.
No coding to satisfy the test suite was done.
* Looked at bug #276267, bug #379178. Generic stylable class is a good idea. Will return to those once the MathML DOM implementation will get to that point (I intend to work on test cases and the actual implementation in parallel).
Small clarification - the inability to run 64-bit xppro on xppro32 host is due to my particular hardware configuration (vmware does not want to run 64-bit OS on 32-bit host if there is no hardware support for Intel VT, which is exactly my case - the older Pentium D box). With a different (newer) processor everything should work just fine.
* Implement DOMImplementation::hasFeature() reporting for "org.w3c.dom.mathml", so that the test in attachment #265757 [details] [diff] [review] is satisfied.
* Implement more test cases for MathML DOM, in particular, for this week:
* Add the test case for hasFeature() in XHTML (the one in #265757 is for HTML).
* Test cases for MathMLDOMImplementation:
1. Test case for the availability of the interface at the global level, just like DOMImplementation is available.
2. Test cases for document.implementation being an instance of MathMLDOMImplementation (i.e. document.implementation instanceof MathMLDOMImplementation) in HTML and XHTML.
3. Test cases for MathMLDOMImplementation::createMathMLDocument() in HTML and XHTML, the following should be tested:
* Method is available.
* Method matches the signature createMathMLDocument() per MathML2, 2nd ed, D.1.1.
* An instance of MathMLDocument is created as a result of calling the method.
* The MathMLDocument that was created is indeed empty, corresponding to a single MathML math element with no child elements and non-default attributes, it is the root element of the document and is accessible via the documentElement attribute of the MathMLDocument.
4. Test cases for MathMLDocument interface in HTML and XHTML:
* MathMLDocument is available at the global scope.
* Test cases for the referrer attribute (is readonly, is a string, is null if navigated directly to the page, otherwise contains the valid referrer URL).
* Test cases for the domain attribute (is readonly, is a string, is null if the domain name cannot be determined.
* Test cases for the URI attribute (is readonly, is a string).
5. Basic test case for MathMLElement (is available at the global scope). More to come.
6. Any extra test cases feasible...
* Implementation patch for the test cases (hard to predict how much time this will take, a lot of exploration to be done), useful pointers are http://www.mozilla.org/docs/dom/mozilla/hacking.html, also bug #33871. Go as far as time allows there.
* Provide the updated patch for DOM test suite and DOM implementation (I think it makes sense to keep these in distinct patches.
Not much progress this week on the tasks within plan (some other activities, Mozilla related though took most of the time, good opportunity to dive into layout and MathML codebase though).
The plan stays for this week.
Created attachment 267242 [details]
The actual proposal.
Just figured that the URL was pointing at the old version of the proposal, the actual negotiated proposal is attached.
It looks like I'm about a week behind schedule :(.
The bi-weekly report will be filed today. Significant progress is made in DOM implementation.
Created attachment 268109 [details] [diff] [review]
MathML DOM patch rev1
A result of continuous work on MathML DOM. Build process integrated, but I'm not sure if it is done properly. A lot of things are there, including a fixture for mochitest to generate stubs in several flavours of XHTML and HTML to test different codepathes, a bunch of tests, IDL's, DOM infos, some content implementations.
More in the upcoming report.
Any reviews welcome.
Heh, there are some non-MathML parts in there, due to hacking of some other stuff in the same tree. Have to get those out...
What was done over the two weeks:
[+] Implement the generator script to be able to automatically generate test boilerplate in HTML, all flavours of XHTML 1.0, XHTML 1.1.
[+] Implement DOMImplementation hasFeature reporting for "org.w3c.dom.mathml" so that the test in attachment 265757 [details] [diff] [review] is satisfied
[+] Make DOM tests conditional on MOZ_MATHML_DOM being enabled
[+] Add the test case for the availability of MathMLDOMImplementation interface at the global level
[+] Add the test case for document.implementation being an instance of MathMLDOMImplementation in HTML and XHTML
[+] Create a test case for MathMLDOMImplementation::createMathMLDocument() being available in HTML and XHTML
[+] Create a test case for MathMLDOMImplementation::createMathMLDocument matching a signature per MathML2 2nd ed D 1.1
[+] Create a test case for an instance of MathMLDocument being created as a result of MathMLDOMImplementation createMathMLDocument being called
[+] Add the IDL code for all MathML interfaces per MathML 2.0 specification (the IDL files for all interfaces was added).
[+] Fix bug 382884 at least on trunk.
[+] Setup the module and linkage for MathML content and document moduli.
[+] Implement the layout submodule and in particular nsMathMLDocument (??) class and
use that to implement createMathMLDocument() method of MathMLDOMImplementation().
[-] keep testcases and DOM code in several patches (will use one patch, can separate later).
[-] Extra testcases for MathMLDocument.
[+] Some other Mozilla/Bugzilla-related work.
Overall: I'm definitely late with delivering MathML DOM code. How late - currently it seems like between one and two weeks. It is a question though if the schedule was realistic, given that the "missing" week was spent mostly on reviewing the Gecko layout code on trunk and branches, which had to be done anyway.. What is inspiring is that a lot of infrastructure and code review was done, so I now add stuff and know where and what to look for, more or less. If I will be able to deliver of 7-10 DOM classes per day, I should be close to be done with DOM at the review level by 25th. Here is the hoping...
One random comment about the patch; it has some tabs, use spaces in
the source files.
Thx, will get rid of tabs.
[ ] Create a test case for MathMLDocument that was created after MathMLDOMImplementation createMathMLDocument indeed being empty corresponding to a single MathML element with no child elements and non-default attributes that is the root element of the document and is accessible via documentElement attribute of the MathMLDocument instance.
[ ] Make sure that tabs are replaced by spaces within the source files (fix modeline string if necessary).
[ ] Write a simple tool to check if there are tabs in the patch being submitted or, better, a greasemonkey script to do it upon submission in bugzilla...
[ ] Create test cases for the referrer attribute of MathMLDocument instance is readonly is a string is null if navigated directly to the page contains a valid referrrer URL otherwise.
[ ] Create test cases for the domain attribute of MathMLDocument instances is readonly is a string is null if the domain name cannot be determined
[ ] Create test cases for the URI attribute of MathMLDocument instances is readonly is a string
[ ] Design the mechanics of how to apply the tests for instances of base class on the descendant instance (i.e. we would probably like to verify that an instance MathMLSemanticsElement is still a MathMLElement)..
[ ] Design and Implement the tests for MathMLElement interface and its instances.
[ ] Design and Implement nsMathMLElement implementation class and classinfo support for MathMLElement so that it satisfies the tests for MathMLElement instances.
[ ] Integrate MathMLElement and MathMLDocument with the layout module (i.e. so that layout's mathml uses these rather than generic element).
[ ] Explore if it is feasible to backport the patch onto 1.8, because MathML is broken on trunk and I have to verify if it does render properly with newer elements from the MathML DOM.
[ ] Move the tests over to the content module (currently they are within layout).
[ ] Remove non-MathML parts from the DOM hacking source tree so that it does not make it into future patches.
[ ] Implement the tests for MathMLContainer interface and instances of it (plenty of tests to be implemented), elaborate more here.
[ ] Implement the wrapper/implementation class for MathMLContainer (so that there is less code needed in its descendants and implementation of some/all methods can be reused).
[ ] Design and implement the tests for MathMLMathElement interface and its instances.
[ ] Implement nsMathMLMathElement class and classinfo for nsIDOMMathMLMathElement so that the tests are satisfied.)
[ ] Proceed with more (at this point the workflow should be more or less stabilized).
(In reply to comment #18)
> modeline string if necessary).
> [ ] Write a simple tool to check if there are tabs in the patch being
> submitted or, better, a greasemonkey script to do it upon submission in
Do you know of jst review?
> [ ] Explore if it is feasible to backport the patch onto 1.8, because
> MathML is broken on trunk and I have to verify if it does render properly with
> newer elements from the MathML DOM.
Isn't the plan to unbreak MathML on trunk?
> Do you know of jst review?
Now I do. Thanks.
> > [ ] Explore if it is feasible to backport the patch onto 1.8, because
> > MathML is broken on trunk and I have to verify if it does render properly with
> > newer elements from the MathML DOM.
> Isn't the plan to unbreak MathML on trunk?
Yes, but per the proposal that goes after the DOM. The idea is/was to do DOM first because it does not require that much of learning curve to go ahead and start doing it. Unbreaking MathML on trunk properly may take well beyond the midterm review and, besides all else, the idea is to have something up by the review =) and the proposal was stated that way. I keep reading Cairo/Thebes docs and looking at the code and, yes, the plan is to unbreak MathML on trunk.
BTW that greasemonkey script is a little outdated and does not work for me.. Trying to fix it real quick..
Created attachment 268271 [details]
The modifed "JST review" greasemonkey script which works for me.
Ok, this one seems to work for me.
Er... so we seem to have some duplication between the work here and the work in bug 276267 on the MathML DOM end of things.... Daniel should probably have been cced here from comment 3 on.
Is there a plan to actually make MathML work on trunk (with cairo) too, by the way?
See comment 20, attachment 267242 [details], presence of bug 324857 among the blockers. The plan is indeed to actually make MathML work on trunk with cairo, but this student was not sure if he can estimate what and how should be done to unbreak MathML on trunk at the time of proposal writing, thus he decided to do MathML DOM while climbing the learning curve and concentrate on unbreaking MathML later.
bug 276267 was idle recently. There was some work also on the common stylable base class in bug 379178 but it did not make its way to the tree.. While there was some duplication, I believe it is fairly minor at the moment... Also, strictly speaking, bug 276267 is NEW at the moment, so it is unclear to me who does what in that bug..
Sure. All I'm saying is that you should coordinate with Daniel to avoid duplication of effort.
Absolutely. I thought that the news actually made it through via bug 143842 though.
Thanks for cc'ing me here. Yes, I'd still like to get the common base into there, if there's a decision on whether to exclude SVG and do it for MathML and HTML alone.
For MathML DOM, I have that rather old patch but no other work done yet; so the only thing at risk to be duplicated here is that stylable thing here.
Created attachment 269626 [details] [diff] [review]
MathML DOM patch rev2
A result of continuous work on MathML DOM (most likely bitrotted against the most recent trunk, I did not check, the patch is against the trunk of Sun May 20 00:28:10 CDT 2007).
As such, the patch implements bug 276267, and also bug 124709 (application/mathml+xml is supported and wired to MathMLDocument - a lot of it had to do with content, layout, parser code, some changes are most likely redundant, but it works for me.)
A lot of test code was written. More details in upcoming progress report.
Having said that.. When it will come to review of this monster, It'll most likely take a while to get all parts of it reviewed (build, layout, content, mathml, parser etc.) and I would want to stay up-to-date with the trunk while all this is happening without painful and time-consuming sync.. I wonder what is the best way to do so? I found these pieces of documentation: http://developer.mozilla.org/en/docs/Using_Monotone_With_Mozilla_CVS and http://developer.mozilla.org/en/docs/Using_SVK_With_Mozilla_CVS
I have virtually no experience working with either, so I wonder what is the best approach.. Or perhaps there is a third option?
Created attachment 269752 [details]
Progress report 06/25/07
The detailed progress report.
To summarize: no, the DOM code is not ready for a review. Integrating MathMLDocument and wiring it to application/mathml+xml took much time which should have gone to writing and testing the left MathML elements code. Now MathMLDocument is actually integrated (actually, there is no point in having Document class if it is not wired to the actual content-type internally) and leaf element code and tests can be written.
The infrastructure of DOM test suite was significantly enhanced and seems to have stabilized. It appears pretty realistic to have review-level DOM code by midterm review date (7/9) if not a week earlier even - yes, this is way behind of what is in the proposal, but it is the way it is and I am actually getting there. Now rather than fighting with content-type I can concentrate on research and code review which needs to be done to unbreak MathML presentation on trunk while wrapping up with DOM.
Created attachment 271506 [details] [diff] [review]
Cumulative MathML DOM patch of 07/09/07
A cumulative patch for MathML DOM. This is roughly what will be cut into reviewable pieces and attached to a bunch of separate bugs shortly.
This will enable incremental updates of leaf dom code which are also en-route.
Core patches (MOZ_MATHML_DOM, classinfo, IDL) uploaded; implementation, bug 276267 (requires sync with trunk) and test suite patches are to be filed shortly.
Created attachment 272606 [details]
Cumulative MathML DOM patch of 07/17/07 gzipped
Last WIP DOM patch. The next step (this week) is posting a bug with implementation of Presentation MathML DOM (Content is still WIP), tests and requesting reviews.
I have to wonder... what is the codesize (compiled binary size) impact of that patch?
Did not measure.. Theoretically should be roughly like the turning SVG off and on at build time, I guess. At least, of that order.
It is quite possible to do a configuration with only MathMLElement/MathMLDocument support, or MathMLElement only. That should be quite small binarily, within bug 276267 scope, I'm long past there, but it is possible to do a define, if that's needed for some build..
vlad, it would be best if you concentrate on the class/style attr
support of bug 276267. This is frankly the most important/useful, and pretty much all this is needed for practical purposes in our native browser context.
It is important to keep in mind the spirit with which the MathML spec was first conceived. For the spec folks (many of whom came from commercial companies other than browser vendors), their guiding principle (those many years ago) was that MathML would be a plugin because math was bound to be a niche that didn't have a chance to be implemented natively in the major browsers (remember, open source that we take for granted today wasn't even in the psyche of most people, and also witness the still closed IE today). So the MathML DOM, which is non-normative, allows a plugin to provide just that without having to implement a whole raft of other things. In our case (built-in), we have the XML DOM which subsumes what the MathML DOM does. Having the MathML DOM (in our native browser context) just for the sake of it brings this huge code size without returning a distinctive value. We are better off just finishing bug 276267, which stands a greater chance to be reviewed & accepted.
err, sure. I'll accelerate with 276267 patch..
At this point, having significant part of the DOM (and test suite..) implemented I wish this discussion started a bit earlier.. There is also an option of getting code in, but having 3 different configurations - no MathML DOM, lightweight MathML DOM (bug 276267) and full MathML DOM.
Also, there is a need for hard data about code size of MathML DOM and without so that we can see how many bytes are there. In that patch, significant part is a test suite which in itself does not add to the binary size.
Building with codesighs... The question is: how bad is too bad, i.e. how much overhead is critical?
Some data (w/o codesighs): total size of dist/bin with the latest MOZ_MATHML_DOM patch is 7,496,251 bytes, versus 7,464,563 w/o (macos build, configuration:
ac_add_options --disable-debug )...
Created attachment 273033 [details] [diff] [review]
Makes codesighs work on OS X for me
Created attachment 273050 [details]
MacOS codesighs data of 07/19/07 (detailed diff, gzipped)
Overall Change in Size
Total: +378592 (+380712/-2120)
Code: +51840 (+51856/-16)
Data: +326752 (+328856/-2104)
What's the total code size reported?
By the way, care to file a separate bug on making codesighs work on OSX and getting it turned on on the tinderboxen? That would be really good data to have in general.
(In reply to comment #43)
> What's the total code size reported?
With MathML DOM:
Without - subtract diff values... Now here I realize that --enable-optimize was not turned on in the build..
> By the way, care to file a separate bug on making codesighs work on OSX and
> getting it turned on on the tinderboxen? That would be really good data to
> have in general.
Will do. I'm not quite sure how to get it up on tinderboxen and who does that, but I will file the bug.
Comment on attachment 273033 [details] [diff] [review]
Makes codesighs work on OS X for me
See bug 388891, this patch was done backwards anyway =)
When using SVK with Mozilla CVS, there is a caveat - SVK forces uniform linestyle on the source code, which sometimes yields \r\n's as newlines, anyway, this breaks build on 1_8 (at least) coz' config/preprocessor.pl does not like that (effectively all symbols are undefined).
TODO: Update http://developer.mozilla.org/en/docs/Using_SVK_With_Mozilla_CVS with this info.
It seems that this bug is now mostly a duplicate of bug 143842?
We should just close this and bug 143842 (WONTFIX for that one). I'm pretty sure we don't want to implement most of the MathML DOM --- the parts that duplicate functionality already available in existing DOM APIs.