859101 - tar Cannot open: File or path name too long when for test_document.getElementsByName-newelements.html.json and test_script-IDL-event-htmlfor.html.json

Status: VERIFIED FIXED

(Core :: DOM: Core & HTML, defect)

Description

[Mark Banner (:standard8)]

When building Thunderbird on Windows on Tinderbox during make package-test we are getting:

tar: tests/dom/imptests/failures/html/html/dom/documents/dom-tree-accessors/document.getElementsByName/test_document.getElementsByName-newelements.html.json: Cannot open: File or path name too long
tar: tests/dom/imptests/failures/html/html/obsolete/requirements-for-implementations/other-elements-attributes-and-apis/test_script-IDL-event-htmlfor.html.json: Cannot open: File or path name too long

Comment 1

[Mark Banner (:standard8)]

Can we just patch this, or do we have to fix the W3C tests first :-( ?

I'm not sure releng are going to be able to shorten our directory names much further...

Comment 2

[Ian Neal]

In:
tests/dom/imptests/failures/html/html/dom/documents/dom-tree-accessors/document.getElementsByName/test_document.getElementsByName-newelements.html.json
Why does the "document.getElementsByName" have to be repeated in the filename?

For the other one, do obsolete tests need to be kept around? If they do, then can't the directories "requirements-for-implementations" and "other-elements-attributes-and-apis" be given shorter names (8-10 characters) and then the description field used? (For both obsolete and non-obsolete paths)

Comment 3

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Feel free to bring that up on the relevant mailing list: http://lists.w3.org/Archives/Public/public-html-testsuite/

Comment 4

[Mark Banner (:standard8)]

Is there any way we can fast-track a solution here, or are we stuck with this until we fix upstream?

Comment 5

[Mark Banner (:standard8)]

I'm perusing a different direction here. Having looked a bit more, I think there's scope for changing TB's objdir name on buildbots and getting us to within 1 char difference of Firefox.

Comment 6

[Mark Banner (:standard8)]

Attachment: Shorten Thunderbird's object directory name

If this is acceptable, then I'm looking to get this deployed asap so that we can fix Thunderbird bustage on trunk which is making it difficult to get the beta release ready.

Although we recently added all the '0's to the directory names that the builds are stored in, they did not take account of the additional mozilla/ that Thunderbird has. As we can't really shorten the directory names of the builds, we need a different alternative.

Tests on try with patches that I probably can't get landed too quickly (due to needing to do them upstream first) indicate that a reduction in 5 characters is just enough.

This patch changes Thunderbird's object directory from objdir-tb to obj-, this gives us 5 more characters and means that we're now within 1 of Firefox:

obj-/mozilla = 12
obj-firefox  = 11

I'm not too keen on the obj- but it is in .hgignore, and gets us enough for now and almost matching FF.

I'd love to drop the mozilla/ directory completely, and we are getting close to that, but we need a bit more time yet.

Comment 7

[Mark Banner (:standard8)]

Attachment: Shorten Thunderbird's object directory name v2

This time with the really short version.

Comment 8

[[e-mail] (:bhearsum)]

I've read over this bug and I still don't understand why the tests with multiple very long path segments can't be renamed. I understand that these are tests imported from upstream, but why can't we rename them in our own codebase? There's very little we can do on the infrastructure side to cope with this at this point. Even if we take Mark's fix, we're only buying time until there's an even longer test name.

Comment 9

[rsx11m]

(In reply to Mark Banner (:standard8) from comment #6)
> Although we recently added all the '0's to the directory names ...
>> e:\builds\moz2_slave\tb-c-cen-w32-d-000000000000000

What are those redundant -d-000000000000000 good for anyway? This adds 18 chars. SeaMonkey doesn't have them, and their Windows builds are fine.

(In reply to Ben Hearsum [:bhearsum] from comment #8)
> Even if we take Mark's fix, we're only buying time until there's an even
> longer test name.

Maybe, but right now Thunderbird development and testing is stalled due to this on trunk, thus taking an immediate fix now to get things running again while keeping this open for a more thorough fix seems to be the right thing to do.

Comment 10

[Mark Banner (:standard8)]

(In reply to rsx11m from comment #9)
> (In reply to Mark Banner (:standard8) from comment #6)
> > Although we recently added all the '0's to the directory names ...
> >> e:\builds\moz2_slave\tb-c-cen-w32-d-000000000000000
> 
> What are those redundant -d-000000000000000 good for anyway? This adds 18
> chars. SeaMonkey doesn't have them, and their Windows builds are fine.

They even up the build directory name lengths, so that when we do releases (which use longer names), we don't suddenly find with exceeded our limits.

Comment 11

[[e-mail] (:bhearsum)]

(In reply to rsx11m from comment #9)
> (In reply to Ben Hearsum [:bhearsum] from comment #8)
> > Even if we take Mark's fix, we're only buying time until there's an even
> > longer test name.
> 
> Maybe, but right now Thunderbird development and testing is stalled due to
> this on trunk, thus taking an immediate fix now to get things running again
> while keeping this open for a more thorough fix seems to be the right thing
> to do.

This is just as easy to fix by renaming the test, and it's a longer term fix to do so. If someone's willing to commit to renaming the tests in the near future (and actually follows up on it) I'm willing to take this fix now.

Comment 12

[rsx11m]

Mark & Ben, makes sense. It is my understanding that bug 854329 imported those tests verbatim from whatever 3rd party, hence referring to upstream fixing above. If possible to do so without impacting future updates on those tests, reductions in filname length as pointed out in comment #2 sure should be the way to go.

Comment 13

[[e-mail] (:bhearsum)]

(In reply to rsx11m from comment #12)
> Mark & Ben, makes sense. It is my understanding that bug 854329 imported
> those tests verbatim from whatever 3rd party, hence referring to upstream
> fixing above. If possible to do so without impacting future updates on those
> tests, reductions in filname length as pointed out in comment #2 sure should
> be the way to go.

I re-opened that bug.

Comment 14

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

If someone want to shorten the paths for now, I won't block that. Unless the importing scripts are changed, though, any such changes will be overwritten on the next sync.

Comment 15

[[e-mail] (:bhearsum)]

(In reply to :Ms2ger from comment #14)
> If someone want to shorten the paths for now, I won't block that. Unless the
> importing scripts are changed, though, any such changes will be overwritten
> on the next sync.

Where are the import scripts, and where should a bug about changing them go?

Comment 16

[rsx11m]

http://mxr.mozilla.org/mozilla-central/source/dom/imptests/README explains it, and there are a bunch of Python scripts in that directory as well, but they may come from some master repository at the W3C site.

Comment 17

[[e-mail] (:bhearsum)]

Attachment: shorten files; try to update import script

I think this patch is rouhly what we need...AFAICT, importTestsuite.py is our own script, so it should be safe to modify (eg, changes to it won't get overridden). I probably won't have time to drive this to completion.

I tried testing the script change, but I'm not sure I was doing it right....when I deleted one of the renamed files and reran the script, it didn't add it back:
➜  imptests  rm html/html/dom/documents/dta/doc.gEBN/test_doc.gEBN-newelements.html 
➜  imptests  python importTestsuite.py webapps.txt                                 
Going to clone https://dvcs.w3.org/hg/webapps to hg-webapps...
Removing webapps...
Removing hg-webapps...
Cloning https://dvcs.w3.org/hg/webapps to hg-webapps...
requesting all changes
adding changesets
adding manifests
adding file changes
added 1080 changesets with 6106 changes to 2867 files
updating to branch default
2245 files updated, 0 files merged, 0 files removed, 0 files unresolved
Going to import ['DOMCore/tests/approved', 'DOMCore/tests/submissions/Ms2ger', 'DOMCore/tests/submissions/Opera', 'WebStorage/tests/submissions', 'XMLHttpRequest/tests/submissions/Ms2ger']...
Copying ['DOMCore/tests/approved', 'DOMCore/tests/submissions/Ms2ger', 'DOMCore/tests/submissions/Opera', 'WebStorage/tests/submissions', 'XMLHttpRequest/tests/submissions/Ms2ger']...
Copying [u'WebStorage/tests/submissions/Infraware', u'WebStorage/tests/submissions/Ms2ger']...
Copying [u'WebStorage/tests/submissions/Infraware/iframe']...
Creating build files...
Creating Makefile.in in webapps/WebStorage/tests/submissions/Infraware/iframe...
Creating moz.build in webapps/WebStorage/tests/submissions/Infraware/iframe...
Creating build files...
Creating Makefile.in in webapps/WebStorage/tests/submissions/Infraware...
Creating moz.build in webapps/WebStorage/tests/submissions/Infraware...
Creating Makefile.in in webapps/WebStorage/tests/submissions/Ms2ger...
Creating moz.build in webapps/WebStorage/tests/submissions/Ms2ger...
Creating build files...
Creating Makefile.in in webapps/DOMCore/tests/approved...
Creating moz.build in webapps/DOMCore/tests/approved...
Creating Makefile.in in webapps/DOMCore/tests/submissions/Ms2ger...
Creating moz.build in webapps/DOMCore/tests/submissions/Ms2ger...
Creating Makefile.in in webapps/DOMCore/tests/submissions/Opera...
Creating moz.build in webapps/DOMCore/tests/submissions/Opera...
Creating Makefile.in in webapps/WebStorage/tests/submissions...
Creating moz.build in webapps/WebStorage/tests/submissions...
Creating Makefile.in in webapps/XMLHttpRequest/tests/submissions/Ms2ger...
Creating moz.build in webapps/XMLHttpRequest/tests/submissions/Ms2ger...
Creating mozbuild...
dom/imptests/webapps.mozbuild already tracked!
hg addremoving...
Removing hg-webapps again...
Done
➜  imptests  ls html/html/dom/documents/dta/doc.gEBN/test_doc.gEBN-newelements.html
ls: cannot access html/html/dom/documents/dta/doc.gEBN/test_doc.gEBN-newelements.html: No such file or directory

Comment 18

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Comment on attachment 736838 [details] [diff] [review]
shorten files; try to update import script

This works, but all the file moves should be registered as such. Try hg addremove --similarity.

Comment 19

[[e-mail] (:bhearsum)]

(In reply to :Ms2ger from comment #18)
> Comment on attachment 736838 [details] [diff] [review]
> shorten files; try to update import script
> 
> This works, but all the file moves should be registered as such. Try hg
> addremove --similarity.

Whoops, I meant to generate this with 'hg diff --git'. I'm going to push this to try to make sure the test still work. I'll fix that up this ends up landing.

Comment 20

[[e-mail] (:bhearsum)]

Attachment: change build files; use proper renames

I had to update a few more files, but it looks like the tests still work...

10:25:25     INFO -  7555 INFO TEST-START | /tests/dom/imptests/html/html/obsolete/implreq/oeaaa/test_document-color-01.html
10:25:25     INFO -  7556 INFO TEST-PASS | /tests/dom/imptests/html/html/obsolete/implreq/oeaaa/test_document-color-01.html | Elided 2 passes or known failures.
10:25:25     INFO -  7557 INFO TEST-END | /tests/dom/imptests/html/html/obsolete/implreq/oeaaa/test_document-color-01.html | finished in 86ms
10:25:25     INFO -  7558 INFO TEST-START | /tests/dom/imptests/html/html/obsolete/implreq/oeaaa/test_document-color-02.html
10:25:25     INFO -  7559 INFO TEST-PASS | /tests/dom/imptests/html/html/obsolete/implreq/oeaaa/test_document-color-02.html | Elided 7 passes or known failures.
10:25:25     INFO -  7560 INFO TEST-END | /tests/dom/imptests/html/html/obsolete/implreq/oeaaa/test_document-color-02.html | finished in 90ms

Comment 21

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Comment on attachment 736869 [details] [diff] [review]
change build files; use proper renames

Review of attachment 736869 [details] [diff] [review]:
-----------------------------------------------------------------

If it passes try, r=me with this.

::: dom/imptests/importTestsuite.py
@@ +55,5 @@
>          return a
>      return "%s/%s" % (a, b)
>  
> +def shorten(path):
> +    path = path.replace('dom-tree-accessors', 'dta')

Add a pointer to this bug here.

Comment 22

[[e-mail] (:bhearsum)]

Comment on attachment 736869 [details] [diff] [review]
change build files; use proper renames

https://hg.mozilla.org/integration/mozilla-inbound/rev/c0bda1b44155

Comment 23

[[e-mail] (:bhearsum)]

Looks like this landed fine for Firefox. I guess we have to wait until it hits mozilla-central to see if it fixed up Thunderbird?

Comment 24

[rsx11m]

Yes, there are no tinderbox build for comm-central with mozilla-inbound, only mozilla-central.

Comment 25

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/c0bda1b44155

Comment 26

[Mark Banner (:standard8)]

Thanks Ben for fixing this :-)