Closed Bug 1622739 Opened 6 years ago Closed 6 years ago

geckodriver build can take arbitrarily long because of cinnabar invocation

Categories

(Testing :: geckodriver, defect, P3)

Product:
Testing
Component:
geckodriver
Version:
Version 3
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: kats, Unassigned)

Details

I did a ./mach build binaries and the build seemed to hang. Running pstree shows this:

 |-+= 01271 kats /Applications/iTerm.app/Contents/MacOS/iTerm2
 | |-+= 92254 kats /Applications/iTerm.app/Contents/MacOS/iTerm2 --server login -fp kats
 | | \-+= 92255 root login -fp kats
 | |   \-+= 92256 kats -bash
 | |     \-+= 64565 kats /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python ./mach build binaries
 | |       |--- 64608 kats /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python ./mach build binaries
 | |       \-+- 64611 kats /Applications/Xcode.app/Contents/Developer/usr/bin/make -j4 -s binaries
 | |         \-+- 64616 kats /Applications/Xcode.app/Contents/Developer/usr/bin/make recurse_compile
 | |           \-+- 64618 kats /Applications/Xcode.app/Contents/Developer/usr/bin/make -C testing/geckodriver target
 | |             \-+- 64621 kats /Users/kats/.rustup/toolchains/stable-x86_64-apple-darwin/bin/cargo rustc --release --frozen --manifest-path /Users/kats/zspace/gecko-wr/testing/geckodriver/Cargo.toml
 | |               \-+- 64947 kats /Users/kats/zspace/gecko-wr/obj-host-opt/testing/geckodriver/./release/build/geckodriver-d885ca8994bf05a2/build-script-build
 | |                 \-+- 64964 kats git cinnabar git2hg 884207b4691ee84824f2ece63fde3c36afb27846
 | |                   \-+- 64966 kats /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python /Users/kats/.mozbuild/git-cinnabar/git-cinnabar git2hg 884207b
 | |                     |--- 64972 kats /Users/kats/.mozbuild/git-cinnabar/git-cinnabar-helper --import
 | |                     \--- 64973 kats (git)

Not entirely sure what git-cinnabar-helper --import does but it seems to be taking a lot of time and is not desirable (to me) for something in the middle of ./mach build binaries.

Also to be clear, when I say "a lot of time" I mean multiple minutes. I got tired of waiting and killed the git-cinnabar-helper --import process which allowed the build to complete, and the build output showed I had been waiting for 10+ minutes:

 0:50.45 toolkit/library/build/XUL
 0:54.41     Finished release [optimized] target(s) in 3.93s
 1:29.21 ld: warning: could not create compact unwind for _ffi_call_unix64: does not use RBP or RSP based frame
12:13.70     Finished release [optimized] target(s) in 12m 09s
12:13.97 338 compiler warnings present.
12:14.46 Overall system resources - Wall time: 732s; CPU: 17%; Read bytes: 4254265344; Write bytes: 2516619264; Read time: 181816; Write time: 66861

Kats, I assume you are using git and not hg? The only place where cinnabar is used is here:

https://searchfox.org/mozilla-central/source/testing/geckodriver/build.rs#106

Maybe rev-parse iterates through all the existent commits? James are you still using git, and can you reproduce that?

Flags: needinfo?(james)
Priority: -- → P3

I am using git with cinnabar. Generally running git cinnabar git2hg works fine for me. In this case the revision it was trying was my HEAD commit which is a local git commit I made that has no hg equivalent. I don't know if that matters.

Oh, could you retry with having master (central) checked out?

FWIW I've been using cinnabar for a while now and this is the first time I've hit this problem. It might just have been random network error that struck in the middle of the cinnabar operation. Generally running git cinnabar git2hg on local-only commits will return 0000000000000000000000000000000000000000. That being said I can't try building master at the moment (fails due to bug 1622256). But I'll keep an eye out for this happening again.

In the meantime maybe glandium can provide some insight into why git cinnabar git2hg might be invoking git-cinnabar-helper --import which seems like a network operation in a place that doesn't seem like it should need one.

I can only thing of two things we could do on the geckodriver side here; either stop adding this information to the binary (but it's useful for user reports) or terminate the process after some timeout and just have None as the value. The latter seems pretty sensible in general, but you want to make sure we don't end up with missing information in release.

Flags: needinfo?(james)

(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #5)

In the meantime maybe glandium can provide some insight into why git cinnabar git2hg might be invoking git-cinnabar-helper --import which seems like a network operation in a place that doesn't seem like it should need one.

Flags: needinfo?(mh+mozilla)

git-cinnabar-helper --import doesn't do anything network related. It's what actually does the work for git cinnabar git2hg reading the git packs and getting the info out. If git cinnabar git2hg is stuck it would be good to know what it's stuck on, because that doesn't seem right.

Flags: needinfo?(mh+mozilla)

Ok. I guess if it happens again I'll attach gdb or something.

I just hit this again, but running moz-phab submit:

 | |-+= 99460 kats /Applications/iTerm.app/Contents/MacOS/iTerm2 --server login -fp kats
 | | \-+= 99461 root login -fp kats
 | |   \-+= 99462 kats -bash
 | |     \-+= 36253 kats /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python /usr/local/bin/moz-phab submit -m Use double quotes --upstr
 | |       \-+- 36263 kats /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python /usr/local/bin/moz-phab submit -m Use double quotes --ups
 | |         \-+- 36288 kats git cinnabar git2hg 964843eda68b06b25744912d89a2dff880490539
 | |           \-+- 36289 kats /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python /Users/kats/.mozbuild/git-cinnabar/git-cinnabar git2hg 964843eda68b06b
 | |             |--- 36294 kats /Users/kats/.mozbuild/git-cinnabar/git-cinnabar-helper --import
 | |             \--- 36295 kats (git)

I attached to pid 36294 using lldb and dumped a backtrace:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff7e6281ba libsystem_kernel.dylib`__read_nocancel + 10
    frame #1: 0x00007fff7e57de2b libsystem_c.dylib`_sread + 16
    frame #2: 0x00007fff7e57d4a3 libsystem_c.dylib`__srefill1 + 24
    frame #3: 0x00007fff7e579937 libsystem_c.dylib`getdelim + 251
    frame #4: 0x0000000100119718 git-cinnabar-helper`strbuf_getwholeline + 136
    frame #5: 0x0000000100119803 git-cinnabar-helper`strbuf_getline + 19
    frame #6: 0x0000000100011ca7 git-cinnabar-helper`main + 10743
    frame #7: 0x0000000100000a04 git-cinnabar-helper`start + 52

Looks like maybe it's stuck waiting for input on stdin?

Flags: needinfo?(mh+mozilla)

... and detaching gdb got it unstuck and moz-phab completed successfully.

Just a theory, but it might be that stdout needs flushing after writing the newline, here.

(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #12)

Just a theory, but it might be that stdout needs flushing after writing the newline, here.

I doubt it. Would you be able to get a python stack trace for the git-cinnabar process when it's stuck?

Flags: needinfo?(mh+mozilla)

So far I haven't run into this again. I'll reopen with a python stack trace if I hit it again.

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WORKSFORME

I was able to reproduce it. It's a problem in git-cinnabar. Filed as https://github.com/glandium/git-cinnabar/issues/244

You need to log in before you can comment on or make changes to this bug.