Open Bug 819003 Opened 7 years ago Updated 5 years ago

compile-test-debug iteration for C++ (and prob xpcshell) tests very slow because pushing 650M debug libxul to android takes quite a while

Categories

(Testing :: Mozbase, defect)

x86
macOS
defect
Not set

Tracking

(Not tracked)

REOPENED

People

(Reporter: dmose, Unassigned)

Details

(Whiteboard: [android-dev-pain])

I haven't timed it; I'd guess 5-10 minutes.  Making it significantly faster could save a lot of engineering time, both per person and in the aggregate.

ekr turned me on to rdiff, a binary patching tool, <http://linux.die.net/man/1/rdiff>.  It's part of the rdiff-backup package, which is somehow related to rsync.

If this makes it significantly faster, it'd rock to have mozbase "adb push" a static copy of this somewhere (/data/local/tests/bin?) and use this when pushing over files.
I think the saner choice here would probably just be to push stripped binaries, no? Most of our test harnesses only run against the test package, which already has stripped binaries. This is an artifact of running against the objdir, which has unstripped binaries.
Stripped binaries aren't debuggable, which would seem to defeat the purpose.
Usually for debugging on mobile you use something like gdbserver, and load the symbols on your desktop machine. JimDB knows how to do most of this.
Oooh, if gdbserver doesn't need access to the symbols, that could help a bunch.
Whiteboard: [android-pain]
Whiteboard: [android-pain] → [android-dev-pain]
Good call, Ted: indeed gdbserver doesn't need to have access to the symbols. I tried stripping the symbols off of the copy of libxul that got pushed, which made it 57 megs and dropped the push time down to 12.5 seconds.  Debugging it with NDK gdb/gdbserver pair works, though that debugger seems to have unreliable breakpoints.(*)

My assumption is that code that makes separate stripped copies of shared libraries wants to live in mozbase, since it's likely to be shared across a variety of scripts, and if that's true, maybe that's really the fix for this bug.  Thoughts?

(*)I have yet to make unit test debugging work with JimDB and it's gdbserver; I've filed bug 818301 and bug 819052.
That sounds pretty reasonable, since you'd hit this same issue running xpcshell tests as well. (Not mochitest/reftest, since they'd use the .apk for running tests, which is packaged and thus stripped.)

Ideally you would cache the stripped binaries somewhere so that you don't have to repeat the process if you rerun the tests without rebuilding, since stripping does take a little bit of time.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WONTFIX
After discussing on irc, reopening with a more descriptive title. This issue is still present with the xpcshell tests, and we should fix it one way or another (probably by stripping libxul.so).
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
Summary: compile-test-debug iteration very slow because pushing 650M debug libxul to android takes quite a while → compile-test-debug iteration for xpcshell tests very slow because pushing 650M debug libxul to android takes quite a while
Summary: compile-test-debug iteration for xpcshell tests very slow because pushing 650M debug libxul to android takes quite a while → compile-test-debug iteration for C++ (and prob xpcshell) tests very slow because pushing 650M debug libxul to android takes quite a while
When I run xpcshell tests via the xpcshell-tests-remote make target, it uses stripped libs.

This code selects the lib location, unless a lib location has been specified:

https://hg.mozilla.org/mozilla-central/file/47713070c007/testing/xpcshell/remotexpcshelltests.py#l379

That uses <objdir>/dist/fennec if present, which contains stripped libs by default.

I'll see if the unit tests can use the same logic.
This will likely break unless someone has run "make package" then, you won't have dist/fennec present. Worse, if you rebuild without repackaging, you'll be running against old libraries.
Right - I always "make package" or use one of the convenience make targets like build_and_deploy. 

Are we saying we would prefer to strip libs, if necessary, at test time?
You need to log in before you can comment on or make changes to this bug.