946002 - Run Valgrind builds on every push

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Nicholas Nethercote [inactive]]

For Valgrind builds to be unhidden, they need to run on every push.

Comment 1

[Nicholas Nethercote [inactive]]

Additional info:  they only need to run on Linux64, which is our most scalable platform.  They currently take about 45 minutes.

Comment 2

[Chris AtLee [:catlee]]

Attachment: valgrind-buildbotcustom.diff

do valgrind builds per push rather than nightly

Comment 3

[Chris AtLee [:catlee]]

Comment on attachment 8343715 [details] [diff] [review]
valgrind-buildbotcustom.diff

https://hg.mozilla.org/build/buildbotcustom/rev/c6536c16d218

this will take effect after the next reconfig

Comment 4

[Nicholas Nethercote [inactive]]

Thanks, catlee!  I give you that gold star now :)

> this will take effect after the next reconfig

When will that be, and is there a way to be notified?

Comment 5

[Nicholas Nethercote [inactive]]

> They currently take about 45 minutes.

FWIW, I did some measurements (with debugging code on the TBPL machines) and found that about 38 minutes of that is setup and build time, and only about 7 minutes is the actual Valgrind run.

Comment 6

[[e-mail] (:bhearsum)]

In production

Comment 7

[Nicholas Nethercote [inactive]]

\o/

Comment 8

[Chris AtLee [:catlee]]

we ended up busting the trees because these jobs time out and leave processes running, and don't reboot. Landed http://hg.mozilla.org/build/buildbotcustom/rev/9c71ca2b0e0b to try and fix.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Specifically:

[[e-mail] ~]$ ps uxwww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
cltbld    1412 17.0  6.4 1459264 990024 ?      Sl   16:28   5:19 /usr/bin/valgrind --error-exitcode=1 --smc-check=all-non-file --gen-suppressions=all --leak-check=full --num-callers=15 --show-possibly-lost=no --track-origins=yes --suppressions=/builds/slave/fx-team-l64-valgrind-000000000/objdir/_valgrind/cross-architecture.sup --suppressions=/builds/slave/fx-team-l64-valgrind-000000000/objdir/_valgrind/x86_64-redhat-linux-gnu.sup /builds/slave/fx-team-l64-valgrind-000000000/objdir/dist/firefox/firefox http://localhost:8888/index.html -profile /tmp/tmpB7OAsw

Comment 10

[Nicholas Nethercote [inactive]]

I'm investigating the timeouts in bug 948145.  They only started happening yesterday, I don't know why.

Comment 11

[Hal Wine [:hwine] use NI!]

in production