1271622 - migrate valgrind build to use mozharness

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Kim Moir [:kmoir] ET]

currently the valgrind build is invoked via a shell script.

https://dxr.mozilla.org/build-central/source/tools/scripts/valgrind/valgrind.sh

It would be good if we could use mozharness instead which would make migrating this build to taskcluster much easier.

Comment 1

[Kim Moir [:kmoir] ET]

perhaps something like how the asan build is currently invoked but modified for valgrind, for example this is how asan is invoked
http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux64-asan-debug/1462883587/mozilla-inbound-linux64-asan-debug-bm74-build1-build1044.txt.gz

You can look at the details for any asan build here
Linux x64 asan

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound

mozharness is a generalized set of scripts for common build tasks like checking out code, uploading artifacts etc.  The scripts are in tree, for example

http://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness

Comment 2

[Kim Moir [:kmoir] ET]

In this build log

http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux64-asan/1462891572/mozilla-inbound-linux64-asan-bm72-build1-build1334.txt.gz

mozharness is invoked like this if you search in the above log for the asan build, you will need to make a new one for the valgrind one and test

tools/buildbot/bin/python scripts/scripts/fx_desktop_build.py --config builds/releng_base_linux_64_builds.py --custom-build-variant-cfg asan --config balrog/production.py --branch mozilla-inbound --build-pool production

Comment 3

[Kim Moir [:kmoir] ET]

Probably a good first step would be to copy 

testing/mozharness/configs/builds/releng_base_linux_64_builds.py and play with it with it to see how the actions run.  Then create a custom build variant for valgrind

Comment 4

[Kim Moir [:kmoir] ET]

Francis, 

further to our earlier vidyo conversation, this is how I ran the build locally without cloning the repos every time. (Obviously this is for asan instead of valgrind)

Kims-MacBook-Pro:mozharness kmoir$ pwd
/Users/kmoir/hg/mozilla-inbound/testing/mozharness


create a buildprops.json.,  You don't need all the properties in the properties.json that are there for the real build.
Kims-MacBook-Pro:mozharness kmoir$ cat buildprops.json
{"properties": {"project": "", "product": "firefox", "script_repo_revision": "production", "scheduler": "mozilla-central-firefox", "repository": "", "buildername": "Linux x86-64 mozilla-central asan build", "buildid": "20160516100347", "branch": "mozilla-central", "slavename": "dev-linux64-ec2-fkang.dev.releng.use1.mozilla.com", "master": "http://buildbot-master72.bb.releng.usw2.mozilla.com:8001"}}

modify 
testing/mozharness/configs/builds/releng_base_linux_64_builds.py
    'vcs_share_base': '/builds/hg-shared',

to something in your home dir
    'vcs_share_base': '/Users/kmoir/builds/hg-shared',


You can comment out the clobber steps etc in the file that is associated with the custom-build-variant-cfg for this case 


testing/mozharness/configs/builds/releng_sub_linux_configs/64_asan.py


@@ -1,17 +1,17 @@
 import os
 
 MOZ_OBJDIR = 'obj-firefox'
 
 config = {
     'default_actions': [
-        'clobber',
-        'clone-tools',
-        'checkout-sources',
+#        'clobber',
+#        'clone-tools',
+#        'checkout-sources',


run the script as follows

python scripts/fx_desktop_build.py --config builds/releng_base_linux_64_builds.py --custom-build-variant-cfg asan --config balrog/production.py --branch mozilla-central --build-pool production

Basically, for builds that specify --custom-build-variant-cfg have configurations here that override some of the other configuration files properties

Kims-MacBook-Pro:mozharness kmoir$ ls  configs/builds/releng_sub_linux_configs/
32_debug.py		64_asan.py		64_code_coverage.py	64_graphene.py		64_mulet.py		64_stat_and_debug.py	64_tsan.py
64_artifact.py		64_asan_and_debug.py	64_debug.py		64_horizon.py		64_source.py		64_stat_and_opt.py


See http://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/mozharness/mozilla/building/buildbase.py#l328 for more details

Comment 5

[Kim Moir [:kmoir] ET]

Attachment: buildbot_properties.json

examples files from 

http://buildbot-master94.bb.releng.use1.mozilla.com:8001/builders/Linux%20x86-64%20mozilla-central%20asan%20build/builds/0 

that can be used to run the build on your aws master

Comment 6

[Kim Moir [:kmoir] ET]

Attachment: buildprops.json

another test file for running on aws loaner

Comment 7

[Kim Moir [:kmoir] ET]

Francis, these are the steps I used to getting the asan build running on my loaner aws instance.  Obviously, you'll want to run valgrind, but it's not on mozharness yet so you can use asan as an example.  Let me know if you have questions, happy to help!

ssh to your aws dev instance as cltbld 

cd /builds/hg-shared
[e-mail] hg-shared]$ mkdir integration
[[e-mail] hg-shared]$ cd integration/
[[e-mail] integration]$ pwd
/builds/hg-shared/integration
hg clone http://hg.mozilla.org/mozilla-central

cd /builds/hg-shared/integration/mozilla-central/testing/mozharness

Copy a buildprops.json and a buildbot_properties.json from an existing build
(see attached)

As root
[[e-mail] ~]# touch /builds/gapi.data
/builds/google-oauth-api.key /builds/mozilla-api.key /builds/mozilla-desktop-geoloc-api.key /builds/mozilla-fennec-geoloc-api.key /builds/adjust-sdk.token

Copy relengapi.tok from a real build machine to the /builds/relengapi.tok this is needed to download the tool tool artifacts

Login as cltlbd again
Cd /builds/hg-shared/integration/mozilla-central/testing/mozharness
run
python scripts/fx_desktop_build.py --config builds/releng_base_linux_64_builds.py --custom-build-variant-cfg asan --config balrog/production.py --branch mozilla-central --build-pool production


To avoid clobber for future runs

You can comment out the clobber steps etc in the file that is associated with the custom-build-variant-cfg for this case 


testing/mozharness/configs/builds/releng_sub_linux_configs/64_asan.py


@@ -1,17 +1,17 @@
 import os
 
 MOZ_OBJDIR = 'obj-firefox'
 
 config = {
     'default_actions': [
-        'clobber',
-        'clone-tools',
-        'checkout-sources',
+#        'clobber',
+#        'clone-tools',
+#        'checkout-sources',


This shouldn’t really matter if you are running a test with the same revision

The build will fail when uploading artifacts to s3 because the credentials aren’t there.  You can comment them out

Comment 8

[Francis Kang]

Attachment: Build log for valgrind build

Comment 9

[Francis Kang]

Attachment: Mozharness log from running mach valgrind-test

Comment 10

[Francis Kang]

I ran a build and test run on TaskCluster, and it seems to have worked ok : https://tools.taskcluster.net/task-inspector/#Eh43tMd4QdCsCp0NGTV1Sg/0

The build was invoked with the in-tree configurations to activate valgrind options, and the test was invoked by adding a new mozharness action in mozilla/building/buildbase.py

Comment 11

[Francis Kang]

https://public-artifacts.taskcluster.net/DP18cpWISEKPBEe8_BMdtg/0/public/logs/live_backing.log

The buildlog for a mozharness valgrind build on tc. It corresponds to a moz-inbound revision that failed on treeherder, but still passed when run on tc.

Comment 12

[Francis Kang]

Attachment: Proposed patch to add valgrind support to mozharness, for use by taskcluster.

Comment 13

[Kim Moir [:kmoir] ET]

francis, not sure if you are ready for feedback/review yet or you are going to try to run this on try first, let me know, happy to help!

Comment 14

[Francis Kang]

Attachment: Patch to add valgrind support to mozharness, and to execute it on taskcluster.

Comment 15

[Francis Kang]

Comment on attachment 8758875 [details] [diff] [review]
Patch to add valgrind support to mozharness, and to execute it on taskcluster.

try run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e52e001f4d20

Comment 16

[Francis Kang]

Attachment: MozReview Request: Add mozharness valgrind script

Review commit: https://reviewboard.mozilla.org/r/56990/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/56990/

Comment 17

[Francis Kang]

Attachment: MozReview Request: Bug 1271622: Add taskcluster taskgraphs for linux64-valgrind build

The graph executes the mozharness script with the custom build variant 'valgrind',
and executes mach's valgrind-test via mozharness.

Review commit: https://reviewboard.mozilla.org/r/56992/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/56992/

Comment 18

[Kim Moir [:kmoir] ET]

Comment on attachment 8758886 [details]
MozReview Request: Add mozharness valgrind script

https://reviewboard.mozilla.org/r/56990/#review53972

Do we need to submit updates for valgrind builds in 

    testing/mozharness/configs/builds/releng_sub_linux_configs/64_valgrind.py
  the update action

Comment 19

[Kim Moir [:kmoir] ET]

Comment on attachment 8758887 [details]
MozReview Request: Bug 1271622: Add taskcluster taskgraphs for linux64-valgrind build

https://reviewboard.mozilla.org/r/56992/#review53974

Looks good.  However, I think valgrind builds are opt not debug so if you could refactor your patches to reflect that it would be great.

Comment 20

[Francis Kang]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2832672bb2b8

Comment 21

[Francis Kang]

Attachment: Bug 1271622: Add taskcluster taskgraphs for linux64-valgrind build

Change valgrind build from debug to opt

Review commit: https://reviewboard.mozilla.org/r/57220/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/57220/

Comment 22

[Francis Kang]

https://reviewboard.mozilla.org/r/56990/#review53972

Nope. Removed. Similarly, do we need to include build stats for a valgrind test?

Comment 23

[Francis Kang]

Attachment: Bug 1271622 - Remove update step from mozharness and tc

The mozharness configuration file for valgrind builds, and the
tc taskdef no longer try to submit to balrog, as valgrind builds
arent published.

Review commit: https://reviewboard.mozilla.org/r/57230/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/57230/

Comment 24

[Francis Kang]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6fe9c5a00595

Comment 25

[Kim Moir [:kmoir] ET]

Comment on attachment 8759189 [details]
Bug 1271622: Add taskcluster taskgraphs for linux64-valgrind build

https://reviewboard.mozilla.org/r/57220/#review54002

Looks good, obviously we need to land the mh stuff first before this.

Comment 26

[Kim Moir [:kmoir] ET]

https://reviewboard.mozilla.org/r/57230/#review54004

You commented out generate build stats too when I think you just want to comment out the updates part

Comment 27

[Francis Kang]

https://reviewboard.mozilla.org/r/57230/#review54004

We don't do it for asan or debug builds though. Is there are particular reason we do it for valgrind?

Comment 28

[Kim Moir [:kmoir] ET]

https://reviewboard.mozilla.org/r/57230/#review54024

You commented out 
 
 #'generate-build-stats',
 #'update',
 
 in 64_valgrind.py but the yml file still references generate-build-stats. I think you just wanted to remove the update?

Comment 29

[Francis Kang]

https://reviewboard.mozilla.org/r/57230/#review54024

Ah that was intentional; I should have made it clear. 

My understanding was that generate-build-stats is only run in automation as it uploads the statistics back to treeherder. Whereas when mozharness is used manually (and thus use the default actions) it wouldn't have access to whatever credentials are required.

Comment 30

[Kim Moir [:kmoir] ET]

Could you clarify what you mean by "run moharness manually"?

In any case, 

opt_linux64_valgrind.yml still lists generate-build-stats as an action in your patch

Comment 31

[Francis Kang]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0d03fd5a11c1

Comment 32

[Kim Moir [:kmoir] ET]

Comment on attachment 8759201 [details]
Bug 1271622 - Remove update step from mozharness and tc

https://reviewboard.mozilla.org/r/57230/#review54076

Let's land these tomorrow morning.

Comment 33

[Francis Kang]

Attachment: Patch for mozharness and tc to handle valgrind builds

Comment 34

[Kim Moir [:kmoir] ET]

Attachment: valgrind.patch

actual patch that landed, previous one was bitrotten from other patches that landed recently

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7f5e5e52d06f
migrate valgrind build to use mozharness r=kmoir

Comment 36

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/7f5e5e52d06f