Closed Bug 1271622 Opened 4 years ago Closed 4 years ago

migrate valgrind build to use mozharness

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(firefox49 fixed)

RESOLVED FIXED
Tracking Status
firefox49 --- fixed

People

(Reporter: kmoir, Assigned: h.franciskang)

References

Details

Attachments

(10 files, 2 obsolete files)

currently the valgrind build is invoked via a shell script.

https://dxr.mozilla.org/build-central/source/tools/scripts/valgrind/valgrind.sh

It would be good if we could use mozharness instead which would make migrating this build to taskcluster much easier.
perhaps something like how the asan build is currently invoked but modified for valgrind, for example this is how asan is invoked
http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux64-asan-debug/1462883587/mozilla-inbound-linux64-asan-debug-bm74-build1-build1044.txt.gz

You can look at the details for any asan build here
Linux x64 asan

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound

mozharness is a generalized set of scripts for common build tasks like checking out code, uploading artifacts etc.  The scripts are in tree, for example

http://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness
Assignee: nobody → fkang
In this build log

http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux64-asan/1462891572/mozilla-inbound-linux64-asan-bm72-build1-build1334.txt.gz

mozharness is invoked like this if you search in the above log for the asan build, you will need to make a new one for the valgrind one and test

tools/buildbot/bin/python scripts/scripts/fx_desktop_build.py --config builds/releng_base_linux_64_builds.py --custom-build-variant-cfg asan --config balrog/production.py --branch mozilla-inbound --build-pool production
Probably a good first step would be to copy 

testing/mozharness/configs/builds/releng_base_linux_64_builds.py and play with it with it to see how the actions run.  Then create a custom build variant for valgrind
Francis, 

further to our earlier vidyo conversation, this is how I ran the build locally without cloning the repos every time. (Obviously this is for asan instead of valgrind)

Kims-MacBook-Pro:mozharness kmoir$ pwd
/Users/kmoir/hg/mozilla-inbound/testing/mozharness


create a buildprops.json.,  You don't need all the properties in the properties.json that are there for the real build.
Kims-MacBook-Pro:mozharness kmoir$ cat buildprops.json
{"properties": {"project": "", "product": "firefox", "script_repo_revision": "production", "scheduler": "mozilla-central-firefox", "repository": "", "buildername": "Linux x86-64 mozilla-central asan build", "buildid": "20160516100347", "branch": "mozilla-central", "slavename": "dev-linux64-ec2-fkang.dev.releng.use1.mozilla.com", "master": "http://buildbot-master72.bb.releng.usw2.mozilla.com:8001"}}

modify 
testing/mozharness/configs/builds/releng_base_linux_64_builds.py
    'vcs_share_base': '/builds/hg-shared',

to something in your home dir
    'vcs_share_base': '/Users/kmoir/builds/hg-shared',


You can comment out the clobber steps etc in the file that is associated with the custom-build-variant-cfg for this case 


testing/mozharness/configs/builds/releng_sub_linux_configs/64_asan.py


@@ -1,17 +1,17 @@
 import os
 
 MOZ_OBJDIR = 'obj-firefox'
 
 config = {
     'default_actions': [
-        'clobber',
-        'clone-tools',
-        'checkout-sources',
+#        'clobber',
+#        'clone-tools',
+#        'checkout-sources',


run the script as follows

python scripts/fx_desktop_build.py --config builds/releng_base_linux_64_builds.py --custom-build-variant-cfg asan --config balrog/production.py --branch mozilla-central --build-pool production

Basically, for builds that specify --custom-build-variant-cfg have configurations here that override some of the other configuration files properties

Kims-MacBook-Pro:mozharness kmoir$ ls  configs/builds/releng_sub_linux_configs/
32_debug.py		64_asan.py		64_code_coverage.py	64_graphene.py		64_mulet.py		64_stat_and_debug.py	64_tsan.py
64_artifact.py		64_asan_and_debug.py	64_debug.py		64_horizon.py		64_source.py		64_stat_and_opt.py


See http://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/mozharness/mozilla/building/buildbase.py#l328 for more details
Depends on: 1273666
No longer depends on: 1273666
Attached file buildprops.json
another test file for running on aws loaner
Francis, these are the steps I used to getting the asan build running on my loaner aws instance.  Obviously, you'll want to run valgrind, but it's not on mozharness yet so you can use asan as an example.  Let me know if you have questions, happy to help!

ssh to your aws dev instance as cltbld 

cd /builds/hg-shared
cltbld@dev-linux64-ec2-kmoir.dev.releng.use1.mozilla.com hg-shared]$ mkdir integration
[cltbld@dev-linux64-ec2-kmoir.dev.releng.use1.mozilla.com hg-shared]$ cd integration/
[cltbld@dev-linux64-ec2-kmoir.dev.releng.use1.mozilla.com integration]$ pwd
/builds/hg-shared/integration
hg clone http://hg.mozilla.org/mozilla-central

cd /builds/hg-shared/integration/mozilla-central/testing/mozharness

Copy a buildprops.json and a buildbot_properties.json from an existing build
(see attached)

As root
[root@dev-linux64-ec2-kmoir.dev.releng.use1.mozilla.com ~]# touch /builds/gapi.data
/builds/google-oauth-api.key /builds/mozilla-api.key /builds/mozilla-desktop-geoloc-api.key /builds/mozilla-fennec-geoloc-api.key /builds/adjust-sdk.token

Copy relengapi.tok from a real build machine to the /builds/relengapi.tok this is needed to download the tool tool artifacts

Login as cltlbd again
Cd /builds/hg-shared/integration/mozilla-central/testing/mozharness
run
python scripts/fx_desktop_build.py --config builds/releng_base_linux_64_builds.py --custom-build-variant-cfg asan --config balrog/production.py --branch mozilla-central --build-pool production


To avoid clobber for future runs

You can comment out the clobber steps etc in the file that is associated with the custom-build-variant-cfg for this case 


testing/mozharness/configs/builds/releng_sub_linux_configs/64_asan.py


@@ -1,17 +1,17 @@
 import os
 
 MOZ_OBJDIR = 'obj-firefox'
 
 config = {
     'default_actions': [
-        'clobber',
-        'clone-tools',
-        'checkout-sources',
+#        'clobber',
+#        'clone-tools',
+#        'checkout-sources',


This shouldn’t really matter if you are running a test with the same revision

The build will fail when uploading artifacts to s3 because the credentials aren’t there.  You can comment them out
I ran a build and test run on TaskCluster, and it seems to have worked ok : https://tools.taskcluster.net/task-inspector/#Eh43tMd4QdCsCp0NGTV1Sg/0

The build was invoked with the in-tree configurations to activate valgrind options, and the test was invoked by adding a new mozharness action in mozilla/building/buildbase.py
https://public-artifacts.taskcluster.net/DP18cpWISEKPBEe8_BMdtg/0/public/logs/live_backing.log

The buildlog for a mozharness valgrind build on tc. It corresponds to a moz-inbound revision that failed on treeherder, but still passed when run on tc.
francis, not sure if you are ready for feedback/review yet or you are going to try to run this on try first, let me know, happy to help!
Comment on attachment 8758875 [details] [diff] [review]
Patch to add valgrind support to mozharness, and to execute it on taskcluster.

try run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e52e001f4d20
Attachment #8758875 - Attachment description: Patch to add valgrind support to mozharness, and to execute it on taskcluster. try run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e52e001f4d20 → Patch to add valgrind support to mozharness, and to execute it on taskcluster.
The graph executes the mozharness script with the custom build variant 'valgrind',
and executes mach's valgrind-test via mozharness.

Review commit: https://reviewboard.mozilla.org/r/56992/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/56992/
Comment on attachment 8758886 [details]
MozReview Request: Add mozharness valgrind script

https://reviewboard.mozilla.org/r/56990/#review53972

Do we need to submit updates for valgrind builds in 

    testing/mozharness/configs/builds/releng_sub_linux_configs/64_valgrind.py
  the update action
Attachment #8758886 - Flags: review?(kmoir)
Comment on attachment 8758887 [details]
MozReview Request: Bug 1271622: Add taskcluster taskgraphs for linux64-valgrind build

https://reviewboard.mozilla.org/r/56992/#review53974

Looks good.  However, I think valgrind builds are opt not debug so if you could refactor your patches to reflect that it would be great.
Attachment #8758887 - Flags: review?(kmoir)
Change valgrind build from debug to opt

Review commit: https://reviewboard.mozilla.org/r/57220/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/57220/
Attachment #8759189 - Flags: review?(kmoir)
https://reviewboard.mozilla.org/r/56990/#review53972

Nope. Removed. Similarly, do we need to include build stats for a valgrind test?
The mozharness configuration file for valgrind builds, and the
tc taskdef no longer try to submit to balrog, as valgrind builds
arent published.

Review commit: https://reviewboard.mozilla.org/r/57230/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/57230/
Attachment #8759201 - Flags: review?(kmoir)
Comment on attachment 8759189 [details]
Bug 1271622: Add taskcluster taskgraphs for linux64-valgrind build

https://reviewboard.mozilla.org/r/57220/#review54002

Looks good, obviously we need to land the mh stuff first before this.
Attachment #8759189 - Flags: review?(kmoir) → review+
https://reviewboard.mozilla.org/r/57230/#review54004

You commented out generate build stats too when I think you just want to comment out the updates part
https://reviewboard.mozilla.org/r/57230/#review54004

We don't do it for asan or debug builds though. Is there are particular reason we do it for valgrind?
https://reviewboard.mozilla.org/r/57230/#review54024

You commented out 
 
 #'generate-build-stats',
 #'update',
 
 in 64_valgrind.py but the yml file still references generate-build-stats. I think you just wanted to remove the update?
https://reviewboard.mozilla.org/r/57230/#review54024

Ah that was intentional; I should have made it clear. 

My understanding was that generate-build-stats is only run in automation as it uploads the statistics back to treeherder. Whereas when mozharness is used manually (and thus use the default actions) it wouldn't have access to whatever credentials are required.
Could you clarify what you mean by "run moharness manually"?

In any case, 

opt_linux64_valgrind.yml still lists generate-build-stats as an action in your patch
Comment on attachment 8759201 [details]
Bug 1271622 - Remove update step from mozharness and tc

https://reviewboard.mozilla.org/r/57230/#review54076

Let's land these tomorrow morning.
Attachment #8759201 - Flags: review?(kmoir) → review+
Attachment #8758875 - Attachment is obsolete: true
Attached patch valgrind.patchSplinter Review
actual patch that landed, previous one was bitrotten from other patches that landed recently
Attachment #8759676 - Flags: checked-in+
Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7f5e5e52d06f
migrate valgrind build to use mozharness r=kmoir
https://hg.mozilla.org/mozilla-central/rev/7f5e5e52d06f
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.