The default bug view has changed. See this FAQ.

migrate valgrind build to use mozharness

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
11 months ago
10 months ago

People

(Reporter: kmoir, Assigned: Francis Kang)

Tracking

unspecified

Firefox Tracking Flags

(firefox49 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(10 attachments, 2 obsolete attachments)

5.96 KB, text/plain
Details
159.32 KB, text/plain
Details
9.19 MB, text/plain
Details
45.49 KB, text/plain
Details
58 bytes, text/x-review-board-request
Details | Review
58 bytes, text/x-review-board-request
Details | Review
58 bytes, text/x-review-board-request
kmoir
: review+
Details | Review
58 bytes, text/x-review-board-request
kmoir
: review+
Details | Review
7.81 KB, patch
Details | Diff | Splinter Review
7.81 KB, patch
kmoir
: checked-in+
Details | Diff | Splinter Review
(Reporter)

Description

11 months ago
currently the valgrind build is invoked via a shell script.

https://dxr.mozilla.org/build-central/source/tools/scripts/valgrind/valgrind.sh

It would be good if we could use mozharness instead which would make migrating this build to taskcluster much easier.
(Reporter)

Comment 1

11 months ago
perhaps something like how the asan build is currently invoked but modified for valgrind, for example this is how asan is invoked
http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux64-asan-debug/1462883587/mozilla-inbound-linux64-asan-debug-bm74-build1-build1044.txt.gz

You can look at the details for any asan build here
Linux x64 asan

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound

mozharness is a generalized set of scripts for common build tasks like checking out code, uploading artifacts etc.  The scripts are in tree, for example

http://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness
(Reporter)

Updated

11 months ago
Assignee: nobody → fkang
(Reporter)

Comment 2

11 months ago
In this build log

http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux64-asan/1462891572/mozilla-inbound-linux64-asan-bm72-build1-build1334.txt.gz

mozharness is invoked like this if you search in the above log for the asan build, you will need to make a new one for the valgrind one and test

tools/buildbot/bin/python scripts/scripts/fx_desktop_build.py --config builds/releng_base_linux_64_builds.py --custom-build-variant-cfg asan --config balrog/production.py --branch mozilla-inbound --build-pool production
(Reporter)

Comment 3

11 months ago
Probably a good first step would be to copy 

testing/mozharness/configs/builds/releng_base_linux_64_builds.py and play with it with it to see how the actions run.  Then create a custom build variant for valgrind
(Reporter)

Comment 4

11 months ago
Francis, 

further to our earlier vidyo conversation, this is how I ran the build locally without cloning the repos every time. (Obviously this is for asan instead of valgrind)

Kims-MacBook-Pro:mozharness kmoir$ pwd
/Users/kmoir/hg/mozilla-inbound/testing/mozharness


create a buildprops.json.,  You don't need all the properties in the properties.json that are there for the real build.
Kims-MacBook-Pro:mozharness kmoir$ cat buildprops.json
{"properties": {"project": "", "product": "firefox", "script_repo_revision": "production", "scheduler": "mozilla-central-firefox", "repository": "", "buildername": "Linux x86-64 mozilla-central asan build", "buildid": "20160516100347", "branch": "mozilla-central", "slavename": "dev-linux64-ec2-fkang.dev.releng.use1.mozilla.com", "master": "http://buildbot-master72.bb.releng.usw2.mozilla.com:8001"}}

modify 
testing/mozharness/configs/builds/releng_base_linux_64_builds.py
    'vcs_share_base': '/builds/hg-shared',

to something in your home dir
    'vcs_share_base': '/Users/kmoir/builds/hg-shared',


You can comment out the clobber steps etc in the file that is associated with the custom-build-variant-cfg for this case 


testing/mozharness/configs/builds/releng_sub_linux_configs/64_asan.py


@@ -1,17 +1,17 @@
 import os
 
 MOZ_OBJDIR = 'obj-firefox'
 
 config = {
     'default_actions': [
-        'clobber',
-        'clone-tools',
-        'checkout-sources',
+#        'clobber',
+#        'clone-tools',
+#        'checkout-sources',


run the script as follows

python scripts/fx_desktop_build.py --config builds/releng_base_linux_64_builds.py --custom-build-variant-cfg asan --config balrog/production.py --branch mozilla-central --build-pool production

Basically, for builds that specify --custom-build-variant-cfg have configurations here that override some of the other configuration files properties

Kims-MacBook-Pro:mozharness kmoir$ ls  configs/builds/releng_sub_linux_configs/
32_debug.py		64_asan.py		64_code_coverage.py	64_graphene.py		64_mulet.py		64_stat_and_debug.py	64_tsan.py
64_artifact.py		64_asan_and_debug.py	64_debug.py		64_horizon.py		64_source.py		64_stat_and_opt.py


See http://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/mozharness/mozilla/building/buildbase.py#l328 for more details
(Reporter)

Updated

11 months ago
Depends on: 1273666
(Reporter)

Updated

11 months ago
No longer depends on: 1273666
(Reporter)

Comment 5

11 months ago
Created attachment 8754411 [details]
buildbot_properties.json

examples files from 

http://buildbot-master94.bb.releng.use1.mozilla.com:8001/builders/Linux%20x86-64%20mozilla-central%20asan%20build/builds/0 

that can be used to run the build on your aws master
(Reporter)

Comment 6

11 months ago
Created attachment 8754416 [details]
buildprops.json

another test file for running on aws loaner
(Reporter)

Comment 7

11 months ago
Francis, these are the steps I used to getting the asan build running on my loaner aws instance.  Obviously, you'll want to run valgrind, but it's not on mozharness yet so you can use asan as an example.  Let me know if you have questions, happy to help!

ssh to your aws dev instance as cltbld 

cd /builds/hg-shared
cltbld@dev-linux64-ec2-kmoir.dev.releng.use1.mozilla.com hg-shared]$ mkdir integration
[cltbld@dev-linux64-ec2-kmoir.dev.releng.use1.mozilla.com hg-shared]$ cd integration/
[cltbld@dev-linux64-ec2-kmoir.dev.releng.use1.mozilla.com integration]$ pwd
/builds/hg-shared/integration
hg clone http://hg.mozilla.org/mozilla-central

cd /builds/hg-shared/integration/mozilla-central/testing/mozharness

Copy a buildprops.json and a buildbot_properties.json from an existing build
(see attached)

As root
[root@dev-linux64-ec2-kmoir.dev.releng.use1.mozilla.com ~]# touch /builds/gapi.data
/builds/google-oauth-api.key /builds/mozilla-api.key /builds/mozilla-desktop-geoloc-api.key /builds/mozilla-fennec-geoloc-api.key /builds/adjust-sdk.token

Copy relengapi.tok from a real build machine to the /builds/relengapi.tok this is needed to download the tool tool artifacts

Login as cltlbd again
Cd /builds/hg-shared/integration/mozilla-central/testing/mozharness
run
python scripts/fx_desktop_build.py --config builds/releng_base_linux_64_builds.py --custom-build-variant-cfg asan --config balrog/production.py --branch mozilla-central --build-pool production


To avoid clobber for future runs

You can comment out the clobber steps etc in the file that is associated with the custom-build-variant-cfg for this case 


testing/mozharness/configs/builds/releng_sub_linux_configs/64_asan.py


@@ -1,17 +1,17 @@
 import os
 
 MOZ_OBJDIR = 'obj-firefox'
 
 config = {
     'default_actions': [
-        'clobber',
-        'clone-tools',
-        'checkout-sources',
+#        'clobber',
+#        'clone-tools',
+#        'checkout-sources',


This shouldn’t really matter if you are running a test with the same revision

The build will fail when uploading artifacts to s3 because the credentials aren’t there.  You can comment them out
(Assignee)

Comment 8

10 months ago
Created attachment 8756383 [details]
Build log for valgrind build
(Assignee)

Comment 9

10 months ago
Created attachment 8756390 [details]
Mozharness log from running mach valgrind-test
(Assignee)

Comment 10

10 months ago
I ran a build and test run on TaskCluster, and it seems to have worked ok : https://tools.taskcluster.net/task-inspector/#Eh43tMd4QdCsCp0NGTV1Sg/0

The build was invoked with the in-tree configurations to activate valgrind options, and the test was invoked by adding a new mozharness action in mozilla/building/buildbase.py
(Assignee)

Comment 11

10 months ago
https://public-artifacts.taskcluster.net/DP18cpWISEKPBEe8_BMdtg/0/public/logs/live_backing.log

The buildlog for a mozharness valgrind build on tc. It corresponds to a moz-inbound revision that failed on treeherder, but still passed when run on tc.
(Assignee)

Comment 12

10 months ago
Created attachment 8758030 [details] [diff] [review]
Proposed patch to add valgrind support to mozharness, for use by taskcluster.
(Reporter)

Comment 13

10 months ago
francis, not sure if you are ready for feedback/review yet or you are going to try to run this on try first, let me know, happy to help!
(Assignee)

Comment 14

10 months ago
Created attachment 8758875 [details] [diff] [review]
Patch to add valgrind support to mozharness, and to execute it on taskcluster.
Attachment #8758030 - Attachment is obsolete: true
(Assignee)

Comment 15

10 months ago
Comment on attachment 8758875 [details] [diff] [review]
Patch to add valgrind support to mozharness, and to execute it on taskcluster.

try run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e52e001f4d20
Attachment #8758875 - Attachment description: Patch to add valgrind support to mozharness, and to execute it on taskcluster. try run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e52e001f4d20 → Patch to add valgrind support to mozharness, and to execute it on taskcluster.
(Assignee)

Comment 16

10 months ago
Created attachment 8758886 [details]
MozReview Request: Add mozharness valgrind script

Review commit: https://reviewboard.mozilla.org/r/56990/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/56990/
Attachment #8758886 - Flags: review?(kmoir)
Attachment #8758887 - Flags: review?(kmoir)
(Assignee)

Comment 17

10 months ago
Created attachment 8758887 [details]
MozReview Request: Bug 1271622: Add taskcluster taskgraphs for linux64-valgrind build

The graph executes the mozharness script with the custom build variant 'valgrind',
and executes mach's valgrind-test via mozharness.

Review commit: https://reviewboard.mozilla.org/r/56992/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/56992/
(Reporter)

Comment 18

10 months ago
Comment on attachment 8758886 [details]
MozReview Request: Add mozharness valgrind script

https://reviewboard.mozilla.org/r/56990/#review53972

Do we need to submit updates for valgrind builds in 

    testing/mozharness/configs/builds/releng_sub_linux_configs/64_valgrind.py
  the update action
Attachment #8758886 - Flags: review?(kmoir)
(Reporter)

Comment 19

10 months ago
Comment on attachment 8758887 [details]
MozReview Request: Bug 1271622: Add taskcluster taskgraphs for linux64-valgrind build

https://reviewboard.mozilla.org/r/56992/#review53974

Looks good.  However, I think valgrind builds are opt not debug so if you could refactor your patches to reflect that it would be great.
Attachment #8758887 - Flags: review?(kmoir)
(Assignee)

Comment 20

10 months ago
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2832672bb2b8
(Assignee)

Comment 21

10 months ago
Created attachment 8759189 [details]
Bug 1271622: Add taskcluster taskgraphs for linux64-valgrind build

Change valgrind build from debug to opt

Review commit: https://reviewboard.mozilla.org/r/57220/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/57220/
Attachment #8759189 - Flags: review?(kmoir)
(Assignee)

Comment 22

10 months ago
https://reviewboard.mozilla.org/r/56990/#review53972

Nope. Removed. Similarly, do we need to include build stats for a valgrind test?
(Assignee)

Comment 23

10 months ago
Created attachment 8759201 [details]
Bug 1271622 - Remove update step from mozharness and tc

The mozharness configuration file for valgrind builds, and the
tc taskdef no longer try to submit to balrog, as valgrind builds
arent published.

Review commit: https://reviewboard.mozilla.org/r/57230/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/57230/
Attachment #8759201 - Flags: review?(kmoir)
(Assignee)

Comment 24

10 months ago
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6fe9c5a00595
(Reporter)

Comment 25

10 months ago
Comment on attachment 8759189 [details]
Bug 1271622: Add taskcluster taskgraphs for linux64-valgrind build

https://reviewboard.mozilla.org/r/57220/#review54002

Looks good, obviously we need to land the mh stuff first before this.
Attachment #8759189 - Flags: review?(kmoir) → review+
(Reporter)

Comment 26

10 months ago
https://reviewboard.mozilla.org/r/57230/#review54004

You commented out generate build stats too when I think you just want to comment out the updates part
(Assignee)

Comment 27

10 months ago
https://reviewboard.mozilla.org/r/57230/#review54004

We don't do it for asan or debug builds though. Is there are particular reason we do it for valgrind?
(Reporter)

Comment 28

10 months ago
https://reviewboard.mozilla.org/r/57230/#review54024

You commented out 
 
 #'generate-build-stats',
 #'update',
 
 in 64_valgrind.py but the yml file still references generate-build-stats. I think you just wanted to remove the update?
(Assignee)

Comment 29

10 months ago
https://reviewboard.mozilla.org/r/57230/#review54024

Ah that was intentional; I should have made it clear. 

My understanding was that generate-build-stats is only run in automation as it uploads the statistics back to treeherder. Whereas when mozharness is used manually (and thus use the default actions) it wouldn't have access to whatever credentials are required.
(Reporter)

Comment 30

10 months ago
Could you clarify what you mean by "run moharness manually"?

In any case, 

opt_linux64_valgrind.yml still lists generate-build-stats as an action in your patch
(Assignee)

Comment 31

10 months ago
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0d03fd5a11c1
(Reporter)

Comment 32

10 months ago
Comment on attachment 8759201 [details]
Bug 1271622 - Remove update step from mozharness and tc

https://reviewboard.mozilla.org/r/57230/#review54076

Let's land these tomorrow morning.
Attachment #8759201 - Flags: review?(kmoir) → review+
(Assignee)

Comment 33

10 months ago
Created attachment 8759666 [details] [diff] [review]
Patch for mozharness and tc to handle valgrind builds
Attachment #8758875 - Attachment is obsolete: true
(Reporter)

Comment 34

10 months ago
Created attachment 8759676 [details] [diff] [review]
valgrind.patch

actual patch that landed, previous one was bitrotten from other patches that landed recently
Attachment #8759676 - Flags: checked-in+

Comment 35

10 months ago
Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7f5e5e52d06f
migrate valgrind build to use mozharness r=kmoir

Comment 36

10 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/7f5e5e52d06f
Status: NEW → RESOLVED
Last Resolved: 10 months ago
status-firefox49: --- → fixed
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.