1049798 - Create Linux64 code coverage builds on Cedar

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

From https://bugzilla.mozilla.org/show_bug.cgi?id=890116#c1

Linux/Linux64:
1. CFLAGS, CXXFLAGS need -fprofile-arcs -ftest-coverage
2. LDFLAGS needs -fprofile-arcs -ftest-coverage -lgcov
3. Upload all .gcno files from the build dir to a package somehow. Directory structure needs preservation.

Comment 1

[Andrew Halberstadt [:ahal]]

I found http://mxr.mozilla.org/build/source/buildbot-configs/mozilla2/linux64/mozilla-central/codecoverage/mozconfig from when we used to run cc builds. Is that mozconfig still relevant? I see at least LDFLAGS needs to be updated.

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: (buildbot-configs) first WIP

Here's my first stab at the buildbot-configs side of things. I copied most of the configuration over and probably got a lot of it wrong. I'll need some feedback from Armen or someone in releng.

For now I figured we should get them running on Cedar first. I don't know if the way I did it is the best way, but it looks like it'll work. Also I'll need to land the in-tree mozconfig first.

Armen, do you mind taking a quick look at this to see if I did something obviously wrong?

Comment 3

[Andrew Halberstadt [:ahal]]

Also this will schedule builds on every push, which we'll need at least on Cedar to get all the tests going. Once this is ready to go live everywhere, I'll need to figure out how to make it periodic.

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: (m-c) Add in-tree code-coverage mozconfig

Basic mozconfig, same as linux64 except exports the proper environment variables.

Comment 5

[Joshua Cranmer [:jcranmer]]

Comment on attachment 8470178 [details] [diff] [review]
(m-c) Add in-tree code-coverage mozconfig

My code-coverage-on-try builds have been using the following additions, fwiw:
ac_add_options --enable-optimize # Debug builds time out
export CFLAGS="--coverage"
export CXXFLAGS="--coverage"
export LDFLAGS="--coverage"
export HOST_LDFLAGS="--coverage" # NSPR Linux32 fix judging from comments

Comment 6

[Andrew Halberstadt [:ahal]]

Actually I've been looking at how the new mozharness build scripts work, and I think it would be easiest to just set the environment in configuration there, rather than make a new mozconfig (though we can certainly do both if needed).

Comment 7

[Andrew Halberstadt [:ahal]]

From a chat in #releng it appears the upload step happens within the build system (i.e in-tree). So I'll file a follow up for that once these builds are scheduled on Cedar. Chat pasted below:

<ahal> so with the new mozharness build scripts, does the uploading of the installer/tests to ftp.m.o happen in mozharness? Or is that done by buildbot
<jlund> ahal: so 'mach build' does almost all the post build things :)
<mshal> well, we haven't enabled the 'mach build'-does-all-the-things yet in m-c :)
<mshal> if it's a new step entirely we'll want to add it to moz-automation.mk
<ahal> mshal: I'm trying to get code coverage builds stood up.. after the build we'll need to walk the directory tree, collect all .gcov files, zip them up and upload them to the build dir
<mshal> ahal: ahh. Well you might want to look at the top-level Makefile.in at a target like buildsymbols
<mshal> ahal: and basically create a new target that does the steps you want. Then we can work that into the automation
<mshal> ahal: you can see the output from buildsymbols (SYMBOL_ARCHIVE_BASENAME.zip) gets added to UPLOAD_FILES in m-c/toolkit/mozapps/installer/packager.mk. You'd probably want the same with your gcov zip file if you want it in the same place
<mshal> (same place on ftp, I mean)

Comment 8

[Armen [:armenzg]]

Comment on attachment 8470142 [details] [diff] [review]
(buildbot-configs) first WIP

Review of attachment 8470142 [details] [diff] [review]:
-----------------------------------------------------------------

It doesn't look like you're doing anything crazy.

::: mozilla/config.py
@@ +765,5 @@
> +
> +            'product_name': 'firefox',
> +            'unittest_platform': 'linux64-cc-opt',
> +            'app_name': 'browser',
> +            'base_name': 'Linux x86-64 %(branch)s',

I think you have to add CC to distinct it from the linux64 builds.

Comment 9

[Andrew Halberstadt [:ahal]]

Attachment: (buildbot-configs) Schedule code coverage builds on Cedar

Hey Ben, feel free to re-direct the review if appropriate. I just copied a lot of the options from elsewhere and don't really understand what they all do.. so please look extra carefully :).

This patch still depends on me landing the "code-coverage" mozconfig on Cedar, will do that in a bit.

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 8470844 [details] [diff] [review]
(buildbot-configs) Schedule code coverage builds on Cedar

I probably can't look at this soon. Jordan, do you feel comfortable reviewing this?

Comment 11

[Jordan Lund (:jlund)]

Comment on attachment 8470844 [details] [diff] [review]
(buildbot-configs) Schedule code coverage builds on Cedar

Review of attachment 8470844 [details] [diff] [review]:
-----------------------------------------------------------------

For not being familiar, you seem to know what you're doing :)

lgtm. This should achieve what you want. If we are going to use mozharness only for doing the builds, we don't actually need most of these config items. They exist in other platforms as we support our legacy buildbot factory way and the new mozharness way.

Mozharness needs things like 'mozharness_desktop_build', 'reboot_command', and 'mozharness_python' and that's pretty much it. Unfortunately, there are things that misc.py and our schedulers need to know: e.g., 'profiled_build', 'base_name', 'slaves' for figuring out what our Builders are made up of.

I can help with this if you like. I'm also OK with landing this and cleaning it up when we address the other platforms after we stop supporting both.

How's the mozharness side of things going? do you need a hand with setting up the config file? The '--custom-build-variant-cfg code-coverage' shortname impl might be a bit confusing but it doesn't need much to stand up. see http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/building/buildbase.py#235 for reference. :)

Comment 12

[Andrew Halberstadt [:ahal]]

(In reply to Jordan Lund (:jlund) from comment #11)
> For not being familiar, you seem to know what you're doing :)

Heh, I mostly just copied Armen's work in the mulet bug.

> lgtm. This should achieve what you want. If we are going to use mozharness
> only for doing the builds, we don't actually need most of these config
> items. They exist in other platforms as we support our legacy buildbot
> factory way and the new mozharness way.

Ok, I certainly don't mind cleaning it up prior to landing if that will make it easier for you later.

> How's the mozharness side of things going? do you need a hand with setting
> up the config file? The '--custom-build-variant-cfg code-coverage' shortname
> impl might be a bit confusing but it doesn't need much to stand up. see
> http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/building/
> buildbase.py#235 for reference. :)

Would you prefer a different shortname? So my understanding is I basically just need to check in a config file into configs/builds/releng_sub_linux_configs and that is almost all that is needed on the mozharness side? I'm guessing that anything defined in this new config should get removed from buildbot-configs? I'll take a look at what some of the other builds are doing and try to copy it.

Comment 13

[Jordan Lund (:jlund)]

> 
> Ok, I certainly don't mind cleaning it up prior to landing if that will make
> it easier for you later.

cool, it all has to be done anyway so I'm fine if you leave it.

> Would you prefer a different shortname? So my understanding is I basically
> just need to check in a config file into
> configs/builds/releng_sub_linux_configs and that is almost all that is
> needed on the mozharness side?

shortname WFM

and yes, you add the config to that specific spot with the naming pattern you define in http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/building/buildbase.py#235

the platform and arch will be figured out for you.

tip: if you want to see what your releng_sub_linux_configs/{config}.py will do, you can always do something like:

python scripts/fx_desktop_build.py --config builds/releng_base_linux_32_builds.py --custom-build-variant-cfg code-coverage --branch cedar --build-pool production --dump-config

or for a config breakdown of what items come from what:

python scripts/fx_desktop_build.py --config builds/releng_base_linux_32_builds.py --custom-build-variant-cfg code-coverage --branch cedar --build-pool production --dump-config-hierarchy

--dump-config-hierarchy and --dump-config should work on any script and they will cause 0 actions to actually be run.

> I'm guessing that anything defined in this
> new config should get removed from buildbot-configs?

I wish that were 100% true. There may be some values that will be new and specific to mozharness, some values that are based off a buildbot factory equivalent but buildbot still needs them too for other reasons (scheduler), and some values that are used by buildbot but not by mozharness as they could be old or used for a different build.

It won't be easy untangling the bag of snakes. One thing you could do is look at the init invocations of NightlyBuildFactory/TryBuildFactory/ReleaseBuildFactory/RepackBuildFactory in misc.py, and if your platform items from mozilla/config.py is only used as init args in those factories, those items can be removed from buildbot-configs

Those factories are whats being replaced by mozharness. Again, leave to me if you're more comfortable.

Comment 14

[Andrew Halberstadt [:ahal]]

Attachment: (mozharness) Add code coverage config to fx_desktop_build.py

This is almost identical to the 64_mulet.py config (except with src_mozconfig and the shortname changed).

I think I will take you up on your offer to leave the buildbot-configs/mozharness duplicate configuration to you :p.

Comment 15

[Andrew Halberstadt [:ahal]]

Attachment: Output of --dump-config-hierarchy

For the record I ran that command you suggested (using a random buildprops.json pulled out of Cedar). Here is the output, though I'm not sure how to interpret that to see if it is correct or not.

Comment 16

[Andrew Halberstadt [:ahal]]

Landed the mozconfig on Cedar:
https://hg.mozilla.org/projects/cedar/rev/be1ff6a6d827

Comment 17

[Andrew Halberstadt [:ahal]]

This bug kind of turned into getting this going on Cedar. I'll file a follow-up for running it periodically across all branches once we have all the tests going as well.

Comment 18

[Jordan Lund (:jlund)]

(In reply to Andrew Halberstadt [:ahal] from comment #17)
> This bug kind of turned into getting this going on Cedar. I'll file a
> follow-up for running it periodically across all branches once we have all
> the tests going as well.

so just a heads up. mozharness builds are only run on cedar right now. I am aiming for trunk branches by the end of next week. If you want to move faster than that, we will need to let this run from buildbot like we do with other platforms.

I'll review the patch by EOD today.

Comment 19

[Jordan Lund (:jlund)]

Comment on attachment 8472399 [details] [diff] [review]
(mozharness) Add code coverage config to fx_desktop_build.py

Review of attachment 8472399 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with removal of tool_manifest_src item altogether and confirmation that we want to do unittests on this build.

I think this looks good. Since it's only going to work on cedar, we can iterate on it there :)

::: configs/builds/releng_sub_linux_configs/64_code_coverage.py
@@ +12,5 @@
> +        # 'generate-build-stats',
> +        'update',  # decided by query_is_nightly()
> +    ],
> +    'stage_platform': 'linux64-cc',
> +    'tooltool_manifest_src': "browser/config/tooltool-manifests/linux64/\

whoops. my bad. this should be 'browser/config/tooltool-manifests/linux64/releng.manifest' (which will come --cfg releng_base_linux_32_builds.py so we don't even need this item in this config). Thanks for pointing that out, I'll fix the mulet equiv.

@@ +16,5 @@
> +    'tooltool_manifest_src': "browser/config/tooltool-manifests/linux64/\
> +asan.manifest",
> +    'platform_supports_post_upload_to_latest': False,
> +    'enable_signing': False,
> +    'enable_talos_sendchange': False,

sanity check: this means that we won't do talos jobs based off this build but we will do unittests. That correct?

Comment 20

[Andrew Halberstadt [:ahal]]

Attachment: (mozharness) Add code coverage config to fx_desktop_build.py

Updated patch to remove the tooltool_manifest_src config. Carry r+ forward.

Yes, that is correct. The goal is to run all unittests, but no perf tests (as those would give false positives in terms of code coverage).

Comment 21

[Andrew Halberstadt [:ahal]]

Comment on attachment 8472952 [details] [diff] [review]
(mozharness) Add code coverage config to fx_desktop_build.py

https://hg.mozilla.org/build/mozharness/rev/f3df8c11524b

Comment 22

[Andrew Halberstadt [:ahal]]

Comment on attachment 8470844 [details] [diff] [review]
(buildbot-configs) Schedule code coverage builds on Cedar

https://hg.mozilla.org/build/buildbot-configs/rev/5e1d83d6e5ee

Comment 23

[Andrew Halberstadt [:ahal]]

(In reply to Jordan Lund (:jlund) from comment #18) 
> so just a heads up. mozharness builds are only run on cedar right now. I am
> aiming for trunk branches by the end of next week. If you want to move
> faster than that, we will need to let this run from buildbot like we do with
> other platforms.

Thanks for the heads up, that timeline works fine with me. I'll need to spend some time getting the right set of unittests running, as well as make some core build config changes to handle the uploading of the .gcno files. This won't get done until some time next week anyway.

Comment 24

[Andrew Halberstadt [:ahal]]

Hmm, the buildbot-config patch seems to be merged to production, but isn't showing up on Cedar (somewhat predictably):
https://tbpl.mozilla.org/?tree=Cedar&rev=aba55d017636

Comment 25

[Andrew Halberstadt [:ahal]]

Oh my mistake, it is running, but it's showing up under Linux x64 Opt. Is this a tbpl change that's needed?

Comment 26

[Andrew Halberstadt [:ahal]]

Ok, it's building. I'll file a follow up to change the symbol to CC or put it on a new line or something.

Comment 27

[Jonathan Griffin (:jgriffin)]

(In reply to Andrew Halberstadt [:ahal] from comment #25)
> Oh my mistake, it is running, but it's showing up under Linux x64 Opt. Is
> this a tbpl change that's needed?

Indeed.  TBPL will default to a generic "Linux x64" build if none of the more specific linux64 build regexes are matched.

Comment 28

[Armen [:armenzg]]

I see code coverage *debug* test jobs but there is no debug build job to trigger it.

Should I remove them?

Comment 29

[Andrew Halberstadt [:ahal]]

If you like. Eventually we might have code coverage debug builds, but we can add them back in when the time comes.