1015018 - Please enable OS X debug static analysis builds on buildbot automation

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[(no longer active)]

We've been running Linux clang static analysis builds on TBPL for a while.  My patch in bug 861425 fixes the Mac build, so we're ready to turn on the Mac debug static analysis build as well.

I can help write the mozconfig for it.  What else do we need to do?

Comment 1

[(no longer active)]

(Note to self: this is the mozconfig for the Linux version of these builds: http://mxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/linux64/debug-static-analysis-clang)

Comment 2

[(no longer active)]

I got to a green build on TBPL: <https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=2d9c0806000d>

I am adding this mozconfig in bug 1115426: browser/config/mozconfigs/macosx64/debug-static-analysis

What is the next step for us to get these builds?  I would like to enable them as soon as possible, and I can help write the releng-side changes if someone points me to what needs to be done.

Thanks!

Comment 3

[(no longer active)]

Attachment: mozharness changes

Please review mercilessly.  I have no idea what I'm doing!

Comment 4

[(no longer active)]

Attachment: buildbot-configs changes

Comment 5

[(no longer active)]

I don't know what else needs to be done...  The above is trying to replicate all of the relevant entries on <http://mxr.mozilla.org/build/search?string=st-an>.

Comment 6

[Jordan Lund (:jlund)]

Comment on attachment 8541330 [details] [diff] [review]
buildbot-configs changes

Review of attachment 8541330 [details] [diff] [review]:
-----------------------------------------------------------------

initial revision looks good. since this is un-tested, we can stand this up on try and iterate until it looks good. we should be able to do this in a 'self serve' manner. 

essentially we would need a patch similar to the mozilla/config.py changes here: https://bug1110769.bugzilla.mozilla.org/attachment.cgi?id=8535630

after that lands, we could do try pushes with a custom user mozharness repo by doing something similar to: https://hg.mozilla.org/try/diff/e2dddcfda2b9/testing/mozharness/mozharness.json

::: mozilla/config.py
@@ +1288,5 @@
> +            'mozharness_desktop_build': {
> +                'script_name': 'scripts/fx_desktop_build.py',
> +                'extra_args': [
> +                    '--config', 'builds/releng_base_mac_64_builds.py',
> +                    '--custom-build-variant-cfg', 'debug',

we want this to be like the linux st-an-debug: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/config.py?rev=390650d81a26#577

iow - s/debug/stat-and-debug/

@@ +2916,5 @@
>  for name, branch in items_before(BRANCHES, 'gecko_version', mc_gecko_version):
>      if 'linux64-st-an-debug' in branch['platforms']:
>          del branch['platforms']['linux64-st-an-debug']
> +    if 'macosx64-st-an-debug' in branch['platforms']:
> +        del branch['platforms']['macosx64-st-an-debug']

rather than having it enabled on m-c all at once, we can iterate on try like mentioned above

Comment 7

[Jordan Lund (:jlund)]

Comment on attachment 8541329 [details] [diff] [review]
mozharness changes

Review of attachment 8541329 [details] [diff] [review]:
-----------------------------------------------------------------

in linux64-st-an-debug we have, platform_supports_post_upload_to_latest:True. we spoke about this over irc, essentially it does the following: http://mxr.mozilla.org/build/source/tools/stage/post_upload.py#463.

IIUC, I'd imagine we would want to set this to False in this file too.

::: configs/builds/releng_sub_mac_configs/64_stat_and_debug.py
@@ +11,5 @@
> +        # 'generate-build-stats',
> +        'update',  # decided by query_is_nightly()
> +    ],
> +    'debug_build': True,
> +    'stage_platform': 'macosx64-debug-nonunified',

s/macosx64-debug-nonunified/macosx64-st-an-debug/

@@ +13,5 @@
> +    ],
> +    'debug_build': True,
> +    'stage_platform': 'macosx64-debug-nonunified',
> +    'purge_minsize': 12,
> +    'tooltool_manifest_src': "browser/config/tooltool-manifests/macosx64/\

this is already the default in configs/builds/releng_base_mac_64_builds.py (the base config that this sub config inherits from) so this line is not needed. we redefine tooltool_manifest_src for linux64-st-an-debug because we point to a custom manifest for that: browser/config/tooltool-manifests/linux64/clang.manifest

not sure if we need a custom tooltool manifest for the osx equivalent?

@@ +18,5 @@
> +releng.manifest",
> +    'enable_signing': False,
> +    'enable_talos_sendchange': False,
> +    'enable_unittest_sendchange': False,
> +    'enable_check_test': False,

in linux64-st-an-debug we have set this to true which makes it so we run 'make -k check'. sanity check that you don't want to run check_test for these?

Comment 8

[(no longer active)]

Attachment: buildbot-configs changes

Comment 9

[(no longer active)]

(In reply to Jordan Lund (:jlund) from comment #7)
> @@ +13,5 @@
> > +    ],
> > +    'debug_build': True,
> > +    'stage_platform': 'macosx64-debug-nonunified',
> > +    'purge_minsize': 12,
> > +    'tooltool_manifest_src': "browser/config/tooltool-manifests/macosx64/\
> 
> this is already the default in configs/builds/releng_base_mac_64_builds.py
> (the base config that this sub config inherits from) so this line is not
> needed. we redefine tooltool_manifest_src for linux64-st-an-debug because we
> point to a custom manifest for that:
> browser/config/tooltool-manifests/linux64/clang.manifest
> 
> not sure if we need a custom tooltool manifest for the osx equivalent?

No, we don't.

> @@ +18,5 @@
> > +releng.manifest",
> > +    'enable_signing': False,
> > +    'enable_talos_sendchange': False,
> > +    'enable_unittest_sendchange': False,
> > +    'enable_check_test': False,
> 
> in linux64-st-an-debug we have set this to true which makes it so we run
> 'make -k check'. sanity check that you don't want to run check_test for
> these?

No my mistake, thanks for catching it!

Comment 10

[(no longer active)]

Attachment: mozharness changes

Comment 11

[Jordan Lund (:jlund)]

Comment on attachment 8542187 [details] [diff] [review]
mozharness changes

ship it! :)

Comment 12

[Jordan Lund (:jlund)]

Attachment: 141229_osx_st_an_debug-bbotcfgs.patch

oops, I forgot we also need this. linux64-cc already had try_by_default:False so that's why it wasn't shown in the patch I linked you to copy from

this is an interdiff from your last patch. r+ if they are both applied

Comment 13

[Jordan Lund (:jlund)]

Comment on attachment 8542183 [details] [diff] [review]
buildbot-configs changes

Review of attachment 8542183 [details] [diff] [review]:
-----------------------------------------------------------------

so I think this will work with my latest attachment. there *might* be some funny behavior as I don't think we have ever had a debug build that is part of the try_by_default:False list. anyway, it's on me :)

Comment 14

[(no longer active)]

Comment on attachment 8542187 [details] [diff] [review]
mozharness changes

https://hg.mozilla.org/build/mozharness/rev/be60b87732a7

Comment 15

[(no longer active)]

Comment on attachment 8542183 [details] [diff] [review]
buildbot-configs changes

https://hg.mozilla.org/build/buildbot-configs/rev/548d99344d7f

Comment 16

[(no longer active)]

How can I test this on try?  I tried both using macosx64-st-an token in the trychooser syntax <https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=e2fcb8dea411> and -p all <https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=d2fcb01eab89> and neither seems to have done the job...

Comment 17

[(no longer active)]

Actually I guess attachment 8542309 [details] [diff] [review] explains -p all not working...

Comment 18

[(no longer active)]

Also tried macosx64-st-an-debug to no avail: <https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=904ebe7793a1>

Comment 19

[Phil Ringnalda (:philor)]

buildbot-config changes don't actually take effect until after a buildbot reconfig - at some random time in the future, someone will (probably, though not quite certainly) comment in the bug saying that revision is "in production" at which point it will actually be deployed.

Comment 20

[Jordan Lund (:jlund)]

(In reply to :Ehsan Akhgari (not reading bugmail, needinfo? me!) from comment #18)
> Also tried macosx64-st-an-debug to no avail:
> <https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=904ebe7793a1>

what philor said. :) 

this bug should be updated and you will see your patch being added here[1]. I think your commit message here: 'try: -b do -p macosx64-st-an-debug -u none -t none' should work. your right in that -p all will not work. you patched it this way so we can test and iterate on this without adding possible burning jobs to every try all push.

[1] https://wiki.mozilla.org/ReleaseEngineering/Maintenance#Reconfigs_.2F_Deployments

Comment 21

[Jordan Lund (:jlund)]

mozharness has been merged to production. patches are live :)

Comment 22

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/548d99344d7f

Comment 23

[(no longer active)]

https://tbpl.mozilla.org/?tree=Try&rev=ad3e5e502276

TreeHerder doesn't seem to display the job: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=ad3e5e502276 which I filed as bug 1117237.

Comment 24

[(no longer active)]

(In reply to :Ehsan Akhgari (not reading bugmail, needinfo? me!) from comment #23)
> https://tbpl.mozilla.org/?tree=Try&rev=ad3e5e502276
> 
> TreeHerder doesn't seem to display the job:
> https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=ad3e5e502276
> which I filed as bug 1117237.

Scratch that!  This is the real try link: <https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=0264914d132b>

Comment 25

[(no longer active)]

From <http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-0264914d132b/try-macosx64-st-an-debug/try-macosx64-st-an-debug-bm83-try1-build1.txt.gz>:

16:10:44     INFO -  cd ../../dist && cd ./firefox && python /builds/slave/try-osx64-st-an-d-000000000000/scripts/release/signing/signtool.py --cachedir /builds/slave/try-osx64-st-an-d-000000000000/signing_cache -t /builds/slave/try-osx64-st-an-d-000000000000/token -n /builds/slave/try-osx64-st-an-d-000000000000/nonce -c /builds/slave/try-osx64-st-an-d-000000000000/scripts/release/signing/host.cert -H gpg:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9110 -H gpg:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9110 -H gpg:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9110 -H dmg:mac-signing2.srv.releng.scl3.mozilla.com:9110 -H dmg:mac-signing3.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9110 -f dmgv2   NightlyDebug.app && cd /builds/slave/try-osx64-st-an-d-000000000000/build/src/obj-firefox/dist && /builds/slave/try-osx64-st-an-d-000000000000/build/src/build/package/mac_osx/pkg-dmg --source 'firefox' --target 'firefox-37.0a1.en-US.mac64.dmg' --volname 'Nightly'  --copy 'branding/dsstore:/.DS_Store' --mkdir /.background --copy 'branding/background.png:/.background' --icon 'branding/disk.icns' --symlink '/Applications:/ '
16:10:44     INFO -  python: can't open file '/builds/slave/try-osx64-st-an-d-000000000000/scripts/release/signing/signtool.py': [Errno 2] No such file or directory
16:10:44     INFO -  make[4]: *** [make-package-internal] Error 2
16:10:44     INFO -  make[3]: *** [make-package] Error 2
16:10:44     INFO -  make[2]: *** [default] Error 2
16:10:44     INFO -  make[1]: *** [package] Error 2
16:10:44     INFO -  make: *** [automation/package] Error 2

Jordan, do you know what this means?

Comment 26

[(no longer active)]

ping?

Comment 27

[(no longer active)]

Ben, can you please tell me who else can help me with this bug?  Thanks!

Comment 28

[[e-mail] (:bhearsum)]

(In reply to :Ehsan Akhgari (not reading bugmail, needinfo? me!) from comment #25)
> From
> <http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/eakhgari@mozilla.
> com-0264914d132b/try-macosx64-st-an-debug/try-macosx64-st-an-debug-bm83-try1-
> build1.txt.gz>:
> 
> 16:10:44     INFO -  cd ../../dist && cd ./firefox && python
> /builds/slave/try-osx64-st-an-d-000000000000/scripts/release/signing/
> signtool.py --cachedir
> /builds/slave/try-osx64-st-an-d-000000000000/signing_cache -t
> /builds/slave/try-osx64-st-an-d-000000000000/token -n
> /builds/slave/try-osx64-st-an-d-000000000000/nonce -c
> /builds/slave/try-osx64-st-an-d-000000000000/scripts/release/signing/host.
> cert -H
> gpg:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing4.srv.releng.scl3.
> mozilla.com:9110 -H
> gpg:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing5.srv.releng.scl3.
> mozilla.com:9110 -H
> gpg:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing6.srv.releng.scl3.
> mozilla.com:9110 -H dmg:mac-signing2.srv.releng.scl3.mozilla.com:9110 -H
> dmg:mac-signing3.srv.releng.scl3.mozilla.com:9110 -H
> dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9110 -H
> dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9110 -H
> dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9110 -H
> dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9110 -f dmgv2  
> NightlyDebug.app && cd
> /builds/slave/try-osx64-st-an-d-000000000000/build/src/obj-firefox/dist &&
> /builds/slave/try-osx64-st-an-d-000000000000/build/src/build/package/mac_osx/
> pkg-dmg --source 'firefox' --target 'firefox-37.0a1.en-US.mac64.dmg'
> --volname 'Nightly'  --copy 'branding/dsstore:/.DS_Store' --mkdir
> /.background --copy 'branding/background.png:/.background' --icon
> 'branding/disk.icns' --symlink '/Applications:/ '
> 16:10:44     INFO -  python: can't open file
> '/builds/slave/try-osx64-st-an-d-000000000000/scripts/release/signing/
> signtool.py': [Errno 2] No such file or directory

It looks like MOZ_SIGN_CMD is looking in the wrong place. In this job, the "scripts" directory is likely mozharness repository, and signtool.py is part of the "tools" repository, which appears to be checked out to /builds/slave/try-osx64-st-an-d-000000000000/build/tools for this job.

I'm guessing that there may be a bug in some of the recent reworking of this code that Jordan did. However, I don't think there's any need for these type of jobs to be signed at all. Removing "dep_signing_servers" and "nightly_signing_servers" might be the best way to go here.

Comment 29

[(no longer active)]

Yes, your suggestion makes perfect sense.  I did that in <https://hg.mozilla.org/build/buildbot-configs/rev/3673b25e6429>.

How can I get a reconfig to retest this?

Comment 30

[[e-mail] (:bhearsum)]

(In reply to :Ehsan Akhgari (not reading bugmail, needinfo? me!) from comment #29)
> Yes, your suggestion makes perfect sense.  I did that in
> <https://hg.mozilla.org/build/buildbot-configs/rev/3673b25e6429>.
> 
> How can I get a reconfig to retest this?

I pinged buildduty, it's going!

Comment 31

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/3673b25e6429

Comment 32

[(no longer active)]

The try push is green! <https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=a585c8ed65a2>

Comment 33

[(no longer active)]

Attachment: Enable macosx64-st-an-debug builds by default

I think this does the right thing.  I am basically redoing the changes from the first iteration of the patch that I undid, and copying how we do things for linux64-st-an-debug.  Please review carefully!

Comment 34

[[e-mail] (:bhearsum)]

Comment on attachment 8548260 [details] [diff] [review]
Enable macosx64-st-an-debug builds by default

Review of attachment 8548260 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ -1311,5 @@
>                  'script_timeout': 3 * 3600,
>                  'script_maxtime': int(5.5 * 3600),
>              },
>  
> -            'try_by_default': False,

Are you sure this should be on try by default? I'm honestly not sure. Looks fine otherwise though.

Comment 35

[(no longer active)]

(In reply to Ben Hearsum [:bhearsum] from comment #34)
> Comment on attachment 8548260 [details] [diff] [review]
> Enable macosx64-st-an-debug builds by default
> 
> Review of attachment 8548260 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: mozilla/config.py
> @@ -1311,5 @@
> >                  'script_timeout': 3 * 3600,
> >                  'script_maxtime': int(5.5 * 3600),
> >              },
> >  
> > -            'try_by_default': False,
> 
> Are you sure this should be on try by default? I'm honestly not sure. Looks
> fine otherwise though.

Yes, since these builds can catch the types of issues that developers don't catch locally.  We should probably do the same thing for other Linux64 as well.

Comment 36

[(no longer active)]

Comment on attachment 8548260 [details] [diff] [review]
Enable macosx64-st-an-debug builds by default

https://hg.mozilla.org/build/buildbot-configs/rev/6dc553dac3da

Should I just wait for a reconfig now?

Comment 37

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/6dc553dac3da

Comment 38

[Jordan Lund (:jlund)]

apologies, I was on PTO

thank you ben for helping. It looks like osx builds are green and running on m-c based branches since this mornings reconfig. Shall we resolve this bug for now and let it ride the trains?

Also, wrt the bug that was happening, if mozharness is running the build, it is in charge of deciding when to sign and does not rely on buildbot (self.config['enable_signing']). For this osx st-an-debug build, mozharness correctly set it to False but buildbot was passing MOZ_SIGN_CMD through os.env which got inherited into mozharness. That happened, as Ben realized, because of dep/nightly signing buildbot config items.

the solution may be to make sure we don't have a MOZ_SIGN_CMD in the env if self.config('enable_signing') is False. I filed 1121745 and will attach a patch there.

Comment 39

[(no longer active)]

Yep, we're done here.  Thanks for your help everyone!