1141745 - do not reach clobberer api for try builds

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

[13:29]	RyanVM|sheriffduty	KWierso: up for an experiment?
[13:30]	KWierso	RyanVM|sheriffduty: eh?
[13:30]	RyanVM|sheriffduty	OSX 10.6 bustage on Try
[13:30]	jcranmer|away	know anyone better for diagnosing?
[13:30]	RyanVM|sheriffduty	not being hit on production
[13:30]	RyanVM|sheriffduty	Callek suspects a clobber might bring it out
[13:30]	Callek	jcranmer|away: of course, note: http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/candidates/ has 1.8b1 from 2012
[13:30]	mrrrgn	Anybody here have a light review queue that would like to help a sista out?
[13:31]	mrrrgn	counts to 10 before choosing a random target
[13:31]	KWierso	RyanVM|sheriffduty: what tree should we clobber for this?
[13:31]	RyanVM|sheriffduty	let's go with m-c to minimize impact
[13:31]	jcranmer|away	Fallen|away: rkent: ^^^^
[13:31]	Callek	jcranmer|away: so maybe the lightning version is conflicting somehow -- note there is also http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/candidates/21.0b1-candidates/ on ftp right now
[13:32]	RyanVM|sheriffduty	and we can trigger new osx builds on tip
[13:32]	KWierso	RyanVM|sheriffduty: clobbered
[13:32]	RyanVM|sheriffduty	Callek: ^
[13:32]	KWierso	though it looks like tomcat clobbered all of m-c this morning
[13:32]	Callek	RyanVM|sheriffduty: thanks
[13:32]	RyanVM|sheriffduty	exciting
[13:33]	RyanVM|sheriffduty	indeed
[13:33]	KWierso	around 5am pacific
[13:33]	Callek	KWierso: was that before or after the suspected patch merged?
[13:33]	KWierso	iunno
[13:33]	RyanVM|sheriffduty	m-c tip osx builds are forced clobbers
[13:33]	RyanVM|sheriffduty	which contains glandium's push
[13:33]	RyanVM|sheriffduty	so yeah....
[13:34]	RyanVM|sheriffduty	Callek: is it possible we're *not* clobbering correctly on Try?
[13:34]	jcranmer|away	Callek: Fallen|away: rkent: it looks like it's trying to upload according to, I'm guessing, the gdata provider version number
[13:34]	KWierso	wonders what the clobberer would do if he hit try with it
[13:34]	Callek	RyanVM|sheriffduty: given the runner bug I just reviewed, sure
[13:34]	Callek	would be really really sad if we're not though
[13:36]	KWierso	Callek, RyanVM|sheriffduty: want me to try clobbering try with the clobberer?
[13:36]	Callek	KWierso: imo can't hurt
[13:36]	KWierso	done
[13:37]	Callek	KWierso: RyanVM|sheriffduty mrrrgn jlund|lunch: well this looks suspect for sure
[13:37]	Callek	https://callek.pastebin.mozilla.org/8825158
[13:37]	Callek	this was one of the OSX builds on a failed job
[13:37]	KWierso	
[13:37]	Callek	note the fact that the *try* job has a last-clobber-date
[13:38]	Callek	of march 6
[13:39]	Callek	KWierso: jlund|lunch coop: ok thats definitely bug worthy ^
[13:39]	Callek	ahal: fyi ^
[13:40]	Callek	wouldn't be shocked if this was regression from mozharness-based-builds
[13:40]	jlund|lunch	was just thinking the same

Comment 1

[Jordan Lund (:jlund)]

14:02:02 <jlund|lunch> catlee: Callek most mh places (desktop mh builds included) use http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/purge.py#120
14:05:12 <jlund|lunch> Callek: catlee: my logic was add --purge, and all is well: http://mxr.mozilla.org/build/source/mozharness/configs/builds/branch_specifics.py#82 http://mxr.mozilla.org/build/source/ash-mozharness/mozharness/base/vcs/hgtool.py#98
14:58:42 <•catlee> jlund|buildduty: it looks like that just forces us to hit the clobberer endpoint instead of just clobbering locally
15:00:15 <jlund|buildduty> ya, I don't think I have special logic for clobberer on try. I do pass --purge to hgtool.py (like we do for MBF) and my understanding was that would wipe the whole src tree
15:00:40 <•catlee> yeah, should do
15:00:47 <•catlee> but weird that we're hitting clobberer API at all for try
15:00:52 <•catlee> doesn't make sense to I don't think

Comment 2

[Ryan VanderMeulen [:RyanVM]]

FWIW, this bug means we basically have no OSX test coverage on Try (since other versions are off by default).

Comment 3

[Jordan Lund (:jlund)]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #2)
> FWIW, this bug means we basically have no OSX test coverage on Try (since
> other versions are off by default).

I am looking into this today

Comment 4

[Justin Wood (:Callek)]

See also https://bugzilla.mozilla.org/show_bug.cgi?id=1142458

Comment 5

[Jordan Lund (:jlund)]

from what I can tell, it appears like this is not the cause of OSX tests failing. Build jobs are using --purge (start clean) with hgtool when checking out source.

Either way, I'll leave this bug open and switch to just doing a rm -rf (to be sure to be sure) before checking out src instead of calling clobberer api

Comment 6

[Chris AtLee [:catlee]]

If we're using --purge, we shouldn't need to rm -rf anything. purge should be getting rid of unwanted files.

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Jorendorff is seeing cgc failures in https://treeherder.mozilla.org/#/jobs?repo=try&revision=b54f48f10454 on try. Ryan chalked these failures up to needs-clobber on inbound.

What's the status of this bug?

Comment 8

[Phil Ringnalda (:philor)]

That would be a thing separate from this thing which is a thing separate from what it was filed about.

In the case of spidermonkey shell builds, they used to not clobber on try, then sfink fixed that by making them clobber everywhere all the time, then that got fixed so that local devs could pass --noclean to the script to say not to clobber, then the bulk of the script moved into the tree, and in the in-tree script clobbering only happens if you pass --clobber, but the old script which calls the in-tree script thinks that it should instead be passing --dep for those people who asked to not have a clobber, instead of realizing that it should be passing --clobber except in the case when it has been told --noclean.

Or, shorter version, you need to file a bug against sfink to make it again the case that spidermonkey shell builds in automation clobber always. Or that spidermonkey shell builds on try clobber always, if he feels like doing just that, this time.

Comment 9

[Jordan Lund (:jlund)]

ya, I don't think related to this. I've changed the bug summary to reflect current issue and clear up confusion.

essentially, desktop ff builds use --purge which should start us from a clean slate every time

Comment 10

[Jordan Lund (:jlund)]

Attachment: 150414_1141745_dont_clobberer_try_builds.patch

this makes it so we default to calling clobberer when clobberer_url is in config but we can explicitly not use it.

here is the self.config changes of this patch

before patch:
   * non-try script run:
      * cmd: python scripts/fx_desktop_build.py --config builds/releng_base_mac_64_builds.py --branch mozilla-central --build-pool production --developer-run --cfg balrog/production.py --dump-config-hierarchy
      * config from cmd: http://people.mozilla.org/~jlund/before_patch_mc
   * try script run:
      * cmd: python scripts/fx_desktop_build.py --config builds/releng_base_mac_64_builds.py --branch try --build-pool production --developer-run --cfg balrog/production.py --dump-config-hierarchy
      * config from cmd: http://people.mozilla.org/~jlund/before_patch_try

after patch:
   * non-try script run:
      * cmd: python scripts/fx_desktop_build.py --config builds/releng_base_mac_64_builds.py --branch mozilla-central --build-pool production --developer-run --cfg balrog/production.py --dump-config-hierarchy
      * config from cmd: http://people.mozilla.org/~jlund/after_patch_mc
   * try script run:
      * cmd: python scripts/fx_desktop_build.py --config builds/releng_base_mac_64_builds.py --branch try --build-pool production --developer-run --cfg balrog/production.py --dump-config-hierarchy
      * config from cmd: http://people.mozilla.org/~jlund/after_patch_try


and just the diffs:

jlund@Hastings163:/tmp
> diff -pU 15 before_patch_mc after_patch_mc
--- before_patch_mc     2015-04-14 19:17:14.000000000 +0000
+++ after_patch_mc      2015-04-14 19:20:37.000000000 +0000
@@ -164,30 +164,31 @@
             "linux64",
             "win32",
             "win64"
         ],
         "platform_supports_post_upload_to_latest": true,
         "release_to_try_builds": false,
         "repo_base": "https://hg.mozilla.org",
         "stage_product": "firefox",
         "stage_ssh_key": "ffxbld_rsa",
         "stage_username": "ffxbld",
         "taskcluster_credentials_file": "oauth.txt",
         "tinderbox_build_dir": null,
         "to_tinderbox_dated": true,
         "tools_repo": "https://hg.mozilla.org/build/tools",
         "tooltool_url": "http://tooltool.pvt.build.mozilla.org/build",
+        "use_clobberer": true,
         "virtualenv_modules": [
             "requests==2.2.1",
             "PyHawk-with-a-single-extra-commit==0.1.5",
             "taskcluster==0.0.15"
         ],
         "virtualenv_path": "venv",
         "volatile_config": {
             "actions": null,
             "add_actions": null,
             "no_actions": null
         },
         "who": "",
         "work_dir": "build"
     }
 }
\ No newline at end of file

jlund@Hastings163:/tmp
> diff -pU 15 before_patch_try after_patch_try
--- before_patch_try    2015-04-14 19:16:55.000000000 +0000
+++ after_patch_try     2015-04-14 19:16:16.000000000 +0000
@@ -32,31 +32,32 @@
                 "ignore_failures": true
             }
         ]
     },
     "builds/branch_specifics.py": {
         "branch_supports_uploadsymbols": false,
         "clone_by_revision": true,
         "clone_with_purge": true,
         "include_post_upload_builddir": true,
         "release_to_try_builds": true,
         "repo_path": "try",
         "stage_ssh_key": "trybld_dsa",
         "stage_username": "trybld",
         "tinderbox_build_dir": "%(who)s-%(got_revision)s",
         "to_tinderbox_dated": false,
-        "use_branch_in_symbols_extra_buildid": false
+        "use_branch_in_symbols_extra_buildid": false,
+        "use_clobberer": false
     },
     "builds/build_pool_specifics.py": {
         "clobberer_url": "https://api.pub.build.mozilla.org/clobberer/lastclobber",
         "graph_server": "graphs.mozilla.org",
         "hgtool_base_bundle_urls": [
             "https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles"
         ],
         "sendchange_masters": [
             "buildbot-master81.build.mozilla.org:9301"
         ],
         "stage_server": "stage.mozilla.org",
         "symbol_server_host": "symbolpush.mozilla.org"
     },
     "builds/releng_base_mac_64_builds.py": {
         "app_ini_path": "%(obj_dir)s/dist/bin/application.ini",

Comment 11

[Jordan Lund (:jlund)]

Comment on attachment 8592397 [details] [diff] [review]
150414_1141745_dont_clobberer_try_builds.patch

remote:   https://hg.mozilla.org/build/mozharness/rev/4d4626f5221e

Comment 12

[Jordan Lund (:jlund)]

try push without using clobberer: https://treeherder.mozilla.org/logviewer.html#?job_id=7951430&repo=try