964411 - Create hg share directory for gaia-central for test slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

The gaia repo is big, and unfortunately we need to clone it on every test slave that runs a gaia test job.

Also unfortunately, this clone has a tendency to hang, see bug 960814 and bug 920153.

From what I hear, releng already has some capacity to access a shared hg repo on build slaves...can we set up a similar share for test slaves, and would it solve this problem?  I.e., if two different slaves need different revisions of the shared repo...has does that work?

Comment 1

[Chris AtLee [:catlee]]

puppet-wise we'll need to manage the hg-shared directory (/builds/hg-shared on the builders), and install hgtool.

I don't know how much tooling changes we need to use the share and hgtool rather than straight up hg though.

Comment 2

[Aki Sasaki (not active)]

Pretty sure MercurialVCS in mozharness will use the hg share base dir if it's set, even without hgtool, so just creating the dir and setting HG_SHARE_BASE_DIR and/or self.config['vcs_share_base'] should do the trick.

Comment 3

[Chris AtLee [:catlee]]

we'll probably want a bundle created too

Comment 4

[Chris AtLee [:catlee]]

Attachment: create hg-shared/git-shared dirs on test slaves

I think this should just work since all the slave have dirs::builds already.

Comment 5

[Chris AtLee [:catlee]]

This was deployed to production earlier today. We should be able to modify the scripts to make use of it.

Comment 6

[Jonathan Griffin (:jgriffin)]

Great, thanks!

Comment 7

[[e-mail] (:bhearsum)]

Looks like we just need to set HG_SHARE_BASE_DIR now...I'll verify that and get this finished up.

Comment 8

[[e-mail] (:bhearsum)]

It looks like there are some tests that still run on talos-r3-fed slaves. Specifically, Marionette tests on b2g18 branches:
https://tbpl.mozilla.org/php/getParsedLog.php?id=33754691&tree=Mozilla-B2g18&full=1. They share a config with a bunch of others (that run on ec2 machines), so we'll need to create /builds/hg-shared on them before deploying anything.

There's also comments that suggest that Marionette tests run on try (https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla-tests/config.py#L1054), but I don't think that's actually true. In any case, they use a different config so I'm just going to ignore them for the purpose of this bug.

The full list of configs that need vcs_share_base set is:
marionette/gaia_ui_test_prod_config.py
marionette/automation_emulator_config.py
b2g/gaia_unit_production_config.py
marionette/prod_config.py

Once I figure out the /builds/hg-shared situation on the rev 3 machines I'll post a patch for the configs.

Comment 9

[Phil Ringnalda (:philor)]

That outdated comment about running marionette on try meant that we used to run marionette on Windows on try, until bug 924245 when we stopped. We do run it on Linux, but on ec2.

Comment 10

[[e-mail] (:bhearsum)]

Attachment: turn on hg share for things that use gaia-central

So, this seems to work:
13:12:17     INFO - hg share works.
13:12:17     INFO - Updating shared repo
13:12:17     INFO - Cloning https://hg.mozilla.org//integration/gaia-central to /builds/hg-shared/integration/gaia-central.
13:12:17     INFO - mkdir: /builds/hg-shared/integration
13:12:17     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'clone', 'https://hg.mozilla.org//integration/gaia-central', '/builds/hg-shared/integration/gaia-central']
13:12:17     INFO - Copy/paste: hg --config ui.merge=internal:merge clone https://hg.mozilla.org//integration/gaia-central /builds/hg-shared/integration/gaia-central
13:18:07     INFO -  requesting all changes
13:18:07     INFO -  adding changesets
13:18:07     INFO -  adding manifests
13:18:07     INFO -  adding file changes
13:18:07     INFO -  added 26889 changesets with 63222 changes to 12648 files (+1 heads)
13:18:07     INFO -  updating to branch default
13:18:07     INFO -  6361 files updated, 0 files merged, 0 files removed, 0 files unresolved
13:18:07     INFO - Return code: 0
13:18:07     INFO - Updating /builds/hg-shared/integration/gaia-central.
13:18:07     INFO - Getting output from command: ['hg', '--config', 'ui.merge=internal:merge', 'branch'] in /builds/hg-shared/integration/gaia-central
13:18:07     INFO - Copy/paste: hg --config ui.merge=internal:merge branch
13:18:07     INFO - Reading from file tmpfile_stdout
13:18:07     INFO - Output received:
13:18:07     INFO -  default
13:18:07     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1
13:18:07     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1
13:18:07     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'update', '-C'] in /builds/hg-shared/integration/gaia-central
13:18:07     INFO - Copy/paste: hg --config ui.merge=internal:merge update -C
13:18:08     INFO -  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
13:18:08     INFO - Return code: 0
13:18:08     INFO - Getting output from command: ['hg', '--config', 'ui.merge=internal:merge', 'parent', '--template', '{node|short}'] in /builds/hg-shared/integration/gaia-central
13:18:08     INFO - Copy/paste: hg --config ui.merge=internal:merge parent --template {node|short}
13:18:08     INFO - Reading from file tmpfile_stdout
13:18:08     INFO - Output received:
13:18:08     INFO -  621040ee199c
13:18:08     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1
13:18:08     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1
13:18:08     INFO - Trying to share /builds/hg-shared/integration/gaia-central to /builds/slave/test/gaia
13:18:08     INFO - Sharing /builds/hg-shared/integration/gaia-central to /builds/slave/test/gaia.
13:18:08     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'share', '-U', '/builds/hg-shared/integration/gaia-central', '/builds/slave/test/gaia']
13:18:08     INFO - Copy/paste: hg --config ui.merge=internal:merge share -U /builds/hg-shared/integration/gaia-central /builds/slave/test/gaia
13:18:08     INFO - Return code: 0
13:18:08     INFO - Updating /builds/slave/test/gaia revision 8afa9636490a2b8dce22a5f165a9e6d49d30ac9a.
13:18:08     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'update', '-C', '-r', '8afa9636490a2b8dce22a5f165a9e6d49d30ac9a'] in /builds/slave/test/gaia
13:18:08     INFO - Copy/paste: hg --config ui.merge=internal:merge update -C -r 8afa9636490a2b8dce22a5f165a9e6d49d30ac9a
13:18:34     INFO -  6361 files updated, 0 files merged, 0 files removed, 0 files unresolved
13:18:34     INFO - Return code: 0
13:18:34     INFO - Getting output from command: ['hg', '--config', 'ui.merge=internal:merge', 'parent', '--template', '{node|short}'] in /builds/slave/test/gaia
13:18:34     INFO - Copy/paste: hg --config ui.merge=internal:merge parent --template {node|short}
13:18:34     INFO - Reading from file tmpfile_stdout
13:18:34     INFO - Output received:
13:18:34     INFO -  8afa9636490a
13:18:34     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1
13:18:34     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1
13:18:34     INFO - Changing directory to /builds/slave/test.
13:18:34     INFO - Pull has nothing to do!
13:18:34     INFO - Running post-action listener: _resource_record_post_action



I'm in the process of creating /builds/hg-shared on all of the rev3 machines. I should be able to finish that tonight and land this tomorrow.

talos-r3-fed-098 and talos-r3-fed64-058 will need /builds/hg-shared created on them after they come back from being offline.

Comment 11

[Aki Sasaki (not active)]

Comment on attachment 8367581 [details] [diff] [review]
turn on hg share for things that use gaia-central

I don't think this is a comprehensive list (e.g. talos), but this lgtm.

Comment 12

[Aki Sasaki (not active)]

Oh, gaia-central. N/m :)

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 8367581 [details] [diff] [review]
turn on hg share for things that use gaia-central

Landed on default+production (didn't merge mozharness because there was a bunch of changes I was uncomfortable with).

Comment 14

[[e-mail] (:bhearsum)]

I found a gaia ui test on mozilla-inbound that's now using hg share, huzzuh!

Comment 15

[Ed Morley [:emorley]]

Awesome, thank you everyone! :-)