[Tracking bug] Create new project branch for private repo

RESOLVED FIXED

Status

Release Engineering
General
P1
major
RESOLVED FIXED
8 years ago
2 years ago

People

(Reporter: chris hofmann, Assigned: lsblakk)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [sg:nse])

Attachments

(16 attachments, 16 obsolete attachments)

8.20 KB, patch
catlee
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
11.07 KB, patch
catlee
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
1.19 KB, text/plain
Details
6.32 KB, patch
bhearsum
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
9.83 KB, patch
catlee
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
1.10 KB, patch
catlee
: review+
Details | Diff | Splinter Review
5.14 KB, patch
catlee
: review+
Details | Diff | Splinter Review
2.10 KB, patch
nthomas
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
2.50 KB, patch
nthomas
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
2.00 KB, patch
nthomas
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
4.88 KB, patch
nthomas
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
3.41 KB, patch
catlee
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
1.22 KB, patch
Details | Diff | Splinter Review
5.17 KB, patch
lsblakk
: checked-in+
Details | Diff | Splinter Review
1.94 KB, patch
catlee
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
9.23 KB, patch
catlee
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
(Reporter)

Description

8 years ago
see info on rational for this in top level dependency bug.
(Reporter)

Updated

8 years ago
Group: core-security
(Reporter)

Updated

8 years ago
Assignee: nobody → joduinn
(Reporter)

Comment 1

8 years ago
will will need to connect private security repo to an expanded test farm.

    * first existing ref, unit, and all automated tests
    * then add existing test system to regular automation -
    **  jesse's fuzzers, 
    **  blcary/tomcat top site testing.
(Reporter)

Updated

8 years ago
Assignee: joduinn → nobody
Component: Build Config → Release Engineering: Custom Builds
Product: Firefox → mozilla.org
QA Contact: build.config → custom-builds
Version: 3.5 Branch → other
Component: Release Engineering: Custom Builds → Release Engineering
OS: Mac OS X → All
QA Contact: custom-builds → release
Hardware: x86 → All
Whiteboard: [sg:nse]
Off to John for prioritization
Assignee: nobody → joduinn
discussing scope and "definition of success" offline.
(Reporter)

Comment 4

8 years ago
from the meeting on 3/15 I think we agreed that what this bug meas is setting up the following set of functions on access controlled systems where the results are not publically visable

# XP Firefox Builds (RelEng)
# Unit Tests (RelEng)
# Talos (RelEng) 

and for pushing 
FTP for builds (IT) (problem verification by small trusted test group)
(In reply to comment #4)
> and for pushing 
> FTP for builds (IT) (problem verification by small trusted test group)

This should be an IT bug.
(Reporter)

Updated

8 years ago
Summary: set up full test infrastructure for security patch private repo → set up builds and full automated test infrastructure for security patch private repo
Depends on: 553728
dveditz investigating changing scope last week and again this week.
The scope remains pretty much as chofmann described: we need builds and test runs going off the private security hg repo (to be set up in bug 551750).

The builds themselves need to be delivered to a private server (ftp or otherwise) protected by LDAP access.

The test results don't need to be private initially, they can be output to the normal tinderbox build reports and graph servers as long as the builds themselves and the hg/source/checkin-comment information is private.
Severity: normal → major
Depends on: 551750
Priority: -- → P1
(In reply to comment #7)
> The scope remains pretty much as chofmann described: we need builds and test
> runs going off the private security hg repo (to be set up in bug 551750).

dveditz pointed out private repo setup is completed in bug#551752. Bug#551750 is not related, so removed from dep.bugs.

lsblakk: Note the private repo is on a different hg server. It can be found at https://hgpvt.mozilla.org/shadow-central


> The builds themselves need to be delivered to a private server (ftp or
> otherwise) protected by LDAP access.
This is being tracked in bug#553728 and is still in progress. 


> The test results don't need to be private initially, they can be output to the
> normal tinderbox build reports and graph servers as long as the builds
> themselves and the hg/source/checkin-comment information is private.
Per meeting with dveditz, clyon, we're posting the results of builds, tests, talos to the usual tinderbox server and graphserver. 

lsblakk: if possible to call this "shadow-central", great. However, I know the name of the project branch ties into the logic for the name of the tinderbox, so let me know if there are any problems, and we can rename.
Assignee: joduinn → lsblakk
Depends on: 551752
No longer depends on: 551750
Summary: set up builds and full automated test infrastructure for security patch private repo → Create new project branch for private repo
(Assignee)

Updated

8 years ago
Depends on: 574757
(Assignee)

Updated

8 years ago
Depends on: 575692
(Assignee)

Updated

8 years ago
Depends on: 580118
(Assignee)

Comment 9

8 years ago
Created attachment 459021 [details] [diff] [review]
talos configs for staging/production
Created attachment 459027 [details] [diff] [review]
patch for graphserver data.sql
Attachment #459027 - Flags: review?(catlee)
Created attachment 459028 [details] [diff] [review]
patch for graphserver data.sql

and this time without those slaves being added at the same time :)
Attachment #459027 - Attachment is obsolete: true
Attachment #459028 - Flags: review?(catlee)
Attachment #459027 - Flags: review?(catlee)
Created attachment 459039 [details]
sql insert statement for shadow central
SQL pushed onto production graph DB server successfully.
Created attachment 459075 [details] [diff] [review]
shadow-central support in post_upload.py

tested in staging, the only thing i'm not 100% on is if the .mar lines are needed - might end up needing it later?
Attachment #459075 - Flags: review?(bhearsum)
Comment on attachment 459075 [details] [diff] [review]
shadow-central support in post_upload.py

A couple things:
* Please use the /home/ftp/pub/... style path to be consistent with other entries
* ReleaseToShadowCentralBuilds mostly copies ReleaseToTinderboxBuilds. Can you factor out the common parts to a base? Something like:

def ReleaseToBlahBlah(tinderbox_builds_path, tinderbox_url, options, upload_dir, files, dated):
  stuff

and then ShadowCentral and Tinderbox builds could call it.
Attachment #459075 - Flags: review?(bhearsum) → review-

Updated

8 years ago
Attachment #459028 - Flags: review?(catlee) → review+
(Assignee)

Updated

8 years ago
Attachment #459021 - Flags: review?(catlee)
Created attachment 459091 [details] [diff] [review]
shadow-central buildbot-configs for production

from testing in staging, i was able to run this completely but it didn't use an https://ftp location for package download, there is no staging env setup with the kind of auth i need to be able to start testing this in production. i have permission from joduinn and dveditz to put this in production even if it burns and put out the fires from there.
Attachment #459091 - Flags: review?(catlee)
Created attachment 459092 [details] [diff] [review]
shadow-central buildbotcustom changes

also tested well in staging, need to return the complete repo path when it starts with http or ssh so that i am able to play around with the checkout method.  the slaves will have a .ssh/config file with the ffxbld key location so that this works.
Attachment #459092 - Flags: review?(catlee)
Created attachment 459160 [details] [diff] [review]
shadow-central support in post_upload.py
Attachment #459075 - Attachment is obsolete: true
Attachment #459160 - Flags: review?(bhearsum)
Comment on attachment 459160 [details] [diff] [review]
shadow-central support in post_upload.py

tested in staging.
Comment on attachment 459021 [details] [diff] [review]
talos configs for staging/production

Are we going to be doing releases off this branch?  If not, please remove the release_tests lines.  Looks good otherwise.
Attachment #459021 - Flags: review?(catlee) → review+
Comment on attachment 459091 [details] [diff] [review]
shadow-central buildbot-configs for production

>+######## shadow-central
>+# custom settings for shadow-central repo
>+BRANCHES['shadow-central']['hgurl'] = 'https://hgpvt.mozilla.org/'
>+BRANCHES['shadow-central']['hghost'] = 'ssh://ffxbld@hgpvt.mozilla.org'
>+BRANCHES['shadow-central']['stage_base_path'] = '/mnt/eql/builds/firefox/pvt-builds'
>+BRANCHES['shadow-central']['stage_ssh_key'] = 'stg_ffxbld/ffxbld_dsa'
>+# we have to override the config repo path if the configs will live in our build repo
>+BRANCHES['shadow-central']['config_repo_path'] = 'http://hg.mozilla.org/users/lsblakk_mozilla.com/buildbot-configs'

This should be pointing at the real buildbot-configs

>diff --git a/mozilla/production_config.py b/mozilla/production_config.py
>--- a/mozilla/production_config.py
>+++ b/mozilla/production_config.py
>@@ -69,16 +69,25 @@ SYMBOL_SERVER_HOST = 'dm-symbolpush01.mo
> # Local branch overrides
> BRANCHES = {
>     'mozilla-central': {
>         'packaged_unittest_tinderbox_tree': 'Firefox',
>         'tinderbox_tree': 'Firefox',
>         'mobile_tinderbox_tree': 'Mobile',
>         'mobile_build_failure_emails': ['mobile-build-failures@mozilla.org'],
>     },
>+    'mozilla-central': {
>+        'packaged_unittest_tinderbox_tree': 'Shadow-Central',
>+        'tinderbox_tree': 'Shadow-Central',
>+        'mobile_tinderbox_tree': 'Shadow-Central',
>+        'mobile_build_failure_emails': ['mobile-build-failures@mozilla.org'],
>+        'download_base_url': 'https://ftp.mozilla.org/pvt-builds/firefox/shadow-central-builds',
>+        'build_tools_repo_path' : 'http://hg.mozilla.org/users/stage-ffxbld/tools',
>+        'package_url': 'https://ftp.mozilla.org/pvt-builds/firefox/shadow-central-builds',
>+    },

itym 'shadow-central': ...
Attachment #459091 - Flags: review?(catlee) → review-
Comment on attachment 459160 [details] [diff] [review]
shadow-central support in post_upload.py

Looks good to me.
Attachment #459160 - Flags: review?(bhearsum) → review+
Created attachment 459189 [details] [diff] [review]
shadow-central buildbot-configs for production

addressed previous review comments.
Attachment #459091 - Attachment is obsolete: true
Attachment #459189 - Flags: review?(catlee)
Comment on attachment 459092 [details] [diff] [review]
shadow-central buildbotcustom changes

>diff --git a/process/factory.py b/process/factory.py
>--- a/process/factory.py
>+++ b/process/factory.py
>@@ -322,16 +322,18 @@ class MozillaBuildFactory(BuildFactory):
>          workdir='.'
>         ))
> 
>     def getRepoName(self, repo):
>         return repo.rstrip('/').split('/')[-1]
> 
>     def getRepository(self, repoPath, hgHost=None, push=False):
>         assert repoPath
>+        if repoPath.startswith('http' or 'ssh'):
>+            return repoPath

This isn't doing what you think it's doing.  Test what this function returns when you pass it an ssh url.

>     def addSourceSteps(self):
>-        self.addStep(Mercurial,
>-         name='hg_update',
>-         mode='update',
>-         baseURL='http://%s/' % self.hgHost,
>-         defaultBranch=self.repoPath,
>-         timeout=60*60, # 1 hour
>-        )
>+        if self.hgHost.startswith('ssh'):
>+            self.addStep(Mercurial(
>+             name='hg_ssh_clone',
>+             mode='update',
>+             baseURL= '%s/' % self.hgHost,
>+             defaultBranch=self.repoPath,
>+             timeout=60*60, # 1 hour
>+            ))
>+        else:
>+            self.addStep(Mercurial,
>+             name='hg_update',
>+             mode='update',
>+             baseURL='http://%s/' % self.hgHost,
>+             defaultBranch=self.repoPath,
>+             timeout=60*60, # 1 hour
>+            )

What's the rationale here?

>-        self.addStep(ShellCommand,
>-         name='get_codesize_log',
>-         command=['wget', '-O', 'codesize-auto-old.log',
>-          'http://%s/pub/mozilla.org/%s/%s/codesize-auto.log' % \
>-           (self.stageServer, self.productName, self.logUploadDir)],
>-         workdir='.',
>-         env=self.env
>-        )
>+        if self.branchName == 'shadow-central':
>+          # skipping for now, dveditz is looking into the need for this and if it can be in public
>+          pass
>+        else:
>+            self.addStep(ShellCommand,
>+             name='get_codesize_log',
>+             command=['wget', '-O', 'codesize-auto-old.log',
>+              'http://%s/pub/mozilla.org/%s/%s/codesize-auto.log' % \
>+               (self.stageServer, self.productName, self.logUploadDir)],
>+             workdir='.',
>+             env=self.env
>+            )
>         if self.mozillaDir == '':
>             codesighsObjdir = self.objdir
>         else:
>             codesighsObjdir = '../%s' % self.mozillaObjdir
> 
>         self.addStep(Codesighs,
>          name='get_codesighs_diff',
>          objdir=codesighsObjdir,
>@@ -1065,25 +1080,29 @@ class MercurialBuildFactory(MozillaBuild
>              branch=self.graphBranch,
>              resultsname=self.baseName
>             )
>         self.addStep(ShellCommand,
>          name='echo_codesize_log',
>          command=['cat', '../codesize-auto-diff.log'],
>          workdir='build%s' % self.mozillaDir
>         )
>-        self.addStep(ShellCommand,
>-         name='upload_codesize_log',
>-         command=['scp', '-o', 'User=%s' % self.stageUsername,
>-          '-o', 'IdentityFile=~/.ssh/%s' % self.stageSshKey,
>-          '../codesize-auto.log',
>-          '%s:%s/%s' % (self.stageServer, self.stageBasePath,
>-                        self.logUploadDir)],
>-         workdir='build%s' % self.mozillaDir
>-        )
>+        if self.branchName == 'shadow-central':
>+          # skipping for now, dveditz is looking into the need for this and if it can be in public
>+          pass
>+        else:
>+            self.addStep(ShellCommand,
>+             name='upload_codesize_log',
>+             command=['scp', '-o', 'User=%s' % self.stageUsername,
>+              '-o', 'IdentityFile=~/.ssh/%s' % self.stageSshKey,
>+              '../codesize-auto.log',
>+              '%s:%s/%s' % (self.stageServer, self.stageBasePath,
>+                            self.logUploadDir)],
>+             workdir='build%s' % self.mozillaDir
>+            )

I'm not a big fan of special-casing based on branchName.  Is there a better way to do this?

>@@ -1947,20 +1966,26 @@ class NightlyBuildFactory(MercurialBuild
>             uploadEnv['UPLOAD_SSH_KEY'] = '~/.ssh/%s' % self.stageSshKey
> 
>         # Always upload builds to the dated tinderbox builds directories
>         if self.tinderboxBuildsDir is None:
>             tinderboxBuildsDir = "%s-%s" % (self.branchName, self.platform)
>         else:
>             tinderboxBuildsDir = self.tinderboxBuildsDir
>         postUploadCmd =  ['post_upload.py']
>-        postUploadCmd += ['--tinderbox-builds-dir %s' % tinderboxBuildsDir,
>-                          '-i %(buildid)s',
>-                          '-p %s' % self.productName,
>-                          '--release-to-tinderbox-dated-builds']
>+        if self.branchName == 'shadow-central':
>+            postUploadCmd += ['--tinderbox-builds-dir %s' % tinderboxBuildsDir,
>+                              '-i %(buildid)s',
>+                              '-p %s' % self.productName,
>+                              '--release-to-shadow-central-builds']
>+        else:
>+            postUploadCmd += ['--tinderbox-builds-dir %s' % tinderboxBuildsDir,
>+                              '-i %(buildid)s',
>+                              '-p %s' % self.productName,
>+                              '--release-to-tinderbox-dated-builds']

ditto.
Attachment #459092 - Flags: review?(catlee) → review-
Comment on attachment 459189 [details] [diff] [review]
shadow-central buildbot-configs for production

>+    'shadow-central': {
>+        'packaged_unittest_tinderbox_tree': 'Shadow-Central',
>+        'tinderbox_tree': 'Shadow-Central',
>+        'mobile_tinderbox_tree': 'Shadow-Central',
>+        'mobile_build_failure_emails': ['mobile-build-failures@mozilla.org'],
>+        'download_base_url': 'https://ftp.mozilla.org/pvt-builds/firefox/shadow-central-builds',
>+        'build_tools_repo_path' : 'http://hg.mozilla.org/users/stage-ffxbld/tools',

This needs to be fixed up too.  Looks good otherwise.
Attachment #459189 - Flags: review?(catlee) → review+
Created attachment 459319 [details] [diff] [review]
shadow-central buildbotcustom changes

ok so instead of looking for self.branchName == "shadow-central" I'm making everything about a secure repo branch test against the hgHost starting with 'ssh' since the hg host for all our normal branches doesn't have a prefix at all.

also the codesighs upload/download steps for right now are commented out since they go to a public server (hardcoded http://) at the moment and there is a time crunch on getting this up so dveditz is happier to see this in production than to have the codesighs steps right off the bat.  they will need to set up a secure location for logs at some point and more refactoring for this will probably be needed.

also - with the getRepository function, I changed how I'm checking for 'http' or 'ssh' but as far as i can tell it only is affected by my configs for shadow-central where I have to specify the ssh:// for the repo to get a secure hg clone from the private repo (there is a config file in ~/.ssh on the slave) and also the build tools repo has to be explicitly stated because right now what is passed in is built from the hgHost -- this would lead to 'ssh://ffxbld@hgpvt.mozilla.org/build/tools' being the location that the builder looks to for the build-tools
Attachment #459092 - Attachment is obsolete: true
Attachment #459319 - Flags: review?(catlee)
(Assignee)

Updated

8 years ago
Blocks: 579092
Comment on attachment 459160 [details] [diff] [review]
shadow-central support in post_upload.py

http://hg.mozilla.org/build/tools/rev/b5b74daa2863
Attachment #459160 - Flags: checked-in+
Comment on attachment 459189 [details] [diff] [review]
shadow-central buildbot-configs for production

http://hg.mozilla.org/build/buildbot-configs/rev/524a79e748b6
(with change to config repo path so it's not my user repo)
Attachment #459189 - Flags: checked-in+
Comment on attachment 459028 [details] [diff] [review]
patch for graphserver data.sql

http://hg.mozilla.org/graphs/rev/f48dbf993b8c
Attachment #459028 - Flags: checked-in+
Comment on attachment 459021 [details] [diff] [review]
talos configs for staging/production

http://hg.mozilla.org/build/buildbot-configs/rev/36ef285be631

(took out the release-specific configs)
Attachment #459021 - Flags: checked-in+
(Assignee)

Updated

8 years ago
Depends on: 581030
(Assignee)

Updated

8 years ago
Depends on: 581106
Created attachment 459500 [details] [diff] [review]
shadow-central buildbotcustom changes

taking out the codesighs steps entirely for now as a temporary measure. this patch also features better getRepository() handling
Attachment #459319 - Attachment is obsolete: true
Attachment #459500 - Flags: review?(catlee)
Attachment #459319 - Flags: review?(catlee)
Created attachment 459501 [details] [diff] [review]
turn off codesighs for shadow-central in config.py

this goes along with the buildbotcustom changes to codesighs
Attachment #459501 - Flags: review?(catlee)
Created attachment 459506 [details] [diff] [review]
shadow-central buildbotcustom changes

checks against config.get('enable_codesighs', True) instead of pf so that it's not needed in the configs for other branches.
Attachment #459500 - Attachment is obsolete: true
Attachment #459506 - Flags: review?(catlee)
Attachment #459500 - Flags: review?(catlee)
Comment on attachment 459506 [details] [diff] [review]
shadow-central buildbotcustom changes

I don't really like how postUploadCmd behaviour is determined right now, but I don't have a better idea.
Attachment #459506 - Flags: review?(catlee) → review+

Updated

8 years ago
Attachment #459501 - Flags: review?(catlee) → review+
(Assignee)

Updated

8 years ago
Depends on: 583900
(Assignee)

Updated

8 years ago
Depends on: 584496
Created attachment 462982 [details] [diff] [review]
update to post_upload.py to send urls back that talos/unittest can download
Attachment #462982 - Flags: review?(nrthomas)
Created attachment 462985 [details] [diff] [review]
update to production configs for shadow-central to fix url paths

changing the stage_base_path to be like the mozilla-central path, fixing the download_base_url and package_url to remove extra firefox in path and to match post_upload.py update.

also, setting 'build_tools_repo_path' to 'http://hg.mozilla.org/build/tools' because with the defaults it ends up looking to https://hgpvt.mozilla.org/build/tools
Attachment #462985 - Flags: review?(nrthomas)
Attachment #462982 - Flags: review?(nrthomas) → review+
Created attachment 462994 [details] [diff] [review]
update to production configs for shadow-central to fix url paths

took out both download_base_url (no l10n for shadow-central so not needed) and package_url (only used by try)
Attachment #462985 - Attachment is obsolete: true
Attachment #462994 - Flags: review?(nrthomas)
Attachment #462985 - Flags: review?(nrthomas)
Attachment #462994 - Flags: review?(nrthomas) → review+
Comment on attachment 462982 [details] [diff] [review]
update to post_upload.py to send urls back that talos/unittest can download

http://hg.mozilla.org/build/tools/rev/43a9ae023d36
Attachment #462982 - Flags: checked-in+
Comment on attachment 462994 [details] [diff] [review]
update to production configs for shadow-central to fix url paths

http://hg.mozilla.org/build/buildbot-configs/rev/51b3c8aab426
Attachment #462994 - Flags: checked-in+
(Assignee)

Updated

8 years ago
Depends on: 585615
(Assignee)

Updated

8 years ago
Depends on: 585619
(Assignee)

Updated

7 years ago
Summary: Create new project branch for private repo → [Tracking bug] Create new project branch for private repo
(Assignee)

Updated

7 years ago
Depends on: 592055
(Assignee)

Updated

7 years ago
Depends on: 592060
Created attachment 476967 [details] [diff] [review]
change post_upload.py SHADOW_CENTRAL_URL_PATH to use https://ftp.m.o

now that bug 585615 is closed, we'll need this change in order to get the tests running.
Attachment #476967 - Flags: review?(nrthomas)
Attachment #476967 - Flags: review?(nrthomas) → review+
Comment on attachment 476967 [details] [diff] [review]
change post_upload.py SHADOW_CENTRAL_URL_PATH to use https://ftp.m.o

http://hg.mozilla.org/build/tools/rev/25cbf2a13c85
Attachment #476967 - Flags: checked-in+
(Assignee)

Updated

7 years ago
Depends on: 602634
(Assignee)

Updated

7 years ago
Depends on: 602640
Depends on: 606050
(Assignee)

Updated

7 years ago
Depends on: 607246
Created attachment 486527 [details] [diff] [review]
updated post_upload.py for new storage location dm-pvtbuild01

based on the change to storage location in https://bugzilla.mozilla.org/show_bug.cgi?id=585615#c49
Attachment #486527 - Flags: review?(nrthomas)
Comment on attachment 486527 [details] [diff] [review]
updated post_upload.py for new storage location dm-pvtbuild01

Please make sure the copy of post_upload.py on the private server is somewhere we can update it later. eg symlink from somewhere on the path to ~ffxbld/tools/stage/
Attachment #486527 - Flags: review?(nrthomas) → review+
Created attachment 486809 [details] [diff] [review]
post_upload.py now with staging SHADOW_CENTRAL paths

carrying forward nthomas' r on this.
Attachment #486527 - Attachment is obsolete: true
Lukas, catlee hit me up for a similar review on bug 606376 and I realised the %(product)s isn't necessary. I'd like to keep shadow-central and fuzzing at the same level, so please remove the four %(product)s/ on check in. Thanks.
Created attachment 486844 [details] [diff] [review]
post_upload.py now with more generic paths

So I actually made a few more changes here than what you asked for.  In order to keep the generic ReleaseToBuildDir which uses product & tinderbox_builds_dir I left %(product)s in but in the call to ReleaseToShadowCentralBuilds I override options.product.  This would leave room for ReleaseToFuzzerBuilds where product = fuzzer and other fun options with the PVT_BUILD_DIR
Attachment #486809 - Attachment is obsolete: true
Attachment #486844 - Flags: review?(nrthomas)
Created attachment 486845 [details] [diff] [review]
post_upload.py now with more generic paths [take 2]

same as previous patch except with an errant 'firefox' removed.  i will ask aravind tomorrow to create a dir on dm-pvtbuild01 for shadow-central with ffxbld ownership
Attachment #486844 - Attachment is obsolete: true
Attachment #486845 - Flags: review?(nrthomas)
Attachment #486844 - Flags: review?(nrthomas)
(Assignee)

Updated

7 years ago
Depends on: 608193
Created attachment 486851 [details] [diff] [review]
[needs testing] config changes for new shadow-central location
Attachment #486845 - Flags: review?(nrthomas) → review+
Created attachment 488141 [details] [diff] [review]
[tested] factory.py changes for shadow-central post_upload command

This will need to be landed at the same time as attachment 486845 [details] [diff] [review]
Attachment #488141 - Flags: review?(catlee)
Created attachment 488142 [details] [diff] [review]
[tested] config changes for new shadow-central location

includes settings for staging s-c as well as the server name change for production.
Attachment #486851 - Attachment is obsolete: true
Attachment #488142 - Flags: review?(catlee)
Comment on attachment 488141 [details] [diff] [review]
[tested] factory.py changes for shadow-central post_upload command

>diff --git a/process/factory.py b/process/factory.py
>--- a/process/factory.py
>+++ b/process/factory.py
>@@ -105,24 +105,26 @@ def postUploadCmdPrefix(upload_dir=None,
>     list of arguments.  Some arguments may be WithProperties instances.
> 
>     If as_list is False, the command will be returned as a WithProperties
>     instance representing the entire command line as a single string.
> 
>     It is expected that the returned value is augmented with the list of files
>     to upload, and where to upload it.
>     """
>-
>-    cmd = ["post_upload.py"]
>+    if to_shadow:
>+      cmd = ["$HOME/bin/post_upload.py"]
>+    else:
>+      cmd = ["post_upload.py"]

What's the point of this?  Can't we put $HOME/bin in $PATH on the new server?
Comment on attachment 488142 [details] [diff] [review]
[tested] config changes for new shadow-central location

production/staging settings should go into the production_ / staging_config.py files.
Attachment #488142 - Flags: review?(catlee) → review-
> What's the point of this?  Can't we put $HOME/bin in $PATH on the new server?

ssh to the new server to run post_upload.py does not get the post_upload.py that is in $HOME/bin (which is already in $PATH for both stage-ffxbld and ffxbld) because it's added in .bash_profile. in order to be able to have it in $PATH from ssh without specifying $HOME, I need to have a symlink in /usr/local/bin (like on staging-stage,stage,preproduction-stage) but this a) requires root access and b) because we only have dm-pvtbuild01 and not a staging server for it means we would use the same post_upload for staging/production.

So this solution kills both birds - we can control the location of the symlink without having to pester IT for every little change and also can keep a staging version and production version separate.
(In reply to comment #52)
> Comment on attachment 488142 [details] [diff] [review]
> [tested] config changes for new shadow-central location
> 
> production/staging settings should go into the production_ / staging_config.py
> files.

i can take out the line for staging hghost but it cannot go in staging_config.py since it is not able to override the setting.  the setting in config.py is the one that gets used for that particular variable.
(In reply to comment #54)
> (In reply to comment #52)
> > Comment on attachment 488142 [details] [diff] [review] [details]
> > [tested] config changes for new shadow-central location
> > 
> > production/staging settings should go into the production_ / staging_config.py
> > files.
> 
> i can take out the line for staging hghost but it cannot go in
> staging_config.py since it is not able to override the setting.  the setting in
> config.py is the one that gets used for that particular variable.

unless...I can try taking it out of config.py and putting it in production/staging respectively so that it's not overridden but actually set in each.
Created attachment 488184 [details] [diff] [review]
[tested] config changes for new shadow-central location now split in prod/stg configs properly

ok this works too and follows the convention better.
Attachment #488142 - Attachment is obsolete: true
Attachment #488184 - Flags: review?(catlee)
Comment on attachment 488184 [details] [diff] [review]
[tested] config changes for new shadow-central location now split in prod/stg configs properly

>diff --git a/mozilla/production_config.py b/mozilla/production_config.py
>--- a/mozilla/production_config.py
>+++ b/mozilla/production_config.py
>@@ -78,16 +78,18 @@ BRANCHES = {
>         'mobile_build_failure_emails': ['mobile-build-failures@mozilla.org'],
>     },
>     'shadow-central': {
>         'packaged_unittest_tinderbox_tree': 'Shadow-Central',
>         'tinderbox_tree': 'Shadow-Central',
>         'mobile_tinderbox_tree': 'Shadow-Central',
>         'mobile_build_failure_emails': ['mobile-build-failures@mozilla.org'],
>         'build_tools_repo_path' : 'http://hg.mozilla.org/build/tools',
>+        'stage_server' : 'dm-pvtbuild01.mozilla.org',
>+        'hghost' : 'ssh://stage-ffxbld@hgpvt.mozilla.org',

Looks good except for this hghost setting for production; also doesn't 'stage_base_path' need to be refactored out of config.py as well?
Attachment #488184 - Flags: review?(catlee) → review-
Created attachment 488269 [details] [diff] [review]
config changes for new shadow-central take 3

good catch on the stage_base_path
Attachment #488184 - Attachment is obsolete: true
Attachment #488269 - Flags: review?(catlee)
Comment on attachment 488269 [details] [diff] [review]
config changes for new shadow-central take 3

You have 'stage_base_path' twice in the staging configs.  Looks good otherwise.
Attachment #488269 - Flags: review?(catlee) → review+
(Assignee)

Updated

7 years ago
Depends on: 609728

Updated

7 years ago
Attachment #488141 - Flags: review?(catlee)
Created attachment 489081 [details] [diff] [review]
[tested] factory.py changes for shadow-central post_upload command

just setting the product for proper directory setting in post_upload.py
Attachment #488141 - Attachment is obsolete: true
Attachment #489081 - Flags: review?(catlee)
(Assignee)

Updated

7 years ago
Flags: needs-reconfig?
(Assignee)

Updated

7 years ago
Depends on: 610575
Comment on attachment 489081 [details] [diff] [review]
[tested] factory.py changes for shadow-central post_upload command

Is this still needed?  post_upload.py looks like it handles setting product to shadow-central already.
Attachment #489081 - Flags: review?(catlee)
(In reply to comment #61)
> Comment on attachment 489081 [details] [diff] [review]
> [tested] factory.py changes for shadow-central post_upload command
> 
> Is this still needed?  post_upload.py looks like it handles setting product to
> shadow-central already.

doh! yes, you're right, post_upload.py is inserting the shadow-central product.
Comment on attachment 486845 [details] [diff] [review]
post_upload.py now with more generic paths [take 2]

http://hg.mozilla.org/build/tools/rev/f1cfb7097212
Attachment #486845 - Flags: checked-in+
Comment on attachment 488269 [details] [diff] [review]
config changes for new shadow-central take 3

http://hg.mozilla.org/build/buildbot-configs/rev/9712bc1b2338
Attachment #488269 - Flags: checked-in+
Created attachment 489314 [details] [diff] [review]
[tested] set ignore_certs to True for shadow-central DownloadFile() on tests/talos packages

This is mostly a bustage fix for win32 and macosx, which are failing tests/talos on lacking --no-check-certificate. The other platforms have been fine without it.  Tested in mozilla-tests1 in WINNT 6.1.
Attachment #489314 - Flags: review?(nrthomas)
Comment on attachment 489314 [details] [diff] [review]
[tested] set ignore_certs to True for shadow-central DownloadFile() on tests/talos packages

I'd either globally ignore certs (not so great) or evaluate 
 self.branchName.lower().startswith('shadow')
once.
Attachment #489314 - Flags: review?(nrthomas) → review-
Created attachment 489398 [details] [diff] [review]
ignore certs v.2

Evaluating once (per factory due to inheritance not lining up between Test factory and Talos factory).
Attachment #489314 - Attachment is obsolete: true
Attachment #489398 - Flags: review?(nrthomas)
Comment on attachment 489398 [details] [diff] [review]
ignore certs v.2

Looks OK, works in staging ?
Attachment #489398 - Flags: review?(nrthomas) → review+
Hey, maybe I'm missing something mentioned already in a comment, but why are we disabling cert checking?
A couple of the platforms (macosx 10.5.8, win32) are not able to download the packages needed for talos/tests from the https:// location because of doing wget without --no-check-certificates
Created attachment 489505 [details] [diff] [review]
ignore certs v.3

Carrying forward nthomas' r+. Tested in staging and learned that I had to move the lines in MozillaTestFactory to after the MozillaBuildFactory is initialized to access branchName, and in TalosFactory had to move down a few lines to be under self.branchName getting set.
Attachment #489398 - Attachment is obsolete: true
(Assignee)

Updated

7 years ago
Flags: needs-reconfig?
Comment on attachment 489505 [details] [diff] [review]
ignore certs v.3

http://hg.mozilla.org/build/buildbotcustom/rev/7ba9b1f5a04a
Attachment #489505 - Flags: checked-in+
(Assignee)

Updated

7 years ago
Depends on: 611245
Created attachment 490023 [details] [diff] [review]
[tested] config changes for enabling leak test log steps in shadow-central

ran this on sm01, and did a leak test build in mozilla-central too to make sure that it isn't adversely affected.
Attachment #490023 - Flags: review?(catlee)
Created attachment 490024 [details] [diff] [review]
[tested] shadow-central logBaseUrl override added to MercurialBuildFactory in order to enable LeakTestSteps
Attachment #490024 - Flags: review?(catlee)
(In reply to comment #70)
> A couple of the platforms (macosx 10.5.8, win32) are not able to download the
> packages needed for talos/tests from the https:// location because of doing
> wget without --no-check-certificates

Why don't they have the right certs? I think disabling cert checking is just hiding some future problem. We should just add the right certs on that box so cert checking works.

Updated

7 years ago
Attachment #490024 - Flags: review?(catlee) → review+

Updated

7 years ago
Attachment #490023 - Flags: review?(catlee) → review+
Comment on attachment 490024 [details] [diff] [review]
[tested] shadow-central logBaseUrl override added to MercurialBuildFactory in order to enable LeakTestSteps

http://hg.mozilla.org/build/buildbotcustom/rev/94c2c863d60f
Attachment #490024 - Flags: checked-in+
Comment on attachment 490023 [details] [diff] [review]
[tested] config changes for enabling leak test log steps in shadow-central

http://hg.mozilla.org/build/buildbot-configs/rev/dc3e5259438b
Attachment #490023 - Flags: checked-in+
Codesighs steps have been fixed, leak builds are working now too. This branch is live and ready for use.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
(Assignee)

Updated

7 years ago
Attachment #489505 - Attachment is patch: true
Attachment #489505 - Attachment mime type: application/octet-stream → text/plain
Product: mozilla.org → Release Engineering

Updated

2 years ago
Group: core-security → core-security-release
Group: core-security-release
You need to log in before you can comment on or make changes to this bug.