563081 - deb repo updates with both chinook and fremantle

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P4)

Description

[Aki Sasaki (not active)]

And fremantle-qt, if applicable.

Comment 1

[Aki Sasaki (not active)]

Currently rewriting the (single locale, single platform) makefile in python+json.

Comment 2

[Aki Sasaki (not active)]

Attachment: run script in buildbot

tested staging trunk on smm:8011.
Bye bye funky for loop!

Comment 3

[Aki Sasaki (not active)]

Attachment: nuke the obsolete makefile from build/tools

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 452147 [details] [diff] [review]
run script in buildbot

>+def isstr(arg):
>+    try:
>+        if str(arg) == arg:
>+            return True
>+    except:
>+        pass
>+    return False

This method is not needed.  A replacement is 'if type(arg) is str:' instead of 'if isstr(arg):'.  To deal with the exceedingly rare case of a string subclass you could use issubclass(c,b).

>>> class str2(str):
...   def __str__(self):
...     return 'Not the real representation'
...
>>> s=str2('john')
>>> str(s) == s
False
>>> issubclass(type(s), str)
True


> class DebRepoSign(BuildFactory):
>+    def __init__(self, configFile, scriptRepo, clobber=True,
>+                 scriptName='mozharness/scripts/signdebs.py', locales=None,
>+                 env=None, platforms=None, workDir=None,
>+                 logLevel='info',
>+                 baseWorkDir="/scratchbox/users/cltbld/home/cltbld",

I would like to move towards using the bind mount and would prefer that this default for the baseworkdir not be set to a scratchbox related path.  It should be set in the config files if it is going to be scratchbox installation specific imo.

>+        if workDir:
>+            self.workDir = workDir
>+        else:
>+            self.workDir = 'deb-%s' % branch

i don't know where 'branch' is supposed to come from.

>+        self.absWorkDir = '%s/%s' % (baseWorkDir, self.workDir)
>+        self.repoCommand = [self.scriptName, '--configFile', configFile,
>+                            '--workDir', self.workDir,
>+                            '--logLevel', logLevel]
>+        if locales:
>+            if isstr(locales):

as above, should be
              if type(locales) is str:
>+                self.repoCommand.extend(['--locale', locales])
>+            else:
>+                self.repoCommand.extend(['--locale', ','.join(locales)])
>+        if platforms:
>+            if isstr(platforms):

again, should be
              if type(platforms) is str:
>+                self.repoCommand.extend(['--platform', platforms])
>+            else:
>+                self.repoCommand.extend(['--platform', ','.join(platforms)])

>+        self.addStep(ShellCommand(
>+         command=['sh', '-c', 'if [ -d mozharness ] ; then ' +
>+                  'hg -R mozharness pull -r default ; else ' +
>+                  'hg clone ' + self.scriptRepo + ' mozharness ; ' +
>+                  'fi && hg -R mozharness update -r default'],
>+         env=self.env,
>+         workdir=self.absWorkDir,
>+         haltOnFailure=True,
>+         description=['clone', 'script', 'repo'],
>+        ))

Is there any reason why you aren't using the Mercurial class?  If you really do prefer using the shell commands, the pull doesn't need to use the -r flag, the clone should use --noupdate and the update should likely use -C.  The tag/revision should be configurable from the config file.

>+for branch in sorted(BRANCHES.keys()):
>+    bc = BRANCHES[branch]

can we call this something more descriptive than bc?  What about:

for branch_name in sorted(BRANCHES.keys()):
    branch = BRANCHES[branch]

>+    nightlyBuilderName = "%s deb sign nightly" % branch
>     nightlyDebFactory = DebRepoSign(
>-        branchNick=branchNick,
>-        buildToolsRepo=DEBSIGN_CONFIG['tools_repo_path'],
>-        env=branch['nightly']['env'].copy(),
>-        extraDebsList=branch['nightly']['extra_debs_list']
>+        configFile=bc['nightly_config_file'],
>+        scriptRepo=bc.get('scriptRepo', 'http://hg.mozilla.org/users/asasaki_mozilla.com/mozharness'),

If we are going to run out of a user repo, lets tag it and have the tag in the config files.  Alternately, can we check it into the tools repository?

>+        locales=bc.get('nightly_locales', None),
>+        env=bc.get('env', None),

I assume that the script will be responsible for setting up the environment it needs.  Is this correct?

>+    c['schedulers'].append(Nightly(
>+        name="%s deb nightly scheduler" % branch,
>+        branch=branch,
>+        hour=[4],
>         builderNames=[nightlyBuilderName],
>     ))

If a nightly build finishes after 4 am, how will we ensure that it is processed?  If we rekick a nightly build, will we have to reconfig this master?

Comment 5

[Aki Sasaki (not active)]

(In reply to comment #4)
> This method is not needed.  A replacement is 'if type(arg) is str:' instead of
> 'if isstr(arg):'.

Ok, wfm

> I would like to move towards using the bind mount and would prefer that this
> default for the baseworkdir not be set to a scratchbox related path.  It should
> be set in the config files if it is going to be scratchbox installation
> specific imo.

The bind mount isn't on staging-mobile-master. Until we have deb-signing specific slaves, I think we should leave it.

> i don't know where 'branch' is supposed to come from.

Good point.

> Is there any reason why you aren't using the Mercurial class?

Yup, that's what I had before. No real objection to using the Mercurial class though.

> can we call this something more descriptive than bc?  What about:

Heh. Sure.

> If we are going to run out of a user repo, lets tag it and have the tag in the
> config files.  Alternately, can we check it into the tools repository?

Sure, I'll tag until it's mature enough to go into tools or elsewhere.
It won't just be the one script though; it would need to be the entire repo.

> I assume that the script will be responsible for setting up the environment it
> needs.  Is this correct?

Correct.

> If a nightly build finishes after 4 am, how will we ensure that it is
> processed?  If we rekick a nightly build, will we have to reconfig this master?

Nope, force build works unless we change directories/branches/platforms etc.

Comment 6

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #5)
> > If a nightly build finishes after 4 am, how will we ensure that it is
> > processed?  If we rekick a nightly build, will we have to reconfig this master?
> 
> Nope, force build works unless we change directories/branches/platforms etc.

is that force build on the debsign master or the master that did the build?

Comment 7

[Aki Sasaki (not active)]

Debsign master. Same as it is today, except there's no trigger builders, just the signing builders.

In the future, a sendchange or a trigger/dependency in the schedulerdb should do it.

Comment 8

[Aki Sasaki (not active)]

Attachment: with jhford's fixes

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 452765 [details] [diff] [review]
with jhford's fixes

>+BRANCHES["mozilla-central"]["script_repo_tag"] = 'PRODUCTION'

+        scriptRepoPath=branch.get('scriptRepo', 'http://hg.mozilla.org/users/asasaki_mozilla.com/mozharness'),
+        scriptRepoTag=branch.get('scriptRepoTag', 'default'),


Are these supposed to be mismatched?  r+ with s/scriptRepo/script_repo/ and s/scriptRepoTag/script_repo_tag/.

I am going to look at the PRODUCTION tag of the repository as part of my review for the Makefile change.

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 452765 [details] [diff] [review]
with jhford's fixes

http://hg.mozilla.org/build/buildbot-configs/rev/47ec014dbdff

Comment 11

[Aki Sasaki (not active)]

Attachment: Once bug 563770 is fixed, add fremantle-qt to the list of trunk platforms

Comment 12

[Aki Sasaki (not active)]

Since jhford wants to look at the PRODUCTION tag of mozharness, that's currently here:

http://hg.mozilla.org/users/asasaki_mozilla.com/mozharness/file/679dd1bd203b

Comment 13

[Aki Sasaki (not active)]

Hm, this works perfectly with "force build" but the Mercurial step dies on the Nightly scheduler =(

I'll have to debug and add a new patch.

Comment 14

[Aki Sasaki (not active)]

Comment on attachment 452935 [details] [diff] [review]
Once bug 563770 is fixed, add fremantle-qt to the list of trunk platforms

http://hg.mozilla.org/build/buildbot-configs/rev/6226e9eb4cd6

Comment 15

[Aki Sasaki (not active)]

This is live and working.
jhford still wants to look at my scripts, but agrees that that can be done in a side bug with lower priority.

Resolving.