373995 - rewrite update/l10n verification scripts

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Robert Helmer [:rhelmer]]

The current l10n and update verification scripts in mozilla/testing/release are written in bash and are kind of cumbersome to use; they also duplicate a lot of logic that we have elsewhere (e.g. unpack a build), we could actually parse the update.xml for real, and do more complex tests than I am comfortable writing in bash.

A description of what they do now and what I'd like them to be able to do is here:
http://wiki.mozilla.org/User:Rhelmer:Verification_proposals

It's probably worth considering our overall release verification process, as we could use some refactoring across the board.

One goal should be to make it easy to run these on our standard build VM, per platform, so e.g. l10n can run right after l10n repacks are done (before pushing to stage), and updates can run on the platform they are intended for (win32 on win32, linux on linux).

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

needed for "release automation", hence marking as critical.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

assignee changed during triage.

Comment 3

[Robert Helmer [:rhelmer]]

This has been moved to the default assignee for Core/Testing; I don't have time to work on it right now. Halp wanted :)

Comment 4

[Rob Campbell [:rc] (:robcee)]

yeah, this has been on my todo list for months now, but I haven't had time to work on it. Accepting patches! :)

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to comment #4)
> yeah, this has been on my todo list for months now, but I haven't had time to
> work on it. Accepting patches! :)

I think one of the goals here should be to not need configuration; the scripts can figure out reasonable defaults. There will probably be edge cases we will need to override, but let's worry about the most common case for now.

It might help to go back to fundamentals, and ignore the way the current stuff is written; this might actually be a very simple project:

Both l10n and update verify should share this first step:

1) given a set of (RC) binaries, determine the previous version equivalents and download them

I think the requirements for l10n are:
1) " "
2) compare contents of current version/previous version, for each locale

We can have rules about what's ok/not ok. "metadiff" might be useful for informational purposes, but I really want this to be a PASS/FAIL test.

I think the requirements for update are:
1) " "
2) construct AUS URL based on previous build ID, version and product name
3) download and verify MARs, and attempt to apply update
4) (same as l10n #2)

So, they can certainly share a lot of code.

robcee, what are you using to do unpacking and grabbing the build ID for Talos? I think we should use that here, or use one of the earlier pre-existing methods.

I can at least get this started, l10n in particular should be dead simple. 

Comment 6

[Rob Campbell [:rc] (:robcee)]

Attachment: PerfConfigurator.py

yeah, I like the sound of this.

I've got a python script, "PerfConfigurator.py" that just grabs the build id from the talkback master.ini and then updates a YAML file with it and a few other variables. That needs to change for trunk as we're removing talkback, but, pending bsmedberg's patches to add the build id to an external file, we'll be able to update it. In the meantime, take a look at how I'm grabbing it from the patch. As usual, it might be easier to just lift out the parts you need.

Comment 7

[Robert Helmer [:rhelmer]]

Currently, update verification works like this (this assumes that you know all the info needed to build an AUS URL, that is build platform/locale/version/build ID):

1) download the latest release build
2) contact AUS, download the MARs, verify the size/checksum of each
3) attempt to apply updates

If the update failed to apply, then it's a FAIL. If it did apply, then for each 
previous release, then:

4) contact AUS for every previous release, download and compare checksum/size for all MARs against update.xml

This is good, but it doesn't test that every previous release points to the latest complete MAR, when we know it should. Also, it spends time downloading and checksumming the MARs when it's not necessary.

We can both check for this class of problems as well as spend less time, by changing step 4 and doing a couple more things:

4) store the checksum/size info for the complete MAR

For each previous release:

5) contact AUS, compare checksum/size to output of step 4
6) do an HTTP HEAD and verify the Content-Length of the MAR itself

Comment 8

[Robert Helmer [:rhelmer]]

We need this for release automation; I am going to start moving this into Bootstrap.

Comment 9

[Robert Helmer [:rhelmer]]

Attachment: installer build unpacker for MozBuild::Util

Here's a rough first stab; tested on exe/tar.gz/dmg files. Tried to keep the API very simple; let me know what you think.

Comment 10

[J. Paul Reed [:preed]]

Comment on attachment 274874 [details] [diff] [review]
installer build unpacker for MozBuild::Util


This is pretty good... some thoughts...


>+##
>+# Unpacks Mozilla installer builds.
>+#
>+# Arguments:
>+#     file    - file to unpack
>+#     unpackDir - dir to unpack into
>+##
>+
>+sub UnpackBuild {
>+    my %args = @_;
>+
>+    my $file = $args{'file'};
>+    my $unpackDir = $args{'unpackDir'};
>+
>+    if (! defined($file) ) {
>+        die("UnpackBuild: file is a required argument: $!");

I usually mark these lines with ASSERT, a la; die("ASSERT: UnpackBuild: blah");

>+    if ($suffix eq 'dmg') {
>+        my $mntDir = './mnt';
>+        if (! -d $mntDir) {
>+            mkdir($mntDir) || die("UnpackBuild: cannot create mntdir: $!");
>+        }
>+        system("echo \"y\" | PAGER=\"/bin/cat\" hdiutil attach -quiet -puppetstrings -noautoopen -mountpoint ./mnt \"$file\"") || die ("UnpackBuild: Cannot unpack $file: $!");

This is ugly, but you explained why you need it... you might comment in here why you're not using RunShellCommand.

>+        my $rv = RunShellCommand(command => 'rsync',

So, for all of these sub-commands you run, you should follow the form of the signing tools; they all use variables like $rsync for the tools they call, and they have statement to check the environment for a path to the tool, and if it's not there, just use "rsync" or "7z." This has proven invaluable when trying to use the same script among multiple versions of the same tool (for old branches and stuff).

>+                                 args => ['-av', $mntDir, $unpackDir],
>+                                 output => 1);

Libraries shouldn't output stuff to stdout.

>+        my $oldpwd = $ENV{'PWD'};

Use getcwd() for this, not %ENV.

Other than that, looks good; I would be interested in porting the signing tools to use this when it's in.

Comment 11

[Robert Helmer [:rhelmer]]

Attachment: address preed's comments

Comment 12

[Bob Clary [:bc] (inactive)]

I think you may have problems with using echo "Y" | hdiutil in a fully patched OS X. I used to use that until it broke recently (~month) after a security update although I didn't use the exact set of arguments you are using. I had to use expect to answer the prompt.

Comment 13

[Robert Helmer [:rhelmer]]

(In reply to comment #12)
> I think you may have problems with using echo "Y" | hdiutil in a fully patched
> OS X. I used to use that until it broke recently (~month) after a security
> update although I didn't use the exact set of arguments you are using. I had to
> use expect to answer the prompt.

Yeah I know that this is a problem in newer versions.. I'm going to check in what I've got and add more workarounds as-needed :)

Comment 14

[Robert Helmer [:rhelmer]]

That bit's landed:
Checking in MozBuild/Util.pm;
/cvsroot/mozilla/tools/release/MozBuild/Util.pm,v  <--  Util.pm
new revision: 1.19; previous revision: 1.18
done

Now I just need to write code that actually uses it. I'm basically going to port the contents of mozilla/testing/release/, both l10n and update verification, directly into bootstrap, as a first step.

Comment 16

[Robert Helmer [:rhelmer]]

Attachment: first stab at update verify rewrite

Here's a first step towards replacing the core update verification functionality (download build N-1, apply update, compare to build N).

The more I work on this, the more things I see to refactor :) For example the OS mapping stuff and especially the "fetch *info.txt from FTP" bits should be common functions.

However, I wanted to put this out there so someone can correct me if I seem to be going off the rails.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 278545 [details] [diff] [review]
first stab at update verify rewrite

We talked about some of this stuff on IRC already but repeating here for completeness.

>Index: Bootstrap/Step/Updates.pm
...
>+    } elsif ($os eq 'win32') {
>+        $suffix = 'mac.dmg';
>+        $bouncerName = 'osx';
>+        $platform = 'Darwin_Universal-gcc3';

win32 vs mac mismatch here, same for following elsif. 

I agree about factoring some of this into utility functions, similar to what's at the top of Bootstrap/Util.pm. Ideally we wouldn't have differences between strings used for os, shipped-locales & bouncer, but we only have leverage on shipped-locales.

>+    # Walk through locale manifest, only include valid locales
>+    my @locales = ();
>+    my $localeInfo = $config->GetLocaleInfo();
>+    foreach my $locale (sort(keys(%{$localeInfo}))) {
>+        my $exceptions = $localeInfo->{$locale};
>+        for my $exception (@$exceptions) {
>+            if ($bouncerName eq $exception) {
>+                push(@locales, $locale);

The variable name $exceptions threw me, AFAICT GetLocaleInfo returns a list of shipped locales rather than unshipped. Also, don't you want the locale list from the previous list ? Might have to make GetLocaleInfo use a tag if it's passed.

>+    foreach my $locale (@locales) {
>+
>+        # grab os-specific buildID file on FTP

Factoring this into a separate util function is definitely a good idea, similar to how you made StoreBuildID in the Build step. Also, all the locales have the same BuildID so this can be done outside the locale loop.

>+        my $candidateDir = '/home/ftp/pub/' .  $product .  '/nightly/' . 
>+                           $oldVersion . '-candidates/rc' . $oldRc;

You could use GetFtpCandidatesDir(bitsUnsigned => 0) here, then substitute oldVersion for version, oldRc for Rc. Then we'd only look up this location in one place.

>+            cmd => 'scp',
>+            cmdArgs => [$stagingUser . '@' . $stagingServer . ':' . 
>+                        $candidateDir .'/' . $os . '_info.txt',
>+                        $buildIDTempFile],

We've used rsync to push/pull files alot elsewhere in the code. You have scp here and http a bit lower down.  I don't think there's much between them, but I'm curious why you've done it this way.

>+        if (! $buildID =~ /^\d+$/) {
>+            die("ASSERT: BumpPatcherConfig: $buildID is non-numerical");

Could check for the right number of digits here, or even better in Build:StoreBuildID.

>+        my $oldUrl =  'http://' . $stagingServer . '/pub/mozilla.org/' . 
>+                      $product .  '/nightly/' . 
>+                      $oldVersion . '-candidates/rc' . $oldRc . '/' .
>+                      $product . '-' . $oldVersion . '.' . $locale . '.' . 
>+                      $suffix;

Hm, perhaps it's time to teach GetFtpCandidatesDir how to do http:// as well as paths, previous version. All these manually constructed URLs make me a bit nervous.

>+        # FIXME hardcoded product name
>+        my $ausUrl = 'https://aus2.mozilla.org/update/2/Firefox/' . $oldVersion . '/' . $buildID . '/' . $platform . '/' . $locale . '/' . $channel . '/update.xml';

Probably also worth commenting that we're using the 2nd version of the URL scheme rather than the third, and throw a ?force=1 on the end for good measure.

>+    UnpackBuild(file => catfile($verifyDir, 'target-' . $suffix), 
>+              unpackDir => catfile($verifyDir, 'target'));

On windows, UnpackBuild() doesn't yet support combining nonlocalized, localized, and optional dirs after calling 7z to extract, so the updater would fail. The idea of running the installer has merit if you can convince it to install somewhere disposable. And if we're taking a step in that direction, it's worth remembering that the majority of windows users will have updates applied out of C:\Docs & Settings\... rather than C:\Program Files\Mozilla Firefox\updates\0. Hard to know how far to go with mimicking users.

>+    # FIXME really parse update.xml to determine marUrl
>+    # FIXME cycle through partial and complete mars

Aye :-)

>+    # Apply update 
>+    $this->Log('msg' => 'Attempting to apply update...');
>+        cmd => './updater',
>+        cmdArgs => ['../', '0'],
>+        logFile => $verifyLog,
>+        dir => catfile($verifyDir,'source', 'firefox'),
>     );

Other platforms reminder.
 
>+    # Recursive comparison of source and target

There are a few exclusions to the diff in the existing code which would be worth re-evaluating.

Comment 18

[Robert Helmer [:rhelmer]]

Attachment: cleanup and refactor, work in progress

Not quite ready for formal review, but wanted to get this out there, if anyone has comments.

This actually works cross-platform, refactored out the buildID stuff to GetBuildID().

I've been thinking about what's missing here.. mostly we don't have complete information about the list of past releases. We could use the patcher.cfg for this, but I was thinking of a simpler way that I think is a better test - why not try each previous release e.g. 2.0.0.6, 2.0.0.5, download all builds for the relevant OS, extract the buildID from them and check that they are pointing to the expected update path?

Then we need no special information such as build ID, locale, etc. It's all extracted from the builds themselves, which seems like a good thing to test.

Comment 19

[Robert Helmer [:rhelmer]]

Attachment: refactor into Bootstrap::Build class

Let me know what you think of this approach; the basic idea is that you can instantiate an object of type Bootstrap::Build, which stands for a particular file. For example:

my $build = Bootstrap::Build(product='fx', version='2', rc='1', osname='win32', locale='en-US');

Now you can do things like:

my $filename = $build->GetFileName();
my $buildID = $build->GetBuildID();
$build->Unpack(fromDir => '1', dest => '2');

This makes Updates.pm way more readable, and I think this'd be useful in other areas of the code. It does whatever is necessary to get this info, e.g. download the build, and keeps it around for as long as the object is alive.

Comment 20

[Robert Helmer [:rhelmer]]

Attachment: disable stage_depscheduler until auto-update-verify is fixed

We actually need to automatically deploy Updates as well, but until we can auto-configure updateverify it's a moot point.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 282844 [details] [diff] [review]
refactor into Bootstrap::Build class

Sorry for the delay - overall this looks pretty good. At first I was a bit concerned about having two packages called Build, but I think I got used to it. Anyways, no obvious alternative to Bootstrap::Build occurred to me, and renaming Step::Build to (eg) Step::Compile seemed a bit painful. A few things I noticed are inline.

>Index: bootstrap.cfg.example
>==================================================================

Could remove verifyDir from this

Index: Bootstrap/Build.pm
>===================================================================

>+# TODO support application.ini for 1.9+
>+sub GetBuildID { 

Could we avoid having two GetBuildID functions by calling them different names ? Maybe GetBuildIDFromServer & GetBuildIDFromInstaller, a bit long but clearer.

>+sub Unpack {
...
>+    my $config = new Bootstrap::Config();
>+    my $stagingServer = $config->Get('var' => 'stagingServer');
>+    my $product = $config->Get('var' => 'product');

Left over from development ? Could use $this->{'product'}

>+    UnpackBuild(file => catfile($fromDir, $this->GetFileName()), 
>+                unpackDir => $dest)
>+     or die("Could not unpack build: $!");
>+
>+    if ($this->{'osname'} eq 'win32') {
>+        $this->Shell(
>+          cmd => 'rsync',              
>+          cmdArgs => ['-av',
>+                      catfile($dest, 'source') . '/nonlocalized/',
>+                      catfile($dest, 'source', $product), 
>+                      ],  
>+          dir => $dest,
>+        );  
>+        $this->Shell(
>+          cmd => 'rsync',
>+          cmdArgs => ['-av',
>+                      catfile($dest, 'source') . '/localized/',
>+                      catfile($dest, 'source', $product), 
>+                      ],  
>+          dir => $dest,
>+        );  
>+        $this->Shell(
>+          cmd => 'rsync',
>+          cmdArgs => ['-av',
>+                      catfile($dest, 'source') . '/optional/',
>+                      catfile($dest, 'source', $product), 
>+                      ],  
>+          dir => $dest,

Could we role this into MozBuild::Util::UnpackBuild ? Also, how does the 'source' part of this work if $dest is $verifyDir/target ?

>+sub ApplyUpdate {
...
>+    if ($this->{'osname'} eq 'linux') {
>+        $dirPrefix = '';
>+        $updater = 'updater';
>+    } elsif ($this->{'osname'} eq 'macosx') {
>+        $dirPrefix = 'Contents/MacOS';
>+        $updater = 'updater.app/Contents/MacOS/updater';
>+    } elsif ($this->{'osname'} eq 'win32') {
>+        $dirPrefix = '';
>+        $updater = 'updater';
>+    } else {
>+        die('Unsupported OS' . $this->{'osname'});
>+    }
>+    if ($osname eq 'macosx') {
>+    } elsif ($osname eq 'win32') {
>+        $updater = 'updater.exe';
>+    }

Can tidy the second if block into the first.

>+# Static method
>+sub GetLocalesFromFTP {
>+    my %args = @_;
>+
>+    my $version = $args{'version'};
>+    my $osname = $args{'osname'};
>+
>+    my $osDir;
>+    if ($osname eq 'linux') {
>+        $osDir = 'linux-i686';
>+    } elsif ($osname eq 'macosx') {
>+        $osDir = 'mac';
>+    } elsif ($osname eq 'win32') {
>+        $osDir = 'win32';
>+    } else {
>+        die("Unsupported OS $osname");
>+    }

There are similar transformations in GetFileName and GetInstallerURL, maybe farm it out to a separate function in Bootstrap::Util.

>+    foreach my $entry (@entries) {
>+        if ($entry->type() eq "[DIR]") {
>+          my $filename = $entry->filename();
>+          $filename =~ s/\/$//;
>+          next if ($filename =~ /^\/pub/);

I'm curious what this last line guards against.

>Index: Bootstrap/Step.pm
>===================================================================
...
>+sub GetBuildID() {

See comments above, and I'm not sure whether this belongs here or in Bootstrap::Util. What are your thoughts ?

>Index: Bootstrap/Step/Updates.pm
>===================================================================
...
>+    foreach my $locale (@{$locales}) {
>+        my $verifyDir = tempdir(CLEANUP => 1);
>+        my $newBuild = new Bootstrap::Build(
>+         product => $product,
>+         version => $version,
>+         rc => 1,

$rc instead of 1.

>+        foreach my $marType ('complete', 'partial') {
>+            my $oldBuild = new Bootstrap::Build(
>+             product => $product,
>+             version => $oldVersion,
>+             osname => $osname,
>+             locale => $locale,
>+             channel => 'betatest',
>+            );

Any thoughts on what we'll do to verify releasetest/release later on ? Earlier in the bug you mentioned checking completeness of the update snippets. Do you hope to get that in this round or later ? Should be pretty easy to make a GetVersionsFromFTP function.

>-    # Customize updates.cfg to contain the channels you are interested in 
>-    # testing.
>+            my $oldUrl = $oldBuild->GetInstallerURL();
>+        
>+            $this->Log('msg' => 'Downloading old build from ' . $oldUrl);
>+            DownloadFile(url => $oldUrl, 
>+                         dest => catfile($verifyDir, $oldBuild->GetFileName()));
>+        
>+            $this->Log('msg' => 'Unpack old build');
>+            $oldBuild->Unpack(fromDir => $verifyDir,
>+                              dest => catfile($verifyDir, 'source'));

Can't see where the emptiness of this dir is checked.
        
>+            my $ausUrl = $oldBuild->GetAusURL();
>     
>-    my $verifyLog = catfile($logDir, 'updates_verify.log');
>-    $this->Shell(
>-      cmd => './verify.sh', 
>-      cmdArgs => ['-c', $verifyConfig],
>-      logFile => $verifyLog,
>-      dir => catfile($verifyDirVersion, 'updates'),
>-      timeout => 36000,
>-    );
>+            $this->Log('msg' => 'Downloading update.xml from ' . $ausUrl);
>+            my $updateFile = catfile($verifyDir, 'update.xml');
>+            DownloadFile(url => $ausUrl, dest => $updateFile);

Can everything from "foreach my $marType" to here (except the Unpack) go outside the loop ? Or is there a problem with existing vars on the object the second time round ?

Comment 22

[Robert Helmer [:rhelmer]]

Attachment: simple stopgap way to get auto-config

As we discussed in irc, here's a simple way to get this going right away: leave all current update verification tools in place, and simply bump the config (as we do for Tinderbox and Patcher).

I used part of attachment 282844 [details] [diff] [review] - refactor GetBuildID() into a common shared method.

I do think it's worth doing a full rewrite, but it's more complex than I originally thought, and I don't want to block fully automated releases on it any longer.

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 284343 [details] [diff] [review]
simple stopgap way to get auto-config

>Index: Bootstrap/Step.pm
...
>+ sub GetBuildID() {
>+    my $this = shift;

As we discussed on IRC, calling this function GetBuildIDFromFTP might save on confusion with TinderLogParse::GetBuildID.

>+    if (! $buildID =~ /^\d+$/) {
>+        die("ASSERT: BumpPatcherConfig: $buildID is non-numerical");

Any reason this can't be /^\d{8}$/ ?

>Index: Bootstrap/Step/Updates.pm
...
>+sub BumpVerifyConfig {
>+    my $this = shift;
>+    my %args = @_;
>+
>+    my $configFile = $args{'configFile'};

Leftovers ready for the chop ? (args and configFile lines)

...
>+    if ($origFile[0] =~ $version) {
>+            $this->Log('msg' => "verifyConfig $configFile already bumped");
>+            return;
>+    }

I'm a bit confused by this. The first line is a typically comment referring to the previous release, so you'd want $oldVersion in the test. Or you could check the next line for $version, where you're matching on the "to" definition.

...
>+    my $localeFileTag = uc($product).'_'.$oldVersion.'_RELEASE';
>+    $localeFileTag =~ s/\./_/g;
>+    print "TAG: $localeFileTag\n";

Remove this now ?

>+    my $localeManifest = GetLocaleManifest(app => $appName,
>+                                           cvsroot => $mozillaCvsroot,
>+                                           tag => $localeFileTag);
>+
>+    use Data::Dumper;

Leftover ?

...
>+    # add data for latest release
>+    my @data = ("# $version $osname\n",
>+                'release="'.$version.'" product="'.ucfirst($product).'" platform="'.$buildTarget.'" build_id="'.$buildID.'" locales="'.join(' ', sort(@locales)).'" channel="'.$channel.'" from="/'.$product.'/releases/'.$oldVersion.'/'.$ftpOsname.'/%locale%/'.$releaseFile.'" to="/'.$product.'/nightly/'.$version.'-candidates/rc'.$rc.'/'.$nightlyFile.'"'."\n");

Would to great to break up this line for (code) readability.

>+
>+    open(FILE, "> $configFile") or die ("Could not open file $configFile: $!");
>+    print FILE @data;
>+    print FILE @strippedFile;
>+    close(FILE) or die ("Could not close file $configFile: $!");
>+
>+    ## TODO check in and tag file when done

Please update the comment. We haven't been tagging this file, so I'm ok with not getting that now.

r+ with those changes.

Comment 24

[Robert Helmer [:rhelmer]]

Attachment: [checked in] simple stopgap way to get auto-config [as landed]

Checking in Bootstrap/Step.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step.pm,v  <--  Step.pm
new revision: 1.15; previous revision: 1.14
done
Checking in Bootstrap/Step/PatcherConfig.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/PatcherConfig.pm,v  <--  PatcherConfig.pm
new revision: 1.8; previous revision: 1.7
done
Checking in Bootstrap/Step/Updates.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Updates.pm,v  <--  Updates.pm
new revision: 1.17; previous revision: 1.16
done

Comment 25

[Robert Helmer [:rhelmer]]

Attachment: [checked in]set aus_server and ftp_server in verify config

Followup to attachment 284491 [details] [diff] [review] - for staging purposes, it'd be nice to change the aus and ftp servers to point to staging equivalents. We can override these settings in the verify config.

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 284493 [details] [diff] [review]
[checked in]set aus_server and ftp_server in verify config

Looks good. We'll need to remember to change ausServer in the bootstrap.cfg for staging once AUS is set up.

Comment 27

[Robert Helmer [:rhelmer]]

Comment on attachment 284493 [details] [diff] [review]
[checked in]set aus_server and ftp_server in verify config

Checking in Bootstrap/Step/Updates.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Updates.pm,v  <--  Updates.pm
new revision: 1.19; previous revision: 1.18
done

Comment 28

[Robert Helmer [:rhelmer]]

Attachment: [checked in] need to append /pub/mozilla.org to ftp_server

Sorry, missed a detail in attachment 284493 [details] [diff] [review] - verify.sh expects ftp_server to be e.g. "stage.mozilla.org/pub/mozilla.org"

Comment 29

[Robert Helmer [:rhelmer]]

Comment on attachment 284660 [details] [diff] [review]
[checked in] need to append /pub/mozilla.org to ftp_server

Checking in Bootstrap/Step/Updates.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Updates.pm,v  <--  Updates.pm
new revision: 1.21; previous revision: 1.20
done

Comment 30

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: [checked in] fix rc-vs-oldRc bug in Updates.pm

This caused update verification to fail during FF2009... changing from rc to oldRc should do the trick.

Comment 31

[Robert Helmer [:rhelmer]]

Comment on attachment 286324 [details] [diff] [review]
[checked in] fix rc-vs-oldRc bug in Updates.pm

Checking in Bootstrap/Step/Updates.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Updates.pm,v  <--  Updates.pm
new revision: 1.25; previous revision: 1.24
done

Comment 32

[Robert Helmer [:rhelmer]]

This bug has gotten a bit out of hand. I still want to rewrite these scripts, but this is too big of a task to stick into one bug.

I'm going to WONTFIX this and file more specific followup bugs, that are achievable in a reasonable amount of time.