Closed Bug 975034 Opened 10 years ago Closed 10 years ago

Upgrade ec2 test instances mesa versions from 8.0.2-0ubuntu3(32-bit hosts)/8.0.4-0ubuntu0.2(64-bit hosts) to 8.0.4-0ubuntu0.6mozilla1

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

x86
macOS
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jrmuizel, Assigned: armenzg)

References

Details

Attachments

(5 files, 4 obsolete files)

3.64 KB, patch
Details | Diff | Splinter Review
226.09 KB, patch
rail
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
2.45 KB, patch
rail
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
41.73 KB, patch
rail
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
1.50 KB, patch
rail
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
Attached patch Fix llvm pipeSplinter Review
We need the following patch to fix llvmpipe in the version of mesa that we are using.

It should be relatively easy to build new deb files with this patch.
Blocks: 818968
I hope to investigate this next week.
Assignee: nobody → armenzg
I should be able to look at this tomorrow.

Do you have build instructions for it?
apt-get build-dep mesa
apt-get source mesa
cd mesa-8.0.4
patch -p1 < $patch$
dpkg-buildpackage
I used the pbuilder approach (https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_DEBs#Old_Stuff) on "my laptop" (don't worry this approach is safe, everything happens in chroot environment).

# grab the latest package for precise, see links at the bottom of http://packages.ubuntu.com/source/precise-updates/mesa
dget http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/mesa_8.0.4-0ubuntu0.7.dsc
# unpack it
dpkg-source -x mesa_8.0.4-0ubuntu0.7.dsc

# download and enable the patch
cd mesa-8.0.4/
wget -O debian/patches/moz-fix-llvmpipe https://bug975034.bugzilla.mozilla.org/attachment.cgi?id=8379163
echo moz-fix-llvmpipe >> debian/patches/series

# Bump the version
DEBFULLNAME="Rail Aliiev" DEBEMAIL="rail@mozilla.com" dch -l mozilla --distribution precise-updates "Fix llvm pipe"

# rebuild the source package
dpkg-source -b .
cd ..

# build the binaries
sudo pbuilder --build --distribution precise --architecture amd64 --basetgz /var/cache/pbuilder/base-precise-amd64.tgz --buildresult out64 mesa_8.0.4-0ubuntu0.7mozilla1.dsc
sudo pbuilder --build --distribution precise --architecture i386 --basetgz /var/cache/pbuilder/base-precise-i386.tgz  --buildresult out32 mesa_8.0.4-0ubuntu0.7mozilla1.dsc

You can find the packages in the out64 and out32 directories. Then you need to upload them to the releng-updates repo, see https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#Ubuntu:_Landing_Custom_Repository_Changes

Hope it helps :)
I tried using the puppetagain-build-deb approach and failed.

In case Jeff wonders why this bug is looking so complicated. We need to be able to ensure reproducibility and commit the deb creating code into the puppet repo.
I'm also working on documenting this stuff in a way that helps a newbie like me catch up faster and rely less on others.

NOTE: I'm using a "precise-amd64" basepath instead of "precise-updates-amd64" (since rail built from 0.7.dsc "precise-updates" instead of 0.6.dsc "precise")
http://packages.ubuntu.com/source/precise-updates/mesa (0.7)
http://packages.ubuntu.com/source/precise/mesa (0.6)

My puppetagain-build-deb approach:
> ssh armenzg@ubuntu64packager1.srv.releng.use1.mozilla.com
> # Let's get our environment prepared
> sudo cowbuilder --login --basepath /var/cache/pbuilder/precise-amd64/ --bindmount /tmp
> apt-get install mercurial build-essential devscripts wget debhelper libparse-debianchangelog-perl unzip
> mkdir /tmp/mesa
> cd /tmp/mesa
> 
> # 1) here I start the same steps as rail
> 
> # grab the latest package for precise, see links at the bottom of
> # http://packages.ubuntu.com/source/precise-updates/mesa
> # unpack it
> dpkg-source -x mesa_8.0.4-0ubuntu0.7.dsc
> # download and enable the patch
> cd mesa-8.0.4/
> wget -O debian/patches/moz-fix-llvmpipe https://bug975034.bugzilla.mozilla.org/attachment.cgi?id=8379163
> echo moz-fix-llvmpipe >> debian/patches/series
> # Bump the version
> DEBFULLNAME="Rail Aliiev" DEBEMAIL="rail@mozilla.com" dch -l mozilla --distribution precise-updates "Fix llvm pipe"
> # rebuild the source package
> dpkg-source -b .
> 
> # 2) here my steps diverge
> cd /tmp
> hg clone http://hg.mozilla.org/build/puppet
> cd puppet/modules/packages/manifests
> cp -r /tmp/mesa/mesa-8.0.4/debian mesa-debian
> hg addremove
> hg diff > /tmp/mesa.diff
> cp /tmp/mesa/mesa_8.0.4.orig.tar.gz .
> chmod +x /tmp/puppet/modules/pkgbuilder/files/puppetagain-build-deb
> /tmp/puppet/modules/pkgbuilder/files/puppetagain-build-deb precise amd64 mesa-debian

It fails with this message:
> /var/cache/pbuilder/precise-amd64/ does not exist

Do I have to start the cowbuilder with /var/cache/pbuilder?

If I start puppetagain-build-deb outside of the cowbuiler I can reach /var/cache/pbuilder but I fail to meet the dependencies.

Here is the last end of the log where I get a message to be missing packages:
#############################################################################
Unable to resolve dependencies!  Giving up...
The following NEW packages will be installed:
  bsdmainutils{a} debhelper{a} dh-apparmor{a} diffstat{a} file{a} 
  gettext{a} gettext-base{a} groff-base{a} html2text{a} intltool-debian{a} 
  libcroco3{a} libgettextpo0{a} libmagic1{a} libpipeline1{a} libpopt0{a} 
  libunistring0{a} libxml2{a} man-db{a} pkg-config{a} po-debconf{a} 
  quilt{a} 
0 packages upgraded, 21 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/6202 kB of archives. After unpacking 19.3 MB will be used.
Abort.
 -> Finished parsing the build-deps
Reading package lists...
Building dependency tree...
Reading state information...
fakeroot is already the newest version.
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
 pbuilder-satisfydepends-dummy : Depends: debhelper (>= 8.1.3) but it is not going to be installed
                                 Depends: quilt (>= 0.40) but it is not going to be installed
                                 Depends: pkg-config
                                 Depends: libdrm-dev (>= 2.4.34) but it is not going to be installed
                                 Depends: libx11-dev but it is not going to be installed
                                 Depends: xutils-dev
                                 Depends: x11proto-gl-dev (>= 1.4.14) but it is not going to be installed
                                 Depends: libxxf86vm-dev but it is not going to be installed
                                 Depends: libexpat1-dev but it is not going to be installed
                                 Depends: libxfixes-dev but it is not going to be installed
                                 Depends: libxdamage-dev but it is not going to be installed
                                 Depends: libxext-dev but it is not going to be installed
                                 Depends: autoconf but it is not going to be installed
                                 Depends: automake but it is not going to be installed
                                 Depends: x11proto-dri2-dev (>= 2.6) but it is not going to be installed
                                 Depends: libx11-xcb-dev but it is not going to be installed
                                 Depends: libxcb-dri2-0-dev but it is not going to be installed
                                 Depends: libxcb-glx0-dev but it is not going to be installed
                                 Depends: libxcb-xfixes0-dev but it is not going to be installed
                                 Depends: python-libxml2 but it is not going to be installed
                                 Depends: libudev-dev but it is not going to be installed
                                 Depends: flex
                                 Depends: bison
                                 Depends: llvm-3.0-dev but it is not going to be installed
                                 Depends: libwayland-dev (>= 0.85.0) but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
############################################

Here's my second attempt based on rail's approach. I start again but resuming from where we diverge on "#2":
> ssh armenzg@ubuntu64packager1.srv.releng.use1.mozilla.com
> cd /tmp/mesa
> sudo pbuilder --build --distribution precise --architecture amd64 --basetgz /var/cache/pbuilder/base-precise-amd64.tgz --buildresult out64 mesa_8.0.4-0ubuntu0.7mozilla1.dsc

It was working but I aborted it until I hear about what approach I should take.

Is it because we started of with the source code based on "precise-updates" instead of "precise"?

Am I having trouble with 
I see that the last thing that puppetagain-build-deb calls is:
+ sudo -E cowbuilder --execute --bindmounts /tmp/tmp.aox3Y6WSuX --basepath /var/cache/pbuilder/precise-amd64/ --distribution precise --architecture amd64 -- /usr/lib/pbuilder/pdebuild-internal /tmp/tmp.aox3Y6WSuX/Mesa-8.0.4 --debbuildopts '' --debbuildopts '' --uid 1048 --gid 1048 --pbuildersatisfydepends /usr/lib/pbuilder/pbuilder-satisfydepends
FTR, rail says that it generates a lot of packages:
http://people.mozilla.org/~raliiev/sattap/21e5c13d.png
Attachment #8383938 - Flags: feedback?(rail)
Comment on attachment 8383938 [details] [diff] [review]
mesa (8.0.4-0ubuntu0.7mozilla1) precise-updates

lgtm
Attachment #8383938 - Flags: feedback?(rail) → feedback+
I can follow the steps of rail in comment 4 on my local machine.

dustin, what can I do to make the cowbuilder meet the dependencies?

I can continue without following the documentation but I would prefer making things work for the next person.
puppetagain-build-deb is already installed on that host, and it *calls* cowbuilder.  So you shouldn't have to pay much attention to cowbuilder at all - just get the debian directory set up correctly.

That said, as far as dependencies go, it's a little scary that it needs a new version of debhelper.  If it's just mesa that needs to be upgraded, it's probably better to start with the debian package matching that in our puppetagain repo, and just bump the version number of mesa that it builds, rather than trying to use a new package (which may depend on lots of other new things).
This is the version we have installed:
[root@tst-linux32-spot-301.test.releng.usw2.mozilla.com ~]# dpkg -l | grep "mesa"
ii  libgl1-mesa-dri                        8.0.2-0ubuntu3                          free implementation of the OpenGL API -- DRI modules
ii  libgl1-mesa-glx                        8.0.2-0ubuntu3                          free implementation of the OpenGL API -- GLX runtime
ii  libglapi-mesa                          8.0.2-0ubuntu3                          free implementation of the GL API -- shared library
ii  libglu1-mesa                           8.0.2-0ubuntu3                          Mesa OpenGL utility library (GLU)

https://launchpad.net/ubuntu/precise/i386/libegl1-mesa-drivers/8.0.2-0ubuntu3
Should the distribution be "release" instead of "updates"?
Do I have to run pbuilder again but with:
--othermirror "deb http://archive.ubuntu.com/ubuntu precise-release main restricted universe multiverse"
and a different basepath?

Currently building locally:
dget http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/mesa_8.0.2-0ubuntu3.dsc
# unpack it
dpkg-source -x mesa_8.0.2-0ubuntu3.dsc

# download and enable the patch
cd mesa-8.0.2/
wget -O debian/patches/moz-fix-llvmpipe https://bug975034.bugzilla.mozilla.org/attachment.cgi?id=8379163
echo moz-fix-llvmpipe >> debian/patches/series

# Bump the version
DEBFULLNAME="Rail Aliiev" DEBEMAIL="rail@mozilla.com" dch -l mozilla --distribution precise-updates "Fix llvm pipe"

# rebuild the source package
dpkg-source -b .
cd ..

# build the binaries
sudo pbuilder --build --distribution precise --architecture amd64 --basetgz /var/cache/pbuilder/base-precise-amd64.tgz --buildresult out64 mesa_8.0.2-0ubuntu3mozilla1.dsc
sudo pbuilder --build --distribution precise --architecture i386 --basetgz /var/cache/pbuilder/base-precise-i386.tgz  --buildresult out32 mesa_8.0.2-0ubuntu3mozilla1.dsc
Now building it as well on ubuntu64packager1.

cd puppet/modules/packages/manifests
patch -p1 < ~/mesa.diff
wget https://launchpad.net/ubuntu/+archive/primary/+files/mesa_8.0.2.orig.tar.gz
puppetagain-build-deb precise amd64 mesa-debian

Let's see how it goes.
jrmuizel: would a patched version of 8.0.2 work for you?
(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #14)
> jrmuizel: would a patched version of 8.0.2 work for you?

Certainly.
ubuntu64packager1 seems to have succeeded for amd64. I'm now building for i386.

I got a build failure for 8.0.2 when building locally:
In file included from nouveau_screen.c:27:0:
nouveau_driver.h:41:28: fatal error: nouveau_device.h: No such file or directory
compilation terminated.
make[7]: *** [nouveau_screen.o] Error 1
make[7]: Leaving directory `/tmp/buildd/mesa-8.0.2/build/dri/src/mesa/drivers/dri/nouveau'
make[6]: *** [lib] Error 2
make[6]: Leaving directory `/tmp/buildd/mesa-8.0.2/build/dri/src/mesa/drivers/dri/nouveau'
make[5]: *** [subdirs] Error 1
make[5]: Leaving directory `/tmp/buildd/mesa-8.0.2/build/dri/src/mesa/drivers/dri'
make[4]: *** [default] Error 1
make[4]: Leaving directory `/tmp/buildd/mesa-8.0.2/build/dri/src/mesa/drivers'
make[3]: *** [driver_subdirs] Error 2
make[3]: Leaving directory `/tmp/buildd/mesa-8.0.2/build/dri/src/mesa'
make[2]: *** [subdirs] Error 1
make[2]: Leaving directory `/tmp/buildd/mesa-8.0.2/build/dri/src'
make[1]: *** [default] Error 1
make[1]: Leaving directory `/tmp/buildd/mesa-8.0.2/build/dri'
make: *** [debian/stamp/x86_64-linux-gnu-build-dri] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
E: Failed autobuilding of package
Summary: Deploy patched version of Mesa 8.0.4 → Patch our current version of Mesa (8.0.2) for our EC2 test machines
Comment on attachment 8386234 [details] [diff] [review]
add to puppet mesa-debian - mesa (8.0.2-0ubuntu3mozilla1) precise-updates; urgency=low

Rubber stamp
Attachment #8386234 - Flags: review?(rail) → review+
Following the documentation:
https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#Ubuntu:_Landing_Custom_Repository_Changes

and I'm hitting this (using --ignore=wrongdistribution does not fix it):
root@ubuntu64packager1:/data/repos/apt# reprepro -V --basedir . include precise /tmp/mesa-precise-amd64/*.changes
.changes put in a distribution not listed within it!
To ignore use --ignore=wrongdistribution.
There have been errors!

I will start from scratch.
I'm re-running steps from comment 12 by changing from "precise-updates" to "precise":
DEBFULLNAME="Rail Aliiev" DEBEMAIL="rail@mozilla.com" dch -l mozilla --distribution precise "Fix llvm pipe"

The first line of debian/changelog should read like this:
mesa (8.0.2-0ubuntu3mozilla1) precise; urgency=low
instead of this:
mesa (8.0.2-0ubuntu3mozilla1) precise-updates; urgency=low
I'm not deploying to the in-house Ubuntu machines.
The in-house machines are using this version 9.0.0-0ubuntu1~precise1.
In other words, EC2 machines are running 12.04.04 and in-house are running 12.10. Quite unexpected.

The patch still need to be tested.
Attachment #8386327 - Flags: feedback?(rail)
Attachment #8386327 - Flags: feedback?(dustin)
Comment on attachment 8386234 [details] [diff] [review]
add to puppet mesa-debian - mesa (8.0.2-0ubuntu3mozilla1) precise-updates; urgency=low

Landed with changelog saying "precise" instead of "precise-updates".
https://hg.mozilla.org/build/puppet/rev/d6aac1ea887f
Attachment #8386234 - Flags: checked-in+
I take it back (only the package version is different):
[root@tst-linux64-spot-460.test.releng.usw2.mozilla.com ~]# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04 LTS

[cltbld@talos-linux64-ix-001.test.releng.scl3.mozilla.com ~]$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04 LTS
Comment on attachment 8386327 [details] [diff] [review]
deploy patched version of mesa 8.0.2 to EC2 test machines

This will need the debian build dir checked in as well.  Also, a comment modules/gui/manifests/init.pp would be helpful to remind us why this is only no non-GPU machines.
Attachment #8386327 - Flags: feedback?(dustin) → feedback+
So I'm confused - what *is* different?  Different versions of mesa?
Comment on attachment 8386327 [details] [diff] [review]
deploy patched version of mesa 8.0.2 to EC2 test machines

Review of attachment 8386327 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/packages/manifests/mesa.pp
@@ +7,5 @@
> +            package {
> +                # This package is a recompiled version of
> +                # https://launchpad.net/ubuntu/+source/mesa
> +                "mesa":
> +                    ensure => '8.0.2-0ubuntu3mozilla1';

This won't work since there is no such package. You have to specify exact names of the binary packages here.

You can list the package names you compiled with something like this:

ls *deb | awk -F_ '{print $1}' | xargs
# copy the list of names and run the following on the target machine:

dpkg -l <packages from the output above> 2>/dev/null | grep ^ii | awk '{print $2}'

I just ran this on one of the test slaves and got this:

libgl1-mesa-dri
libgl1-mesa-glx
libglapi-mesa
libglu1-mesa
libxatracker1

So,
package {
  ["libgl1-mesa-dri", "libgl1-mesa-glx", "libglapi-mesa", "libglu1-mesa", "libxatracker1"]:
    ensure => ....

should work.

Otherwise looks good.
Attachment #8386327 - Flags: feedback?(rail) → feedback+
dustin, rail: I'm having a problem finding the .orig.tar.gz in the right place and I believe I fix it by running a wget step. I don't see any of this mentioned in the documentation; is this a valid approach?

root@ubuntu64packager1:/data/repos/apt# reprepro -V --basedir . include precise /tmp/mesa-precise-amd64/*.changes
Unable to find pool/main/m/mesa/mesa_8.0.2.orig.tar.gz needed by mesa_8.0.2-0ubuntu3mozilla1.dsc!
Perhaps you forgot to give dpkg-buildpackage the -sa option,
 or you could try --ignore=missingfile to guess possible files to use.
39 files were added but not used.
The next deleteunreferenced call will delete them.
There have been errors!

To fix it:
root@ubuntu64packager1:/data/repos/apt# wget -Oreleng/pool/main/m/mesa/mesa_8.0.2.orig.tar.gz https://launchpad.net/ubuntu/+archive/primary/+files/mesa_8.0.2.orig.tar.gz
root@ubuntu64packager1:/data/repos/apt# reprepro -V --basedir . include precise /tmp/mesa-precise-amd64/*.changes
Exporting indices...
 looking for changes in 'precise|main|amd64'...
  replacing './releng/dists/precise/main/binary-amd64/Packages' (uncompressed,bzip2ed)
 looking for changes in 'precise|main|i386'...
 looking for changes in 'precise|main|source'...
  replacing './releng/dists/precise/main/source/Sources' (gzipped)

##########################################

Documentation up to now:

0) Clobber
rm -rf /tmp/mesa*

1) Create the debian directory and patch it
dget http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/mesa_8.0.2-0ubuntu3.dsc
dpkg-source -x mesa_8.0.2-0ubuntu3.dsc
cd mesa-8.0.2/
wget -O debian/patches/moz-fix-llvmpipe https://bug975034.bugzilla.mozilla.org/attachment.cgi?id=8379163
echo moz-fix-llvmpipe >> debian/patches/series
DEBFULLNAME="Rail Aliiev" DEBEMAIL="rail@mozilla.com" dch -l mozilla --distribution precise "Fix llvm pipe"

NOTE: Running pbuilder actually failed. Using puppetagain-build-deb works. I didn't look more into it.

2) Create the puppet change
cp -r mesa-8.0.2/debian ~/puppet/modules/packages/manifests/mesa-debian
cd ~/puppet
hg addremove
hg diff > ~/mesa.diff

3) Create the deb files [1]
cd modules/packages/manifests
wget https://launchpad.net/ubuntu/+archive/primary/+files/mesa_8.0.2.orig.tar.gz
puppetagain-build-deb precise amd64 mesa-debian
puppetagain-build-deb precise i386 mesa-debian

4) Prepare custom repository [2]
sudo su
rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/releng/ /data/repos/apt/releng/
rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/conf/ /data/repos/apt/conf/
rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/db/ /data/repos/apt/db/
cd /data/repos/apt
reprepro -V --basedir . include precise /tmp/mesa-precise-amd64/*.changes

[1] https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_DEBs#Building_from_orig.tar.gz.2C_debian.tar.gz_and_dsc
[2] https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#Ubuntu:_Landing_Custom_Repository_Changes
(In reply to Dustin J. Mitchell [:dustin] (I ignore NEEDINFO) from comment #25)
> So I'm confused - what *is* different?  Different versions of mesa?

Different versions of mesa:
9.0.0-0ubuntu1~precise1.
VS
8.0.2-0ubuntu3
I'm proceeding with the assumption that this step was a kosher one:
wget -Oreleng/pool/main/m/mesa/mesa_8.0.2.orig.tar.gz https://launchpad.net/ubuntu/+archive/primary/+files/mesa_8.0.2.orig.tar.gz

root@ubuntu64packager1:/data/repos/apt# reprepro -V --basedir . include precise /tmp/mesa-precise-amd64/*.changes
Exporting indices...
 looking for changes in 'precise|main|amd64'...
  replacing './releng/dists/precise/main/binary-amd64/Packages' (uncompressed,bzip2ed)
 looking for changes in 'precise|main|i386'...
 looking for changes in 'precise|main|source'...
  replacing './releng/dists/precise/main/source/Sources' (gzipped)
root@ubuntu64packager1:/data/repos/apt# reprepro -V --basedir . includedeb precise /tmp/mesa-precise-i386/*.deb
/tmp/mesa-precise-i386/libegl1-mesa_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libegl1-mesa-dbg_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libegl1-mesa-dev_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libegl1-mesa-drivers_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libegl1-mesa-drivers-dbg_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgbm1_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgbm1-dbg_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgbm-dev_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgl1-mesa-dev_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgl1-mesa-dri_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgl1-mesa-dri-dbg_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgl1-mesa-dri-experimental_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgl1-mesa-dri-experimental-dbg_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgl1-mesa-glx_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgl1-mesa-glx-dbg_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgl1-mesa-swx11_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgl1-mesa-swx11-dbg_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgl1-mesa-swx11-dev_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libglapi-mesa_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libglapi-mesa-dbg_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgles1-mesa_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgles1-mesa-dbg_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgles1-mesa-dev_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgles2-mesa_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgles2-mesa-dbg_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libgles2-mesa-dev_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libglu1-mesa_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libglu1-mesa-dev_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libopenvg1-mesa_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libopenvg1-mesa-dbg_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libopenvg1-mesa-dev_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libosmesa6_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libosmesa6-dev_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libxatracker1_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libxatracker1-dbg_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/libxatracker-dev_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
/tmp/mesa-precise-i386/mesa-common-dev_8.0.2-0ubuntu3mozilla1_i386.deb: component guessed as 'main'
Exporting indices...
 looking for changes in 'precise|main|amd64'...
 looking for changes in 'precise|main|i386'...
  replacing './releng/dists/precise/main/binary-i386/Packages' (uncompressed,bzip2ed)
 looking for changes in 'precise|main|source'...
root@ubuntu64packager1:/data/repos/apt#  rsync -av /data/repos/apt/releng/ releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/releng/
root@releng-puppet2.srv.releng.scl3.mozilla.com's password: 
sending incremental file list
dists/precise/
dists/precise/Release
dists/precise/main/binary-amd64/
dists/precise/main/binary-amd64/Packages
dists/precise/main/binary-amd64/Packages.bz2
dists/precise/main/binary-i386/
dists/precise/main/binary-i386/Packages
dists/precise/main/binary-i386/Packages.bz2
dists/precise/main/source/
dists/precise/main/source/Sources.gz
pool/main/m/mesa/
pool/main/m/mesa/libegl1-mesa-dbg_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libegl1-mesa-dbg_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libegl1-mesa-dev_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libegl1-mesa-dev_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libegl1-mesa-drivers-dbg_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libegl1-mesa-drivers-dbg_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libegl1-mesa-drivers_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libegl1-mesa-drivers_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libegl1-mesa_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libegl1-mesa_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgbm-dev_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgbm-dev_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgbm1-dbg_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgbm1-dbg_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgbm1_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgbm1_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgl1-mesa-dev_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgl1-mesa-dev_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgl1-mesa-dri-dbg_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgl1-mesa-dri-dbg_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgl1-mesa-dri-experimental-dbg_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgl1-mesa-dri-experimental-dbg_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgl1-mesa-dri-experimental_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgl1-mesa-dri-experimental_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgl1-mesa-dri_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgl1-mesa-dri_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgl1-mesa-glx-dbg_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgl1-mesa-glx-dbg_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgl1-mesa-glx_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgl1-mesa-glx_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgl1-mesa-swx11-dbg_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgl1-mesa-swx11-dbg_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgl1-mesa-swx11-dev_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgl1-mesa-swx11-dev_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgl1-mesa-swx11_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgl1-mesa-swx11_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libglapi-mesa-dbg_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libglapi-mesa-dbg_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libglapi-mesa_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libglapi-mesa_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgles1-mesa-dbg_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgles1-mesa-dbg_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgles1-mesa-dev_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgles1-mesa-dev_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgles1-mesa_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgles1-mesa_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgles2-mesa-dbg_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgles2-mesa-dbg_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgles2-mesa-dev_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgles2-mesa-dev_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libgles2-mesa_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libgles2-mesa_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libglu1-mesa-dev_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libglu1-mesa-dev_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libglu1-mesa_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libglu1-mesa_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libopenvg1-mesa-dbg_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libopenvg1-mesa-dbg_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libopenvg1-mesa-dev_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libopenvg1-mesa-dev_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libopenvg1-mesa_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libopenvg1-mesa_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libosmesa6-dev_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libosmesa6-dev_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libosmesa6_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libosmesa6_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libxatracker-dev_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libxatracker-dev_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libxatracker1-dbg_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libxatracker1-dbg_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/libxatracker1_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/libxatracker1_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/mesa-common-dev_8.0.2-0ubuntu3mozilla1_amd64.deb
pool/main/m/mesa/mesa-common-dev_8.0.2-0ubuntu3mozilla1_i386.deb
pool/main/m/mesa/mesa_8.0.2-0ubuntu3mozilla1.diff.gz
pool/main/m/mesa/mesa_8.0.2-0ubuntu3mozilla1.dsc
pool/main/m/mesa/mesa_8.0.2.orig.tar.gz

sent 82605617 bytes  received 2453 bytes  1776517.63 bytes/sec
total size is 1335036054  speedup is 16.16

root@ubuntu64packager1:/data/repos/apt#  rsync -av /data/repos/apt/db/ releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/db/
root@releng-puppet2.srv.releng.scl3.mozilla.com's password: 
sending incremental file list
./
checksums.db
packages.db
references.db
release.caches.db
version

sent 263639 bytes  received 2906 bytes  25385.24 bytes/sec
total size is 659488  speedup is 2.47
It seems that linux64 machines run a higher version of the mesa libraries. The b2g reftests run on the 32bit machines; should I limit the change to the 32bit machines? They already have different versions so I would not be worried of doing something wrong.

8.0.4-0ubuntu0.2 - linux64 machines
8.0.2-0ubuntu3   - linux32 machines

The current versions are probably coming from here:
http://puppetagain.pub.build.mozilla.org/data/repos/apt/ubuntu/pool/main/m/mesa/
The new ones that I want to deploy are in here:
http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng/pool/main/m/mesa/

Following on comment 26 (I just wanted to reproduce the steps):
root@ubuntu64packager1:~# cd /data/repos/apt/releng/pool/main/m/mesa/
root@ubuntu64packager1:/data/repos/apt/releng/pool/main/m/mesa# ls *deb | awk -F_ '{print $1}' | xargs
libegl1-mesa libegl1-mesa libegl1-mesa-dbg libegl1-mesa-dbg libegl1-mesa-dev libegl1-mesa-dev libegl1-mesa-drivers libegl1-mesa-drivers libegl1-mesa-drivers-dbg libegl1-mesa-drivers-dbg libgbm1 libgbm1 libgbm1-dbg libgbm1-dbg libgbm-dev libgbm-dev libgl1-mesa-dev libgl1-mesa-dev libgl1-mesa-dri libgl1-mesa-dri libgl1-mesa-dri-dbg libgl1-mesa-dri-dbg libgl1-mesa-dri-experimental libgl1-mesa-dri-experimental libgl1-mesa-dri-experimental-dbg libgl1-mesa-dri-experimental-dbg libgl1-mesa-glx libgl1-mesa-glx libgl1-mesa-glx-dbg libgl1-mesa-glx-dbg libgl1-mesa-swx11 libgl1-mesa-swx11 libgl1-mesa-swx11-dbg libgl1-mesa-swx11-dbg libgl1-mesa-swx11-dev libgl1-mesa-swx11-dev libglapi-mesa libglapi-mesa libglapi-mesa-dbg libglapi-mesa-dbg libgles1-mesa libgles1-mesa libgles1-mesa-dbg libgles1-mesa-dbg libgles1-mesa-dev libgles1-mesa-dev libgles2-mesa libgles2-mesa libgles2-mesa-dbg libgles2-mesa-dbg libgles2-mesa-dev libgles2-mesa-dev libglu1-mesa libglu1-mesa libglu1-mesa-dev libglu1-mesa-dev libopenvg1-mesa libopenvg1-mesa libopenvg1-mesa-dbg libopenvg1-mesa-dbg libopenvg1-mesa-dev libopenvg1-mesa-dev libosmesa6 libosmesa6 libosmesa6-dev libosmesa6-dev libxatracker1 libxatracker1 libxatracker1-dbg libxatracker1-dbg libxatracker-dev libxatracker-dev mesa-common-dev mesa-common-dev


[root@tst-linux64-spot-460.test.releng.usw2.mozilla.com ~]# dpkg -l libegl1-mesa libegl1-mesa libegl1-mesa-dbg libegl1-mesa-dbg libegl1-mesa-dev libegl1-mesa-dev libegl1-mesa-drivers libegl1-mesa-drivers libegl1-mesa-drivers-dbg libegl1-mesa-drivers-dbg libgbm1 libgbm1 libgbm1-dbg libgbm1-dbg libgbm-dev libgbm-dev libgl1-mesa-dev libgl1-mesa-dev libgl1-mesa-dri libgl1-mesa-dri libgl1-mesa-dri-dbg libgl1-mesa-dri-dbg libgl1-mesa-dri-experimental libgl1-mesa-dri-experimental libgl1-mesa-dri-experimental-dbg libgl1-mesa-dri-experimental-dbg libgl1-mesa-glx libgl1-mesa-glx libgl1-mesa-glx-dbg libgl1-mesa-glx-dbg libgl1-mesa-swx11 libgl1-mesa-swx11 libgl1-mesa-swx11-dbg libgl1-mesa-swx11-dbg libgl1-mesa-swx11-dev libgl1-mesa-swx11-dev libglapi-mesa libglapi-mesa libglapi-mesa-dbg libglapi-mesa-dbg libgles1-mesa libgles1-mesa libgles1-mesa-dbg libgles1-mesa-dbg libgles1-mesa-dev libgles1-mesa-dev libgles2-mesa libgles2-mesa libgles2-mesa-dbg libgles2-mesa-dbg libgles2-mesa-dev libgles2-mesa-dev libglu1-mesa libglu1-mesa libglu1-mesa-dev libglu1-mesa-dev libopenvg1-mesa libopenvg1-mesa libopenvg1-mesa-dbg libopenvg1-mesa-dbg libopenvg1-mesa-dev libopenvg1-mesa-dev libosmesa6 libosmesa6 libosmesa6-dev libosmesa6-dev libxatracker1 libxatracker1 libxatracker1-dbg libxatracker1-dbg libxatracker-dev libxatracker-dev mesa-common-dev mesa-common-dev 2> /dev/null | grep ^ii | awk '{print $2}'
libgl1-mesa-dri
libgl1-mesa-glx
libglapi-mesa
libglu1-mesa
libxatracker1
[root@tst-linux64-spot-460.test.releng.usw2.mozilla.com ~]# dpkg -l | grep "libxatracker"
ii  libxatracker1                          8.0.2-0ubuntu3                          X acceleration library -- runtime
[root@tst-linux64-spot-460.test.releng.usw2.mozilla.com ~]# dpkg -l | grep "mesa"
ii  libgl1-mesa-dri                        8.0.4-0ubuntu0.2                        free implementation of the OpenGL API -- DRI modules
ii  libgl1-mesa-dri:i386                   8.0.4-0ubuntu0.2                        free implementation of the OpenGL API -- DRI modules
ii  libgl1-mesa-glx                        8.0.4-0ubuntu0.2                        free implementation of the OpenGL API -- GLX runtime
ii  libgl1-mesa-glx:i386                   8.0.4-0ubuntu0.2                        free implementation of the OpenGL API -- GLX runtime
ii  libglapi-mesa                          8.0.4-0ubuntu0.2                        free implementation of the GL API -- shared library
ii  libglapi-mesa:i386                     8.0.4-0ubuntu0.2                        free implementation of the GL API -- shared library
ii  libglu1-mesa                           8.0.4-0ubuntu0.2                        Mesa OpenGL utility library (GLU)
ii  libglu1-mesa:i386                      8.0.4-0ubuntu0.2                        Mesa OpenGL utility library (GLU)

[root@tst-linux32-spot-400.test.releng.usw2.mozilla.com ~]# dpkg -l libegl1-mesa libegl1-mesa libegl1-mesa-dbg libegl1-mesa-dbg libegl1-mesa-dev libegl1-mesa-dev libegl1-mesa-drivers libegl1-mesa-drivers libegl1-mesa-drivers-dbg libegl1-mesa-drivers-dbg libgbm1 libgbm1 libgbm1-dbg libgbm1-dbg libgbm-dev libgbm-dev libgl1-mesa-dev libgl1-mesa-dev libgl1-mesa-dri libgl1-mesa-dri libgl1-mesa-dri-dbg libgl1-mesa-dri-dbg libgl1-mesa-dri-experimental libgl1-mesa-dri-experimental libgl1-mesa-dri-experimental-dbg libgl1-mesa-dri-experimental-dbg libgl1-mesa-glx libgl1-mesa-glx libgl1-mesa-glx-dbg libgl1-mesa-glx-dbg libgl1-mesa-swx11 libgl1-mesa-swx11 libgl1-mesa-swx11-dbg libgl1-mesa-swx11-dbg libgl1-mesa-swx11-dev libgl1-mesa-swx11-dev libglapi-mesa libglapi-mesa libglapi-mesa-dbg libglapi-mesa-dbg libgles1-mesa libgles1-mesa libgles1-mesa-dbg libgles1-mesa-dbg libgles1-mesa-dev libgles1-mesa-dev libgles2-mesa libgles2-mesa libgles2-mesa-dbg libgles2-mesa-dbg libgles2-mesa-dev libgles2-mesa-dev libglu1-mesa libglu1-mesa libglu1-mesa-dev libglu1-mesa-dev libopenvg1-mesa libopenvg1-mesa libopenvg1-mesa-dbg libopenvg1-mesa-dbg libopenvg1-mesa-dev libopenvg1-mesa-dev libosmesa6 libosmesa6 libosmesa6-dev libosmesa6-dev libxatracker1 libxatracker1 libxatracker1-dbg libxatracker1-dbg libxatracker-dev libxatracker-dev mesa-common-dev mesa-common-dev 2> /dev/null | grep ^ii | awk '{print $2}'
libgl1-mesa-dri
libgl1-mesa-glx
libglapi-mesa
libglu1-mesa
libxatracker1
[root@tst-linux32-spot-400.test.releng.usw2.mozilla.com ~]# dpkg -l | grep "mesa"
ii  libgl1-mesa-dri                        8.0.2-0ubuntu3                          free implementation of the OpenGL API -- DRI modules
ii  libgl1-mesa-glx                        8.0.2-0ubuntu3                          free implementation of the OpenGL API -- GLX runtime
ii  libglapi-mesa                          8.0.2-0ubuntu3                          free implementation of the GL API -- shared library
ii  libglu1-mesa                           8.0.2-0ubuntu3                          Mesa OpenGL utility library (GLU)
[root@tst-linux32-spot-400.test.releng.usw2.mozilla.com ~]# dpkg -l | grep "libxatracker"
ii  libxatracker1                          8.0.2-0ubuntu3                          X acceleration library -- runtime
rail, do you see anything wrong with this patch?
It is not actually doing anything.
Attachment #8386327 - Attachment is obsolete: true
Attachment #8386968 - Flags: feedback?(rail)
The difference with the previous patch: s/i386/i686/ [1]

It seems that the packages update [2] but I see more changes happening.
a) some packages are installed that were not specified
b) libglu1-mesa was set to be 8.0.2-0ubuntu3mozilla1, however, it ended up with 8.0.4-0ubuntu0.2

                   Before          After 
                   #######         ############
libgl1-mesa-dri    8.0.2-0ubuntu3  8.0.2-0ubuntu3mozilla1
libgl1-mesa-glx    8.0.2-0ubuntu3  8.0.2-0ubuntu3mozilla1
libgl1-mesa-swx11  n/a             8.0.4-0ubuntu0.2
libglapi-mesa      8.0.2-0ubuntu3  8.0.2-0ubuntu3mozilla1
libglu1-mesa       8.0.2-0ubuntu3  8.0.4-0ubuntu0.2
libosmesa6         n/a             8.0.4-0ubuntu0.2
libxatracker1      8.0.2-0ubuntu3  8.0.2-0ubuntu3mozilla1

New aim:
1) get a no-op puppet patch
2) don't deploy the change to the 32-bit machine
3) get the change deployed to the 64-bit machines

[1]
[root@tst-linux32-ec2-armenzg.test.releng.use1.mozilla.com ~]# facter | grep "hardwaremodel"
hardwaremodel => i686

[2]
Notice: /Stage[main]/Packages::Mesa/Package[libxatracker1]/ensure: ensure changed '8.0.2-0ubuntu3' to '8.0.2-0ubuntu3mozilla1'
Notice: /Stage[main]/Packages::Mesa/Package[libglu1-mesa]/ensure: ensure changed '8.0.2-0ubuntu3' to '8.0.2-0ubuntu3mozilla1'
Notice: /Stage[main]/Packages::Mesa/Package[libglapi-mesa]/ensure: ensure changed '8.0.2-0ubuntu3' to '8.0.2-0ubuntu3mozilla1'
Notice: /Stage[main]/Packages::Mesa/Package[libgl1-mesa-dri]/ensure: ensure changed '8.0.2-0ubuntu3' to '8.0.2-0ubuntu3mozilla1'
Notice: /Stage[main]/Packages::Mesa/Package[libgl1-mesa-glx]/ensure: ensure changed '8.0.2-0ubuntu3' to '8.0.2-0ubuntu3mozilla1'

[3]
Attachment #8386968 - Attachment is obsolete: true
Attachment #8386968 - Flags: feedback?(rail)
Attachment #8386992 - Attachment is obsolete: true
Comment on attachment 8387037 [details] [diff] [review]
pin mesa versions for test EC2 32-bit and 64-bit instances (no-op)

I don't think you need to bump $repoflag for a noop change. Otherwise looks good.
Attachment #8387037 - Flags: review?(rail) → review+
dustin: how can I proceed? puppetagain-build-deb as-is is not being able to regenerate the 0.8.4 version of mesa.

(In reply to Dustin J. Mitchell [:dustin] (I ignore NEEDINFO) from comment #10)
> puppetagain-build-deb is already installed on that host, and it *calls*
> cowbuilder.  So you shouldn't have to pay much attention to cowbuilder at
> all - just get the debian directory set up correctly.
> 
> That said, as far as dependencies go, it's a little scary that it needs a
> new version of debhelper.  If it's just mesa that needs to be upgraded, it's
> probably better to start with the debian package matching that in our
> puppetagain repo, and just bump the version number of mesa that it builds,
> rather than trying to use a new package (which may depend on lots of other
> new things).

I'm back to comment 9.
Aiming now 0.8.4 instead of 0.8.2 since the Linux64 machines are already on 0.8.4.

##########################

0) Clobber
rm -rf /tmp/mesa*

1) Create the debian directory and patch it
dget http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/mesa_8.0.4-0ubuntu0.6.dsc
dpkg-source -x mesa_8.0.4-0ubuntu0.6.dsc
cd mesa-8.0.4/
wget -O debian/patches/moz-fix-llvmpipe https://bug975034.bugzilla.mozilla.org/attachment.cgi?id=8379163
echo moz-fix-llvmpipe >> debian/patches/series
DEBFULLNAME="Rail Aliiev" DEBEMAIL="rail@mozilla.com" dch -l mozilla --distribution precise "Fix llvm pipe"

2) Create the puppet change
cp -r debian/* ~/puppet/modules/packages/manifests/mesa-debian
cd ~/puppet
hg addremove
hg diff > ~/mesa.diff

3) Create the deb files [1]
cd modules/packages/manifests
wget https://launchpad.net/ubuntu/+archive/primary/+files/mesa_8.0.4.orig.tar.gz
puppetagain-build-deb precise amd64 mesa-debian

Reading package lists...
Building dependency tree...
Reading state information...
passwd is already the newest version.
The following extra packages will be installed:
  debootstrap libidn11 wget
Suggested packages:
  pbuilder-uml gdebi-core
Recommended packages:
  sudo devscripts
The following NEW packages will be installed:
  debootstrap libidn11 pbuilder wget
debconf: delaying package configuration, since apt-utils is not installed
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/780 kB of archives.
After this operation, 2281 kB of additional disk space will be used.
Selecting previously unselected package libidn11.
(Reading database ... 13955 files and directories currently installed.)
Unpacking libidn11 (from .../libidn11_1.23-2_amd64.deb) ...
Selecting previously unselected package wget.
Unpacking wget (from .../wget_1.13.4-2ubuntu1_amd64.deb) ...
Selecting previously unselected package debootstrap.
Unpacking debootstrap (from .../debootstrap_1.0.39_all.deb) ...
Selecting previously unselected package pbuilder.
Unpacking pbuilder (from .../pbuilder_0.208ubuntu1_all.deb) ...
Setting up libidn11 (1.23-2) ...
Setting up wget (1.13.4-2ubuntu1) ...
Setting up debootstrap (1.0.39) ...
Setting up pbuilder (0.208ubuntu1) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
I: Setting DEBBUILDOPTS=
I: Setting DEBBUILDOPTS=
W: no hooks of type D found -- ignoring
ccache: failed to create /home/armenzg/.ccache (No such file or directory)
dpkg-architecture: warning: Couldn't determine gcc system type, falling back to default (native compilation)
 -> Attempting to satisfy build-dependencies
ccache: failed to create /home/armenzg/.ccache (No such file or directory)
ccache: failed to create /home/armenzg/.ccache (No such file or directory)
ccache: failed to create /home/armenzg/.ccache (No such file or directory)
ccache: failed to create /home/armenzg/.ccache (No such file or directory)
ccache: failed to create /home/armenzg/.ccache (No such file or directory)
ccache: failed to create /home/armenzg/.ccache (No such file or directory)
ccache: failed to create /home/armenzg/.ccache (No such file or directory)
ccache: failed to create /home/armenzg/.ccache (No such file or directory)
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: amd64
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper (>= 8.1.3), quilt (>= 0.40), pkg-config, libdrm-dev (>= 2.4.34), libx11-dev, xutils-dev, x11proto-gl-dev (>= 1.4.14), libxxf86vm-dev, libexpat1-dev, dpkg-dev (>= 1.15.6), libxfixes-dev, libxdamage-dev, libxext-dev, autoconf, automake, x11proto-dri2-dev (>= 2.6), linux-libc-dev (>= 2.6.31), libx11-xcb-dev, libxcb-dri2-0-dev, libxcb-glx0-dev, libxcb-xfixes0-dev, python-libxml2, libudev-dev, flex, bison, llvm-3.0-dev, libwayland-dev (>= 0.85.0)
dpkg-deb: building package `pbuilder-satisfydepends-dummy' in `/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 14126 files and directories currently installed.)
Unpacking pbuilder-satisfydepends-dummy (from .../pbuilder-satisfydepends-dummy.deb) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper (>= 8.1.3); however:
  Package debhelper is not installed.
 pbuilder-satisfydepends-dummy depends on quilt (>= 0.40); however:
  Package quilt is not installed.
 pbuilder-satisfydepends-dummy depends on pkg-config; however:
  Package pkg-config is not installed.
 pbuilder-satisfydepends-dummy depends on libdrm-dev (>= 2.4.34); however:
  Package libdrm-dev is not installed.
 pbuilder-satisfydepends-dummy depends on libx11-dev; however:
  Package libx11-dev is not installed.
 pbuilder-satisfydepends-dummy depends on xutils-dev; however:
  Package xutils-dev is not installed.
 pbuilder-satisfydepends-dummy depends on x11proto-gl-dev (>= 1.4.14); however:
  Package x11proto-gl-dev is not installed.
 pbuilder-satisfydepends-dummy depends on libxxf86vm-dev; however:
  Package libxxf86vm-dev is not installed.
 pbuilder-satisfydepends-dummy depends on libexpat1-dev; however:
  Package libexpat1-dev is not installed.
 pbuilder-satisfydepends-dummy depends on libxfixes-dev; however:
  Package libxfixes-dev is not installed.
 pbuilder-satisfydepends-dummy depends on libxdamage-dev; however:
  Package libxdamage-dev is not installed.
 pbuilder-satisfydepends-dummy depends on libxext-dev; however:
  Package libxext-dev is not installed.
 pbuilder-satisfydepends-dummy depends on autoconf; however:
  Package autoconf is not installed.
 pbuilder-satisfydepends-dummy depends on automake; however:
  Package automake is not installed.
 pbuilder-satisfydepends-dummy depends on x11proto-dri2-dev (>= 2.6); however:
  Package x11proto-dri2-dev is not installed.
 pbuilder-satisfydepends-dummy depends on libx11-xcb-dev; however:
  Package libx11-xcb-dev is not installed.
 pbuilder-satisfydepends-dummy depends on libxcb-dri2-0-dev; however:
  Package libxcb-dri2-0-dev is not installed.
 pbuilder-satisfydepends-dummy depends on libxcb-glx0-dev; however:
  Package libxcb-glx0-dev is not installed.
 pbuilder-satisfydepends-dummy depends on libxcb-xfixes0-dev; however:
  Package libxcb-xfixes0-dev is not installed.
 pbuilder-satisfydepends-dummy depends on python-libxml2; however:
  Package python-libxml2 is not installed.
 pbuilder-satisfydepends-dummy depends on libudev-dev; however:
  Package libudev-dev is not installed.
 pbuilder-satisfydepends-dummy depends on flex; however:
  Package flex is not installed.
 pbuilder-satisfydepends-dummy depends on bison; however:
  Package bison is not installed.
 pbuilder-satisfydepends-dummy depends on llvm-3.0-dev; however:
  Package llvm-3.0-dev is not installed.
 pbuilder-satisfydepends-dummy depends on libwayland-dev (>= 0.85.0); however:
  Package libwayland-dev is not installed.
Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
The following NEW packages will be installed:
  bsdmainutils{a} debhelper{a} dh-apparmor{a} diffstat{a} file{a} 
  gettext{a} gettext-base{a} groff-base{a} html2text{a} intltool-debian{a} 
  libcroco3{a} libgettextpo0{a} libmagic1{a} libpipeline1{a} libpopt0{a} 
  libunistring0{a} libxml2{a} man-db{a} pkg-config{a} po-debconf{a} 
  quilt{a} 
0 packages upgraded, 21 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/6202 kB of archives. After unpacking 19.3 MB will be used.
The following packages have unmet dependencies:
 pbuilder-satisfydepends-dummy : Depends: libdrm-dev (>= 2.4.34) but it is not going to be installed.
                                 Depends: libx11-dev but it is not going to be installed.
                                 Depends: xutils-dev but it is not going to be installed.
                                 Depends: x11proto-gl-dev (>= 1.4.14) but it is not going to be installed.
                                 Depends: libxxf86vm-dev but it is not going to be installed.
                                 Depends: libexpat1-dev but it is not going to be installed.
                                 Depends: libxfixes-dev but it is not going to be installed.
                                 Depends: libxdamage-dev but it is not going to be installed.
                                 Depends: libxext-dev but it is not going to be installed.
                                 Depends: autoconf but it is not going to be installed.
                                 Depends: automake but it is not going to be installed.
                                 Depends: x11proto-dri2-dev (>= 2.6) but it is not going to be installed.
                                 Depends: libx11-xcb-dev but it is not going to be installed.
                                 Depends: libxcb-dri2-0-dev but it is not going to be installed.
                                 Depends: libxcb-glx0-dev but it is not going to be installed.
                                 Depends: libxcb-xfixes0-dev but it is not going to be installed.
                                 Depends: python-libxml2 but it is not going to be installed.
                                 Depends: libudev-dev but it is not going to be installed.
                                 Depends: flex but it is not going to be installed.
                                 Depends: bison but it is not going to be installed.
                                 Depends: llvm-3.0-dev but it is not going to be installed.
                                 Depends: libwayland-dev (>= 0.85.0) but it is not going to be installed.
Unable to resolve dependencies!  Giving up...
The following NEW packages will be installed:
  bsdmainutils{a} debhelper{a} dh-apparmor{a} diffstat{a} file{a} 
  gettext{a} gettext-base{a} groff-base{a} html2text{a} intltool-debian{a} 
  libcroco3{a} libgettextpo0{a} libmagic1{a} libpipeline1{a} libpopt0{a} 
  libunistring0{a} libxml2{a} man-db{a} pkg-config{a} po-debconf{a} 
  quilt{a} 
0 packages upgraded, 21 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/6202 kB of archives. After unpacking 19.3 MB will be used.
Abort.
 -> Finished parsing the build-deps
Reading package lists...
Building dependency tree...
Reading state information...
fakeroot is already the newest version.
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
 pbuilder-satisfydepends-dummy : Depends: debhelper (>= 8.1.3) but it is not going to be installed
                                 Depends: quilt (>= 0.40) but it is not going to be installed
                                 Depends: pkg-config
                                 Depends: libdrm-dev (>= 2.4.34) but it is not going to be installed
                                 Depends: libx11-dev but it is not going to be installed
                                 Depends: xutils-dev
                                 Depends: x11proto-gl-dev (>= 1.4.14) but it is not going to be installed
                                 Depends: libxxf86vm-dev but it is not going to be installed
                                 Depends: libexpat1-dev but it is not going to be installed
                                 Depends: libxfixes-dev but it is not going to be installed
                                 Depends: libxdamage-dev but it is not going to be installed
                                 Depends: libxext-dev but it is not going to be installed
                                 Depends: autoconf but it is not going to be installed
                                 Depends: automake but it is not going to be installed
                                 Depends: x11proto-dri2-dev (>= 2.6) but it is not going to be installed
                                 Depends: libx11-xcb-dev but it is not going to be installed
                                 Depends: libxcb-dri2-0-dev but it is not going to be installed
                                 Depends: libxcb-glx0-dev but it is not going to be installed
                                 Depends: libxcb-xfixes0-dev but it is not going to be installed
                                 Depends: python-libxml2 but it is not going to be installed
                                 Depends: libudev-dev but it is not going to be installed
                                 Depends: flex
                                 Depends: bison
                                 Depends: llvm-3.0-dev but it is not going to be installed
                                 Depends: libwayland-dev (>= 0.85.0) but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
I: Copying back the cached apt archive contents
I: unmounting /var/cache/pbuilder/ccache filesystem
I: unmounting /tmp/tmp.R3Jyet6Zx9 filesystem
I: unmounting dev/pts filesystem
I: unmounting proc filesystem
 -> Cleaning COW directory
  forking: rm -rf /var/cache/pbuilder/build//cow.1970
I *think* I can skip using puppetagain-build-deb.
So far the deb files are being generated.

> cd ~/tmp/mesa-8.0.4
> dpkg-source -b .
> cd ..
> sudo pbuilder --build --distribution precise --architecture amd64 \
> --basetgz /var/cache/pbuilder/base-precise-amd64.tgz --buildresult out64 mesa_8.0.4-0ubuntu0.6mozilla1.dsc
> sudo pbuilder --build --distribution precise --architecture i386 \
> --basetgz /var/cache/pbuilder/base-precise-i386.tgz  --buildresult out32 mesa_8.0.4-0ubuntu0.6mozilla1.dsc
Completed the testing process:

1) Create the debian directory and patch it
mkdir ~/tmp && cd ~/tmp
dget http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/mesa_8.0.4-0ubuntu0.6.dsc
dpkg-source -x mesa_8.0.4-0ubuntu0.6.dsc
cd mesa-8.0.4/
wget -O debian/patches/moz-fix-llvmpipe https://bug975034.bugzilla.mozilla.org/attachment.cgi?id=8379163
echo moz-fix-llvmpipe >> debian/patches/series
DEBFULLNAME="Rail Aliiev" DEBEMAIL="rail@mozilla.com" dch -l mozilla --distribution precise "Fix llvm pipe"

2) Create the puppet change

NOTE: This is probably unncessary since puppetagain-build-dep does not work for it

cp -r debian/* ~/puppet/modules/packages/manifests/mesa-debian
cd ~/puppet
hg addremove
hg diff > ~/mesa.diff

3) Create the deb files [1] (Using pbuilder instead of puppetagain-build-deb)
cd ~/tmp/mesa-8.0.4
dpkg-source -b .
cd ..
sudo pbuilder --build --distribution precise --architecture amd64 \
 --basetgz /var/cache/pbuilder/base-precise-amd64.tgz \
 --buildresult out64 mesa_8.0.4-0ubuntu0.6mozilla1.dsc
sudo pbuilder --build --distribution precise --architecture i386 \
 --basetgz /var/cache/pbuilder/base-precise-i386.tgz \
 --buildresult out32 mesa_8.0.4-0ubuntu0.6mozilla1.dsc


4) Prepare custom repository [2]
sudo su
rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/releng/ /data/repos/apt/releng/
rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/conf/ /data/repos/apt/conf/
rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/db/ /data/repos/apt/db/
cd /data/repos/apt
cp ~armenzg/tmp/mesa_8.0.4.orig.tar.gz releng/pool/main/m/mesa
reprepro -V --basedir . include precise ~armenzg/tmp/out64/*.changes
reprepro -V --basedir . includedeb precise ~armenzg/tmp/out32/*.deb

5) Sync back
rsync -av /data/repos/apt/releng/ releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/releng/
rsync -av /data/repos/apt/db/ releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/db/

6) Fix the permissions
ssh root@releng-puppet2.srv.releng.scl3.mozilla.com
puppetmaster-fixperms

The new deb files show up in here:
http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng/pool/main/m/mesa/?C=M;O=D

6.b) wait for the other puppet masters to sync up
My puppet runs on step 7.6 were failing because the packages were missing at:
http://puppetagain-apt.pvt.build.mozilla.org/repos/apt/releng/pool/main/m/mesa/?C=M;O=D

7) Test & write the puppet changes
7.1) start/create two test instances & ssh as root:
tst-linux{32,64}-ec2-armenzg

7.2) prepare user environment
ssh armenzg@releng-puppet2.srv.releng.scl3.mozilla.com
cd /etc/puppet/environments/armenzg/env
hg pull -u && hg st

7.3) take note of the library versions on the test slaves

7.4) run a no-op test sync
puppet agent --test --environment=armenzg --server=releng-puppet2.srv.releng.scl3.mozilla.com &> 1

7.5) on the puppet master, bump the versions and the repoflag [3]

7.6) run puppet syncs again on the test instances & watch for the changes on the pupet output [4][5]
puppet agent --test --environment=armenzg --server=releng-puppet2.srv.releng.scl3.mozilla.com &> 2

7.7) review the package versions [6][7]

7.8) test the rollback scenario
* revert your changes on 7.5
* bump the repoflag
* run 7.6 and 7.7 again [8][9]

8) clean up ubuntu64packager1 and shut it off
* as root, rm ~armenzg/tmp

[1] https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Build_DEBs#Building_a_package
[2] https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#Ubuntu:_Landing_Custom_Repository_Changes
[3] diff -r 33ac05c490c0 modules/packages/manifests/mesa.pp
--- a/modules/packages/manifests/mesa.pp	Fri Mar 07 15:19:15 2014 +1300
+++ b/modules/packages/manifests/mesa.pp	Fri Mar 07 07:55:33 2014 -0800
@@ -14,1 +14,1 @@ class packages::mesa {
-                            ensure => '8.0.2-0ubuntu3';
+                            ensure => '8.0.4-0ubuntu0.6mozilla1';
@@ -23,1 +23,1 @@ class packages::mesa {
-                            ensure => '8.0.4-0ubuntu0.2';
+                            ensure => '8.0.4-0ubuntu0.6mozilla1';
diff -r 33ac05c490c0 modules/packages/manifests/setup.pp
--- a/modules/packages/manifests/setup.pp	Fri Mar 07 15:19:15 2014 +1300
+++ b/modules/packages/manifests/setup.pp	Fri Mar 07 07:55:33 2014 -0800
@@ -61,1 +61,1 @@ class packages::setup {
-            $repoflag = 13
+            $repoflag = 14

[4]
Notice: /Stage[main]/Packages::Mesa/Package[libxatracker1]/ensure: ensure changed '8.0.4-0ubuntu0.2' to '8.0.4-0ubuntu0.6mozilla1'
Notice: /Stage[main]/Packages::Mesa/Package[libglu1-mesa]/ensure: ensure changed '8.0.4-0ubuntu0.2' to '8.0.4-0ubuntu0.6mozilla1'
Notice: /Stage[main]/Packages::Mesa/Package[libglapi-mesa]/ensure: ensure changed '8.0.4-0ubuntu0.2' to '8.0.4-0ubuntu0.6mozilla1'
Notice: /Stage[main]/Packages::Mesa/Package[libgl1-mesa-glx]/ensure: ensure changed '8.0.4-0ubuntu0.2' to '8.0.4-0ubuntu0.6mozilla1'

[5]
Notice: /Stage[main]/Packages::Mesa/Package[libglu1-mesa]/ensure: ensure changed '8.0.2-0ubuntu3' to '8.0.4-0ubuntu0.6mozilla1'
Notice: /Stage[main]/Packages::Mesa/Package[libgl1-mesa-dri]/ensure: ensure changed '8.0.2-0ubuntu3' to '8.0.4-0ubuntu0.6mozilla1'

[6]
> 1,5c1,5
> < ii  libgl1-mesa-dri                        8.0.2-0ubuntu3                          free implementation of the OpenGL API -- DRI modules
> < ii  libgl1-mesa-glx                        8.0.2-0ubuntu3                          free implementation of the OpenGL API -- GLX runtime
> < ii  libglapi-mesa                          8.0.2-0ubuntu3                          free implementation of the GL API -- shared library
> < ii  libglu1-mesa                           8.0.2-0ubuntu3                          Mesa OpenGL utility library (GLU)
> < ii  libxatracker1                          8.0.2-0ubuntu3                          X acceleration library -- runtime
---
> > ii  libgl1-mesa-dri                        8.0.4-0ubuntu0.6mozilla1                free implementation of the OpenGL API -- DRI modules
> > ii  libgl1-mesa-glx                        8.0.4-0ubuntu0.6mozilla1                free implementation of the OpenGL API -- GLX runtime
> > ii  libglapi-mesa                          8.0.4-0ubuntu0.6mozilla1                free implementation of the GL API -- shared library
> > ii  libglu1-mesa                           8.0.4-0ubuntu0.6mozilla1                Mesa OpenGL utility library (GLU)
> > ii  libxatracker1                          8.0.4-0ubuntu0.6mozilla1                X acceleration library -- runtime

[7]
--- before	2014-03-07 07:51:33.642006000 -0800
+++ after	2014-03-07 08:13:21.982677002 -0800
@@ -1,9 +1,9 @@
-ii  libgl1-mesa-dri                        8.0.4-0ubuntu0.2                        free implementation of the OpenGL API -- DRI modules
-ii  libgl1-mesa-dri:i386                   8.0.4-0ubuntu0.2                        free implementation of the OpenGL API -- DRI modules
-ii  libgl1-mesa-glx                        8.0.4-0ubuntu0.2                        free implementation of the OpenGL API -- GLX runtime
-ii  libgl1-mesa-glx:i386                   8.0.4-0ubuntu0.2                        free implementation of the OpenGL API -- GLX runtime
-ii  libglapi-mesa                          8.0.4-0ubuntu0.2                        free implementation of the GL API -- shared library
-ii  libglapi-mesa:i386                     8.0.4-0ubuntu0.2                        free implementation of the GL API -- shared library
-ii  libglu1-mesa                           8.0.4-0ubuntu0.2                        Mesa OpenGL utility library (GLU)
-ii  libglu1-mesa:i386                      8.0.4-0ubuntu0.2                        Mesa OpenGL utility library (GLU)
-ii  libxatracker1                          8.0.4-0ubuntu0.2                        X acceleration library -- runtime
+ii  libgl1-mesa-dri                        8.0.4-0ubuntu0.6mozilla1                free implementation of the OpenGL API -- DRI modules
+ii  libgl1-mesa-dri:i386                   8.0.4-0ubuntu0.6mozilla1                free implementation of the OpenGL API -- DRI modules
+ii  libgl1-mesa-glx                        8.0.4-0ubuntu0.6mozilla1                free implementation of the OpenGL API -- GLX runtime
+ii  libgl1-mesa-glx:i386                   8.0.4-0ubuntu0.6mozilla1                free implementation of the OpenGL API -- GLX runtime
+ii  libglapi-mesa                          8.0.4-0ubuntu0.6mozilla1                free implementation of the GL API -- shared library
+ii  libglapi-mesa:i386                     8.0.4-0ubuntu0.6mozilla1                free implementation of the GL API -- shared library
+ii  libglu1-mesa                           8.0.4-0ubuntu0.6mozilla1                Mesa OpenGL utility library (GLU)
+ii  libglu1-mesa:i386                      8.0.4-0ubuntu0.6mozilla1                Mesa OpenGL utility library (GLU)
+ii  libxatracker1                          8.0.4-0ubuntu0.6mozilla1                X acceleration library -- runtime

[8]
Notice: /Stage[main]/Packages::Mesa/Package[libxatracker1]/ensure: ensure changed '8.0.4-0ubuntu0.6mozilla1' to '8.0.2-0ubuntu3'
Notice: /Stage[main]/Packages::Mesa/Package[libglu1-mesa]/ensure: ensure changed '8.0.4-0ubuntu0.6mozilla1' to '8.0.2-0ubuntu3'
Notice: /Stage[main]/Packages::Mesa/Package[libglapi-mesa]/ensure: ensure changed '8.0.4-0ubuntu0.6mozilla1' to '8.0.2-0ubuntu3'
Notice: /Stage[main]/Packages::Mesa/Package[libgl1-mesa-dri]/ensure: ensure changed '8.0.4-0ubuntu0.6mozilla1' to '8.0.2-0ubuntu3'
Notice: /Stage[main]/Packages::Mesa/Package[libgl1-mesa-glx]/ensure: ensure changed '8.0.4-0ubuntu0.6mozilla1' to '8.0.2-0ubuntu3'

[9]
Notice: /Stage[main]/Packages::Mesa/Package[libxatracker1]/ensure: ensure changed '8.0.4-0ubuntu0.6mozilla1' to '8.0.4-0ubuntu0.2'
Notice: /Stage[main]/Packages::Mesa/Package[libglu1-mesa]/ensure: ensure changed '8.0.4-0ubuntu0.6mozilla1' to '8.0.4-0ubuntu0.2'
Notice: /Stage[main]/Packages::Mesa/Package[libglapi-mesa]/ensure: ensure changed '8.0.4-0ubuntu0.6mozilla1' to '8.0.4-0ubuntu0.2'
Notice: /Stage[main]/Packages::Mesa/Package[libgl1-mesa-dri]/ensure: ensure changed '8.0.4-0ubuntu0.6mozilla1' to '8.0.4-0ubuntu0.2'
Notice: /Stage[main]/Packages::Mesa/Package[libgl1-mesa-glx]/ensure: ensure changed '8.0.4-0ubuntu0.6mozilla1' to '8.0.4-0ubuntu0.2'
Attachment #8387037 - Flags: checked-in+
The process of generating this patch is described on steps 1 & 2 from comment 37.
Attachment #8387663 - Flags: review?(rail)
Attachment #8387037 - Attachment description: mesa no-op → pin mesa versions for test EC2 32-bit and 64-bit instances (no-op)
The rollback scenario has been tested as mentioned on comment 37.

Do you think we're ready to roll this on Monday?

Or would you like me to pull a couple of production instances, update them and run them through staging? (using my dev instances would be a pain to setup)
Attachment #8387667 - Flags: review?(rail)
Summary: Patch our current version of Mesa (8.0.2) for our EC2 test machines → Upgrade ec2 test instances mesa versions from 8.0.2-0ubuntu3(32-bit hosts)/8.0.4-0ubuntu0.2(64-bit hosts) to 8.0.4-0ubuntu0.6mozilla1
Attachment #8387663 - Flags: review?(rail) → review+
Attachment #8387667 - Flags: review?(rail) → review+
Attachment #8387663 - Flags: checked-in+
Comment on attachment 8387667 [details] [diff] [review]
upgrade ec2 test instances mesa versions from 8.0.2-0ubuntu3(32-bit hosts)/8.0.4-0ubuntu0.2(64-bit hosts) to 8.0.4-0ubuntu0.6mozilla1

Live in production.

Let's see if we need to backout or not.
Attachment #8387667 - Flags: checked-in+
I'm worried about a single test failure on mozilla-inbound for b2g desktop on Linux64.
I don't see it on b2g-inbound but it could be a matter of timing.
Let's re-trigger to get more results.
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=b2g_ubuntu64.*gaia
https://tbpl.mozilla.org/?tree=B2g-Inbound&jobname=b2g_ubuntu64.*gaia

We should know in 30 minutes.
The test failures are not related to our deployment. It was a bad merge.
It looks good so far.
Let's hope that no one finds anything in the next day.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.