1024938 - Puppet run fails in 'apt-get update' for specific distributions

Status: VERIFIED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Yesterday we brought some Ubuntu 14.04 nodes online by kickstarting them. More details see bug 999435. After the box went online I got the following email for each of them:

Puppet Report for mm-ub-1404-64-1.qa.scl3.mozilla.com

Fri Jun 13 07:50:29 -0700 2014 /Stage[packagesetup]/Packages::Setup/Exec[apt-get-update] (err): Failed to call refresh: /usr/bin/apt-get update returned 100 instead of one of [0]
Fri Jun 13 07:50:29 -0700 2014 /Stage[packagesetup]/Packages::Setup/Exec[apt-get-update] (err): /usr/bin/apt-get update returned 100 instead of one of [0]
Fri Jun 13 07:50:29 -0700 2014 Puppet (err): /usr/bin/apt-get update returned 100 instead of one of [0]
Fri Jun 13 07:50:29 -0700 2014 /Stage[packagesetup]/Packages::Setup/Exec[apt-get-update-daily]/returns (err): change from notrun to 0 failed: /usr/bin/apt-get update returned 100 instead of one of [0]

Not sure if this is critical or not, but most likely worth to have a look at to get it fixed. On IRC Rail mentioned it could have been that another apt-get task was running at this time. So maybe we should add a check for that, so that no failure email will be sent out.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Doing a manual puppet run on the hosts I see:

Notice: /Stage[packagesetup]/Packages::Setup/Exec[apt-get-update]/returns: W: GPG error: http://repos trusty Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 1054B7A24BD6EC30
Notice: /Stage[packagesetup]/Packages::Setup/Exec[apt-get-update]/returns: W: Failed to fetch http://repos/repos/apt/releng-updates/dists/trusty-updates/all/binary-i386/Packages  404  Not Found
Notice: /Stage[packagesetup]/Packages::Setup/Exec[apt-get-update]/returns: 
Notice: /Stage[packagesetup]/Packages::Setup/Exec[apt-get-update]/returns: W: Failed to fetch http://repos/repos/apt/xorg-edgers/dists/trusty/main/binary-i386/Packages  404  Not Found
Notice: /Stage[packagesetup]/Packages::Setup/Exec[apt-get-update]/returns: 
Notice: /Stage[packagesetup]/Packages::Setup/Exec[apt-get-update]/returns: E: Some index files failed to download. They have been ignored, or old ones used instead.
Error: /Stage[packagesetup]/Packages::Setup/Exec[apt-get-update]: Failed to call refresh: /usr/bin/apt-get update returned 100 instead of one of [0]
Error: /Stage[packagesetup]/Packages::Setup/Exec[apt-get-update]: /usr/bin/apt-get update returned 100 instead of one of [0]

Looks like the GPG key is wrong and needs an update.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I wonder if that breaks us from installing additional packages on Ubuntu. Dustin, what do you think?

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

The GPG warnings are harmless, actually -- apt-get update's output is very confusing.

It's the 404's that are problematic.  The specific problem is that they're looking for the `trusty-updates` dists directory in the releng-updates and xorg-edgers repos, and it's not there.

We'll need to put a trusty package into releng-updates, and we'll need to mirror trusty for xorg-edgers.

Jake, did you have this problem?

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

A latest manual run via the terminal shows me:

Err http://repos trusty-updates/all i386 Packages
  404  Not Found
Ign http://repos trusty-updates/all Translation-en_US
Ign http://repos trusty-updates/all Translation-en
Ign http://repos trusty/main Translation-en_US
Ign http://repos trusty/main Translation-en
Err http://repos trusty/main i386 Packages
  404  Not Found

So it's a bit different than the above one done via puppet.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Yeah, and that output omits the middle of the URLs, so of course you can't tell what it's doing.  It's the same errors, though.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

That's not really an accurate title.  It's failing in apt-get update due to missing distributions in certain repos.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Well, feel free to use a better name. :)

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Dustin, I have seen you did some work here. The number of failures went down, but I still get errors for:

Err http://repos trusty-updates/all i386 Packages
Err http://repos trusty/main i386 Packages

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

I didn't do anything yet!

I tried to mirror xorg-edgers, but I need to get the GPG signatures aligned.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

OK, I added a trusty repo to releng-updates with the for loop in

https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#Adding_a_Single_Package

and I sync'd xorg-edgers with --nocleanup, so it should have the existing packages plus the new trusty repo.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

So, once that syncs to your puppetmaster (which it should email you about), you should be good to go.  Re-open if not.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

apt-get update is working fine now, and doesn't complain about missing distributions anymore. Thanks Dustin.