1038697 - Sync Ubuntu updates on a frequent basis to our local mirror

Status: RESOLVED WONTFIX

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Right now if a new Ubuntu version comes out, a full sync will happen. That means all packages are available from a local repository. The bad thing is that usually we never update this mirror for new security related updates. For the moment we simply pull in necessary updates into the releng-updates repo.

So what we could do (as discussed on IRC with dustin and jmaher) is to pull in updates of Ubuntu in a 6 weeks period. As best in the middle of our release cycle, so that none of the installed OS updates would affect performance tests.

Therefore we would have a structure similar to http://archive.ubuntu.com/ubuntu/dists/, which could look like:

ubuntu
  dists
    trusty (only dist right now)
    trusty-backports (needed?)
    trusty-proposed (needed?)
    trusty-security
    trusty-updates
  pool
    main
    multiverse
    universe

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This would be done manually, and just update trusty-security and trusty-updates at that time.  We'd need to re-generate AWS AMIs at that point, and also trigger apt-get to install the latest version of everything currently installed on existing machines.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

this wouldn't update our machines, just make updates available?

I would see us needing to run tests on all branches prior to updating machines.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

No it would update all machines at the same time.  So this would basically be a flag day at some particular position in the release cycle at which time system updates would ship, and we'd expect to clean up any test failures or performance changes that result.

We could do some amount of testing in relabs, but we only have a few hosts there, and they'd have to run on a staging master.  What kind of breakage do you expect?  How much work would it be to perform that testing given, say, three Ubuntu hosts?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

for unittests we could have some breakage- just changing a single library caused us 2 weeks of work while switching to ubuntu originally.  We were all green and the 1 library delayed us a lot.

For talos, just ensuring the tests runs is all we need, the numbers can be documented as shifted.

What makes this interesting is that we need to do this for all branches since each branch has different features and tests.

Will this be for hardware machines only, aws only, or both?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Both.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

ok, then we would need to do all our tests and view the results- maybe a custom try pool that we could do this on?  We just need a couple of days to get the test results and verify all is well- if it isn't well, then it could take a few days to a few weeks to get green again.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

OK, well, just to be clear on the status of this bug -- cool idea, but not something we're planning on doing while we're still running Precise (12.04), and not something anyone is committed to doing for Trusty (14.04), but definitely worth considering.

In the interim, QA is the only one using Trusty, so re-mirroring trusty-updates and trusty-security periodically is just fine.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Dustin J. Mitchell [:dustin] from comment #7)
> In the interim, QA is the only one using Trusty, so re-mirroring
> trusty-updates and trusty-security periodically is just fine.

Sounds good. Do we have scripts for that? Would this re-mirroring be triggered by you?

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

No, no scripts, just the stuff in https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages.  We haven't done these sync's regularly, so there's been no need to automated.

For the moment, since the authoritative set of repos is on the moco puppetmasters, that will have to be someone from releng/relops.  We may be able to have you update and then I sync from your master.  Alternately, maybe we should start thinking about using 'git annex' to track the repos, so that we could just inter-merge them the way we do the puppet repo.  So many ideas!

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So I'm a bit nervous to actually run such a command against /data/repos/apt/*. Can't we add script files for all those commands listed on the wiki and put them directly under /repos/apt? Those will be synced across all puppetmaster instances. So whoever will get such an update started, he/she only has to execute that file. No setup steps or whatever necessary.

git-annex sounds good and it would be indeed something which can help us. Maybe we get some of the scripts working first?

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

I'm not sure why running a shell script with those commands in it makes you less nervous, but you're welcome to cook something up.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I believe this is a wontfix for puppet. Not sure about taskcluster, but that is a different story.