963304 - Move ccache config out of in-tree mozconfigs

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Mike Hommey [:glandium]]

Current in-tree mozconfigs contain something like:
   ac_add_options --with-ccache=/usr/bin/ccache

The problem is that the use of ccache is a (more or less) global policy, and that doesn't work well with it being defined (and changeable) per push. While there *is* value is being able to tweak this per-push, especially on try, it doesn't allow for global changes such as the deployment of alternative cache solutions (see bug 940788 and my recent blog posts)

I think a way to deal with this is to append some mozconfig snippets defined by buildbot-side configuration at the end of .mozconfig after it's copied from the in-tree mozconfig during the build.

Comment 1

[[e-mail] (:bhearsum)]

mshal and I were just talking about this and realized that we're not fully sure what you're asking for. It's clear that you want ccache (or whatever we're using in place of it) defined in Buildbot. I think that makes sense.

It also sounds like you're suggesting that we remove the ability to override it in-tree. If that's the case, can you expand on why a bit more? I think we'd be in a worse place if we did that because it wouldn't be possible for developers to eg, test a new ccache replacement on try without RelEng.

If it's okay to keep it overridable in-tree but set by Buildbot I think the best way to deal with this is to set an environment variable in Buildbot, and have the mozconfig conditionally run that ac_add_options. Eg:
CCACHE_BINARY=/usr/bin/ccache (or /usr/bin/fancy_cacher)

And then:

if [ ! -z $CCACHE_BINARY ]; then
  ac_add_options --with-ccache=$CCACHE_BINARY
fi

We use a similar model to set the update channel: https://mxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/linux32/common-opt#3

Comment 2

[Mike Hommey [:glandium]]

I still want it to be overridable in-tree, but I don't want to require changes to the current mozconfigs for it to work. Sure we'll need to remove the current --with-ccache, but if we make it an inserted mozconfig snippet, we can also add --without-ccache as necessary for shared cache. The problem with relying on changes to in-tree mozconfigs is that it makes it unreliable whether it's used on e.g. try. Especially if we end up needing more changes afterwards.
My current thinking is that we could append a mozconfig snippet wrapped in a
  if test -z "$NO_BUILDBOT_MOZCONFIG"; then
  fi
or something like that, which would allow anything running before it to disable it without having to appear after it. Do I make sense?

Comment 3

[Michael Shal [:mshal]]

(In reply to Mike Hommey [:glandium] from comment #2)
> I still want it to be overridable in-tree, but I don't want to require
> changes to the current mozconfigs for it to work. Sure we'll need to remove
> the current --with-ccache, but if we make it an inserted mozconfig snippet,
> we can also add --without-ccache as necessary for shared cache. 

For the shared cache, you're planning to have --without-ccache and then add some other (separate) option to enable the shared cache? I was thinking you might do --with-ccache=/path/to/shared_cache to just give it a different binary for your caching implementation instead of ccache.

Comment 4

[Mike Hommey [:glandium]]

--with-ccache fails if you give it something like "python script.py". Which you can work around by using a hashbang in the script, except that will fail on windows (the medium term plan being to also use the cache on windows). Also, even if the hashbang worked on windows, our CC_WRAPPER stuff would make it not work. So we need different smarts for different platforms, and none of them need to rely on in-tree changes.

Comment 5

[Mike Hommey [:glandium]]

> and none of them need to rely on in-tree changes.

I mean all of them need to not rely on in-tree changes.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Mike Hommey [:glandium] from comment #4)
> --with-ccache fails if you give it something like "python script.py". Which
> you can work around by using a hashbang in the script, except that will fail
> on windows (the medium term plan being to also use the cache on windows).
> Also, even if the hashbang worked on windows, our CC_WRAPPER stuff would
> make it not work. So we need different smarts for different platforms, and
> none of them need to rely on in-tree changes.

Could you spell out the precise things we want to set for different platforms? It's not clear to me if this is --enable-cache=X (where X is the same or different per platform) or some other flag...or something else altogether.

(In reply to Mike Hommey [:glandium] from comment #2)
> The problem
> with relying on changes to in-tree mozconfigs is that it makes it unreliable
> whether it's used on e.g. try.
> Especially if we end up needing more changes
> afterwards.

You mean because of people pushing older things (like mozilla-aurora or beta) to try? Or something else? The former is easy to mitigate.

Comment 7

[Mike Hommey [:glandium]]

(In reply to Ben Hearsum [:bhearsum] from comment #6)
> (In reply to Mike Hommey [:glandium] from comment #4)
> > --with-ccache fails if you give it something like "python script.py". Which
> > you can work around by using a hashbang in the script, except that will fail
> > on windows (the medium term plan being to also use the cache on windows).
> > Also, even if the hashbang worked on windows, our CC_WRAPPER stuff would
> > make it not work. So we need different smarts for different platforms, and
> > none of them need to rely on in-tree changes.
> 
> Could you spell out the precise things we want to set for different
> platforms? It's not clear to me if this is --enable-cache=X (where X is the
> same or different per platform) or some other flag...or something else
> altogether.

I'm currently thinking augmenting CC on linux/mac and defining CC_WRAPPER on windows.
There would also need to be an environment variable set with a bucket name depending on the slave host name.

> (In reply to Mike Hommey [:glandium] from comment #2)
> > The problem
> > with relying on changes to in-tree mozconfigs is that it makes it unreliable
> > whether it's used on e.g. try.
> > Especially if we end up needing more changes
> > afterwards.
> 
> You mean because of people pushing older things (like mozilla-aurora or
> beta) to try? Or something else? The former is easy to mitigate.

That, and the fact that cross-branch changes would need landing on each branch. There is no reason ccache (or its replacement) setup needs landings on every single branch. And there is no reason not to have the same setup on all branches.

Comment 8

[[e-mail] (:bhearsum)]

(In reply to Mike Hommey [:glandium] from comment #7)
> (In reply to Ben Hearsum [:bhearsum] from comment #6)
> > (In reply to Mike Hommey [:glandium] from comment #4)
> > > --with-ccache fails if you give it something like "python script.py". Which
> > > you can work around by using a hashbang in the script, except that will fail
> > > on windows (the medium term plan being to also use the cache on windows).
> > > Also, even if the hashbang worked on windows, our CC_WRAPPER stuff would
> > > make it not work. So we need different smarts for different platforms, and
> > > none of them need to rely on in-tree changes.
> > 
> > Could you spell out the precise things we want to set for different
> > platforms? It's not clear to me if this is --enable-cache=X (where X is the
> > same or different per platform) or some other flag...or something else
> > altogether.
> 
> I'm currently thinking augmenting CC on linux/mac and defining CC_WRAPPER on
> windows.
> There would also need to be an environment variable set with a bucket name
> depending on the slave host name.
> 
> > (In reply to Mike Hommey [:glandium] from comment #2)
> > > The problem
> > > with relying on changes to in-tree mozconfigs is that it makes it unreliable
> > > whether it's used on e.g. try.
> > > Especially if we end up needing more changes
> > > afterwards.
> > 
> > You mean because of people pushing older things (like mozilla-aurora or
> > beta) to try? Or something else? The former is easy to mitigate.
> 
> That, and the fact that cross-branch changes would need landing on each
> branch. There is no reason ccache (or its replacement) setup needs landings
> on every single branch. And there is no reason not to have the same setup on
> all branches.

I think this is a little short sighted. We've regretted it nearly every other time we've enforced something as global in Buildbot - which is why more and more things are being moved in tree. I can easily see some change to this landing in 6 months to a year, and it being incompatible with esr, or release, or something else.

If it makes a difference, we'd be happy to help with the backports.

Comment 9

[Mike Hommey [:glandium]]

(In reply to Ben Hearsum [:bhearsum] from comment #8)
> I think this is a little short sighted. We've regretted it nearly every
> other time we've enforced something as global in Buildbot - which is why
> more and more things are being moved in tree.

I think you're overreacting. I understand where you're coming from, but there are things that are just too painful in-tree.

> I can easily see some change
> to this landing in 6 months to a year, and it being incompatible with esr,
> or release, or something else.

I can't.

> If it makes a difference, we'd be happy to help with the backports.

And twigs, and inbound*s*, and project branches...

Comment 10

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #9)
> > I can easily see some change
> > to this landing in 6 months to a year, and it being incompatible with esr,
> > or release, or something else.
> 
> I can't.

I'll expand here: if anything, it would be the opposite that would be (much) more likely to happen: changes landing on m-c or inbound that make it stop working.

Comment 11

[[e-mail] (:bhearsum)]

We talked about this on IRC and came to begrudging agreement that this will continue to be done in tree.

Comment 12

[(dormant account)]

TLDR: sccache should be configured out of tree because experience shows that the alternative is unworkable.

Stuff like bug 1024651(eg we can't point older trees at new buckets), work to make sccache more resilient(like bug 1020827) necessitate out-of-tree sccache configuration. bug 1007981 is already a step in this direction.If we stick to intree configs we can't upgrade our backing stores(eg S3 or varnish) for sccache for old trees. This sucks. 

We should configure sscache like jacuzzi configuration where machines don't carry jacuzzi configuration intree.