1483228 - Standardize place to store srcdir specific state

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, enhancement)

Description

[Andrew Halberstadt [:ahal]]

Many things store state in ~/.mozbuild. The downside to this is that if a user has multiple srcdirs, the state will apply to all of them (they may wish to have separate state from one srcdir to the other).

Several tools just leave files lying around in the srcdir (and hopefully remember to add them to the .hgignore/.gitignore). Though I don't personally think this is ideal as it leaves files scattered all over the place and we often forget to ignore them on both hg and git.

In bug 1480187 we started to store some srcdir specific state in ~/.mozbuild by hashing the path to the srcdir. This was done in a rather one-off way though.

I propose we add a "srcdir=False" flag to mozboot.get_state_dir() which will return the global ~/.mozbuild state dir when False, and a srcdir specific state dir when True.

I can think of two viable options for the srcdir specific directory:
<topsrcdir>/.mozbuild
~/.mozbuild/<topsrcdir-basename>-<topsrcdir-hash>

I don't really have a strong preference for one or the other.

p.s Ideally (IMO) people would use mozilla-unified and not have multiple source checkouts, but I don't think convincing everyone to change their workflows is going to be possible, so this is still a problem worth addressing.

Comment 1

[Gabriel Vîjială [:gabriel-v]]

(In reply to Andrew Halberstadt [:ahal] from comment #0)
> p.s Ideally (IMO) people would use mozilla-unified and not have multiple
> source checkouts, but I don't think convincing everyone to change their
> workflows is going to be possible, so this is still a problem worth
> addressing.

In my case, I do use mozilla-unified. I need multiple source checkouts because I work on different areas at once:

- fennec
- tryselect
- mozharness
- cocoa (for a pet peeve on weekends)

Each one of those areas has a different set of mozconfigs and a different build. Using a single repository with `hg shelve` seems a gigantic hassle.

I also prefer to use multiple source directories in order to mitigate the long time `hg push review`  / `arc diff` and `./mach try X` take to run, time during which they lock up the mercurial repository.


> I can think of two viable options for the srcdir specific directory:
> <topsrcdir>/.mozbuild
> ~/.mozbuild/<topsrcdir-basename>-<topsrcdir-hash>

I prefer the <topsrcdir>/.mozbuild directory because:

- it survives when the <topsrcdir> gets moved or renamed
- it gets deleted when the <topsrcdir> is deleted

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1483228 - [mozboot] Simplify get_state_dir()'s return value

mozboot.util.get_state_dir() returns a tuple of (<path>, <bool). The bool
denotes whether or not the state dir came from an environment variable.

But this value is only used in a single place, and is very easy to test for
anyway. It's not worth the added complexity it imposes on all other consumers
of this function. Let's just make this function return the path.

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1483228 - [mozboot] Add ability to get a srcdir specific state dir

Sometimes we want to store state that only applies to a particular srcdir, but
there isn't a standard directory where this lives. Let's add an argument to
'get_state_dir()' to provide an "official" place.

The new API to get the local state dir is 'get_state_dir(srcdir=True)'. Like
the global state dir, this directory is not guaranteed to exist. A reference to
this value can also be obtained via 'self._mach_context.local_state_dir' from
within a mach command (in this case it will be created automatically if it
doesn't exist).

Note: we should probably just make sure both exist at mach startup, but it felt
outside the scope of this change.

Depends on D15723

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1483228 - [tryselect] Use get_state_dir(srcdir=True) to determine directory to store history state

This was previously implemented by creating the local state dir within the
global state dir (using a hash of the path in the name). This moves the
|mach try again| history state to the new local state dir.

It also updates the migration code to be a little more robust, ensuring we
don't accidentally lose people's history. This migration should be removed
at some point in the future (2020 seemed like a safe enough window).

Depends on D15724

Comment 5

[Andrew Halberstadt [:ahal]]

Nick, does my reply in https://phabricator.services.mozilla.com/D15724 answer your question? Are changes still needed to proceed?

Mike, removed you as a reviewer but do you agree with putting this state dir alongside the srcdir? (As opposed to hashed inside ~/.mozbuild)

Comment 6

[Mike Hommey [:glandium]]

Mmmm I think we should avoid writing in the source directory. I think it's better if we fill state stuff in a single central location that users can look at directly. However, as there can be multiple working trees, it might be better to use ~/.mozbuild/state/name-hash, rather than ~/.mozbuild/name-hash.

Comment 7

[Andrew Halberstadt [:ahal]]

Wfm. I'll update the patch.

Just pointing out that once a user deletes a srcdir we won't be able to cleanup its state dir anymore. I don't think this will be a huge problem though.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/205bf106e084
[mozboot] Simplify get_state_dir()'s return value r=nalexander
https://hg.mozilla.org/integration/autoland/rev/c64aeee4b5e0
[mozboot] Add ability to get a srcdir specific state dir r=nalexander
https://hg.mozilla.org/integration/autoland/rev/de78b968e178
[tryselect] Use get_state_dir(srcdir=True) to determine directory to store history state r=nalexander

Comment 9

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/205bf106e084
https://hg.mozilla.org/mozilla-central/rev/c64aeee4b5e0
https://hg.mozilla.org/mozilla-central/rev/de78b968e178