1282256 - Frequent windows mozbuild.mozconfig.MozconfigLoadException: Evaluation of your mozconfig exited with an error. This could be triggered

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Treeherder Bug Filer]

Filed by: [e-mail]

https://treeherder.mozilla.org/logviewer.html#?job_id=10101839&repo=fx-team

http://archive.mozilla.org/pub/firefox/tinderbox-builds/fx-team-win64/1466891003/fx-team-win64-bm94-build1-build8.txt.gz

By which it means:

15:26:22     INFO -  c:\builds\moz2_slave\fx-team-w64-000000000000000000\build\src\python\mozbuild\mozbuild\mozconfig.py:248: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
15:26:22     INFO -    index = lines.index('------END_BEFORE_SOURCE')
15:26:22     INFO -  Traceback (most recent call last):
15:26:22     INFO -    File "c:/builds/moz2_slave/fx-team-w64-000000000000000000/build/src/config/expandlibs_gen.py", line 11, in <module>
15:26:22     INFO -      import expandlibs_config as conf
15:26:22     INFO -    File "c:\builds\moz2_slave\fx-team-w64-000000000000000000\build\src\build\mach_bootstrap.py", line 380, in __call__
15:26:22     INFO -      module = self._original_import(name, globals, locals, fromlist, level)
15:26:22     INFO -    File "c:\builds\moz2_slave\fx-team-w64-000000000000000000\build\src\config\expandlibs_config.py", line 5, in <module>
15:26:22     INFO -      from buildconfig import substs
15:26:22     INFO -    File "c:\builds\moz2_slave\fx-team-w64-000000000000000000\build\src\build\mach_bootstrap.py", line 380, in __call__
15:26:22     INFO -      module = self._original_import(name, globals, locals, fromlist, level)
15:26:22     INFO -    File "c:\builds\moz2_slave\fx-team-w64-000000000000000000\build\src\build\buildconfig.py", line 9, in <module>
15:26:22     INFO -      config = MozbuildObject.from_environment()
15:26:22     INFO -    File "c:\builds\moz2_slave\fx-team-w64-000000000000000000\build\src\python\mozbuild\mozbuild\base.py", line 169, in from_environment
15:26:22     INFO -      config = loader.read_mozconfig(mozconfig, moz_build_app=current_project)
15:26:22     INFO -    File "c:\builds\moz2_slave\fx-team-w64-000000000000000000\build\src\python\mozbuild\mozbuild\mozconfig.py", line 253, in read_mozconfig
15:26:22     INFO -      raise MozconfigLoadException(path, MOZCONFIG_BAD_EXIT_CODE, lines)

Comment 1

[Gregory Szorc [:gps]]

Intermittent?!

Comment 2

[Intermittent Failures Robot]

33 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 20
* autoland: 7
* fx-team: 4
* try: 1
* ash: 1

Platform breakdown:
* windows8-64: 17
* windowsxp: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1282256&startday=2016-06-27&endday=2016-07-03&tree=all

Comment 3

[Carsten Book [:Tomcat]]

(In reply to Gregory Szorc [:gps] from comment #1)
> Intermittent?!

well more frequent

Comment 4

[Intermittent Failures Robot]

40 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 20
* fx-team: 11
* mozilla-central: 3
* autoland: 3
* try: 2
* ash: 1

Platform breakdown:
* windows8-64: 27
* windowsxp: 13

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1282256&startday=2016-07-04&endday=2016-07-10&tree=all

Comment 5

[Intermittent Failures Robot]

19 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 6
* autoland: 5
* fx-team: 4
* mozilla-central: 3
* try: 1

Platform breakdown:
* windows8-64: 11
* windowsxp: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1282256&startday=2016-07-11&endday=2016-07-17&tree=all

Comment 6

[Intermittent Failures Robot]

16 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 6
* fx-team: 4
* autoland: 3
* ash: 3

Platform breakdown:
* windowsxp: 9
* windows8-64: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1282256&startday=2016-07-18&endday=2016-07-24&tree=all

Comment 7

[Intermittent Failures Robot]

17 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 6
* mozilla-inbound: 5
* fx-team: 4
* try: 1
* ash: 1

Platform breakdown:
* windows8-64: 10
* windowsxp: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1282256&startday=2016-07-25&endday=2016-07-31&tree=all

Comment 8

[Mike Hommey [:glandium]]

Attachment: Bug 1282256 - Avoid loading mozconfig in MozbuildObject.from_environment.

We've been reading the mozconfig in MozbuildObject.from_environment to
check whether the mozconfig topobjdir matches the detected topobjdir.

Since bug 1278415, everything using the buildconfig python module now
calls MozbuildObject.from_environment, which reads the mozconfig. A lot
of things to that during the build. But none of them actually need the
data from the mozconfig, and the topobjdir match test has been breaking
things randomly on multiple occasions.

The topobjdir match test, however, really only needs to happen once:
when a mach command starts. So we can move the test to MachCommandBase,
where it belongs, and anything actively using MozbuildObject.mozconfig
will have the mozconfig read, but everything else won't.

On a Linux64 opt build, this brings down the number of times the
mozconfig is read during `mach build` from 979 to 9.

Review commit: https://reviewboard.mozilla.org/r/69168/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/69168/

Comment 9

[Mike Hommey [:glandium]]

Let's pile up some clean up as well.

Comment 10

[Mike Hommey [:glandium]]

Attachment: Bug 1282256 - Remove MozbuildObject._config_guess.

Back when it was added, it was used, but it is not anymore, outside
test_base.py.

Review commit: https://reviewboard.mozilla.org/r/69174/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/69174/

Comment 11

[Mike Hommey [:glandium]]

Attachment: Bug 1282256 - Make MozbuildObject.resolve_mozconfig_topobjdir an instance method.

The only use that didn't have an existing instance was just removed.

Review commit: https://reviewboard.mozilla.org/r/69176/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/69176/

Comment 12

[Mike Hommey [:glandium]]

Attachment: Bug 1282256 - Make MozbuildObject.resolve_config_guess an instance method.

The last use that didn't have an existing instance was just removed.

Review commit: https://reviewboard.mozilla.org/r/69178/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/69178/

Comment 13

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #8)
> On a Linux64 opt build, this brings down the number of times the
> mozconfig is read during `mach build` from 979 to 9.

The remaining 9 come from:
- 5 times `mach environment --format=client.mk`
- twice `mach environment --format=json -o $topobjdir/.mozconfig.json`
- twice `configure.py`

That's suboptimal, but that can be left for a followup. Or we could declare that we don't care about the first 2 items, because they come from client.mk and we'll eventually get rid of it. As for configure.py, it reads it twice because once before checking PYTHON, and once after restarting in the virtualenv.

Comment 15

[Gregory Szorc [:gps]]

Comment on attachment 8777651 [details]
Bug 1282256 - Avoid loading mozconfig in MozbuildObject.from_environment.

https://reviewboard.mozilla.org/r/69168/#review66808

I didn't realize we were evaluating the mozconfig so often during a build. That was horrible.

Comment 16

[Gregory Szorc [:gps]]

Comment on attachment 8777654 [details]
Bug 1282256 - Remove MozbuildObject._config_guess.

https://reviewboard.mozilla.org/r/69174/#review66810

Comment 17

[Gregory Szorc [:gps]]

Comment on attachment 8777655 [details]
Bug 1282256 - Make MozbuildObject.resolve_mozconfig_topobjdir an instance method.

https://reviewboard.mozilla.org/r/69176/#review66812

Comment 18

[Gregory Szorc [:gps]]

Comment on attachment 8777656 [details]
Bug 1282256 - Make MozbuildObject.resolve_config_guess an instance method.

https://reviewboard.mozilla.org/r/69178/#review66814

Comment 19

[Intermittent Failures Robot]

38 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 16
* autoland: 11
* fx-team: 5
* try: 2
* mozilla-central: 2
* mozilla-aurora: 1
* larch: 1

Platform breakdown:
* windowsxp: 19
* windows8-64: 18
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1282256&startday=2016-08-01&endday=2016-08-07&tree=all

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9774cf27be69
Avoid loading mozconfig in MozbuildObject.from_environment. r=gps
https://hg.mozilla.org/integration/autoland/rev/fbeb48318c1e
Remove MozbuildObject._config_guess. r=gps
https://hg.mozilla.org/integration/autoland/rev/58bdfaad94aa
Make MozbuildObject.resolve_mozconfig_topobjdir an instance method. r=gps
https://hg.mozilla.org/integration/autoland/rev/61a4dd746681
Make MozbuildObject.resolve_config_guess an instance method. r=gps

Comment 21

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/9774cf27be69
https://hg.mozilla.org/mozilla-central/rev/fbeb48318c1e
https://hg.mozilla.org/mozilla-central/rev/58bdfaad94aa
https://hg.mozilla.org/mozilla-central/rev/61a4dd746681

Comment 22

[Ryan VanderMeulen [:RyanVM]]

Please request Aurora approval on this when you get a chance.

Comment 23

[Mike Hommey [:glandium]]

Does it happen enough on aurora to take the risk? For one, there's at least one followup bug this caused. I'm also not entirely sure how independent it is from other changes to the build system. So I'd rather not try to uplift if it's not too much of a problem on aurora.

Comment 24

[Ryan VanderMeulen [:RyanVM]]

It happens on Aurora as well, but not at a high enough frequency to care if there's regression possibilities. On the other hand, it could at least in theory be a bigger problem when 50 goes to Beta if a broken build delays a go to build.

Comment 25

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #23)
> Does it happen enough on aurora to take the risk? For one, there's at least
> one followup bug this caused.

I might have misremembered here, because I couldn't find a trace of such a followup when looking at the history following this bug. However *this* bug is *also* a fixup for previous bugs, some of which are in 50, while this one is in 51 only. I'm suspecting this is the cause for bug 1303926 (and try confirms that uplifting this to beta fixes that bug)

Comment 26

[Carsten Book [:Tomcat]]

(In reply to Mike Hommey [:glandium] from comment #25)
> (In reply to Mike Hommey [:glandium] from comment #23)
> > Does it happen enough on aurora to take the risk? For one, there's at least
> > one followup bug this caused.
> 
> I might have misremembered here, because I couldn't find a trace of such a
> followup when looking at the history following this bug. However *this* bug
> is *also* a fixup for previous bugs, some of which are in 50, while this one
> is in 51 only. I'm suspecting this is the cause for bug 1303926 (and try
> confirms that uplifting this to beta fixes that bug)

mike, sylvestre, can we have approval for beta for this changes to fix bug 1303926 ?

Comment 27

[Carsten Book [:Tomcat]]

s/approval/ approval or the approval request

Comment 28

[Sylvestre Ledru [:Sylvestre]]

Approved :)

Comment 29

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/5a648a56c45e
https://hg.mozilla.org/releases/mozilla-beta/rev/99b6469ac099
https://hg.mozilla.org/releases/mozilla-beta/rev/4891b4b3ebb0
https://hg.mozilla.org/releases/mozilla-beta/rev/6be9a5c9c8cd