Intermittent mozmake.EXE[4]: *** [symbolsfullarchive] Error 18

RESOLVED FIXED in Firefox 58

Status

()

Core
Build Config
RESOLVED FIXED
a year ago
3 months ago

People

(Reporter: aryx, Assigned: marco)

Tracking

({intermittent-failure})

unspecified
mozilla58
intermittent-failure
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox58 fixed)

Details

Attachments

(1 attachment)

https://treeherder.mozilla.org/logviewer.html#?job_id=39103333&repo=mozilla-inbound

21:19:11     INFO -    adding: xul.pdb/0C50CE1269DD411EB0A93E9EAA4E37FF1/xul.pd_ (176 bytes security) (deflated 1%)
21:19:11     INFO -    adding: xul.pdb/0C50CE1269DD411EB0A93E9EAA4E37FF1/xul.sym (176 bytes security) (deflated 84%)
21:19:11     INFO -  Makefile:337: recipe for target 'symbolsfullarchive' failed
21:19:11     INFO -  mozmake.EXE[4]: *** [symbolsfullarchive] Error 18
21:19:11     INFO -  mozmake.EXE[4]: Leaving directory 'c:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/src/obj-firefox'
21:19:11     INFO -  c:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/src/build/moz-automation.mk:122: recipe for target 'automation/buildsymbols' failed
21:19:11     INFO -  mozmake.EXE[3]: *** [automation/buildsymbols] Error 2
Depends on: 1314713

Comment 1

a year ago
5 failures in 715 pushes (0.007 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 3
* try: 1
* mozilla-central: 1

Platform breakdown:
* windows8-64: 3
* windowsxp: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2016-11-14&endday=2016-11-20&tree=all

Comment 2

a year ago
6 failures in 623 pushes (0.01 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* autoland: 3
* mozilla-inbound: 2
* mozilla-aurora: 1

Platform breakdown:
* windowsxp: 3
* windows8-64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2016-11-21&endday=2016-11-27&tree=all

Comment 3

a year ago
5 failures in 694 pushes (0.007 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-aurora: 2
* autoland: 2
* mozilla-inbound: 1

Platform breakdown:
* windows8-64: 3
* windowsxp: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2016-11-28&endday=2016-12-04&tree=all

Comment 4

a year ago
6 failures in 289 pushes (0.021 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 3
* autoland: 2
* mozilla-central: 1

Platform breakdown:
* windowsxp: 4
* windows8-64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2016-12-05&endday=2016-12-11&tree=all

Comment 5

8 months ago
1 failures in 814 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows2012-64-noopt: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-06-12&endday=2017-06-18&tree=all

Comment 6

8 months ago
11 failures in 656 pushes (0.017 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 7
* mozilla-inbound: 4

Platform breakdown:
* windows2012-32: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-07-03&endday=2017-07-09&tree=all

Comment 7

7 months ago
12 failures in 720 pushes (0.017 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 7
* mozilla-inbound: 4
* oak: 1

Platform breakdown:
* windows2012-32: 12

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-07-10&endday=2017-07-16&tree=all

Comment 8

7 months ago
7 failures in 822 pushes (0.009 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 4
* mozilla-inbound: 3

Platform breakdown:
* windows2012-32: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-07-17&endday=2017-07-23&tree=all

Comment 9

7 months ago
16 failures in 1008 pushes (0.016 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 14
* try: 1
* mozilla-inbound: 1

Platform breakdown:
* windows2012-32: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-07-24&endday=2017-07-30&tree=all

Comment 10

7 months ago
21 failures in 888 pushes (0.024 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 12
* mozilla-inbound: 5
* mozilla-central: 3
* try: 1

Platform breakdown:
* windows2012-32: 21

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-07-31&endday=2017-08-06&tree=all

Comment 11

7 months ago
24 failures in 901 pushes (0.027 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 13
* mozilla-inbound: 11

Platform breakdown:
* windows2012-32: 24

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-08-07&endday=2017-08-13&tree=all

Comment 12

6 months ago
15 failures in 949 pushes (0.016 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 8
* mozilla-inbound: 6
* try: 1

Platform breakdown:
* windows2012-32: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-08-14&endday=2017-08-20&tree=all

Comment 13

6 months ago
16 failures in 908 pushes (0.018 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 8
* mozilla-inbound: 7
* mozilla-central: 1

Platform breakdown:
* windows2012-32: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-08-21&endday=2017-08-27&tree=all
this is specific to windowns 7 taskcluster builds in debug mode- :ted, is there anything you can think of to fix this or reduce this failure?
Flags: needinfo?(ted)
10:33:14     INFO -    File "z:\build\build\src\python\mozbuild\mozpack\mozjar.py", line 748, in crc32
10:33:14     INFO -      return zlib.crc32(self._data.getvalue()) & 0xffffffff
10:33:14     INFO -  MemoryError

We're hitting an OOM trying to create the symbols zip file, presumably because the content is very large and the Python we're using on Windows is 32-bit. Specifically, this line is trying to calculate the CRC32 checksum for the zip file, for which it needs the entire uncompressed contents of what it's compressing, which is failing here.

There are a few things we could do to fix this:
1) Switch our Windows builds to use a 64-bit Python.
2) Fix this mozjar code to calculate the crc32 as data is written to the object instead of all-at-once at the end. The zlib.crc32 function we're using says in its docs that you can use it incrementally:
https://docs.python.org/2/library/zlib.html#zlib.crc32

We'd have to add a field to the `Deflater` object to hold the running crc32, then in `write`:
https://dxr.mozilla.org/mozilla-central/rev/13d241d08912be31884f9d0d0e805b25343d6c0a/python/mozbuild/mozpack/mozjar.py#705

do something like:
args = (data,) if self._crc is None else (data, self._crc)
self._crc = zlib.crc32(*args)

(assuming we initialized self._crc to None).
Flags: needinfo?(ted)

Comment 16

6 months ago
15 failures in 939 pushes (0.016 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 9
* mozilla-inbound: 4
* mozilla-central: 2

Platform breakdown:
* windows2012-32: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-08-28&endday=2017-09-03&tree=all

Comment 17

6 months ago
16 failures in 924 pushes (0.017 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 10
* mozilla-inbound: 4
* mozilla-central: 2

Platform breakdown:
* windows2012-32: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-09-04&endday=2017-09-10&tree=all

Comment 18

5 months ago
16 failures in 1032 pushes (0.016 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 10
* try: 3
* mozilla-inbound: 3

Platform breakdown:
* windows2012-32: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-09-11&endday=2017-09-17&tree=all

Comment 19

5 months ago
19 failures in 943 pushes (0.02 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 12
* mozilla-central: 3
* mozilla-inbound: 2
* try: 1
* mozilla-beta: 1

Platform breakdown:
* windows2012-32: 19

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-09-18&endday=2017-09-24&tree=all

Comment 20

5 months ago
18 failures in 885 pushes (0.02 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 8
* mozilla-inbound: 5
* try: 2
* mozilla-central: 2
* mozilla-beta: 1

Platform breakdown:
* windows2012-32: 18

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-09-25&endday=2017-10-01&tree=all

Comment 21

5 months ago
23 failures in 824 pushes (0.028 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 10
* autoland: 10
* try: 2
* mozilla-beta: 1

Platform breakdown:
* windows2012-32: 23

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-10-02&endday=2017-10-08&tree=all

Comment 22

4 months ago
15 failures in 947 pushes (0.016 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 11
* mozilla-inbound: 4

Platform breakdown:
* windows2012-32: 13
* osx-10-10: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-10-09&endday=2017-10-15&tree=all
:ted- this is trending to be a more frequent failure- possibly this is something you can look at
Flags: needinfo?(ted)

Comment 24

4 months ago
22 failures in 864 pushes (0.025 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 13
* mozilla-inbound: 9

Platform breakdown:
* windows2012-32: 22

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-10-16&endday=2017-10-22&tree=all
I put a summary of possible fixes in comment 15, we just need to pick one and execute it.
Flags: needinfo?(ted)
:ted, is this something you would do- I am trying to determine who is doing the next step here and both of the options looks like build system tooling.
Flags: needinfo?(ted)

Comment 27

4 months ago
17 failures in 912 pushes (0.019 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 10
* mozilla-inbound: 4
* try: 2
* mozilla-central: 1

Platform breakdown:
* windows2012-32: 17

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-10-23&endday=2017-10-29&tree=all
(Assignee)

Updated

4 months ago
Assignee: nobody → mcastelluccio
Blocks: 1413015
Status: NEW → ASSIGNED
(Assignee)

Comment 28

4 months ago
I'm trying out option 2, but option 1 will probably be needed to fully fix bug 1413015.
(Assignee)

Comment 29

4 months ago
Created attachment 8924283 [details] [diff] [review]
Patch
Attachment #8924283 - Flags: review?(ted)

Comment 30

4 months ago
12 failures in 857 pushes (0.014 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 9
* mozilla-inbound: 2
* try: 1

Platform breakdown:
* windows2012-32: 12

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-10-30&endday=2017-11-05&tree=all

Comment 31

4 months ago
Comment on attachment 8924283 [details] [diff] [review]
Patch

Review of attachment 8924283 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM. I'll land this.
Attachment #8924283 - Flags: review?(ted) → review+

Comment 32

4 months ago
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f47cb574c96e
Calculate crc32 incrementally; r=gps
Flags: needinfo?(ted)

Comment 33

4 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/f47cb574c96e
Status: ASSIGNED → RESOLVED
Last Resolved: 4 months ago
status-firefox58: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla58

Comment 34

3 months ago
7 failures in 849 pushes (0.008 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 4
* mozilla-inbound: 2
* mozilla-central: 1

Platform breakdown:
* windows2012-32: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317041&startday=2017-11-06&endday=2017-11-12&tree=all
You need to log in before you can comment on or make changes to this bug.