Closed Bug 1317041 Opened 8 years ago Closed 7 years ago

Intermittent mozmake.EXE[4]: *** [symbolsfullarchive] Error 18

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox58 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla58
Tracking Flags:
Tracking Status
firefox58 --- fixed

People

(Reporter: aryx, Assigned: marco)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

Patch
3.42 KB, patch
gps
: review+
Details | Diff | Splinter Review 
https://treeherder.mozilla.org/logviewer.html#?job_id=39103333&repo=mozilla-inbound

21:19:11     INFO -    adding: xul.pdb/0C50CE1269DD411EB0A93E9EAA4E37FF1/xul.pd_ (176 bytes security) (deflated 1%)
21:19:11     INFO -    adding: xul.pdb/0C50CE1269DD411EB0A93E9EAA4E37FF1/xul.sym (176 bytes security) (deflated 84%)
21:19:11     INFO -  Makefile:337: recipe for target 'symbolsfullarchive' failed
21:19:11     INFO -  mozmake.EXE[4]: *** [symbolsfullarchive] Error 18
21:19:11     INFO -  mozmake.EXE[4]: Leaving directory 'c:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/src/obj-firefox'
21:19:11     INFO -  c:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/src/build/moz-automation.mk:122: recipe for target 'automation/buildsymbols' failed
21:19:11     INFO -  mozmake.EXE[3]: *** [automation/buildsymbols] Error 2
Depends on: 1314713
this is specific to windowns 7 taskcluster builds in debug mode- :ted, is there anything you can think of to fix this or reduce this failure?
Flags: needinfo?(ted)
10:33:14     INFO -    File "z:\build\build\src\python\mozbuild\mozpack\mozjar.py", line 748, in crc32
10:33:14     INFO -      return zlib.crc32(self._data.getvalue()) & 0xffffffff
10:33:14     INFO -  MemoryError

We're hitting an OOM trying to create the symbols zip file, presumably because the content is very large and the Python we're using on Windows is 32-bit. Specifically, this line is trying to calculate the CRC32 checksum for the zip file, for which it needs the entire uncompressed contents of what it's compressing, which is failing here.

There are a few things we could do to fix this:
1) Switch our Windows builds to use a 64-bit Python.
2) Fix this mozjar code to calculate the crc32 as data is written to the object instead of all-at-once at the end. The zlib.crc32 function we're using says in its docs that you can use it incrementally:
https://docs.python.org/2/library/zlib.html#zlib.crc32

We'd have to add a field to the `Deflater` object to hold the running crc32, then in `write`:
https://dxr.mozilla.org/mozilla-central/rev/13d241d08912be31884f9d0d0e805b25343d6c0a/python/mozbuild/mozpack/mozjar.py#705

do something like:
args = (data,) if self._crc is None else (data, self._crc)
self._crc = zlib.crc32(*args)

(assuming we initialized self._crc to None).
Flags: needinfo?(ted)
:ted- this is trending to be a more frequent failure- possibly this is something you can look at
Flags: needinfo?(ted)
I put a summary of possible fixes in comment 15, we just need to pick one and execute it.
Flags: needinfo?(ted)
:ted, is this something you would do- I am trying to determine who is doing the next step here and both of the options looks like build system tooling.
Flags: needinfo?(ted)
Assignee: nobody → mcastelluccio
Blocks: 1413015
Status: NEW → ASSIGNED
I'm trying out option 2, but option 1 will probably be needed to fully fix bug 1413015.
Attached patch Patch — — Splinter Review 

        Attachment #8924283 -
        Flags: review?(ted)

    
    

    
      
  

    
    

    
  
Comment on attachment 8924283 [details] [diff] [review]
Patch

Review of attachment 8924283 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM. I'll land this.

        Attachment #8924283 -
        Flags: review?(ted) → review+

    
    

    
  
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f47cb574c96e
Calculate crc32 incrementally; r=gps
Flags: needinfo?(ted)

    
    

    
  
https://hg.mozilla.org/mozilla-central/rev/f47cb574c96e
Status: ASSIGNED → RESOLVED
Closed: 7 years ago

          status-firefox58:
          --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla58

    
    

    
      
  

    
  
Product: Core → Firefox Build System

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: