428650 - patcher in faster mode uses a lot of space in /tmp

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

I hit this on fx-linux-1.9-slave1, one of the staging machines (was testing bug 428063 at the time but I believe that it's unrelated. 

Log excerpt:
Extracting /builds/updates/firefox-3.0b5/patcher/temp/firefox/3.0b4/ftp/firefox-3.0b4.lt.win32.complete.mar to /tmp/tmpqMEnlT/104/from
Traceback (most recent call last):
  File "/builds/updates/firefox-3.0b5/patcher/mozilla/tools/update-packaging/make_incremental_updates.py", line 440, in <module>
    main(sys.argv[1:])
  File "/builds/updates/firefox-3.0b5/patcher/mozilla/tools/update-packaging/make_incremental_updates.py", line 437, in main
    create_partial_patches(patches)
  File "/builds/updates/firefox-3.0b5/patcher/mozilla/tools/update-packaging/make_incremental_updates.py", line 366, in create_partial_patches
    extract_mar(from_filename,work_dir_from)
  File "/builds/updates/firefox-3.0b5/patcher/mozilla/tools/update-packaging/make_incremental_updates.py", line 167, in extract_mar
    exec_shell_cmd("mar -x "+filename)    
  File "/builds/updates/firefox-3.0b5/patcher/mozilla/tools/update-packaging/make_incremental_updates.py", line 134, in exec_shell_cmd
    raise Exception, "cmd failed "+cmd
Exception: cmd failed mar -x /builds/updates/firefox-3.0b5/patcher/temp/firefox/3.0b4/ftp/firefox-3.0b4.lt.win32.complete.mar

Prior to this, fastmode built partial updates for linux and mac, but its failing partway through windows. Couldn't reproduce the extraction failure on the command line so the mar is fine.

Took a bit of debugging to figure out what was happening, as make_incremental_updates.py cleans up its temp files when it fails,
but discovered that /tmp was steadily filling up thru the run. It started from 3GB free, which is less than other machines we've used the fast mode on.

Haven't really dug into the code yet, but I'm wondering if we can cleanup as we go rather than after all the patches
 http://mxr.mozilla.org/seamonkey/source/tools/update-packaging/make_incremental_updates.py#409
or put work_dir_root somewhere with more space (eg in the same dir as patcher is called from, /builds/updates/$product-$version/patcher, rather than /tmp).

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Use cwd to create temp directory

This creates a dir tmpXXXXXX-fastmode in the current working directory.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 315276 [details] [diff] [review]
Use cwd to create temp directory

coop, what do you think of this as an approach ? 

From a quick look the code, we might need some patches from a bunch of directories (even if the vast majority are from the first partial).

Comment 3

[Chris Cooper [:coop] (he/him)]

Comment on attachment 315276 [details] [diff] [review]
Use cwd to create temp directory

Makes sense to me. We have nagios helping to monitor disk usage on this drive, so that's better than filling up /tmp.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Checking in make_incremental_updates.py;
/cvsroot/mozilla/tools/update-packaging/make_incremental_updates.py,v  <--  make_incremental_updates.py
new revision: 1.6; previous revision: 1.5
done

This will be one of the changes for UPDATE_PACKAGING_R4, which will get tagged in bug 428063.