920403 - Broken runs due to failing to remove, via rm, certain directories, on-device

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Sadly, we're still sometimes running out of disk space on Hamachi, master:

http://qa-selenium.mv.mozilla.com:8080/view/B2G%20Hamachi/job/b2g.hamachi.mozril.gaia.master.ui/75/console

<snip>

18:33:30 + Deleting any old gaia and profiles
18:33:30 rm failed for /cache/lost+found, Directory not empty
18:33:30 rm failed for /data/b2g/*, No such file or directory
18:33:30 rm failed for /data/local/webapps, No such file or directory
18:33:30 rm failed for /data/local/user.js, No such file or directory
18:33:30 rm failed for /data/local/permissions.sqlite*, No such file or directory
18:33:30 rm failed for /data/local/OfflineCache, No such file or directory
18:33:30 rm failed for /data/local/indexedDB, No such file or directory
18:33:30 rm failed for /data/local/debug_info_trigger, No such file or directory

<snip>

18:33:43 push: gaia/profile/webapps/cubevid.gaiamobile.org/manifest.webapp -> /system/b2g/webapps/cubevid.gaiamobile.org/manifest.webapp
18:33:43 push: gaia/profile/webapps/cubevid.gaiamobile.org/application.zip -> /system/b2g/webapps/cubevid.gaiamobile.org/application.zip
18:33:45 failed to copy 'gaia/profile/webapps/cubevid.gaiamobile.org/application.zip' to '/system/b2g/webapps/cubevid.gaiamobile.org/application.zip': No space left on device
18:33:45 Build step 'Execute managed script' marked build as failure

Raymond and I looked at this earlier today, and until we |adb remount|ed, were unable -- on the device, using ssh/adb shell to get there -- to delete the directories, too, so we might need to sprinkle that before the appropriate rm -r commands

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

These rm's are part of the flash script, which already appears to do a remount:

> + ./fullflash_gecko_ril_gaia.sh -n
> Not installing Commercial Ril, should be using mozril
> + gaining root access
> adbd is already running as root
> + Waiting for adb to come back up
> + remounting the system partition      <--- remount
> remount succeeded
> + Waiting for adb to come back up
> + Stopping b2g
> + Installing new b2g
> push: b2g/dictionaries/en-US.aff -> /system/b2g/dictionaries/en-US.aff
> push: b2g/dictionaries/en-US.dic -> /system/b2g/dictionaries/en-US.dic
> push: b2g/ua-update.json -> /system/b2g/ua-update.json
> push: b2g/b2g -> /system/b2g/b2g
> push: b2g/libsoftokn3.so -> /system/b2g/libsoftokn3.so
> push: b2g/crashreporter.ini -> /system/b2g/crashreporter.ini
> push: b2g/libnssckbi.so -> /system/b2g/libnssckbi.so
> push: b2g/libnss3.so -> /system/b2g/libnss3.so
> push: b2g/updater -> /system/b2g/updater
> push: b2g/libmozglue.so -> /system/b2g/libmozglue.so
> push: b2g/run-mozilla.sh -> /system/b2g/run-mozilla.sh
> push: b2g/dependentlibs.list -> /system/b2g/dependentlibs.list
> push: b2g/removed-files -> /system/b2g/removed-files
> push: b2g/omni.ja -> /system/b2g/omni.ja
> push: b2g/platform.ini -> /system/b2g/platform.ini
> push: b2g/Throbber-small.gif -> /system/b2g/Throbber-small.gif
> push: b2g/application.ini -> /system/b2g/application.ini
> push: b2g/plugin-container -> /system/b2g/plugin-container
> push: b2g/libfreebl3.so -> /system/b2g/libfreebl3.so
> push: b2g/precomplete -> /system/b2g/precomplete
> push: b2g/libxul.so -> /system/b2g/libxul.so
> 21 files pushed. 0 files skipped.
> 3776 KB/s (33093726 bytes in 8.556s)
> + Done installing Gecko!
> + Uninstalling old RIL
> rm failed for /system/b2g/distribution/bundles/libqc_b2g_location, No such file or directory
> rm failed for /system/b2g/distribution/bundles/libqc_b2g_ril, No such file or directory
> + Removing incompatible extensions
> + Adjusting user.js
> + Deleting any old gaia and profiles
> rm failed for /cache/lost+found, Directory not empty
> rm failed for /data/b2g/*, No such file or directory
> rm failed for /data/local/webapps, No such file or directory
> rm failed for /data/local/user.js, No such file or directory
> rm failed for /data/local/permissions.sqlite*, No such file or directory
> rm failed for /data/local/OfflineCache, No such file or directory
> rm failed for /data/local/indexedDB, No such file or directory
> rm failed for /data/local/debug_info_trigger, No such file or directory

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Hey Naoki, I believe you're familiar with these flash scripts.. any clues here?

Comment 3

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I am still trying to investigate where we need to delete further.  Apparantly we need to delete some files as there are residues of the last build remaining somewhere.

Comment 4

[Teodosia Pop]

This issue is happening on aurora too.

Gecko  http://hg.mozilla.org/releases/mozilla-aurora/rev/fb764e648a8f
Gaia  1205faaebe0133333a8a14b9b3d895156438645e
BuildID 20130925004005
Version 26.0a2

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #3)
> I am still trying to investigate where we need to delete further. 
> Apparantly we need to delete some files as there are residues of the last
> build remaining somewhere.

10:43 dhylands: So I don't think fullflash cleans out old updates. Personally, I think we should be rm -r /system/b2g  and not just /system/b2g/webapps in the fullflash script

Comment 6

[Dave Hylands [:dhylands]]

It would also add a "adb shell df /system" after doing the rm -r /system/b2g (and probably again at the end of the script). This would allow you to monitor if you're running out of space.

Right now, if there was a failed/incomplete/unapplied update, it sits in /system/b2g/updated and doesn't get removed. That can wind up consuming space.

Since we're supposedly flashing everything thats needed in /system/b2g it should be good to wipe it out so that there isn't anything left over from before.

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

Dave, here's the shell script we're currently using:

https://github.com/nhirata/fullflash/blob/master/fullflash_gecko_ril_gaia.sh

Comment 8

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Doesn't it require the b2g/system to maintain the adb connection?  Or is it safe to remove because it's in memory?

Comment 9

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I think we have some fixes in the fullflash script that may resolve this issue.
https://github.com/nhirata/fullflash

I've been using it and it seems to work ok for me when deleting the profile data.

Comment 10

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Just to note as progress, we are in the midst of testing this for a little while before we place this in the build.

Comment 11

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Removing the persistant data seemed to resolve this issue :
adb shell rm -r /data/local/storage/persistent/*

Removing the profile wouldn't hurt either if you're trying to have a clean setup.
adb shell rm -r /data/b2g/mozilla