Closed Bug 1094828 Opened 10 years ago Closed 7 years ago

B2G messages test is timing out starting a Marionette session

Categories

(Testing Graveyard :: Eideticker, defect)

ARM
Gonk (Firefox OS)
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: davehunt, Unassigned)

References

Details

Attachments

(1 file)

The messages test appears to be timing out when attempting to start a Marionette session after populating the reference workload.

Traceback (most recent call last):
  File "./bin/update-dashboard.py", line 279, in <module>
    main()
  File "./bin/update-dashboard.py", line 256, in main
    eideticker.prepare_test(testkey, options)
  File "/var/jenkins/workspace/b2g.flame-512.b2g-inbound.tests/src/eideticker/eideticker/runtest.py", line 51, in prepare_test
    device.startB2G()
  File "/var/jenkins/workspace/b2g.flame-512.b2g-inbound.tests/src/eideticker/eideticker/device.py", line 522, in startB2G
    self.setupMarionette()
  File "/var/jenkins/workspace/b2g.flame-512.b2g-inbound.tests/src/eideticker/eideticker/device.py", line 474, in setupMarionette
    self.marionette.start_session()
  File "/var/jenkins/workspace/b2g.flame-512.b2g-inbound.tests/.env/local/lib/python2.7/site-packages/marionette/marionette.py", line 816, in start_session
    self.session = self._send_message('newSession', 'value')
  File "/var/jenkins/workspace/b2g.flame-512.b2g-inbound.tests/.env/local/lib/python2.7/site-packages/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/var/jenkins/workspace/b2g.flame-512.b2g-inbound.tests/.env/local/lib/python2.7/site-packages/marionette/marionette.py", line 621, in _send_message
    "Connection timed out", status=errors.ErrorCodes.TIMEOUT)
marionette.errors.TimeoutException: TimeoutException: Connection timed out

I've temporarily removed this test from the Jenkins managed script to see if the suite recovers. If it does I will check in this change to eideticker-ci. I'll see if I can replicate this issue locally.
Flags: needinfo?(dave.hunt)
Here's an excerpt from a logcat generated when we see this issue:

11-06 12:28:23.770  5830  5830 I GeckoDump: Error getting tracking ID [Exception... "Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIPrefBranch.getIntPref]"  nsresult: "0x8000ffff (NS_ERROR_UNEXPECTED)"  location: "JS frame :: chrome://b2g/content/settings.js :: setUpdateTrackingId :: line 273"  data: no]
11-06 12:28:24.010  5830  5830 I Gecko   : MobileConnectionService: init complete
11-06 12:28:24.410  5830  5861 W GeckoConsole: [JavaScript Error: "IndexedDB UnknownErr: ActorsParent.cpp:413"]
11-06 12:28:24.410  5830  5861 W GeckoConsole: [JavaScript Error: "IndexedDB UnknownErr: ActorsParent.cpp:413"]
11-06 12:28:24.410  5830  5861 W GeckoConsole: [JavaScript Error: "IndexedDB UnknownErr: ActorsParent.cpp:413"]
11-06 12:28:24.440  5830  5861 W GeckoConsole: [JavaScript Error: "IndexedDB UnknownErr: ActorsParent.cpp:413"]
11-06 12:28:24.440  5830  5861 W GeckoConsole: [JavaScript Error: "IndexedDB UnknownErr: ActorsParent.cpp:413"]
11-06 12:28:24.460  5830  5830 I Gecko   : -*- SettingsRequestManager: SETTINGS DATABASE ERROR: Cannot make DB connection!
11-06 12:28:24.470  5830  5830 W GeckoConsole: [JavaScript Error: "uncaught exception: 2147500033"]
I don't think this is specific to Messages. From my investigation it appears that we hit this issue when we delete /data/local/storage/persistent before populating the reference workload.
I can replicate this by simply doing:

$ adb shell rm -r /data/local/storage/persistent
$ b2gpopulate --messages=200

I can then restore by doing:

$ adb shell rm -r /data/local/storage/persistent
$ adb shell stop b2g && adb shell start b2g

It would appear that the empty databases are no longer being created if there's already files in the directory. I'm not sure if this is a regression, or if we should adjust our cleanup routine to not remove persistent storage if we're going to need to write to it.
Flags: needinfo?(dave.hunt)
That reminds me of the recent indexedDB-related breakage. Maybe janv knows ?
Flags: needinfo?(Jan.Varga)
b2gpopulate essentially does the same as the make reference workload script, which may be easier to use for your investigation: https://github.com/mozilla-b2g/gaia/blob/master/test_media/reference-workload/makeReferenceWorkload.sh
All origins stored in /data/local/storage/persistent now participate in temporary storage. It means that we do full scan of /data/local/storage/persistent when any of them is accessed for the first time.
Maybe you are removing the persistent directory while b2g is still running which may cause problems.

I don't have enough experience with this but I would try something like this:
adb shell stop b2g
adb shell rm -r /data/local/storage/persistent
b2gpopulate --messages=200
adb shell start b2g
Flags: needinfo?(Jan.Varga)
We are stopping B2G before removing or pushing any files.
Ok, can you send me populated /data/local/storage/persistent ?
Attached file persistent.zip
Attached.
Flags: needinfo?(Jan.Varga)
Hm, it looks ok and chrome dir shouldn't be tracked by quota manager at all.
Is that really everything what is on the device when b2g is started ?
Flags: needinfo?(Jan.Varga)
Yes, that's all that's in /data/storage/persistent. Are you unable to replicate the issue? If the directory is removed then it recovers.
Well, it works ok when I try to open the sms database in firefox.
It's also strange that there's only sms database, but the error in logcat is for settings database.
I think that's because the settings database is no longer being created if it doesn't already exist. That's what used to happen. Can you push the contents of the attachment to your device and see if you can replicate the issue?
Ok, I'll try.
I had similar issues when pushing my own data, and I resolved using Jan's code in comment 6. However I do one thing differently: I only delete the files I replace using:

  rm -r /data/local/storage/persistent/chrome/idb/*ssm*
I know what's going on, will submit a patch soon.
Jan: Any updates?
Flags: needinfo?(Jan.Varga)
Hm, is this still happening ?
Flags: needinfo?(Jan.Varga)
I know I still have the same issues than comment 15, even when stopping b2g before pushing it...
I can still replicate this, although the path has changed:

$ adb shell rm -r /data/local/storage/permanent
$ b2gpopulate --messages=200
Flags: needinfo?(Jan.Varga)
(In reply to Dave Hunt (:davehunt) from comment #20)
> I can still replicate this, although the path has changed:
> 
> $ adb shell rm -r /data/local/storage/permanent
> $ b2gpopulate --messages=200

Dave, I think the problem is that the script pushes just the sqlite file to /data/local/storage/permanent/chrome/idb directory. If the directory was previously deleted and the script pushes just the sqlite file to the directory, then quota manager will fail at origin initialization because it tries to load /data/local/storage/permanent/chrome/idb/.metadata

So, just save the .metadata file from a working device and use it in the script.
Flags: needinfo?(Jan.Varga)
(In reply to Jan Varga [:janv] from comment #21)
> (In reply to Dave Hunt (:davehunt) from comment #20)
> > I can still replicate this, although the path has changed:
> > 
> > $ adb shell rm -r /data/local/storage/permanent
> > $ b2gpopulate --messages=200
> 
> Dave, I think the problem is that the script pushes just the sqlite file to
> /data/local/storage/permanent/chrome/idb directory. If the directory was
> previously deleted and the script pushes just the sqlite file to the
> directory, then quota manager will fail at origin initialization because it
> tries to load /data/local/storage/permanent/chrome/idb/.metadata

this is actually /data/local/storage/permanent/chrome/.metadata

> 
> So, just save the .metadata file from a working device and use it in the
> script.

Note that the .metadata file contains some info such as group and origin string, so you can't use the same .metadata file for /data/local/storage/permanent/chrome and for example /data/local/storage/permanent/moz-safe-about+home or /data/local/storage/default/1007+f+app+++system.gaiamobile.org
I have the same issue but with my script that pushes data [1], but I don't delete the full directory.

[1] https://github.com/julienw/config-files/blob/master/bash_aliases#L89-L108

I'm trying to understand why the issue is intermittent...
I can confirm that the following works for me:

$ adb pull /data/local/storage/permanent/chrome/.metadata
$ adb shell rm -r /data/local/storage/permanent
$ adb push .metadata /data/local/storage/permanent/chrome/.metadata
$ b2gpopulate --messages=200

Unfortunately our cleanup routine currently removes the /data/local/storage/permanent directory without backing up or restoring these .metadata files. Rather than complicate this cleanup, is there any way we can recreate the .metadata files if they've been deleted or corrupted? Alternatively, would it be safe to assume /data/local/storage/permanent/chrome/.metadata will not change and package/push it as part of b2gpopulate?
Flags: needinfo?(Jan.Varga)
(In reply to Dave Hunt (:davehunt) from comment #24)
> I can confirm that the following works for me:

Thanks for confirming the .metadata theory!

> 
> $ adb pull /data/local/storage/permanent/chrome/.metadata
> $ adb shell rm -r /data/local/storage/permanent
> $ adb push .metadata /data/local/storage/permanent/chrome/.metadata
> $ b2gpopulate --messages=200
> 
> Unfortunately our cleanup routine currently removes the
> /data/local/storage/permanent directory without backing up or restoring
> these .metadata files. Rather than complicate this cleanup, is there any way
> we can recreate the .metadata files if they've been deleted or corrupted?
> Alternatively, would it be safe to assume
> /data/local/storage/permanent/chrome/.metadata will not change and
> package/push it as part of b2gpopulate?

/data/local/storage/permanent/chrome/.metadata is special, we don't use the last access time stored in it and you can package it safely as part of b2gpopulate
Flags: needinfo?(Jan.Varga)
Eideticker and Firefox OS are no longer active projects.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WONTFIX
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: