B2G messages test is timing out starting a Marionette session

RESOLVED WONTFIX

Status

Testing Graveyard
Eideticker
RESOLVED WONTFIX
4 years ago
10 months ago

People

(Reporter: davehunt, Unassigned)

Tracking

Details

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
The messages test appears to be timing out when attempting to start a Marionette session after populating the reference workload.

Traceback (most recent call last):
  File "./bin/update-dashboard.py", line 279, in <module>
    main()
  File "./bin/update-dashboard.py", line 256, in main
    eideticker.prepare_test(testkey, options)
  File "/var/jenkins/workspace/b2g.flame-512.b2g-inbound.tests/src/eideticker/eideticker/runtest.py", line 51, in prepare_test
    device.startB2G()
  File "/var/jenkins/workspace/b2g.flame-512.b2g-inbound.tests/src/eideticker/eideticker/device.py", line 522, in startB2G
    self.setupMarionette()
  File "/var/jenkins/workspace/b2g.flame-512.b2g-inbound.tests/src/eideticker/eideticker/device.py", line 474, in setupMarionette
    self.marionette.start_session()
  File "/var/jenkins/workspace/b2g.flame-512.b2g-inbound.tests/.env/local/lib/python2.7/site-packages/marionette/marionette.py", line 816, in start_session
    self.session = self._send_message('newSession', 'value')
  File "/var/jenkins/workspace/b2g.flame-512.b2g-inbound.tests/.env/local/lib/python2.7/site-packages/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/var/jenkins/workspace/b2g.flame-512.b2g-inbound.tests/.env/local/lib/python2.7/site-packages/marionette/marionette.py", line 621, in _send_message
    "Connection timed out", status=errors.ErrorCodes.TIMEOUT)
marionette.errors.TimeoutException: TimeoutException: Connection timed out

I've temporarily removed this test from the Jenkins managed script to see if the suite recovers. If it does I will check in this change to eideticker-ci. I'll see if I can replicate this issue locally.
Flags: needinfo?(dave.hunt)
(Reporter)

Comment 1

4 years ago
Here's an excerpt from a logcat generated when we see this issue:

11-06 12:28:23.770  5830  5830 I GeckoDump: Error getting tracking ID [Exception... "Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIPrefBranch.getIntPref]"  nsresult: "0x8000ffff (NS_ERROR_UNEXPECTED)"  location: "JS frame :: chrome://b2g/content/settings.js :: setUpdateTrackingId :: line 273"  data: no]
11-06 12:28:24.010  5830  5830 I Gecko   : MobileConnectionService: init complete
11-06 12:28:24.410  5830  5861 W GeckoConsole: [JavaScript Error: "IndexedDB UnknownErr: ActorsParent.cpp:413"]
11-06 12:28:24.410  5830  5861 W GeckoConsole: [JavaScript Error: "IndexedDB UnknownErr: ActorsParent.cpp:413"]
11-06 12:28:24.410  5830  5861 W GeckoConsole: [JavaScript Error: "IndexedDB UnknownErr: ActorsParent.cpp:413"]
11-06 12:28:24.440  5830  5861 W GeckoConsole: [JavaScript Error: "IndexedDB UnknownErr: ActorsParent.cpp:413"]
11-06 12:28:24.440  5830  5861 W GeckoConsole: [JavaScript Error: "IndexedDB UnknownErr: ActorsParent.cpp:413"]
11-06 12:28:24.460  5830  5830 I Gecko   : -*- SettingsRequestManager: SETTINGS DATABASE ERROR: Cannot make DB connection!
11-06 12:28:24.470  5830  5830 W GeckoConsole: [JavaScript Error: "uncaught exception: 2147500033"]
(Reporter)

Comment 2

4 years ago
I don't think this is specific to Messages. From my investigation it appears that we hit this issue when we delete /data/local/storage/persistent before populating the reference workload.
(Reporter)

Comment 3

4 years ago
I can replicate this by simply doing:

$ adb shell rm -r /data/local/storage/persistent
$ b2gpopulate --messages=200

I can then restore by doing:

$ adb shell rm -r /data/local/storage/persistent
$ adb shell stop b2g && adb shell start b2g

It would appear that the empty databases are no longer being created if there's already files in the directory. I'm not sure if this is a regression, or if we should adjust our cleanup routine to not remove persistent storage if we're going to need to write to it.
Flags: needinfo?(dave.hunt)
That reminds me of the recent indexedDB-related breakage. Maybe janv knows ?
Flags: needinfo?(Jan.Varga)
(Reporter)

Comment 5

4 years ago
b2gpopulate essentially does the same as the make reference workload script, which may be easier to use for your investigation: https://github.com/mozilla-b2g/gaia/blob/master/test_media/reference-workload/makeReferenceWorkload.sh

Comment 6

4 years ago
All origins stored in /data/local/storage/persistent now participate in temporary storage. It means that we do full scan of /data/local/storage/persistent when any of them is accessed for the first time.
Maybe you are removing the persistent directory while b2g is still running which may cause problems.

I don't have enough experience with this but I would try something like this:
adb shell stop b2g
adb shell rm -r /data/local/storage/persistent
b2gpopulate --messages=200
adb shell start b2g
Flags: needinfo?(Jan.Varga)
(Reporter)

Comment 7

4 years ago
We are stopping B2G before removing or pushing any files.

Comment 8

4 years ago
Ok, can you send me populated /data/local/storage/persistent ?
(Reporter)

Comment 9

4 years ago
Created attachment 8518241 [details]
persistent.zip

Attached.
Flags: needinfo?(Jan.Varga)

Comment 10

4 years ago
Hm, it looks ok and chrome dir shouldn't be tracked by quota manager at all.
Is that really everything what is on the device when b2g is started ?
Flags: needinfo?(Jan.Varga)
(Reporter)

Comment 11

4 years ago
Yes, that's all that's in /data/storage/persistent. Are you unable to replicate the issue? If the directory is removed then it recovers.

Comment 12

4 years ago
Well, it works ok when I try to open the sms database in firefox.
It's also strange that there's only sms database, but the error in logcat is for settings database.
(Reporter)

Comment 13

4 years ago
I think that's because the settings database is no longer being created if it doesn't already exist. That's what used to happen. Can you push the contents of the attachment to your device and see if you can replicate the issue?

Comment 14

4 years ago
Ok, I'll try.
I had similar issues when pushing my own data, and I resolved using Jan's code in comment 6. However I do one thing differently: I only delete the files I replace using:

  rm -r /data/local/storage/persistent/chrome/idb/*ssm*

Comment 16

4 years ago
I know what's going on, will submit a patch soon.
(Reporter)

Comment 17

4 years ago
Jan: Any updates?
Flags: needinfo?(Jan.Varga)

Comment 18

3 years ago
Hm, is this still happening ?
Flags: needinfo?(Jan.Varga)
I know I still have the same issues than comment 15, even when stopping b2g before pushing it...
(Reporter)

Comment 20

3 years ago
I can still replicate this, although the path has changed:

$ adb shell rm -r /data/local/storage/permanent
$ b2gpopulate --messages=200
Flags: needinfo?(Jan.Varga)

Comment 21

3 years ago
(In reply to Dave Hunt (:davehunt) from comment #20)
> I can still replicate this, although the path has changed:
> 
> $ adb shell rm -r /data/local/storage/permanent
> $ b2gpopulate --messages=200

Dave, I think the problem is that the script pushes just the sqlite file to /data/local/storage/permanent/chrome/idb directory. If the directory was previously deleted and the script pushes just the sqlite file to the directory, then quota manager will fail at origin initialization because it tries to load /data/local/storage/permanent/chrome/idb/.metadata

So, just save the .metadata file from a working device and use it in the script.
Flags: needinfo?(Jan.Varga)

Comment 22

3 years ago
(In reply to Jan Varga [:janv] from comment #21)
> (In reply to Dave Hunt (:davehunt) from comment #20)
> > I can still replicate this, although the path has changed:
> > 
> > $ adb shell rm -r /data/local/storage/permanent
> > $ b2gpopulate --messages=200
> 
> Dave, I think the problem is that the script pushes just the sqlite file to
> /data/local/storage/permanent/chrome/idb directory. If the directory was
> previously deleted and the script pushes just the sqlite file to the
> directory, then quota manager will fail at origin initialization because it
> tries to load /data/local/storage/permanent/chrome/idb/.metadata

this is actually /data/local/storage/permanent/chrome/.metadata

> 
> So, just save the .metadata file from a working device and use it in the
> script.

Note that the .metadata file contains some info such as group and origin string, so you can't use the same .metadata file for /data/local/storage/permanent/chrome and for example /data/local/storage/permanent/moz-safe-about+home or /data/local/storage/default/1007+f+app+++system.gaiamobile.org
I have the same issue but with my script that pushes data [1], but I don't delete the full directory.

[1] https://github.com/julienw/config-files/blob/master/bash_aliases#L89-L108

I'm trying to understand why the issue is intermittent...
(Reporter)

Comment 24

3 years ago
I can confirm that the following works for me:

$ adb pull /data/local/storage/permanent/chrome/.metadata
$ adb shell rm -r /data/local/storage/permanent
$ adb push .metadata /data/local/storage/permanent/chrome/.metadata
$ b2gpopulate --messages=200

Unfortunately our cleanup routine currently removes the /data/local/storage/permanent directory without backing up or restoring these .metadata files. Rather than complicate this cleanup, is there any way we can recreate the .metadata files if they've been deleted or corrupted? Alternatively, would it be safe to assume /data/local/storage/permanent/chrome/.metadata will not change and package/push it as part of b2gpopulate?
Flags: needinfo?(Jan.Varga)

Comment 25

3 years ago
(In reply to Dave Hunt (:davehunt) from comment #24)
> I can confirm that the following works for me:

Thanks for confirming the .metadata theory!

> 
> $ adb pull /data/local/storage/permanent/chrome/.metadata
> $ adb shell rm -r /data/local/storage/permanent
> $ adb push .metadata /data/local/storage/permanent/chrome/.metadata
> $ b2gpopulate --messages=200
> 
> Unfortunately our cleanup routine currently removes the
> /data/local/storage/permanent directory without backing up or restoring
> these .metadata files. Rather than complicate this cleanup, is there any way
> we can recreate the .metadata files if they've been deleted or corrupted?
> Alternatively, would it be safe to assume
> /data/local/storage/permanent/chrome/.metadata will not change and
> package/push it as part of b2gpopulate?

/data/local/storage/permanent/chrome/.metadata is special, we don't use the last access time stored in it and you can package it safely as part of b2gpopulate
Flags: needinfo?(Jan.Varga)
(Reporter)

Comment 26

a year ago
Eideticker and Firefox OS are no longer active projects.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → WONTFIX

Updated

10 months ago
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.