1043618 - [B2G][Smart Collection] after repeated python marionette test, smart collection icons change location on each reboot of B2G

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::Homescreen, defect)

Description

[No-Jun Park [:njpark]]

STR:
Flash a new build, and run a python marionette test case, like gaiatest/tests/functional/homescreen/test_homescreen_move_app.py multiple times. (example command: gaiatest --address=localhost:2828 --testvars=gaiatest/$config --restart  --repeat=5 --type=b2g gaiatest/tests/functional/homescreen/test_homescreen_move_app.py)

Expected:
after each reboot, the location of icons should be the same
Actual:
after 2nd or 3rd reboot of the B2G, the location of smart collections icons move to either the top of the vertical homescreen, or at the very bottom of the screen (need to scroll down to see it)

Note:
Repeatedly running adb shell stop b2g and adb shell start b2g does not cause this error.

Version Info:
No-Jun-MacBook-Pro:B2G-flash-tool mozilla$ ./check_versions.sh
Gaia      c72257b2d27135bfcd68e89dd584182797784016
Gecko     https://hg.mozilla.org/mozilla-central/rev/06ac51c2b8a8
BuildID   20140724040205
Version   34.0a1
ro.build.version.incremental=109
ro.build.date=Mon Jun 16 16:51:29 CST 2014
B1TC00011220

CC'ing webqa team for more info on what gaiatest does on each reboot of the device

Comment 1

[No-Jun Park [:njpark]]

Attachment: default layout

Comment 2

[No-Jun Park [:njpark]]

Attachment: altered layout after multiple reset of b2g

Comment 3

[No-Jun Park [:njpark]]

Asking for zac's expertise on gaiatest.  Is there anything else I can simulate the repeated run of gaiatest other than stopping and restarting b2g?

Comment 4

[Zac C (:zac)]

gaiatest deletes the profile and some other files (see the cleanup_data method and others). This means that a new profile is generated as b2g is starting up and as I understand, creating collections is part of that (in that they're not pre-installed in the profile). We've seen a race condition in homescreen creating the icons in the past but it was resolved.. Perhaps there is a slightly more isolated race case. I suppose it's possible this happens on CI but the test deals with it by just finding the icon at the different location.

Was there anything in logcat?

I'll cc kgrandon but not ni him until we think this is a bug.

Comment 5

[Kevin Grandon :kgrandon]

I think we've seen something similar in marionetteJS tests. Currently our smart collections are migrated as part of the FTU - so if you have some tests that startup a clean profile, but not run FTU - they may not show up, or show up at the bottom. This seems to work well for a user, but perhaps not so well for CI.

Comment 6

[No-Jun Park [:njpark]]

Thanks for your inputs :zac and :kgrandon, since the gaiatest run skips the FTU, this is probably why the smart collection icons sometimes appear at the bottom or not appear at all.

I noticed this when I was running my screenshot comparison test, because this made the comparison to fail at random times after I move the icons and take pictures to compare it to the previous runs.  I realize this would be a low priority at this point, but can we regard this as a bug?  (in that case, would this be a marionette bug or FTU bug?) Or if there is a workaround for this that'd be great too.

Comment 7

[Kevin Grandon :kgrandon]

I would ask if it's really necessary - I'm thinking that reftests of an entire app are going to be pretty brittle, and wouldn't recommend this kind of strategy. If it's something we absolutely want to fix, I'd say that we need to run ftu on *every* test that runs these screenshot tests.

I don't really think trying to change gaia at this point is a good idea, there's a risk of upgrade regressions which are poorly tested and scary to tough =/

Comment 8

[No-Jun Park [:njpark]]

(In reply to Kevin Grandon :kgrandon from comment #7)
> I would ask if it's really necessary - I'm thinking that reftests of an
> entire app are going to be pretty brittle, and wouldn't recommend this kind
> of strategy. If it's something we absolutely want to fix, I'd say that we
> need to run ftu on *every* test that runs these screenshot tests.
> 
> I don't really think trying to change gaia at this point is a good idea,
> there's a risk of upgrade regressions which are poorly tested and scary to
> tough =/

Thanks for your input.  actually I'm working on a different kind of image comparison test based on gaiatest, where the reference screenshots are being compared to subsequent runs for correctness. (it's small now, but it appears to be useful) But I agree that we shouldn't poke gaia unless there is a bigger danger.  For the homescreen runs, I'll investigate the option to run the FTU first for now, and see what happens.