1164772 - [SmartCollections] empty icons for Movies & Utilities collections

Status: VERIFIED FIXED

(Firefox OS Graveyard :: Gaia::Everything.me, defect)

Description

[Hermes Cheng (inactive after July 27, 2015)]

Attachment: 2015-05-14-14-41-53.png

As attachment shown, some collections do not have a correct icon.


At v2.2 test env, they are Movies & Utilities.
At master test env, Movies & Photography are incorrect.


* Test Env
**2.2 (with local gecko build)
Build ID               20150513121019
Gaia Revision          e048df68f6f4853b5826a8816e143d95258149de
Gaia Date              2015-05-12 19:10:26
Gecko Revision         n/a
Gecko Version          37.0
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150513.041549
Firmware Date          Wed May 13 04:15:58 EDT 2015
Bootloader             L1TC000118D0

**master
Build ID               20150512160203
Gaia Revision          3654ec1d7ce1e0a56a34d5c3b06f6a9b33ff79ad
Gaia Date              2015-05-12 05:13:33
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/42db79f3cd6b
Gecko Version          41.0a1
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150512.192015
Firmware Date          Tue May 12 19:20:26 EDT 2015
Bootloader             L1TC000118D0

Comment 1

[Gregor Wagner [:gwagner]]

Seems like a server issue.
Is this 100% reproducible? Lets see if we can get a regression window. If its a server issue we probably can't find a window.

Comment 2

[Gregor Wagner [:gwagner]]

NI for Kevin if he knows about any client side changes.

Comment 3

[Kevin Grandon :kgrandon]

No changes on the client side as far as I know. Can we do branch checks?

Comment 4

[Pi Wei Cheng [:piwei] (inactive)]

I think this issue is related to the overall network issue of connecting to smart collections. There is a related meta bug 1163040. While testing I encountered many times of the server telling me I've no internet connection when in fact I am connected to wifi, and this occurred to me regardless of which branch I'm in.

For this bug, it is reproducible on Flame 3.0, 2.2, and 2.1. Empty icons are displayed for certain smart collections. I've observed it happened for Photography on 2.2 and 3.0, and Movies & Photography for 2.1.

Device: Flame 3.0
BuildID: 20150514010203
Gaia: 338f66e6a96491d2f5854b188c6b141ceb690d97
Gecko: 1fab94ad196c
Gonk: 040bb1e9ac8a5b6dd756fdd696aa37a8868b5c67
Version: 41.0a1 (3.0) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:41.0) Gecko/41.0 Firefox/41.0

Device: Flame 2.2
BuildID: 20150514002501
Gaia: aac58a063e3e6acae6ba77fe4cec224fb69450bc
Gecko: 47f1ced9f1d6
Gonk: ab265fb203390c70b8f2a054f38cf4b2f2dad70a
Version: 37.0 (2.2) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:37.0) Gecko/37.0 Firefox/37.0

Device: Flame 2.1
BuildID: 20150514001201
Gaia: c80865cb0bf73f1b97defbc646083b404feb3ac4
Gecko: 23eb9df75991
Gonk: ab265fb203390c70b8f2a054f38cf4b2f2dad70a
Version: 34.0 (2.1) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0

Removing regression keyword since we're not sure if it's a regression.

Comment 5

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

https://github.com/mozilla-b2g/gaia/commit/9d19ae7dd104fd253a8b47f06adacd8cdc0d2281

There's a gaia change in the last 17 days that might have affected it.  I'll have to play around with it.  Rather than reverting, perhaps playing with the numbers might be a good idea.

For the full list, https://github.com/mozilla-b2g/gaia/commits/master/apps/collection
I know I tested Kevin's patch on 4/7 so my guess is autonome's patch.  I think it caused some breaks on other spots.

Needinfo on myself to do this as an action item.

Comment 6

[Michael Henretty [:mikehenrty][:mhenretty]]

Assigning to Naoki while he investigates.

Comment 7

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: logcat_snippit.txt

There's a lot of css property errors, where it falls back to initial.  I'm not sure what the initial values are.  Still need to dig in.  Looking at reverting to the original code for each of these lines to see if that fixes the issue as a test since the variables touch more than just collections.

Comment 8

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

So reverting the lines seemed to help some  ( 
https://github.com/nhirata/mozilla_b2g_gaia/compare/master...nhirata:collection_check )
; whenever it can't draw, I've noticed this in the logcat:
W/Smart Collections(17868): [JavaScript Warning: "An IndexedDB transaction that was not yet complete has been aborted due to page navigation." {file: "app://collection.gaiamobile.org/gaia_build_defer_view.js" line: 301}]

I think we might need a check for that.

The other thing I noticed was : E/(Preallocated app)(17924): [JavaScript Error: "Error: You cannot use the AddonManager in child processes!" {file: "resource://gre/modules/AddonManager.jsm" line: 20}]
which seems to be an preallocated app process issue.  I filed bug 1166545 as it seems to be causing load "lag" and at the same time may be a separate issue to deal with.


Chris, could you take a look please?

Comment 9

[Sam Foster [:sfoster] (he/him)]

I'll look into this.

Comment 10

[Sam Foster [:sfoster] (he/him)]

I did a little logging and poking around. First, all the E.me api requests in apps/collection/js/view_apps.js direct promise rejections to onOffline - so all errors are reported as no-connection errors, which is why we see that message. 

What we are seeing is actually "503 Service Unavailable: Back-end server is at capacity." Here's some logging when reproducing the issue with the Music collection

Collection queueRequest, navigator.onLine:  true view_apps.js:109:6
evme init: noop eme.js:72:8
evme API request: "https://appsearch.services.mozilla.com/partners/1.0/Apps/search/?limit=24&categoryId=142&iconFormat=20&apiKey=<snip>&deviceId=fxos-app://collection.gaiamobile.org/aabad456-e485-4c73-80a6-0f30cb1d16f5&ctx={"lc":"en-US","tz":"-7","v":"2.2.0.0-prerelease","dn":"Flame","cr":"AT&T","ct":"wifi","mcc":"310","mnc":"410","sr":"480x853.5"}&" eme.js:72:8
evme getMozBackground success "https://fxos.cdn.mozilla.net/collection/background/[e-mail]" eme.js:72:8
evme drawBackground drawn blob:app://collection.gaiamobile.org/68ac5293-3eb7-40b0-8ea2-540d5854e7f7 eme.js:72:8
evme rendering icon for Music eme.js:72:8
E.Me onload status:  503 Service Unavailable: Back-end server is at capacity api.js:82:10
E.Me onload httpRequest:  XMLHttpRequest { onreadystatechange: null, readyState: 4, timeout: 0, withCredentials: false, upload: XMLHttpRequestUpload, responseURL: "https://appsearch.services.mozilla.…", status: 503, statusText: "Service Unavailable: Back-end serve…", responseType: "", response: "" } api.js:83:10
Collection onOffline, err:  SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data
Stack trace:
onload@app://collection.gaiamobile.org/shared/js/everythingme/api.js:80:22
 view_apps.js:120:6

Probably we could do better with the error handling on the client-side here, as the action we suggest to the user wont actually help. But the error is server-side.

Comment 11

[Sam Foster [:sfoster] (he/him)]

Can either of you suggest the best person or next steps for this? We're getting a sporadic but frequent "503 Service Unavailable: Back-end server is at capacity." error from E.me

Comment 12

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Ran can you take a look please? 

+jacob as he's the epm.

Comment 13

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Oh wait.  isn't the collection on our end?  I think kgrandon might need to look at this instead of Ran...

Comment 14

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

[ Collection graphics I mean ]

Comment 15

[Kevin Grandon :kgrandon]

Nope, seems this is something with the snapshot server because the url the error is coming from is: https://appsearch.services.mozilla

Comment 16

[Sam Foster [:sfoster] (he/him)]

What do we need to do to get these kind of errors setting off klaxons in someone's inbox? Something in the client code, or external monitoring of the server?

Comment 17

[Ran Ben Aharon [:ranbena] EverythingMe]

I'm looking into our server logs. Will let you know soon.

Comment 18

[Ran Ben Aharon [:ranbena] EverythingMe]

Nothing in logs from past day.
No errors in server nor load balancer.

Perhaps check the proxy server https://appsearch.services.mozilla

Comment 19

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Justin, can you take a look at the server please : appsearch.services.mozilla ?

Comment 20

[Justin Lazaro [:jlaz] (use needinfo)]

It looks like the IP on e.me's servers had changed.  nginx seems to cache the resolved names by default, but I've changed the nginx configuration to have the e.me hostname resolved each time.  Spinning up a new production stack shortly to reflect the new configuration changes.

Comment 21

[Justin Lazaro [:jlaz] (use needinfo)]

The issue should have been resolved now, but can you verify? Much appreciated!

Comment 22

[Sam Foster [:sfoster] (he/him)]

Some quick testing on my end looks good. QA are flagged to verify. Thanks!

Comment 23

[KTucker [:KTucker][Inactive 3/4/2016]]

This issue has been verified as fixed on the Flame 3.0, Flame 2.2 and Flame 2.1

All smart collection icons appear as expected without issue. 

Device: Flame 3.0
BuildID: 20150528010203(Full Flash)(KK)(319mb)
Gaia: 05380df3158fa39e1dde1687c0bf11a71f8c6868
Gecko: baa9c64fea6f
Gonk: 040bb1e9ac8a5b6dd756fdd696aa37a8868b5c67
Version: 41.0a1 (3.0) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:41.0) Gecko/41.0 Firefox/41.0

Device: Flame 2.2
BuildID: 20150528002504(Full Flash)(KK)(319mb)
Gaia: 999bc627063d16c20f703e702f31a5cf0da8b4a6
Gecko: 351101ec82ba
Gonk: bd9cb3af2a0354577a6903917bc826489050b40d
Version: 37.0 (2.2) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:37.0) Gecko/37.0 Firefox/37.0

Device: Flame 2.1
BuildID: 20150528001205 (Full Flash)(KK)(319mb)
Gaia: 2304a1f6327c2ccf35d6995ee16f2231ed1f22a3
Gecko: 894528758073
Gonk: bd9cb3af2a0354577a6903917bc826489050b40d
Version: 34.0 (2.1) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0