[SmartCollections] empty icons for Movies & Utilities collections

VERIFIED FIXED in 2.2 S13 (29may)

Status

defect
VERIFIED FIXED
4 years ago
4 years ago

People

(Reporter: hcheng, Assigned: sfoster)

Tracking

unspecified
2.2 S13 (29may)
ARM
Gonk (Firefox OS)

Firefox Tracking Flags

(blocking-b2g:2.2+, b2g-v2.1 verified, b2g-v2.2 verified, b2g-master verified)

Details

(Whiteboard: [systemsfe])

Attachments

(2 attachments)

As attachment shown, some collections do not have a correct icon.


At v2.2 test env, they are Movies & Utilities.
At master test env, Movies & Photography are incorrect.


* Test Env
**2.2 (with local gecko build)
Build ID               20150513121019
Gaia Revision          e048df68f6f4853b5826a8816e143d95258149de
Gaia Date              2015-05-12 19:10:26
Gecko Revision         n/a
Gecko Version          37.0
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150513.041549
Firmware Date          Wed May 13 04:15:58 EDT 2015
Bootloader             L1TC000118D0

**master
Build ID               20150512160203
Gaia Revision          3654ec1d7ce1e0a56a34d5c3b06f6a9b33ff79ad
Gaia Date              2015-05-12 05:13:33
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/42db79f3cd6b
Gecko Version          41.0a1
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150512.192015
Firmware Date          Tue May 12 19:20:26 EDT 2015
Bootloader             L1TC000118D0
blocking-b2g: --- → 2.2?
Keywords: regression
Whiteboard: [systemsfe]
Seems like a server issue.
Is this 100% reproducible? Lets see if we can get a regression window. If its a server issue we probably can't find a window.
blocking-b2g: 2.2? → 2.2+
Flags: needinfo?(kgrandon)
NI for Kevin if he knows about any client side changes.
No changes on the client side as far as I know. Can we do branch checks?
status-b2g-v2.1: --- → ?
status-b2g-v2.2: --- → ?
Flags: needinfo?(kgrandon)
I think this issue is related to the overall network issue of connecting to smart collections. There is a related meta bug 1163040. While testing I encountered many times of the server telling me I've no internet connection when in fact I am connected to wifi, and this occurred to me regardless of which branch I'm in.

For this bug, it is reproducible on Flame 3.0, 2.2, and 2.1. Empty icons are displayed for certain smart collections. I've observed it happened for Photography on 2.2 and 3.0, and Movies & Photography for 2.1.

Device: Flame 3.0
BuildID: 20150514010203
Gaia: 338f66e6a96491d2f5854b188c6b141ceb690d97
Gecko: 1fab94ad196c
Gonk: 040bb1e9ac8a5b6dd756fdd696aa37a8868b5c67
Version: 41.0a1 (3.0) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:41.0) Gecko/41.0 Firefox/41.0

Device: Flame 2.2
BuildID: 20150514002501
Gaia: aac58a063e3e6acae6ba77fe4cec224fb69450bc
Gecko: 47f1ced9f1d6
Gonk: ab265fb203390c70b8f2a054f38cf4b2f2dad70a
Version: 37.0 (2.2) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:37.0) Gecko/37.0 Firefox/37.0

Device: Flame 2.1
BuildID: 20150514001201
Gaia: c80865cb0bf73f1b97defbc646083b404feb3ac4
Gecko: 23eb9df75991
Gonk: ab265fb203390c70b8f2a054f38cf4b2f2dad70a
Version: 34.0 (2.1) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0

Removing regression keyword since we're not sure if it's a regression.
QA Whiteboard: [QAnalyst-Triage?]
Flags: needinfo?(ktucker)
QA Whiteboard: [QAnalyst-Triage?] → [QAnalyst-Triage+]
Flags: needinfo?(ktucker)
Depends on: 1163040
https://github.com/mozilla-b2g/gaia/commit/9d19ae7dd104fd253a8b47f06adacd8cdc0d2281

There's a gaia change in the last 17 days that might have affected it.  I'll have to play around with it.  Rather than reverting, perhaps playing with the numbers might be a good idea.

For the full list, https://github.com/mozilla-b2g/gaia/commits/master/apps/collection
I know I tested Kevin's patch on 4/7 so my guess is autonome's patch.  I think it caused some breaks on other spots.

Needinfo on myself to do this as an action item.
Flags: needinfo?(nhirata.bugzilla)
Assigning to Naoki while he investigates.
Assignee: nobody → nhirata.bugzilla
Posted file logcat_snippit.txt
There's a lot of css property errors, where it falls back to initial.  I'm not sure what the initial values are.  Still need to dig in.  Looking at reverting to the original code for each of these lines to see if that fixes the issue as a test since the variables touch more than just collections.
So reverting the lines seemed to help some  ( 
https://github.com/nhirata/mozilla_b2g_gaia/compare/master...nhirata:collection_check )
; whenever it can't draw, I've noticed this in the logcat:
W/Smart Collections(17868): [JavaScript Warning: "An IndexedDB transaction that was not yet complete has been aborted due to page navigation." {file: "app://collection.gaiamobile.org/gaia_build_defer_view.js" line: 301}]

I think we might need a check for that.

The other thing I noticed was : E/(Preallocated app)(17924): [JavaScript Error: "Error: You cannot use the AddonManager in child processes!" {file: "resource://gre/modules/AddonManager.jsm" line: 20}]
which seems to be an preallocated app process issue.  I filed bug 1166545 as it seems to be causing load "lag" and at the same time may be a separate issue to deal with.


Chris, could you take a look please?
Flags: needinfo?(nhirata.bugzilla) → needinfo?(chrislord.net)
(Assignee)

Comment 9

4 years ago
I'll look into this.
Assignee: nhirata.bugzilla → sfoster
Flags: needinfo?(chrislord.net)
(Assignee)

Comment 10

4 years ago
I did a little logging and poking around. First, all the E.me api requests in apps/collection/js/view_apps.js direct promise rejections to onOffline - so all errors are reported as no-connection errors, which is why we see that message. 

What we are seeing is actually "503 Service Unavailable: Back-end server is at capacity." Here's some logging when reproducing the issue with the Music collection

Collection queueRequest, navigator.onLine:  true view_apps.js:109:6
evme init: noop eme.js:72:8
evme API request: "https://appsearch.services.mozilla.com/partners/1.0/Apps/search/?limit=24&categoryId=142&iconFormat=20&apiKey=<snip>&deviceId=fxos-app://collection.gaiamobile.org/aabad456-e485-4c73-80a6-0f30cb1d16f5&ctx={"lc":"en-US","tz":"-7","v":"2.2.0.0-prerelease","dn":"Flame","cr":"AT&T","ct":"wifi","mcc":"310","mnc":"410","sr":"480x853.5"}&" eme.js:72:8
evme getMozBackground success "https://fxos.cdn.mozilla.net/collection/background/142@1.5x.jpg" eme.js:72:8
evme drawBackground drawn blob:app://collection.gaiamobile.org/68ac5293-3eb7-40b0-8ea2-540d5854e7f7 eme.js:72:8
evme rendering icon for Music eme.js:72:8
E.Me onload status:  503 Service Unavailable: Back-end server is at capacity api.js:82:10
E.Me onload httpRequest:  XMLHttpRequest { onreadystatechange: null, readyState: 4, timeout: 0, withCredentials: false, upload: XMLHttpRequestUpload, responseURL: "https://appsearch.services.mozilla.…", status: 503, statusText: "Service Unavailable: Back-end serve…", responseType: "", response: "" } api.js:83:10
Collection onOffline, err:  SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data
Stack trace:
onload@app://collection.gaiamobile.org/shared/js/everythingme/api.js:80:22
 view_apps.js:120:6

Probably we could do better with the error handling on the client-side here, as the action we suggest to the user wont actually help. But the error is server-side.
(Assignee)

Comment 11

4 years ago
Can either of you suggest the best person or next steps for this? We're getting a sporadic but frequent "503 Service Unavailable: Back-end server is at capacity." error from E.me
Flags: needinfo?(pdolanjski)
Flags: needinfo?(nhirata.bugzilla)
Ran can you take a look please? 

+jacob as he's the epm.
Flags: needinfo?(ran)
Flags: needinfo?(nhirata.bugzilla)
Flags: needinfo?(jshen)
Oh wait.  isn't the collection on our end?  I think kgrandon might need to look at this instead of Ran...
Flags: needinfo?(kgrandon)
[ Collection graphics I mean ]
Nope, seems this is something with the snapshot server because the url the error is coming from is: https://appsearch.services.mozilla
Flags: needinfo?(kgrandon)
(Assignee)

Comment 16

4 years ago
What do we need to do to get these kind of errors setting off klaxons in someone's inbox? Something in the client code, or external monitoring of the server?
I'm looking into our server logs. Will let you know soon.
Target Milestone: --- → 2.2 S13 (29may)
Nothing in logs from past day.
No errors in server nor load balancer.

Perhaps check the proxy server https://appsearch.services.mozilla
Flags: needinfo?(ran)
Justin, can you take a look at the server please : appsearch.services.mozilla ?
Flags: needinfo?(jlaz)
It looks like the IP on e.me's servers had changed.  nginx seems to cache the resolved names by default, but I've changed the nginx configuration to have the e.me hostname resolved each time.  Spinning up a new production stack shortly to reflect the new configuration changes.
Flags: needinfo?(jlaz)
The issue should have been resolved now, but can you verify? Much appreciated!
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Flags: needinfo?(pdolanjski)
Flags: needinfo?(jshen)
Keywords: qawanted, verifyme
Resolution: --- → FIXED
(Assignee)

Comment 22

4 years ago
Some quick testing on my end looks good. QA are flagged to verify. Thanks!
QA Contact: ktucker
This issue has been verified as fixed on the Flame 3.0, Flame 2.2 and Flame 2.1

All smart collection icons appear as expected without issue. 

Device: Flame 3.0
BuildID: 20150528010203(Full Flash)(KK)(319mb)
Gaia: 05380df3158fa39e1dde1687c0bf11a71f8c6868
Gecko: baa9c64fea6f
Gonk: 040bb1e9ac8a5b6dd756fdd696aa37a8868b5c67
Version: 41.0a1 (3.0) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:41.0) Gecko/41.0 Firefox/41.0

Device: Flame 2.2
BuildID: 20150528002504(Full Flash)(KK)(319mb)
Gaia: 999bc627063d16c20f703e702f31a5cf0da8b4a6
Gecko: 351101ec82ba
Gonk: bd9cb3af2a0354577a6903917bc826489050b40d
Version: 37.0 (2.2) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:37.0) Gecko/37.0 Firefox/37.0

Device: Flame 2.1
BuildID: 20150528001205 (Full Flash)(KK)(319mb)
Gaia: 2304a1f6327c2ccf35d6995ee16f2231ed1f22a3
Gecko: 894528758073
Gonk: bd9cb3af2a0354577a6903917bc826489050b40d
Version: 34.0 (2.1) 
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0
Status: RESOLVED → VERIFIED
Keywords: qawanted, verifyme
QA Contact: ktucker
You need to log in before you can comment on or make changes to this bug.