867859 - Marketplace API returns different number of total apps in JSON meta data at different times

Status: RESOLVED WORKSFORME

(Marketplace Graveyard :: API, defect, P3)

Description

[Parul Mathur]

Attachment: screenshot taken when total number of apps was 666

Steps to reproduce:
1. Search the Marketplace using an API call.
https://marketplace.firefox.com/api/v1/apps/search/

Expected Results:
The total number of apps in the JSON meta data should slowly increase with time.

Actual Result:
The total number of apps in the JSON meta data differ from time to time - sometimes more, sometimes less.
May 01, 2013 6:34 PM PST:  666
(see screenshot attached)

May 01, 2013 7:11 PM PST:  734
Request-Header: GET /api/v1/apps/search/ HTTP/1.1
Response:
"meta": {

    "limit": 20,
    "next": "/api/v1/apps/search/?limit=20&offset=20",
    "offset": 0,
    "previous": null,
    "total_count": 734

}
Response-Headers: HTTP/1.1 200 OK
X-API-Version	1
Server	gunicorn/0.15.0
Vary	X-API-Filter, X-Requested-With, Accept-Language, Cookie, X-Mobile, User-Agent
X-Backend-Server	web5.addons.phx1.mozilla.com
Content-Type	application/json; charset=utf-8
Access-Control-Expose-Headers	X-API-Filter, X-API-Status, X-API-Version
Strict-Transport-Security	max-age=2592000
Date	Thu, 02 May 2013 02:13:20 GMT
Transfer-Encoding	chunked
Access-Control-Allow-Origin	*
Via	Moz-pp-zlb08
Connection	keep-alive
X-API-Filter	carrier=&lang=en-US&region=us
Set-Cookie	lang="en-US\054"; Path=/
Set-Cookie	region=us; Path=/
Access-Control-Allow-Methods	GET, OPTIONS
Access-Control-Allow-Headers	X-HTTP-Method-Override, Content-Type

Comment 1

[Parul Mathur]

Attachment: screenshot taken when total number of apps was 734

Comment 2

[Chuck Harmston [:chuck]]

I'm interested in the value of the X-API-Filter header in the response containing 666 results.

When using the API, we attempt to detect and filter results by region and language. Additionally, it is possible to manually restrict responses by device type and carrier. The API reports which of these filters are implemented via this header.

Documentation on this here: https://firefox-marketplace-api.readthedocs.org/en/latest/topics/overview.html#modifying-results

As your detailed description (thanks!) notes, the 734-result response returned:

X-API-Filter: carrier=&lang=en-US&region=us

My guess is that the X-API-Filter header will report some additional filter in the 666-result response that was not present in the 734-result response. Are you able to reproduce to determine that?

Comment 3

[Parul Mathur]

Attachment: Script to ping Marketplace API every 5 mins and parse the JSON response + read HTTP Response Headers

Unable to reproduce this bug after 1 week of trying.

Steps to reproduce:
1. Write a script to make a GET request to Marketplace API
https://marketplace.firefox.com/api/v1/apps/search/
and parse the JSON response to read the total number of apps in the Marketplace.
Compare with the known the total number of apps and alert if the number drops at any time.
2. Run the script as a cron job every 5 minutes over the span of a week.
(see script attached)

Actual Results:
The total number of apps increased from 744 on May 05, 2013 to 774 on May 12, 2013.
26 different backend web servers responded at random times to the script.
HTTP Response Headers
X-Backend-Server: web[1-26].addons.phx1.mozilla.com

Comment 4

[Matt Basta [:basta]]

This looks like ES simply being out of sync.