Closed Bug 895675 Opened 12 years ago Closed 11 years ago

Input API to insert data

Categories

(Input Graveyard :: Frontend, defect, P1)

Product:
Input Graveyard
Component:
Frontend
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cww, Assigned: willkg)

References

Details

(Whiteboard: u=dev c=api p=3 s=input.2013q3)

Input should have an API for collecting data to support work in Firefox OS: It should accept the following data: Platform (likely "FirefoxOS") Device Manufacturer Version (likely Firefox OS version or build ID) Happy or Sad (default will be sad) Content (open-ended UTF-8 compliant feedback blurb) Email address (optional) Locale (default en-US) User agent (default blank) Browser (default Firefox) Will can spec out the exact syntax and language. It should return some kind of "success" response as well as reasonable error codes. Clients will be responsible for passing in values and displaying sufficiently friendly UI (including thank you pages on success) as well as behaving sensibly when errors are hit (such as retrying or alerting the user).
Flags: needinfo?(willkg)
Pulling some of the description over from bug #799553: """ Requirements: 1. We need to version the API so that we can make changes, but the old API version will continue to work. 2. The API will be for posting data only at this point--no need for getting data, authenticating, etc. 3. We need to think about spam, but it's probably the case that all we can do is throttle incoming data. 4. Probably easiest to do a REST-based API using JSON. """ Having said that, I think it's better to use django-rest-framework. I'll figure that part out later, though. The data requirements are already handled in the response table, so we don't need to spin off new bugs for additional data capture requirements. This bug will additionally cover adding tests and documentation for others to implement clients.
Flags: needinfo?(willkg)
Grabbing it, making it a P1. I'll work through it next week.
Assignee: nobody → willkg
Priority: -- → P1
Hardware: x86 → All
Working on this now. Seems pretty straight-forward. The one part I'm not sure about is the throttling. Throttling is done by ip address. What's a good ballpark number? 20/hour?
Also, as a note, we want to keep track of the number of throttled responses so we can tweak the settings if we see issues. That should be part of the work done here.
Blargh, replied to the wrong thing. I think we'd probably want a way to whitelist domains. The only IP addresses where we are likely to have lots of responses are Mozilla ones. The rest we can throttle pretty aggressively.
I didn't think of Mozilla domains. Beyond that, I'm not sure how to do this because I doubt the list of ip addresses for Mozilla things (remote employees, Mozilla offices, other places that do tests against Input, etc) is remotely static. We do throttling on SUMO--I'll see if we've seen it affect SUMO at all.
Will the API only be called directly from the client/user device? If there are plans of calling this API server to server, then things change a little regarding throttling.
Talked to Ricky on IRC and he brought up CORS. I wasn't going to worry about that until after I rolled out the API and we saw who bumped into CORS issues. It's not hard to work around: http://django-rest-framework.org/topics/ajax-csrf-cors.html#cors
Talked to Ricky on IRC some more. We want to track the number of times the throttling triggers are hit. We can adjust the throttling numbers accordingly. Additionally, we'll keep an eye on incoming feedback and as we bump into spam-related issues, we'll handle those individually. Possibilities include tracking more information about individual responses so we can see the spam trends in aggregate, blacklisting domains/ip addresses (which is a losing proposition), spam detection (maybe run responses through spamassassin or something like that?), ... For now, both of these are outside the scope of this bug.
Cheng: I skimmed through the fields you suggested and I'm concerned about the "browser" field. You have it default to "Firefox". The browser field shows up as the Product facet in the dashboard, so if this is "Firefox" by default, then all input for everything will show up under "Firefox" unless they explicitly set it. Is that ok?
That's where it is now, isn't it? I was trying just to match what we have.
(In reply to Will Kahn-Greene [:willkg] from comment #7) I don't much care about Mozilla remotees hitting a throttle since no one person or small group of people are likely to hit Input that frequently. It's just if we have 100 people dogfooding in an office using the same IP, we're likely to hit a limit.
(In reply to [:Cww] from comment #12) > That's where it is now, isn't it? I was trying just to match what we have. You're the one using the data, so if that's what you want, then that's fine with me. (In reply to [:Cww] from comment #13) > (In reply to Will Kahn-Greene [:willkg] from comment #7) > I don't much care about Mozilla remotees hitting a throttle since no one > person or small group of people are likely to hit Input that frequently. > It's just if we have 100 people dogfooding in an office using the same IP, > we're likely to hit a limit. How about 100/hour?
100/hr is fine. If we suddenly get spam, we'll be more creative in the face of adversity. I don't think I use that field right now (I'm relying on platform = "FirefoxOS") since browser is Firefox, Iceweasel (basically no one) or Unknown (basically Firefox). We can use Browser = API that makes more sense -- which might make that field actually useful.
Just to clarify, I'm planning to roll out this API and then tell the Firefox for Android folks to switch to using it, too. Do you want all users of this API to do Browser = API?
Yeah, let's do that. Since they're currently submitting browser = Firefox and that's not useful for any kind of distinction since Input only handles Firefox feedback. It's currently parsing these from the UA, I thought, so we just have to be sure to change that behavior.
(In reply to [:Cww] from comment #17) > Yeah, let's do that. Since they're currently submitting browser = Firefox > and that's not useful for any kind of distinction since Input only handles > Firefox feedback. I'll hard-code API-submitted feedback to have browser=api. It'll show up weird in the dashboard since the Product facet is based on the browser field, but ... that's fine with me. > It's currently parsing these from the UA, I thought, so we just have to be > sure to change that behavior. I don't know what this means. We don't currently have an API, so I'm not sure what you're referring to here.
How does the product facet work? We have Firefox for Android and Firefox as products but the browser field only has Firefox and Unknown. I'd love for Firefox OS to have its own product facet. My last sentence was just I thought we got browser from the UA string right now and now we're divorcing those two. It might not have made sense.
(In reply to [:Cww] from comment #19) > How does the product facet work? We have Firefox for Android and Firefox as > products but the browser field only has Firefox and Unknown. I'd love for > Firefox OS to have its own product facet. Whoops! I thought we were using the browser field for the Product facet, but we're not. Turns out the values for that is hard-coded and based on whether platform is Android or not. That should be an easy fix. I'll create a separate bug for it. The "user agent" field doesn't have any meaning with the API, so I wasn't going to include it in the API. We use it for the web-based form to infer values for everything so we don't have to pepper the user with questions they don't understand. Clients of the API will know better, so we don't have to infer anything. Sounds like we're all good here.
In a PR: https://github.com/mozilla/fjord/pull/128 It's up for review now. This is mostly to get some code in place so we have something to talk about in regards to implementing clients that use the API. Going forward, after that lands in master and gets pushed to production, I'll talk with Lukas, Margaret and Cheng about it. We'll work out any issues we bump into in new bugs and modify the API accordingly. At some point p, we'll set v1 of the API in stone. Also, this doesn't do much to deal with incoming spam. We'll figure out incoming spam issues as they arise.
Whiteboard: u=dev c=api p=3 s=input.2013q3
HI Will, I am Ivan, the FFOS 1.2 EPM for system platform team. May I know if there is any update about this input API? The bug 851189 depends on this API to proceed with the change on feedback flow. Thank you very much! Ivan Tsay
Flags: needinfo?(willkg)
This bug is waiting on me to land the changes for bug #898417. I finished that work on Tuesday, but needed to wait to push it to production because we just did a Firefox release and thus Input usage spiked. After that lands, I need to tweak the code changes for this one and then I can land this one, too. That's a couple of hours of work--no biggie. I should be done with both bugs this week. Making this block on bug #898417.
Depends on: 898417
Flags: needinfo?(willkg)
Will, Thank you very much for the support.
cc:ing Margaret because I think she'd like to see this. Landed the code in master: https://github.com/mozilla/fjord/commit/ef554c1 https://github.com/mozilla/fjord/commit/13aefed https://github.com/mozilla/fjord/commit/2076472 https://github.com/mozilla/fjord/commit/1c24ea2 I pushed the changes to stage. FxOS and Firefox for Android clients can test out using the API there and let me know if there are bugs and/or requirements-related issues. We can work through them in this bug or spin off new bugs if necessary. When everything is kosher, then we can push the API to prod. Documentation for the API is at: http://fjord.readthedocs.org/en/latest/api.html If there are any questions about that, let me know. I can improve it as we go along, too. I'll leave this bug open until the changes get pushed to production.
Flagging Ivan for needsinfo?. Ivan: Super sorry it took so long to finish this up. Let me know whether it works for your purposes and/or whether there's anything I can do to make it better.
Flags: needinfo?(itsay)
Thanks for cc'ing me, it sounds like we'll have to file a bug to update the feedback page in Firefox for Android. Is there a timeline for when we need to update our client? If the change is minimal, we could probably uplift it to aurora (and maybe beta), but we would obviously prefer to just let it ride the trains (meaning it would reach release in 26 if we landed a patch soon).
Flags: needinfo?(willkg)
Margaret: Everything that works now can continue working for as long as you like--I'm in no rush. Also, if using the API is worse/harder than the existing HTTP POST, let me know. Theoretically, it shouldn't be.
(In reply to Will Kahn-Greene [:willkg] from comment #28) > Margaret: Everything that works now can continue working for as long as you > like--I'm in no rush. Also, if using the API is worse/harder than the > existing HTTP POST, let me know. Theoretically, it shouldn't be. Cool, I'll just file a bug then to use the new API. On quick glance it looks better than what we're currently using :)
Blocks: 799562
(In reply to Will Kahn-Greene [:willkg] from comment #26) > Flagging Ivan for needsinfo?. > > Ivan: Super sorry it took so long to finish this up. Let me know whether it > works for your purposes and/or whether there's anything I can do to make it > better. Excuse me for the late reply. I will check with Arthur to see when he can help on verify it.
Flags: needinfo?(itsay)
I'm not sure why the needsinfo? flag on me wasn't cleared in comment #28. Clearing it now.
Flags: needinfo?(willkg)
Seems like everyone is happy with the API. I need to look into bug #905390 and bug #905372 first, then this can be pushed to production. Hopefully this can all land Thursday this week.
Depends on: 905390, 905372
Landed and pushed to production.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: Input → Input Graveyard
You need to log in before you can comment on or make changes to this bug.