818969 - App status page should indicate expected review waiting time

Status: RESOLVED FIXED

(Marketplace Graveyard :: Developer Pages, enhancement, P5)

Description

[Ian Bicking (:ianbicking)]

My understanding is that the expected review time for apps is about a week.

On submission you just see that your app is "awaiting review" with very little extra information.  It also links to an MDN page: https://developer.mozilla.org/en-US/docs/Apps/Marketplace_review_criteria – as is sensible, this talks about criteria, but not process.

With all this there's no way for a developer to know what to expect.  Same-day review?  A couple days?  A week?  Two weeks?

I think the app status page should include:

- Date the app was submitted for review (or re-entered review-needed)
- What turnaround we are dedicated to providing
- Bonus points: what turnaround we expect to actually provide.  E.g., "you are 15th in line for review." – with or without translating this to an expected time, I think this will greatly improve on people's perception, and make it feel predictable.

If there are multiple queues (e.g., a fast queue for app updates, and another queue for new apps) then they'd need to be formally separated.  It's not so important that you indicate what line you are talking about when you say "15th in line" – only that the position goes down in a somewhat predictable manner.  (Though it might matter if different developers compare their positions.)

Comment 1

[Lisa Brewster [:adora]]

I think these are great ideas.  Adding David Bialer to weigh in from a product management perspective.

Comment 2

[Rob Hudson [:robhudson]]

I've wondered (to myself, up to now) if it'd be interested to record how long an app was in the queue at the time it is approved. If we did so, not only could we have cool historical review-time charts, but we could also calculate the rolling average and use that as an estimate for review time. We could probably also take into account number of devices the app claims to support, if it's new or an update, etc.

Comment 3

[Wil Clouser [:clouserw]]

Queues aren't processed necessarily in order but we can show a rough number.  See bug 546339 for a screenshot of how it is done on add-ons and some discussion.  We can reproduce the same thing in the marketplace.

Edit: More people have replied since I loaded this page in my tab.  If you're looking for something more complicated than what we have here ^ please specify.

Comment 4

[David Bialer [:dbialer]]

I love the idea.
I think we should provide: 
1. Date app entered in queue
2. Queue length and number in queue
3. Expected waiting time (based on 30 day rolling average) and position in queue.

All of this makes our process very transparent - which I see as a positive and will keep us mindful of the developer experience.

Also would be great for our metrics to chart over time.

Let's look at this for Q1.

Comment 5

[Christopher Van Wiemeersch [:cvan]]

For measure, we currently show the queue position on AMO: http://f.cl.ly/items/0D201N283J2P3f0b3u36/Screen%20Shot%202012-12-06%20at%202.58.55%20PM.png

Comment 6

[Andrew Williamson [:eviljeff]]

(In reply to Rob Hudson [:robhudson] from comment #2)
> I've wondered (to myself, up to now) if it'd be interested to record how
> long an app was in the queue at the time it is approved. If we did so, not
> only could we have cool historical review-time charts, but we could also
> calculate the rolling average and use that as an estimate for review time.
> We could probably also take into account number of devices the app claims to
> support, if it's new or an update, etc.

It would be really cool to have this - I've wanted it for ages on AMO too.

Comment 7

[Ian Bicking (:ianbicking)]

Note that while a fancy feature giving position in the queue would be great, simply documenting the average and maximum promised times for review would do a lot to set developer expectations, and avoid the perception of stalled review processes or unexpected delays.

Comment 8

[David Bialer [:dbialer]]

Seems important as I see some complaints but also questions on wait time in Marketplace Feedback.  Also would be good to monitor for ourselves - so that we can set a review time goal.  Is this a S,M,L,XL amount of work?

Comment 9

[Andrew Williamson [:eviljeff]]

(In reply to Ian Bicking (:ianb) from comment #7)
> Note that while a fancy feature giving position in the queue would be great,

this is actually pretty easy - it already exists on AMO.

> simply documenting the average and maximum promised times for review would
> do a lot to set developer expectations, and avoid the perception of stalled
> review processes or unexpected delays.

This is the hard bit - we don't currently have any stats around review times, beyond what we can manually hack together with some sql queries on the db.  Everyone seems to be in agreement its a 'good thing' to have though. 

As for exactly how much work... ? cvan?

Comment 10

[Christopher Van Wiemeersch [:cvan]]

(In reply to Andrew Williamson [:eviljeff] from comment #9)
> (In reply to Ian Bicking (:ianb) from comment #7)
> > Note that while a fancy feature giving position in the queue would be great,
> 
> this is actually pretty easy - it already exists on AMO.
> 
> > simply documenting the average and maximum promised times for review would
> > do a lot to set developer expectations, and avoid the perception of stalled
> > review processes or unexpected delays.
> 
> This is the hard bit - we don't currently have any stats around review
> times, beyond what we can manually hack together with some sql queries on
> the db.  Everyone seems to be in agreement its a 'good thing' to have
> though. 
> 
> As for exactly how much work... ? cvan?

I have the queue position. I can multiply it by how many reviews are ahead of it by some multiplier. Roughly how long would each review take as an estimate?

Comment 11

[David Bialer [:dbialer]]

I think this is a good way to start.

Is it possible to start keeping track of historical data for an app - (ie. time/date of submission, time/date of first review, starting position in queue).  We could then start calculating something like "average time in queue for apps reviewed in previous 30 days".  Or we could do something more sophisticated like keep track of the starting position when submitted to make a prediction based on historical data.  In the future when we collect enough data for a couple of months I mean.

Comment 12

[Andrew Williamson [:eviljeff]]

(In reply to Chris Van Wiemeersch [:cvan] from comment #10)
> I have the queue position. I can multiply it by how many reviews are ahead
> of it by some multiplier. Roughly how long would each review take as an
> estimate?

I wouldn't be able to give that estimate really - and it will change a lot over the next few weeks/months.  We need the historical data tracked like David mentions so we can give a realistic estimate that changes as Marketplace matures.

Comment 13

[Rob Hudson [:robhudson]]

Bram also has some mockups of reviewer tool dashboards that shows some historical review time data that's a nice to have.

Comment 14

[Bram Pitoyo [:bram]]

Attachment: reviewer tools - queue graph

(In reply to Rob Hudson [:robhudson] from comment #13)
> Bram also has some mockups of reviewer tool dashboards that shows some
> historical review time data that's a nice to have.

Rob was referring to the attached graph, which shows the amount of red, yellow and green reviews overtime.

Showing a variant of this graph to developers might give them a better idea of three things:

* What’s the Marketplace’s average processing time?
* Based on time submitted, where does their app belong in the chart (red/yellow/green)?
* Based on the two above, what’s the expected wait time for their particular app?


But on the other hand, Andrew Williamson wrote something good in the Reviewer Tools unbucket email thread:
> App Review isn't like applying for a home loan
> where lots of people do things and your
> application has a 'progress' through different
> departments. Its generally just waiting in a
> queue.


With this process in mind, then, I think that the problem to solve is not so much “How do we accurately display time left in the queue”, but rather, “How do we make the inevitable long queue more bearable and informative?”

Anecdote: I was in a hospital’s emergency section last Monday getting my injured arm treated. What they did was ask me to fill out a bunch of form and then submit it to the receptionist. Upon submission, I was asked to sit down and wait for an unspecified amount of time until a doctor came and got me.

The same thing happened as I moved between people in the same hospital (general practitioner, x-ray mechanics, etc.), I was simply left alone and asked to wait without knowing whether my case was being worked at all.

In hindsight, I asked:
* Would I have liked to see my position in the queue? Yes, but only if I know how long the estimated wait time is
* Would I have liked to see the estimated wait time? Yes, but if it’s too long, it’s only going to make me more frustrated
* Would I have liked to see the hospital’s history of average time taken to treat a case? Probably not, because every case is different

So I thought of a few things I wished I had:
* Position in the queue might be nice to have, but estimated wait time is more important
* Only show me wait time when it’s close by, so I don’t get too frustrated
* Give me a feedback not only when my forms have been handed, but also when somebody is looking at it, so I know that I am getting attention
* Give me a sense of how busy the hospital is at the moment (how many staff is working on how many cases), so I get a sense that you’re doing something behind the ER door

This is how I would translate my experience to the dev dashboard:
* Emphasize position in the queue when it’s close by. When it’s far away, deemphasize it
* Give developers feedback when somebody is reviewing their app (for example, send an email saying “A contributor is looking at your app right now”)
* Give developers a sense of how many reviewers are online at the moment, if the number is positive enough to inspire confidence. If there are too few reviewers who are logged in, show developers the number of registered reviewers instead.
* Estimated wait time is hard to compute and probably won’t be very helpful given time variance between apps. Let’s not do it if it’s too hard of a technical challenge. If we do want to do it, though, let’s compute time on a per-category basis, so the figure is more accurate.

Comment 15

[Ian Bicking (:ianbicking)]

"I was simply left alone and asked to wait without knowing whether my case was being worked at all."

I think this is the biggest problem right now: people don't know if they are really moving through the queue.  Simply giving a queue position would help in this case, because then people can see themselves move up the queue and can have position 1 in sight.  It gives a person the confidence that their app will really be reviewed.

“How do we make the inevitable long queue more bearable and informative?”

Is there any reason a long queue is inevitable?  If everyone gets through the queue eventually, there's no efficiency to a long queue, except as a holding spot for when there are fluctuations in submissions (maybe people submit a lot on Friday) and reviews (reviews happen less on the weekends).  If the queue isn't clearing out regularly it would seem like there's some kind of problem.

Comment 16

[Andrew Williamson [:eviljeff]]

(In reply to Ian Bicking (:ianb) from comment #15)
> “How do we make the inevitable long queue more bearable and informative?”
> 
> Is there any reason a long queue is inevitable?  If everyone gets through
> the queue eventually, there's no efficiency to a long queue, except as a
> holding spot for when there are fluctuations in submissions (maybe people
> submit a lot on Friday) and reviews (reviews happen less on the weekends). 
> If the queue isn't clearing out regularly it would seem like there's some
> kind of problem.

The perception of what is a 'long' queue is entirely relative and dependant on the developer's own preconceptions.  If you're expecting a less than a hour then one day is a long queue. 

We do a problem right now with queues being longer than we want (we're working on it).  Even after the current problem is resolved I'm absolutely certain the queue times will push out again at some point and when it does, we have the same issue as to how to communicate that to the developers.  Planning for the queues to be near zero as the norm is unrealistic.

Comment 17

[Andrew Williamson [:eviljeff]]

(In reply to Bram Pitoyo [:bram] from comment #14)
> In hindsight, I asked:
> * Would I have liked to see my position in the queue? Yes, but only if I
> know how long the estimated wait time is
> * Would I have liked to see the estimated wait time? Yes, but if it’s too
> long, it’s only going to make me more frustrated

I would argue its better to be frustrated and informed up front, than hopelessly optimistic and frustrated later.

> This is how I would translate my experience to the dev dashboard:
> * Emphasize position in the queue when it’s close by. When it’s far away,
> deemphasize it
> * Give developers feedback when somebody is reviewing their app (for
> example, send an email saying “A contributor is looking at your app right
> now”)

I like in theory - in practise though we don't actually know when someone is reviewing an app.  We know when the page is opened - but the page could easily be closed again without a review and my question as a developer would then be 'why?'.  We should avoid generating those kind of progress questions as they cost reviewer time.

> * Give developers a sense of how many reviewers are online at the moment, if
> the number is positive enough to inspire confidence. If there are too few
> reviewers who are logged in, show developers the number of registered
> reviewers instead.

I can see how this may give confidence, though I think its probably false confidence. 

> * Estimated wait time is hard to compute and probably won’t be very helpful
> given time variance between apps. Let’s not do it if it’s too hard of a
> technical challenge. If we do want to do it, though, let’s compute time on a
> per-category basis, so the figure is more accurate.

Despite its difficulty, this is probably the only thing the developer actually wants to know.  So we should do it.

Comment 18

[David Bialer [:dbialer]]

I suggest we don't overthink this or come up with an ideal solution based on information we don't have.

I think we should just start with position in queue, and estimated number of days until review based on (position in queue/average reviews per day).  It will be rough, but a good place to start until there is historical data.  Let's be just as honest as we can based on the information we have.

We can start with an estimate of reviews per day based on average number of reviews per week, and based on a 7 day work week.  This can be a wild guess.  We can improve this in Q3 or Q4 when we have more accurate historical numbers.

We can start collecting data to improve:  Which queue, starting position in queue, date added to queue, and date review started.  That would be our next step to say, if you start at position 78, the average wait for the previous 30 days 12 days.

Comment 19

[Lisa Brewster [:adora]]

Great conversation happening here, and I agree right now that we shouldn't overthink this.  I just want to throw out another data point we might be able to use that's good enough and way easier to implement...what if we just average the wait time for the last 5-10 apps that were reviewed, rounded to the nearest week?  

For now, I'd like the expectation to be set at a granularity of "about 2 weeks" instead of days, so that way developers don't start emailing when it's one day after the estimate.  =]  Our processing rate is highly variable right now.

Comment 20

[Bram Pitoyo [:bram]]

I like Lisa’s idea on granularity. “About 2 weeks” is good, and when the time is more precise, we can show something like “3–5 workdays”. I use workday rather than day as a unit, because it’ll give developers a better idea that we don’t review on the weekends.

Showing position on the line is good.

David’s idea about collecting data is good. I prefer that we collect data first before displaying anything that’s more detailed or accurate.

As far as placement goes, we can place this information on the Manage My Submissions and Manage Status pages.

Comment 21

[Bram Pitoyo [:bram]]

Attachment: DevHub - queue position - 01

This is a mockup of how position in the queue can be incorporated into the “My Submissions” and “Manage Status” pages we have today. It involves adding another field or a new paragraph dedicated to queue position.

Comment 22

[Bram Pitoyo [:bram]]

Attachment: DevHub - queue position - 02

In the future, it’d be great to organize the “My Submissions” page by:
1) Giving each piece of information more space
2) Moving the “actions” navigation to the side of the app name

We’re already using this table-based interface in the “edit listing” page, and it’s pretty extensible. For example, we can add the estimated waiting time whenever we’re ready to do so, without clogging the layout.

Comment 23

[David Bialer [:dbialer]]

This looks good!

Comment 24

[Christopher Van Wiemeersch [:cvan]]

https://github.com/cvan/zamboni/compare/position

Needs tests

Comment 25

[Andrew Williamson [:eviljeff]]

Btw, bug 866760 is connected with the original request, to show estimated waiting time, as it would allow an informed guestimate of waiting time based on past performance.

Comment 26

[Christopher Van Wiemeersch [:cvan]]

https://github.com/mozilla/zamboni/commit/0c9e69e
https://github.com/mozilla/zamboni/commit/e9acaf3