Closed Bug 882864 Opened 11 years ago Closed 8 years ago

Improve estimated waiting time calculations

Categories

(Marketplace Graveyard :: Developer Pages, defect, P4)

defect

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: cvan, Unassigned)

Details

(Whiteboard: [marketplace-transition])

10:00 < eviljeff> cvan: the biggest problem is it doesn't take into account how many apps are remaining in the queue. I'm not sure if the logic should be changed to do something like, for example '300 submisson reviews were completed in 30 days so 10 were done a day so if you are at position 80 it will take 8 days'
10:00 < eviljeff> cvan: unfortunately that may end up with the average time not changing
10:00 < eviljeff> the estimated time not changing, I mean
10:02 < eviljeff> the current estimate assumes we are working to keep the queue at a certain level, so the average time would be a good estimate of time left - whereas at the moment we are working at capacity so the number of apps ahead of you is a better estimate

Let's figure out what it will take to get a more accurate estimated waiting time for developers. Here's the code in question:

https://github.com/mozilla/zamboni/blob/master/apps/editors/helpers.py#L358
https://github.com/mozilla/zamboni/blob/master/apps/editors/helpers.py#L301
Lisa, when I mentioned this issue to you shortly after discussing it on irc (above) you had some other ideas iirc.  I remember them being sensible but not the content!
Flags: needinfo?(adora)
Priority: -- → P4
Lisa, Andrew, can we get some specifics here of how this should work?
(In reply to Christopher Van Wiemeersch [:cvan] from comment #2)
> Lisa, Andrew, can we get some specifics here of how this should work?

Lisa had some ideas, but is ignoring her need-info! :p
I thought I had a good idea, but based on the feedback we're getting from developers, I don't think it would have been the magic bullet I thought it would be.

The low hanging fruit right now is to have separate calculations for priv and non-priv apps, since priv apps go through an additional code review and will take longer to complete.

I just submitted an app on prod, and the estimation is queue position 45 out of 45, and 20 business days.  But since it's a non-priv app and we have plenty of reviewers right now, it'll be done in less than 3 days.  I also have no idea where 45 is coming from, since it's the 29th app in the main queue, with 7 re-reviews, 13 updates, and 13 escalations...but that discrepancy isn't a major concern right now.

What if we averaged the wait time for the oldest 5 apps in the main queue, grouped by priv and non-priv and excluding outliers (like apps waiting more than a week longer, meaning there's probably a blocker of some sort)?  This is what I do in my head when I want to estimate our wait times, but I'm not sure if there are situations where this math falls apart.
Flags: needinfo?(adora)
I'm going to go out on a limb here and ask Annie if she has someone on her team who could help us plan a better system and hopefully do this once instead of just trying new things.  Annie - is there someone we can contact with the context and details around our app reviews and who could help us determine a good algorithm for predicting wait times?
Flags: needinfo?(aelliott)
Hi Wil -

The metrics team was split up into the data scientists and the BI+DW folks. For something like this, it is right in line with the data scientists' training and capabilities.

Please contact John Jensen, whom I have cc:ed here.
Hi all,

This sounds like a neat problem and we could certainly help. It sounds, however, like a whiteboard session or virtual equivalent is required. Wil, can you ping me via email/IRC and we'll set up a short discussion. I am pretty sure who the right person in our team is to handle this, but would like a quick talk first.
(In reply to John Jensen from comment #7)
> Hi all,
> 
> This sounds like a neat problem and we could certainly help. It sounds,
> however, like a whiteboard session or virtual equivalent is required. Wil,
> can you ping me via email/IRC and we'll set up a short discussion. I am
> pretty sure who the right person in our team is to handle this, but would
> like a quick talk first.

you can include Lisa or myself in the discussion if you need any further details about the problem we're trying to solve or how the review queues are operated.
Flags: needinfo?(aelliott)
Given the wait time is consistently very short, like 2 days, do we still need to come up with a better estimate?
Flags: needinfo?(awilliamson)
(In reply to David Bialer [:dbialer] from comment #9)
> Given the wait time is consistently very short, like 2 days, do we still
> need to come up with a better estimate?

yes.  The problem is most apps are reviewed very quickly, and others take much longer so we end up showing 1 day waiting time.  If the app is privileged then its realistically going to take a few days longer but when we've "promised" 1 day then even 3 days is three times longer.  Some developers do complain after 1 day.
Flags: needinfo?(awilliamson)
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
Whiteboard: [marketplace-transition]
You need to log in before you can comment on or make changes to this bug.