1178475 - /reports/index loading UI is terrible

Status: RESOLVED DUPLICATE of bug 1262252

(Socorro :: Webapp, task)

Description

[Robert Helmer [:rhelmer]]

https://i.imgur.com/my5r2xY.png

Allow me to enumerate the ways:

1) our average process time is 3-5s, making people wait at least 30s makes us seem slow
2) if for some reason we do take over 30s (e.g. severe backlog), then this UI inserts a new priority processing job every 30s which makes things worse

#2 is somewhat unavoidable since we can't introspect RabbitMQ queues, but maybe we can just make this loading screen smarter so it polls more frequently (websockets, long-polling, or more frequent HEAD requests), and only inserts the priority job once.

Comment 1

[Peter Bengtsson [:peterbe]]

Websockets/Event Source/long-polling are all awesomeness but it's a complexity hell. For one we'd need a non-wsgi server that handles the sockets. E.g. tornado or nodejs. 

There's fanout.io which might make things easier. It's proprietary but it basically allows a server to send a message (there's a `pip install fanout`) and you point your socket.js client instance to their server. 

However, I would much prefer that just change 30 seconds down to 5 seconds and write a highly optimized django view that does extremely little other than just checking if it exists. 

We could also make it a proper progress bar that moves from left to right. We can make it move 10% for every 1 second. That means that it gets to 50% after the first cycle and 100% after the second cycle. And if the total time has elapsed *over* 10 seconds we add a message saying. "Sorry. It appears to have taken longer than expected" and we start the loop with 5% for every 1 second instead.

Comment 2

[Robert Helmer [:rhelmer]]

I agree that websockets is likely overkill, but just wanted to point out that uWSGI does have websockets support:

https://uwsgi.readthedocs.org/en/latest/WebSockets.html

Making the UI poll every 5s (and not insert a new priority processing job each time) as well as making the progress bar better would indeed be welcome improvements.

Comment 3

[Peter Bengtsson [:peterbe]]

I didn't know about the websocket support in uWSGI but it's still not making it easier. Node, tornado or uWSGI you still need to lob data over to it some way. The example based on that uWSGI support use a Redis PubSub. 

So, is there ever any use case for putting the job back in again? We can simply use the django caching framework to remember that we've already "re-submitted" a particular UUID. Is that good enough?

I'd be happy to work on this bug.

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to Peter Bengtsson [:peterbe] from comment #3)
> I didn't know about the websocket support in uWSGI but it's still not making
> it easier. Node, tornado or uWSGI you still need to lob data over to it some
> way. The example based on that uWSGI support use a Redis PubSub. 
> 
> So, is there ever any use case for putting the job back in again? We can
> simply use the django caching framework to remember that we've already
> "re-submitted" a particular UUID. Is that good enough?

On-demand priority processing has always been an important feature, but it's a bit confusing... it's possible that processing will fail in a transient way, and none of this can be presented in a really good way to the user, as it stands.

Maybe making the initial hit to /report/index insert a priority job, and then perhaps polling could happen to a different endpoint (or with an HTTP param to prevent inserting more priority jobs?)

Comment 6

[Peter Bengtsson [:peterbe]]

This got fixed when the way fetching crashdata was completely refactored.