920022 - Images are not loaded in the Media section from the app Edit Listing page on MP-stage

Status: VERIFIED FIXED

(Marketplace Graveyard :: Developer Pages, defect, P1)

Description

[Iulian Timis]

Steps to reproduce:
1. Submit a hosted or packaged app on MP-stage https://marketplace.allizom.org/developers/submit/
2. Load the Edit Listing page of the submitted app.

Expected results:
The app icon and screenshots are loaded with no issues.

Actual results:
"Image changes being processed" error is displayed and images are stuck on continuous loading. Also, the app icons are missing in the Manage My Submissions page.

Notes/Issues:
Verified on FF27 (Win 7). Issue is not reproducing on -dev.
Screencast for this issue http://screencast.com/t/3nqQjLYI3aE

Comment 1

[Matt Basta [:basta]]

Sounds like celery?

Comment 3

[Victor Carciu]

This is now visible on Production

Comment 4

[Christopher Van Wiemeersch [:cvan]]

I cannot get icons to ever get extracted: http://basdfudgerigar6299.testmanifest.com/manifest.webapp

This could be related to http://sentry.mktmon.services.phx1.mozilla.com/mkt/marketplacefirefoxcom/group/215/

Comment 5

[Bob Silverberg [:bsilverberg]]

We are seeing this in our automated tests on staging as well.

Comment 6

[Stephen Donner [:stephend] Not actively reading bugmail]

:basta, can you help out, here?

Comment 7

[Matt Basta [:basta]]

(In reply to Stephen Donner [:stephend] from comment #6)
> :basta, can you help out, here?

There's not a lot I can do. It really sounds like bug 889924.

I was thinking of adding some really greppable log statements around the tasks that generate images, but I'm on PTO until Friday. If robhudson hasn't saved the day by then, I'll take a code walk and finally install the VPN software so I can look at the logs.

Comment 8

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Matt Basta [:basta] from comment #7)
> (In reply to Stephen Donner [:stephend] from comment #6)
> > :basta, can you help out, here?
> 
> There's not a lot I can do. It really sounds like bug 889924.
> 
> I was thinking of adding some really greppable log statements around the
> tasks that generate images, but I'm on PTO until Friday. If robhudson hasn't
> saved the day by then, I'll take a code walk and finally install the VPN
> software so I can look at the logs.

Thx; turns out we're still experiencing this issue, so if you could take a look, that'd be great!

Comment 9

[Matt Basta [:basta]]

I'll take a look as soon as I can get my environment set back up (or if you manage to find my computer, I'll take a look at it immediately!)

Comment 10

[Stephen Donner [:stephend] Not actively reading bugmail]

:robhudson, mind taking a look?

Comment 11

[Wil Clouser [:clouserw]]

Basta has a loaner machine now.  Basta: let us know what you find, this is weird.  Thanks.

Comment 12

[Matt Basta [:basta]]

I added a heck-ton of logging:

https://github.com/mozilla/zamboni/commit/ecbfdcdcdd5fd313f916cd43e94da648ccbe1d7f

The primary thing I'm interested in is making sure tasks a.) are triggered and b.) run until completion. I'll probe around in the logs a bit later to see what's up.

Comment 13

[Matt Basta [:basta]]

Since this issue also involves icons, I focused on that since it's easier to test. One issue that cvan and I found was that the `fetch_icon` method was getting called, which started a task to resize the icons and save the resized version into the destination location. The second task was never happening (a log immediately before the `delay()` call was firing but a log immediately within the task wasn't). Since we're in a task anyway, we removed the delay and just resized the icon then and there:

https://github.com/mozilla/zamboni/commit/40033878e7f6cfddf342c010d6138f222a54d8ae

We're about to try again and see if this makes any difference.

Comment 14

[Matt Basta [:basta]]

The issue surrounding icons seems to be fixed, but still exists with previews. I'll sit down with cvan tomorrow and take a closer look and try to figure out why celery is failing.

The issue seems to be that the tasks that are delayed simply never get fired. Somewhere between the .delay() call and the task server, they're simply getting lost.

Comment 15

[Matt Basta [:basta]]

I've narrowed it down to the "images" queue. Celery routes in other queues (namely "devhub") all work fine, but the ones that run in "images" are simply not getting executed. At last check on ops, they were restarting the workers. Will report back with more.

Comment 16

[Matt Basta [:basta]]

Images are loading now for me. It seems the tubes have been unclogged. I'll talk to Jason and see if there's anything that we can do to make this not happen in the future.

Comment 17

[Stephen Donner [:stephend] Not actively reading bugmail]

I'll let Iulian mark it officially verified, but our automation passed, again, as soon as :basta resolved this (and still is):

http://qa-selenium.mv.mozilla.com:8080/job/marketplace.stage.developer_hub.saucelabs/300/console

XPASS tests/desktop/developer_hub/test_developer_hub.py::TestDeveloperHub::()::test_that_checks_editing_basic_info_for_a_free_app Bug 920022 - Images are not loaded in the Media section from the app Edit Listing page on MP-stage
XPASS tests/desktop/developer_hub/test_developer_hub.py::TestDeveloperHub::()::test_that_checks_editing_support_information_for_a_free_app Bug 920022 - Images are not loaded in the Media section from the app Edit Listing page on MP-stage
XPASS tests/desktop/developer_hub/test_developer_hub.py::TestDeveloperHub::()::test_that_checks_that_manifest_url_cannot_be_edited_via_basic_info_for_a_free_app Bug 920022 - Images are not loaded in the Media section from the app Edit Listing page on MP-stage
XPASS tests/desktop/developer_hub/test_developer_hub.py::TestDeveloperHub::()::test_that_checks_required_field_validations_on_basic_info_for_a_free_app Bug 920022 - Images are not loaded in the Media section from the app Edit Listing page on MP-stage
XPASS tests/desktop/developer_hub/test_developer_hub.py::TestDeveloperHub::()::test_that_a_screenshot_can_be_added Bug 921036 - [stage] Image changes are not being processed when adding a new app screenshot
XPASS tests/desktop/developer_hub/test_developer_hub.py::TestDeveloperHub::()::test_that_a_screenshot_cannot_be_added_via_an_invalid_file_format Bug 920022 - Images are not loaded in the Media section from the app Edit Listing page on MP-stage
XPASS tests/desktop/developer_hub/test_developer_hub.py::TestDeveloperHub::()::test_that_an_icon_cannot_be_added_via_an_invalid_file_format Bug 920022 - Images are not loaded in the Media section from the app Edit Listing page on MP-stage

Comment 18

[Iulian Timis]

Verified as fixed in https://marketplace.allizom.org/developers/ on FF28 (Win 7).
Postfix screencast http://screencast.com/t/sftAKrA0
Closing bug.