816213 - No new data on dashboard since 2012-11-26

Status: VERIFIED FIXED

(Input Graveyard :: Dashboard, defect)

Description

[Michael Cooper [:mythmon]]

The dashboard is not updating, and has not been for about two days. :sheeri tells me that there was a DB update recently. :Cww tells me that the data is still in the dump files, so it doesn't seem like anything is getting lost.

Comment 1

[Sheeri Cabral [:sheeri]]

For reference, what we did was failed over to a slave, which had been upgraded. However, the data was checked and the data was the same. So I can't see anything different.

Is is possible the failover disconnected a connection? If so, let us know what that is so we can add it to monitoring.

I haven't yet failed back to the master, it's all upgraded now but I'm waiting for the checksum to finish before I fail back. (again, ensuring data integrity before failing back).

Comment 2

[Michael Cooper [:mythmon]]

I believe that the front page is generated entirely by a search system. I think it uses Sphinx, but there is a chance it uses Elastic. I'm not 100% sure either way since I'm not too familiar with the code base.

Maybe the search system was looking at the master database, and didn't get failed over? Maybe going back to the master will just fix this, but I'm not sure.

Comment 3

[Sheeri Cabral [:sheeri]]

Well, we do everything via the virtual IPs, which is what apps should use, and we pointed the virtual IP at the new failed over slave.

If the app is directly pointing at the old master, that should have still been able to access the database most of the time in the past few days, other than a few hours here and there when we upgraded it.

I'll let you know when it's been failed back, but I am not sure failing back will fix it.

Comment 4

[Sheeri Cabral [:sheeri]]

I have just failed back to the master. Can you re-check?

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Sheeri Cabral [:sheeri] from comment #4)
> I have just failed back to the master. Can you re-check?

As long as we've been receiving and storing it, https://input.mozilla.org/en-US/?product=firefox&version=7.0&date_start=2012-11-28 should have recent data.

So too, should https://input.mozilla.org/en-US/?q=&product=firefox&version=7.0&date_start=2012-11-28&date_end=2012-11-27

Comment 6

[Michael Cooper [:mythmon]]

Those links are for firefox version 7, which for some reason the site defaults to. This link will show all versions, so is probably a better smoke test: https://input.mozilla.org/en-US/?product=firefox&version=--&date_start=2012-11-28.

That being said, there is still no new data on the home page. Maybe we need to wait for a cron job that will reindex things? Or maybe we need to trigger a new one?

Comment 7

[Jake Maul [:jakem]]

The sphinx re-index cron was failing because:

ERROR: index 'opinions': sql_query_pre[1]: Query cache is disabled; restart the server with query_cache_type=1 to enable it (DSN=mysql://input_user:***@10.8.70.90:3306/input_mozilla_com).

There is (was, I removed it) a line in the config file for sphinx:

sql_query_pre           = SET SESSION query_cache_type=OFF

Apparently the query cache is now disabled server-wide, and the reindex job just fails outright when trying to disable it for its session. As long as this really is the case (that it's off for reals), it's safe to omit this line. If the query cache comes back on, we might need to put this line back in.

Leaving open, pending confirmation from :sheeri.

Comment 8

[Sheeri Cabral [:sheeri]]

We want it off for reals. The query cache stinks in most (but not all) cases - http://www.pythian.com/news/4142/is-the-query-cache-useful/ 

If there's a reason to have it on, I'm happy to turn it on, but by default it's off, and should stay off. (the former default was on, sitewide at Mozilla, but that's not a good idea these days, hence the change)

Resolving, please re-open if it's not actually resolved.

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED on https://input.mozilla.org/en-US/?q=&product=firefox&version=--&date_start=2012-12-11&date_end=

Comment 10

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: Post-fix screenshot