Closed Bug 521895 Opened 16 years ago Closed 14 years ago

graphs-old historical data gone

Categories

(mozilla.org Graveyard :: Server Operations: Projects, task)

Product:
mozilla.org Graveyard
Component:
Server Operations: Projects
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: mozilla, Assigned: justdave)

References

Details

(Whiteboard: [post 3.5.5/Bugzilla 3.4])

It appears that only new data in graphs-old.m.o is there (mobile, as of Oct 6 or so). All of the historical talos performance data for production (and mobile pre-oct 6) is gone. Could you please investigate and determine the best way to restore this data? - Will this restore wipe the data created between Oct 6 and now? - How long of a downtime will this take? Once we have the answers to the above questions, we can decide when the restore should happen. We may need to wait until the N810s are all reporting to graphs.m.o depending.
graphs-old isn't supposed to be being used anymore. I got permission from rdoherty to nuke it a couple weeks ago, and discovered after the fact that there were still things submitting to it. There was a "final" backup of the database made for posterity on October 2nd and filed. The database was reset again on October 6th trying to figure out where the new data was coming from. Restoring the backup will restore data prior to October 2nd, but everything between the 2nd and the 6th is gone. It should be possible to retain the data from after the 6th when restoring though. This database is consuming a half TB of disk space and has been causing major pain for the backup system. The sooner we can nuke it the better.
I have the data restoring to an alternate database now, it'll probably take a few hours. After it's restored, I'll run the SQL to copy it back into the original database (so the new data will remain online the entire time) and that will likely take another few hours to run.
Assignee: server-ops → justdave
Re comment #1: I'm under the impression that developers want graphs-old around for reference. I can move the N810s to report to graphs.m.o in the next few weeks, however, which would enable us to make graphs-old read-only; would that help? Otherwise, if the end goal is deleting/archiving graphs-old, I think we need to pass that by the developers. And if that happens I'm pretty sure we're going to have to migrate that half TB to graphs.m.o, which would a) be a pain and b) slow down graphs.m.o and c) most likely not really save you much disk, in the end. Re comment #2, I'd honestly feel much safer if the restore happened after graphs-old stops receiving new data (since mobile needs the data from this week), but if you feel confident I don't want to make you have to do this all over again later.
If you want me to wait, I'm happy waiting. I'm in no rush. :) Also, as another option, it would actually be pretty painless to set up another copy of graphs.m.o and point it at the restored database without merging it with the recent data.
Ok. Let's hold off, and I'll work on getting the mobile devices to stop reporting to graphs-old. Marking dependent on the graph server "migration" bug and the corresponding mobile bug.
Depends on: 487335, 487329
I already had the database restoring last night before we decided to wait... I left it running because... why not? :) But anyway, the restore got about 50% of the way done 9 hours in, and had to be aborted because we ran out of disk space. It also caused the B01 slave to be lagged by about an hour and a half for almost 6 hours. We're going to have to find somewhere different to restore this to on the next attempt, as I can't be causing that kind of damage to MDC and Litmus and so friends (which are on the same server). Since this data is mostly historical, and at the point we're using it again it won't be getting updated, I'm thinking perhaps we add some disk space to dm-graphs01 and run mysql locally on that machine for this data. For performance reasons, the live data that's still getting updated can continue to live on the dedicated MySQL servers.
(In reply to comment #1) > graphs-old isn't supposed to be being used anymore. I got permission from > rdoherty to nuke it a couple weeks ago, and discovered after the fact that > there were still things submitting to it. > > There was a "final" backup of the database made for posterity on October 2nd > and filed. The database was reset again on October 6th trying to figure out > where the new data was coming from. Restoring the backup will restore data > prior to October 2nd, but everything between the 2nd and the 6th is gone. It > should be possible to retain the data from after the 6th when restoring though. > > This database is consuming a half TB of disk space and has been causing major > pain for the backup system. The sooner we can nuke it the better. justdave: 1) thanks for the repair work. Sadly, we do need this data for the mobile developers, so do need to restore it. 2) when doing this restore, its important to not lose the newer data that is still being inserted.
The reason for my concern: 1. The old links for mobile devices' graphs no longer work. This that a) the old links use something like a database ID to determine what graph(s) to show. 2. The mobile devices started sending data that we need to preserve (with new IDs). 3. I'm not sure if the restored data will change to use the new IDs or use the old IDs. If the latter, in the best case we'll have 2 sets of mobile devices with the same names. However, I'd guess it's likely that data from the old IDs would start to correspond with the wrong device/machine. 4. We have old links to the desktop machines' graphs. These will most likely not work either. In the above best case, we can change all these links manually. In the worst case, total confusion/corruption. I'm not that familiar with this db but I think I know enough not to trust it at all. My recommendation is to hold off on the restore.
(In reply to comment #8) > The reason for my concern: > > 1. The old links for mobile devices' graphs no longer work. This that a) the > old links use something like a database ID to determine what graph(s) to show. Can I get an example of a link you are using? Are these ids of the test/platform or of a specific test date? > 3. I'm not sure if the restored data will change to use the new IDs or use the > old IDs. I'm pretty sure it would be the old ids. > 4. We have old links to the desktop machines' graphs. These will most likely > not work either. In the above best case, we can change all these links > manually. In the worst case, total confusion/corruption. Let me know what those are and we might be able to figure out some redirects.
(In reply to comment #9) > (In reply to comment #8) > > The reason for my concern: > > > > 1. The old links for mobile devices' graphs no longer work. This that a) the > > old links use something like a database ID to determine what graph(s) to show. > > Can I get an example of a link you are using? Are these ids of the > test/platform or of a specific test date? https://wiki.mozilla.org/Buildbot/Talos/Machines#Mobile_.28Mobile.29 has the list I created; the rest of the page has links for other legacy perf data. The mobile list has been growing and I haven't been updating the links, but http://people.mozilla.com/~mfinkle/graphs/html/?tree=mobile queries graphs-old and gets mobile data. It's now empty, but there is mobile data there. This tells me that the deleting, then the subsequent re-creation of the mobile devices/branches via new submissions has changed their IDs. I'm not sure what the query mfinkle's page uses; Mark? > > 3. I'm not sure if the restored data will change to use the new IDs or use the > > old IDs. > > I'm pretty sure it would be the old ids. Doesn't this mean that there's a risk of old data (for, say, a Tiger talos box) would be marked as previous maemo-n810-01 data, for example? > > 4. We have old links to the desktop machines' graphs. These will most likely > > not work either. In the above best case, we can change all these links > > manually. In the worst case, total confusion/corruption. > > Let me know what those are and we might be able to figure out some redirects.
http://graphs-historical.mozilla.org/ is now up and pointing at the old database, which took 32 hours to restore. It doesn't seem to be working though, and I'm not sure why. I cloned the graphs-old docroot and changed the database config to point at the other database, and that was the only changes I made. Does something else need to be done?
It might be working, but so slow it seems broken. I'll take a look today/tmrw.
ryan, any updates? last comment was that you were going to look at it.
Also, I should stop writing to graphs-old late next week, if that helps.
I've just stopped writing to graphs-old.
Next steps: 1. merge data Do you need this right away? Hoping to hit this after 3.5.5 and after Bugzilla.
(In reply to comment #16) > Do you need this right away? Hoping to hit this after 3.5.5 and after > Bugzilla. We can wait. The new graphs data is working OK and is good enough for our current needs.
Whiteboard: [post 3.5.5/Bugzilla 3.4]
This bug is up for triage again... what's the action at this point, or do we care anymore?
I haven't found myself missing the data. So, IMO, we can drop this.
This affects all of Firefox's historical data as well. Maybe we should bring this up in a platform meeting?
From my side, I think the two best options are 1) do nothing (possibly with a current graphs-old backup/archive) 2) restore graphs-old as a read-only database that doesn't need further backup. The latter would be more thorough, but no one on the Firefox side has noticed that graphs-old data has been missing for nearly two months, so I don't know if it's necessary. I don't think I should be the one to make that decision, however.
#2 has already been done, the remaining action item on this bug was attempting to merge it with the current production database for graphs-old instead of leaving it there. It's currently being served on http://graphs-historical.mozilla.org/
...which apparently doesn't quite work at the moment still. See comment 11.
If mobile doesn't care about the merge (basically archive & drop the data between Oct 6 and the move to graphs.m.o) then graphs-historical should be fine as is, once it's working =) The alternative would be to merge the graphs-old data into graphs-historical... which should work better than the other way around.
Assignee: justdave → mrz
Component: Server Operations → Server Operations: Projects
Assignee: mrz → justdave
This has been sitting here for over a year, and I haven't heard anyone complain... do we still need this? The database is still there, the app fronting it just doesn't work. If nobody's asked in a year, I'm sure going and pulling some data out of the database directly on a one-off wouldn't be a big deal if someone needed something.
I personally have no need for this, but again don't feel that I can make this call. I will post to the newsgroups.
Nobody seems to have spoken up in the last 3 months since that was posted. How "archived" do we want it? If I toss it into our general database backups pool it'll be on tape for a year and then disposed of. I could essentially leave it where it is forever if we wanted (mysql database directory on a standlone server), which would keep it at ready access if anyone ever did want something out of it (it would take a couple days to restore if we went the backup route), since nothing else seems to be clamoring for that disk space at the moment on that box.
We are still getting Talos runs today for fennec-1 and fennec-1.1 so I think we have a good baseline. Please feel free to move the data to the general database backups pool at your convenience.
Leaving it seems preferable to us... powered on or off. This has a lot of historical Firefox desktop performance data. How much disk is it? A sql dump elsewhere is acceptable as well, but leaving it seems like the path of least effort atm.
Actually have it in two places right now. database backup server has a mysqldump of it: > -rw-r--r--+ 1 root root 5.5G Oct 2 2009 b01.graphs_mozilla_org.2009.10.02.FINAL.sql.gz live copy of the data lives on dm-graphs01: > 884K mysql > 127G mysql-innodb
The backups exist, and the box is question has actually already been decommissioned, so this is gone already anyway.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → WONTFIX
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.