528087 - Update calculated file location of processed dump files

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Frank Griswold [:griswolf] [:fgriswold]]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
Build Identifier: 

Files uuid.dump, uuid.json and uuid.jsonz are looked up from within the web app.
The locations of those files changes per Bug #523660

The new style layout is described in Bug #523660, will presumably show up in the wike 'soon'.

If we continue to have php look for files directly, then this file (perhaps others) needs to be updated:

http://code.google.com/p/socorro/source/browse/trunk/webapp-php/application/libraries/CrashReportDump.php

In that file a $dir is passed to method crashFilename($dir, $uuid). Variable $dir appears to hold the string "/some/absolute/path/to/storage"."/name"
Either the '/name' part must be removed or the caller must send it with that part removed. Call the remaining part $root: The file location that holds the subdirectory 'name'. 

Pull $uuid apart to get:
 - $calculatedDay = "20".substr($uuid,-6); 
 - $radix0 = substr($uuid, 0, 2);
 - $radix1 = substr($uuid, 2, 2);
Then create the full path as:
$path = implode("/", array($root, $calculatedDay, 'name', $radix0, $radix1).".jsonz";

NOTE that this is pseudo code, for exposition only: It has not been compiled, and it may not be optimally efficient.

NOTE that instead of calculating the file path in php, the work could/should instead be easily handled by apache rewrite rules, and probably that should happen 'real soon now'.


Reproducible: Always





See http://code.google.com/p/socorro/wiki/DumpingDumpTables
See http://code.google.com/p/socorro/wiki/ProcessedDumpStorage for (as of 2009/11/11) the old-style layout
See http://code.google.com/p/socorro/wiki/JsonDumpStorage for (as of 2009/11/11) the old style layout

Comment 1

[Austin King [:ozten]]

Okay so for a crash uuid 8bf1e883-cf69-4583-8e07-059bf2091122

Old Path:
$config_dir/8b/f1/8bf1e883-cf69-4583-8e07-059bf2091122.jsonz
New Path:
$config_dir/20091122/name/f1/8bf1e883-cf69-4583-8e07-059bf2091122.jsonz

Is that correct?

Comment 2

[Frank Griswold [:griswolf] [:fgriswold]]

This is not quite correct for two reasons:
1: In the old path the value of $config_dir had to end in '/name' which is (very likely) wrong for the new path. Breaking old $config_dir down to something pseudo-likely, it probably looks like "/mnt/socorro/jsonstore/name". For the new path, it should then look like "/mnt/socorro/jsonstore"
2: typo lost the first level of the radix in the new path.

So the example in comment #1 becomes:

for crash bp-8bf1e883-cf69-4583-8e07-059bf2091122

Old Path:
// $old_config_dir ends in '/name'
$old_config_dir/8b/f1/8bf1e883-cf69-4583-8e07-059bf2091122.jsonz
New Path:
// $new_config_dir does not end in '/name'
$new_config_dir/20091122/name/8b/f1/8bf1e883-cf69-4583-8e07-059bf2091122.jsonz

Comment 3

[Austin King [:ozten]]

Adding to 1.2 release and picking up.

Comment 4

[Austin King [:ozten]]

I can't find the comment anywhere, so I'll restate the changes I'm making.

1) renaming dumpPaths to dumpPath in application.php
2) Updating directory formatting method to use dumpPath + date + 'name' + old style directory per Comment#2

dumpPath will be a string instead of a list of strings. The string is the root directory ($new_config_dir) in Comment#2.

Comment 5

[Austin King [:ozten]]

Attachment: First stab at updating file path calculation

This patch changes dumpPaths from an array into "dumpPath" a string which is the root of the dumps directory.
It removes looping over this config.
It grabs the date off the UUID and uses it to prefix the path.

To test - use 1fc2aed1-18d0-4170-9920-2de9a2091001
dumpPath = '/home/aking/fake_dumps'
and see /home/aking/fake_dumps/20091001/name/1f/c2/1fc2aed1-18d0-4170-9920-2de9a2091001.jsonz on khan

Comment 6

[Aravind Gottipati [:aravind]]

I don't want the webapp looking for anything on the file system.  I want the freedom to redo the backend as I see fit and I don't want to go back to the webapp everytime I make a change on the backend.

Comment 7

[Austin King [:ozten]]

(In reply to comment #6)
This is a major architectural change. We are happy to do it, but it has to be prioritized and weighted against other work. The patch was literally 5 minutes of work and 10 minutes of testing.

Comment 8

[Michael Morgan [:morgamic]]

Filesystems have been around for a long time, I don't have anything against using one.

Comment 9

[Michael Morgan [:morgamic]]

We need to talk about it at 1pm, because I don't know what we need.

Comment 10

[Michael Morgan [:morgamic]]

Need to know:
- what you're thinking about as far as the rewrites, etc. -- present to team
- when this abstraction would be available and how to point to it
- how we'll document and add it to the project

Could you provide more details about what you have in mind re: comment #6?

Comment 11

[Frank Griswold [:griswolf] [:fgriswold]]

Comment on attachment 413108 [details] [diff] [review]
First stab at updating file path calculation

function isReportAvailable(...):
Aravind is concerned about every file system access. If possible, we should assume the file exists and catch the exception on failure rather than test first.
Low priority to fix, since this all goes away 'soon'
Otherwise, it looks good (to an eyeball check: Did not run it)

Comment 12

[Aravind Gottipati [:aravind]]

(In reply to comment #10)
> Need to know:
> - what you're thinking about as far as the rewrites, etc. -- present to team

We already have rewrite rules in place because I have crash jsonz files on two different file systems.  We also have two dump locations configured in the webapp.  The current rewrite rules look like this.

  RewriteCond %{REQUEST_URI} /dumps/(\w\w)(\w\w)((?:\w|-)+).jsonz
  RewriteCond /mnt/old_bp_jsonz/name/$1/$2/$1$2$3.jsonz -f
  ReWriteRule /dumps/(\w\w)(\w\w)((?:\w|-)+).jsonz /mnt/old_bp_jsonz/name/$1/$2/$1$2$3.jsonz [L]

  RewriteCond %{REQUEST_URI} /dumps/(\w\w)(\w\w)((?:\w|-)+).jsonz
  RewriteCond /mnt/processed_bp_jsons/name/$1/$2/$1$2$3.jsonz -f [OR]
  RewriteCond /mnt/processed_bp_jsons/name/$1/$2/$1$2$3.jsonz -l
  ReWriteRule /dumps/(\w\w)(\w\w)((?:\w|-)+).jsonz /mnt/processed_bp_jsons/name/$1/$2/$1$2$3.jsonz [L]

When we do move to a per day type storage I will change these rules accordingly.  The overlying goal is that the client should be able to request http://crash-stats.mozilla.com/dumps/uuid.jsonz and get either the dump or a 404.  How anyone maps the url to the dump in the filesystem layer is really not defined by the application.

> - when this abstraction would be available and how to point to it

The abstraction is the json dump url listed above.  The idea is to get rid of the storage location from the application and leave it to apache to provide the file.

> - how we'll document and add it to the project

Not sure what I can help with here.

> 
> Could you provide more details about what you have in mind re: comment #6?

Please let me know if you have any more questions and I can answer them.

Comment 13

[Austin King [:ozten]]

Attachment: Removes file system access from PHP app

The attached patch:
* Adds two configuration variables to application.php
** local url for accessing jsonz files
** public url for accessing jsonz files
** this gives IT complete control over urls and disk location of jsonz files
* Removes File system access
* Adds code to access the gzip compressed json encoded crash files

Comment 14

[Austin King [:ozten]]

Updated http://code.google.com/p/socorro/wiki/SocorroUpgrade

Comment 15

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Comment on attachment 414417 [details] [diff] [review]
Removes file system access from PHP app

The code changes look great, except for one thing.  The pending reports ajax call is no longer working because it got axed in this update.  Add that back in, and update that method to look for the file in its new place, and that should take care of it.

Comment 16

[Austin King [:ozten]]

Attachment: Same patch, but with an updated pending_ajax method

Updated pending_ajax to use new "no filesystem access" method

Comment 17

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Comment on attachment 414585 [details] [diff] [review]
Same patch, but with an updated pending_ajax method

Looks good.  Pending call is working again.

Comment 18

[Austin King [:ozten]]

Committed r1515