Closed Bug 926449 Opened 11 years ago Closed 10 years ago

Implement a storage inspector

Categories

(DevTools :: Storage Inspector, defect)

Product:
DevTools
Component:
Storage Inspector
Platform:
x86
All
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: paul, Unassigned)

References

(Depends on 2 open bugs)

Details

(Whiteboard: [storage][meta]) 

We might want to cover:
- IndexedDB
- Cookies
- LocalStorage
- DataStores
- AppCache

Minimum required for V1 will be:
- IndexedDB
- LocalStorage
- DataStores

V1 can be read-only.
This is important for Gaia developers (IndexedDB is used a lot in Gaia apps and soon Gaia will rely on the DataStore API).

Myself and Optimizer will take a look at this.
I wrote some user stories to cover the basics last quarter:

https://github.com/mozilla/devtools-reqs/blob/master/data-resource/user-stories.md

Ideally each user story would be covered as a bug, and the bugs grouped by a whiteboard tag or marked as blocking this bug - I have no strong preference. Also ( ideally ) if you have additional use cases not covered by the user stories, please add them to the user stories doc.
(In reply to Paul Rouget [:paul] from comment #0)
> We might want to cover:
> - IndexedDB
> - Cookies
> - LocalStorage
> - DataStores
> - AppCache

That looks like a great list.

> Minimum required for V1 will be:
> - IndexedDB
> - LocalStorage
> - DataStores

I think AppCache is maybe more important than DataStores for most web developers. We have the gcli comand but when we show that to developers they complain it isn't discoverable, and they would prefer a more GUI-oriented tool ( possible with the commands exposed ).

Do you see having a cli-oriented query tool like the GCLI / Appcache commands? I would assume some apps would load in a lot of data, and a more mysql/cli interface might be preferable for some users even for querying.

> V1 can be read-only.

Sure. For a V2+ where you can write/delete/modify data, developers might want to be able to import data from a json file or similar. Do you have any ideas about the additional import/export functionality people expect from data tools?
Another question, regarding integration with the rest of the toolbox: do we see this as being a separate additional tool? The toolbox is crowded already. Chrome has the resources tab which includes all page assets, but has a sort-of uncomfortable overlap with the sources tab. cc'ing Joe & Nick for input.
(In reply to Jeff Griffiths (:canuckistani) from comment #3)
> I wrote some user stories to cover the basics last quarter:
> 
> https://github.com/mozilla/devtools-reqs/blob/master/data-resource/user-
> stories.md

That's very useful. Thanks.

> Ideally each user story would be covered as a bug, and the bugs grouped by a
> whiteboard tag or marked as blocking this bug - I have no strong preference.
> Also ( ideally ) if you have additional use cases not covered by the user
> stories, please add them to the user stories doc.

Yes. This bug is a meta bug.

(In reply to Jeff Griffiths (:canuckistani) from comment #4)
> I think AppCache is maybe more important than DataStores for most web
> developers.

You're right. App Cache is a nightmare, and we should support that in the V1.

DataStore is important too for apps. It will become a central piece of Gaia
starting very soon.

> Do you see having a cli-oriented query tool like the GCLI / Appcache
> commands? I would assume some apps would load in a lot of data, and a more
> mysql/cli interface might be preferable for some users even for querying.

We designed a DataBase API. IndexedDB. So if we ever implement a cli to request
data, I think it should be based on the synchronous IndexedDB API.

> > V1 can be read-only.
> 
> Sure. For a V2+ where you can write/delete/modify data, developers might
> want to be able to import data from a json file or similar. Do you have any
> ideas about the additional import/export functionality people expect from
> data tools?

Not yet.

> Another question, regarding integration with the rest of the toolbox: do we
> see this as being a separate additional tool? The toolbox is crowded
> already. Chrome has the resources tab which includes all page assets, but
> has a sort-of uncomfortable overlap with the sources tab. cc'ing Joe & Nick
> for input.

Yes. An additional panel. I don't know how we will scale yet. I think we can address this issues later. But indeed, at some point, we probably want to have a "Resource" tab where we'd get the Data Inspector, + all the files used by the pages.
Some of my thoughts :

- For web devs, Cookies might be the most requested things (if not the most important)
- IndexedDB inspection might really not be that useful for web devs. I know FxOS uses it very much, but on web, IndexedDB is mainly used to store offline data like images, textures, maps etc.
- We definitely need one extra panel for this tool as a whole. Its true that Style Editor and Debugger can be merged (but that is an another discussion).
- AppCache is not only a nightmare for web devs, but for us too. It "might" be a bottleneck for V1 ;) [This is just an assumption]
Summary: Implement a tool to inspector offline data → Implement a tool to inspect offline data
(In reply to Girish Sharma [:Optimizer] from comment #8)
> - For web devs, Cookies might be the most requested things (if not the most
> important)

It's also very simple. So let's do it.

> - IndexedDB inspection might really not be that useful for web devs. I know
> FxOS uses it very much, but on web, IndexedDB is mainly used to store
> offline data like images, textures, maps etc.

This is the main DB used in Firefox OS. We need to support it.

> - AppCache is not only a nightmare for web devs, but for us too. It "might"
> be a bottleneck for V1 ;) [This is just an assumption]

It's not hard to inspect:

See `about:cache?device=offline` and:
nsIApplicationCache
nsIApplicationCacheNamespace
nsIApplicationCacheContainer
nsIApplicationCacheChannel
nsIApplicationCacheService
nsIDOMOfflineResourceList
https://developer.mozilla.org/en-US/docs/Application_cache_implementation_overview
(In reply to Paul Rouget [:paul] from comment #9)
> (In reply to Girish Sharma [:Optimizer] from comment #8)
> > - For web devs, Cookies might be the most requested things (if not the most
> > important)
> 
> It's also very simple. So let's do it.

+1.

> > - IndexedDB inspection might really not be that useful for web devs. I know
> > FxOS uses it very much, but on web, IndexedDB is mainly used to store
> > offline data like images, textures, maps etc.
> 
> This is the main DB used in Firefox OS. We need to support it.

I agree we need to support it, I think for anyone to take this tool seriously we need to provide basic access in V1. To me for each item ( appcache, localstorage, indexedDB ) the prioritization could look like this:

V1: let developers view the data! In some case, maybe allow basic operations like flushing appcache. When we announce, we also show a roadmap for proposed enhancements and ask for feedback.

V2 -> Vn: let the developers do appropriate actions for each type of data. Get feedback from developers based on V1 to inform V2+. Developers might want to do all sorts of things like:

 - save binary blobs from indexedDB to the filesystem
 - edit cookies or localstorage in a property editor similar to the manifest editor in App Manager
 - delete data based on a query, or other command-line interactions

We should see what they ask for and weigh those requests.

Fair question: should V1 be an add-on?
(In reply to Jeff Griffiths (:canuckistani) from comment #11)
> Fair question: should V1 be an add-on?

I like that. Would it be possible to design the whole tool as an addon, and then, ship it as a builtin feature? How would we do that? What about tests? Is the devtools code ready for that?

Apparently, you are working with Irakli on a devtools API, would we use this API?

Actors will have to be built in though.
(In reply to Paul Rouget [:paul] from comment #12)
> (In reply to Jeff Griffiths (:canuckistani) from comment #11)
> > Fair question: should V1 be an add-on?
> 
> I like that. Would it be possible to design the whole tool as an addon, and
> then, ship it as a builtin feature? How would we do that? What about tests?
> Is the devtools code ready for that?

I don't think we're ready for that.

> Apparently, you are working with Irakli on a devtools API, would we use this
> API?

Down the road, for a tool that is maybe less core to what developers expect from us.

> Actors will have to be built in though.

I've been thinking a lot more about the tools as client-server apps. I think the really cool thing will be when add-ons are created that re-mix our existing server capabilities in interesting ways?

For this tool in particular consider my add-on comment as idle distraction. We know we need this tool and it shouldn't bee too much of a mystery how to build it. Let's just build it!
Can we please call this the "storage" tool instead of the "resource" tool? I feel very strongly about this. Everything is a resource; that term is incredibly vague. I think that "storage" better describes what we are shooting for.

I agree, this should be a new panel, but we only need one panel for all storage mediums. I think everyone is on the same page here, right?

I'm imagining a sidebar with the storage type you want to inspect (cookies, appcache, indexeddb, etc) that has its own view when selected with more details that are specific to that type of storage. It would be really cool if we had an api to add new types of storage types that addons could use. That way, libs like lawchair[0], pouchdb[1], jstorage[2], store.js[3], or whatever the cool kids are using these days could provide addons for inspecting/displaying/editing their specific storage stuff.

Definitely not a v1 thing, but I think it is something to keep in the back of our minds while architecting the code.

[0] http://brian.io/lawnchair/
[1] http://pouchdb.com/
[2] http://www.jstorage.info/
[3] https://github.com/marcuswestin/store.js
I agree with Nick. A problem with the term "resource" is that this also includes every files used on the page, including .html files, .js files, and image files. As evidence for this, if you look at Chrome's "resource" tab, you'll find this all-encompassing monstrosity that includes essentially a file explorer (of the above types) combined with stuff like local storage, indexeddb, etc. Things that don't really belong in the same tool.
Sure, "storage" works for me. And all in one panel. Like Chrome.
(In reply to Paul Rouget [:paul] from comment #17)
> Sure, "storage" works for me. And all in one panel. Like Chrome.

Like Chrome in that it's one panel. Not like Chrome in that we don't like the "Frames" section and we don't need the WebSQL section.

Also most websites are just going to be using one or maybe 2 of the storage options, the tree presentation rather assumes that you'll be have many of them to switch between. I'd be interested to see just the detail pane with everything on. Empty storage systems would be included with a simple "No Cookies" type message.
For the record, we have been looking at the option of generating the whole HTML report on the server side. But we'll need to be more granular than that as a DB can hold several dozens of Mb.
I just talked to Paul, we think the absolutely critical bits for v1 can be reduced to:

Storage Types:

* DataStores ( for FxOS & Apps developers )
* IndexedDB
* Localstorage

 - Data will be read-only.
 - We will need some sort of rudimentary search / filtering

V2+ then looks like this:

* Cookies
* Appcache
* Session data
* Read / write access
* Search improvements
* follow-ups based on user feedback
(In reply to Paul Rouget [:paul] from comment #22)
> Also, let's not forget that Indexdb and DataStore will soon be accessible
> from workers:

Is there a bug for being able to connect a toolbox to a worker? The Gaia team really wants this.
(In reply to Jeff Griffiths (:canuckistani) from comment #23)
> (In reply to Paul Rouget [:paul] from comment #22)
> > Also, let's not forget that Indexdb and DataStore will soon be accessible
> > from workers:
> 
> Is there a bug for being able to connect a toolbox to a worker? The Gaia
> team really wants this.

There's bug 757133.
Depends on: 930931
Summary: Implement a tool to inspect offline data → Implement a storage inspector
DataStores use IndexDB on the backend.
Depends on: 797639
(In reply to Paul Rouget [:paul] from comment #25)
> DataStores use IndexDB on the backend.

Then v1 could drop DataStores? v2 can have AppCache and dedicated DataStore access.

I suspect that read-write is less work than AppCache & DataStores (please correct me!), so maybe you can look into that for v1 again (though I agree that read access is far more important than write access).
See Also: → 912562
(In reply to Florian Bender from comment #26)
> (In reply to Paul Rouget [:paul] from comment #25)
> > DataStores use IndexDB on the backend.
> 
> Then v1 could drop DataStores? v2 can have AppCache and dedicated DataStore
> access.
> 
> I suspect that read-write is less work than AppCache & DataStores (please
> correct me!), so maybe you can look into that for v1 again (though I agree
> that read access is far more important than write access).

Write/Delete is not a big task as per the UI, but protocol wise and backend wise, it is not that straight forward. Specially given that we still do not have all the apis we require.

This is not even in its prototype stage. So we will see what happens :)
Depends on: 965872
As for priorities, shouldn't cookies go alongside localStorage in v1 because, you know, lots of code out there uses that old thingie...

My 2 cents.
(In reply to bobi from comment #29)
> As for priorities, shouldn't cookies go alongside localStorage in v1
> because, you know, lots of code out there uses that old thingie...
> 
> My 2 cents.

Please see the dependent bugs. The actor already has cookies and local storage.
Whiteboard: [storage][meta]
Blocks: 971672
Depends on: 1013635
Depends on: 1031189
Depends on: 1031192
Depends on: 1031194
Just to give a bit more info from the Web Compatibility front.
More and more Web sites are using localStorage capabilities for storing information about Web browser profiles. There is no easy way for people in Web compatibility to modify and test the values (except through the console). This interface would be very useful in our work.

Also see http://www.opera.com/dragonfly/documentation/storage/
Depends on: 1049888
Blocks: 1049888
No longer depends on: 1049888
mass component move . filter on #MassComponentMoveStorageInspector
Component: Developer Tools → Developer Tools: Storage Inspector
No longer blocks: 1049888
Depends on: 1049888
Depends on: 1059724
Do we still keep this bug open ?
Ping, should that bug be closed now?

Sebastian
Flags: needinfo?(paul)
Flags: needinfo?(jgriffiths)
Yup, everything else should be follow-ups, would like to see a tracking bug for planned improvements.
Status: NEW → RESOLVED
Closed: 10 years ago
Flags: needinfo?(jgriffiths)
Resolution: --- → FIXED
Flags: needinfo?(paul)
Product: Firefox → DevTools
You need to log in before you can comment on or make changes to this bug.