If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Implement a storage inspector

RESOLVED FIXED

Status

()

Firefox
Developer Tools: Storage Inspector
RESOLVED FIXED
4 years ago
2 years ago

People

(Reporter: paul, Unassigned)

Tracking

(Depends on: 2 bugs)

Trunk
x86
All
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [storage][meta])

(Reporter)

Description

4 years ago
We might want to cover:
- IndexedDB
- Cookies
- LocalStorage
- DataStores
- AppCache

Minimum required for V1 will be:
- IndexedDB
- LocalStorage
- DataStores

V1 can be read-only.
(Reporter)

Updated

4 years ago
Duplicate of this bug: 909714
(Reporter)

Comment 2

4 years ago
This is important for Gaia developers (IndexedDB is used a lot in Gaia apps and soon Gaia will rely on the DataStore API).

Myself and Optimizer will take a look at this.
I wrote some user stories to cover the basics last quarter:

https://github.com/mozilla/devtools-reqs/blob/master/data-resource/user-stories.md

Ideally each user story would be covered as a bug, and the bugs grouped by a whiteboard tag or marked as blocking this bug - I have no strong preference. Also ( ideally ) if you have additional use cases not covered by the user stories, please add them to the user stories doc.
(In reply to Paul Rouget [:paul] from comment #0)
> We might want to cover:
> - IndexedDB
> - Cookies
> - LocalStorage
> - DataStores
> - AppCache

That looks like a great list.

> Minimum required for V1 will be:
> - IndexedDB
> - LocalStorage
> - DataStores

I think AppCache is maybe more important than DataStores for most web developers. We have the gcli comand but when we show that to developers they complain it isn't discoverable, and they would prefer a more GUI-oriented tool ( possible with the commands exposed ).

Do you see having a cli-oriented query tool like the GCLI / Appcache commands? I would assume some apps would load in a lot of data, and a more mysql/cli interface might be preferable for some users even for querying.

> V1 can be read-only.

Sure. For a V2+ where you can write/delete/modify data, developers might want to be able to import data from a json file or similar. Do you have any ideas about the additional import/export functionality people expect from data tools?
Another question, regarding integration with the rest of the toolbox: do we see this as being a separate additional tool? The toolbox is crowded already. Chrome has the resources tab which includes all page assets, but has a sort-of uncomfortable overlap with the sources tab. cc'ing Joe & Nick for input.
(Reporter)

Updated

4 years ago
Duplicate of this bug: 927051
(Reporter)

Comment 7

4 years ago
(In reply to Jeff Griffiths (:canuckistani) from comment #3)
> I wrote some user stories to cover the basics last quarter:
> 
> https://github.com/mozilla/devtools-reqs/blob/master/data-resource/user-
> stories.md

That's very useful. Thanks.

> Ideally each user story would be covered as a bug, and the bugs grouped by a
> whiteboard tag or marked as blocking this bug - I have no strong preference.
> Also ( ideally ) if you have additional use cases not covered by the user
> stories, please add them to the user stories doc.

Yes. This bug is a meta bug.

(In reply to Jeff Griffiths (:canuckistani) from comment #4)
> I think AppCache is maybe more important than DataStores for most web
> developers.

You're right. App Cache is a nightmare, and we should support that in the V1.

DataStore is important too for apps. It will become a central piece of Gaia
starting very soon.

> Do you see having a cli-oriented query tool like the GCLI / Appcache
> commands? I would assume some apps would load in a lot of data, and a more
> mysql/cli interface might be preferable for some users even for querying.

We designed a DataBase API. IndexedDB. So if we ever implement a cli to request
data, I think it should be based on the synchronous IndexedDB API.

> > V1 can be read-only.
> 
> Sure. For a V2+ where you can write/delete/modify data, developers might
> want to be able to import data from a json file or similar. Do you have any
> ideas about the additional import/export functionality people expect from
> data tools?

Not yet.

> Another question, regarding integration with the rest of the toolbox: do we
> see this as being a separate additional tool? The toolbox is crowded
> already. Chrome has the resources tab which includes all page assets, but
> has a sort-of uncomfortable overlap with the sources tab. cc'ing Joe & Nick
> for input.

Yes. An additional panel. I don't know how we will scale yet. I think we can address this issues later. But indeed, at some point, we probably want to have a "Resource" tab where we'd get the Data Inspector, + all the files used by the pages.
Some of my thoughts :

- For web devs, Cookies might be the most requested things (if not the most important)
- IndexedDB inspection might really not be that useful for web devs. I know FxOS uses it very much, but on web, IndexedDB is mainly used to store offline data like images, textures, maps etc.
- We definitely need one extra panel for this tool as a whole. Its true that Style Editor and Debugger can be merged (but that is an another discussion).
- AppCache is not only a nightmare for web devs, but for us too. It "might" be a bottleneck for V1 ;) [This is just an assumption]
Summary: Implement a tool to inspector offline data → Implement a tool to inspect offline data
(Reporter)

Comment 9

4 years ago
(In reply to Girish Sharma [:Optimizer] from comment #8)
> - For web devs, Cookies might be the most requested things (if not the most
> important)

It's also very simple. So let's do it.

> - IndexedDB inspection might really not be that useful for web devs. I know
> FxOS uses it very much, but on web, IndexedDB is mainly used to store
> offline data like images, textures, maps etc.

This is the main DB used in Firefox OS. We need to support it.

> - AppCache is not only a nightmare for web devs, but for us too. It "might"
> be a bottleneck for V1 ;) [This is just an assumption]

It's not hard to inspect:

See `about:cache?device=offline` and:
nsIApplicationCache
nsIApplicationCacheNamespace
nsIApplicationCacheContainer
nsIApplicationCacheChannel
nsIApplicationCacheService
nsIDOMOfflineResourceList
https://developer.mozilla.org/en-US/docs/Application_cache_implementation_overview
(Reporter)

Comment 10

4 years ago
http://mxr.mozilla.org/mozilla-central/source/netwerk/cache2/nsICacheStorageService.idl#39
http://mxr.mozilla.org/mozilla-central/source/netwerk/cache2/nsICacheStorage.idl#86
(In reply to Paul Rouget [:paul] from comment #9)
> (In reply to Girish Sharma [:Optimizer] from comment #8)
> > - For web devs, Cookies might be the most requested things (if not the most
> > important)
> 
> It's also very simple. So let's do it.

+1.

> > - IndexedDB inspection might really not be that useful for web devs. I know
> > FxOS uses it very much, but on web, IndexedDB is mainly used to store
> > offline data like images, textures, maps etc.
> 
> This is the main DB used in Firefox OS. We need to support it.

I agree we need to support it, I think for anyone to take this tool seriously we need to provide basic access in V1. To me for each item ( appcache, localstorage, indexedDB ) the prioritization could look like this:

V1: let developers view the data! In some case, maybe allow basic operations like flushing appcache. When we announce, we also show a roadmap for proposed enhancements and ask for feedback.

V2 -> Vn: let the developers do appropriate actions for each type of data. Get feedback from developers based on V1 to inform V2+. Developers might want to do all sorts of things like:

 - save binary blobs from indexedDB to the filesystem
 - edit cookies or localstorage in a property editor similar to the manifest editor in App Manager
 - delete data based on a query, or other command-line interactions

We should see what they ask for and weigh those requests.

Fair question: should V1 be an add-on?
(Reporter)

Comment 12

4 years ago
(In reply to Jeff Griffiths (:canuckistani) from comment #11)
> Fair question: should V1 be an add-on?

I like that. Would it be possible to design the whole tool as an addon, and then, ship it as a builtin feature? How would we do that? What about tests? Is the devtools code ready for that?

Apparently, you are working with Irakli on a devtools API, would we use this API?

Actors will have to be built in though.
(In reply to Paul Rouget [:paul] from comment #12)
> (In reply to Jeff Griffiths (:canuckistani) from comment #11)
> > Fair question: should V1 be an add-on?
> 
> I like that. Would it be possible to design the whole tool as an addon, and
> then, ship it as a builtin feature? How would we do that? What about tests?
> Is the devtools code ready for that?

I don't think we're ready for that.

> Apparently, you are working with Irakli on a devtools API, would we use this
> API?

Down the road, for a tool that is maybe less core to what developers expect from us.

> Actors will have to be built in though.

I've been thinking a lot more about the tools as client-server apps. I think the really cool thing will be when add-ons are created that re-mix our existing server capabilities in interesting ways?

For this tool in particular consider my add-on comment as idle distraction. We know we need this tool and it shouldn't bee too much of a mystery how to build it. Let's just build it!
Duplicate of this bug: 897717
Can we please call this the "storage" tool instead of the "resource" tool? I feel very strongly about this. Everything is a resource; that term is incredibly vague. I think that "storage" better describes what we are shooting for.

I agree, this should be a new panel, but we only need one panel for all storage mediums. I think everyone is on the same page here, right?

I'm imagining a sidebar with the storage type you want to inspect (cookies, appcache, indexeddb, etc) that has its own view when selected with more details that are specific to that type of storage. It would be really cool if we had an api to add new types of storage types that addons could use. That way, libs like lawchair[0], pouchdb[1], jstorage[2], store.js[3], or whatever the cool kids are using these days could provide addons for inspecting/displaying/editing their specific storage stuff.

Definitely not a v1 thing, but I think it is something to keep in the back of our minds while architecting the code.

[0] http://brian.io/lawnchair/
[1] http://pouchdb.com/
[2] http://www.jstorage.info/
[3] https://github.com/marcuswestin/store.js
I agree with Nick. A problem with the term "resource" is that this also includes every files used on the page, including .html files, .js files, and image files. As evidence for this, if you look at Chrome's "resource" tab, you'll find this all-encompassing monstrosity that includes essentially a file explorer (of the above types) combined with stuff like local storage, indexeddb, etc. Things that don't really belong in the same tool.
(Reporter)

Comment 17

4 years ago
Sure, "storage" works for me. And all in one panel. Like Chrome.
(In reply to Paul Rouget [:paul] from comment #17)
> Sure, "storage" works for me. And all in one panel. Like Chrome.

Like Chrome in that it's one panel. Not like Chrome in that we don't like the "Frames" section and we don't need the WebSQL section.

Also most websites are just going to be using one or maybe 2 of the storage options, the tree presentation rather assumes that you'll be have many of them to switch between. I'd be interested to see just the detail pane with everything on. Empty storage systems would be included with a simple "No Cookies" type message.
(Reporter)

Comment 19

4 years ago
For the record, we have been looking at the option of generating the whole HTML report on the server side. But we'll need to be more granular than that as a DB can hold several dozens of Mb.
(Reporter)

Updated

4 years ago
Duplicate of this bug: 924424
I just talked to Paul, we think the absolutely critical bits for v1 can be reduced to:

Storage Types:

* DataStores ( for FxOS & Apps developers )
* IndexedDB
* Localstorage

 - Data will be read-only.
 - We will need some sort of rudimentary search / filtering

V2+ then looks like this:

* Cookies
* Appcache
* Session data
* Read / write access
* Search improvements
* follow-ups based on user feedback
(Reporter)

Comment 22

4 years ago
Also, let's not forget that Indexdb and DataStore will soon be accessible from workers:
https://bugzilla.mozilla.org/show_bug.cgi?id=701634
https://bugzilla.mozilla.org/show_bug.cgi?id=916196
https://bugzilla.mozilla.org/show_bug.cgi?id=916204
(In reply to Paul Rouget [:paul] from comment #22)
> Also, let's not forget that Indexdb and DataStore will soon be accessible
> from workers:

Is there a bug for being able to connect a toolbox to a worker? The Gaia team really wants this.
(In reply to Jeff Griffiths (:canuckistani) from comment #23)
> (In reply to Paul Rouget [:paul] from comment #22)
> > Also, let's not forget that Indexdb and DataStore will soon be accessible
> > from workers:
> 
> Is there a bug for being able to connect a toolbox to a worker? The Gaia
> team really wants this.

There's bug 757133.
(Reporter)

Updated

4 years ago
Depends on: 930931
(Reporter)

Updated

4 years ago
Summary: Implement a tool to inspect offline data → Implement a storage inspector
(Reporter)

Comment 25

4 years ago
DataStores use IndexDB on the backend.
(Reporter)

Updated

4 years ago
Depends on: 797639

Comment 26

4 years ago
(In reply to Paul Rouget [:paul] from comment #25)
> DataStores use IndexDB on the backend.

Then v1 could drop DataStores? v2 can have AppCache and dedicated DataStore access.

I suspect that read-write is less work than AppCache & DataStores (please correct me!), so maybe you can look into that for v1 again (though I agree that read access is far more important than write access).

Updated

4 years ago
See Also: → bug 912562
(In reply to Florian Bender from comment #26)
> (In reply to Paul Rouget [:paul] from comment #25)
> > DataStores use IndexDB on the backend.
> 
> Then v1 could drop DataStores? v2 can have AppCache and dedicated DataStore
> access.
> 
> I suspect that read-write is less work than AppCache & DataStores (please
> correct me!), so maybe you can look into that for v1 again (though I agree
> that read access is far more important than write access).

Write/Delete is not a big task as per the UI, but protocol wise and backend wise, it is not that straight forward. Specially given that we still do not have all the apis we require.

This is not even in its prototype stage. So we will see what happens :)
Duplicate of this bug: 887925
Depends on: 965872

Comment 29

4 years ago
As for priorities, shouldn't cookies go alongside localStorage in v1 because, you know, lots of code out there uses that old thingie...

My 2 cents.
(In reply to bobi from comment #29)
> As for priorities, shouldn't cookies go alongside localStorage in v1
> because, you know, lots of code out there uses that old thingie...
> 
> My 2 cents.

Please see the dependent bugs. The actor already has cookies and local storage.
Whiteboard: [storage][meta]
(Reporter)

Updated

4 years ago
Blocks: 971672
Depends on: 970517
Blocks: 1003860
Duplicate of this bug: 1010785
Depends on: 1013635
Duplicate of this bug: 1018182
Depends on: 1031189
Depends on: 1031192
Depends on: 1031194
Just to give a bit more info from the Web Compatibility front.
More and more Web sites are using localStorage capabilities for storing information about Web browser profiles. There is no easy way for people in Web compatibility to modify and test the values (except through the console). This interface would be very useful in our work.

Also see http://www.opera.com/dragonfly/documentation/storage/
Depends on: 1049888
Blocks: 1049888
No longer depends on: 1049888
mass component move . filter on #MassComponentMoveStorageInspector
Component: Developer Tools → Developer Tools: Storage Inspector
(Reporter)

Updated

3 years ago
Duplicate of this bug: 1056463
No longer blocks: 1049888
Depends on: 1049888
Depends on: 1059724
Do we still keep this bug open ?
Ping, should that bug be closed now?

Sebastian
Flags: needinfo?(paul)
Flags: needinfo?(jgriffiths)
Yup, everything else should be follow-ups, would like to see a tracking bug for planned improvements.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Flags: needinfo?(jgriffiths)
Resolution: --- → FIXED
(Reporter)

Updated

3 years ago
Flags: needinfo?(paul)
No longer depends on: 1031192
You need to log in before you can comment on or make changes to this bug.