Closed Bug 40867 Opened 24 years ago Closed 22 years ago

Need means to reuse/reload current page without refetching from server

Categories

(Core :: DOM: Navigation, defect, P3)

Product:
Core
Component:
DOM: Navigation
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla1.0

People

(Reporter: jmd, Assigned: rpotts)

References

(Blocks 2 open bugs,
URL
)

Details

(5 keywords, Whiteboard: [Hixie-P1] partial fix is checked into 0.9.2 branch)

Attachments

(9 files, 2 obsolete files)

IRC log: dr and dbaron, discussing
1.90 KB, text/plain
Details
patch for Save As... case
12.13 KB, patch
Details | Diff | Splinter Review
trivial testcase (requires php aware server)
657 bytes, text/php
Details
branch patch to add the 'read only attribute postData' to nsIWebNavigation
3.89 KB, patch
Details | Diff | Splinter Review
Test CGI to return date by either GET or POST method.
320 bytes, text/plain
Details
patch to nsGlobalwindow to expose nsIInterfaceRequestor to JS
1.22 KB, patch
Details | Diff | Splinter Review
Don't expose getInterface() on window, let the caller do an explicit QI to get to it.
2.50 KB, patch
Details | Diff | Splinter Review
consolidated patch to nsGlobalWindow.cpp on exposing interfaces to JS
4.34 KB, patch
Details | Diff | Splinter Review
New patch that addresses the previous comments...
25.04 KB, patch
asa
: approval+
Details | Diff | Splinter Review 
When viewing any page, we need to keep a copy of the source around, instead of
opening again, in the case of the file protocol, or downloading it again, in the
case of the http and ftp protocols.

The reason this must be done is simple. Save As, in other applications, means,
save what I am currently viewing to a file that I will specify. In most cases,
this is what Mozilla does with it's Save As. However, due to the fact that we
are regeting the page, there stands a chance that it has changed, or been
completly removed. This goes completly against the 'save what I am currently
viewing to a file' meaning most users know Save As to mean.

One of the worst consequenses of this is for web developers. Say you are
designing a page, and have a copy open in mozilla. You make an unwanted change
to the source file, and are unable to undo it. Or maybe you even accidently
deleted the source file. No worries, you have a copy cached in Mozilla, right?
Wrong.

Also note that trying to save it, after the file has been deleted, will crash
Mozilla if the file was accessed via ftp or file. I've opened bug 40792 on this
problem. If the file was accessed via http and was deleted, Mozilla will
quietly save the 404 page. This is bad. Very, very bad.

Another problem is pages with dynamic content. You want to save the page you are
viewing. Maybe it was loading 2 hours ago, but thats the page you want saved,
not whatever is there now.

Yet another problem is large pages for users on dialup connections. The Lynx
browser makes you redownload a page when you to view as source. I found this
infuriating when I was a dialup'er, trying to speed up my surfing by using Lynx.
It made no sense to have to redownload what I seemingly was staring right at. 

For the composer, a bug was filed a month ago (Bug 37023), which is related to
this. davidr8 reported that composer was forcing you to save before you could
view source. beppe commented that it was more efficient to read from a file, and
the bug was set to VERIFIED WONTFIX. This is exactly the same problem. Do not
read the source from a file. Say I make a page in composer, save it, then edit
that saved file in another editor. If I switch back to composer, I'm looking at
the old version of the file. If I view source, I want, and expect, to see the
old source. What if i needed to reference it. It's seemingly there, hell, I'm
staring right at it!

Saving the pre-rendered page in memory is a huge bandwidth savings if it was
accessed over http/ftp, and is just plain the right thing to do.
Great bug!  I think it's the same as bug 6119, though.
URL: Everywhere
That is only regarding view source. I proposed that the system be used for Save
As, and view source. I'll put a note on that one.
Somehow, I would think the cache should take care of this, given a "Once Per
Session" recheck pref (in Advanced|Cache in the prefs).

Adding "perf" keyword.
Keywords: perf
Looks similar to bug 6119. Reassigning to Bill law.
Neeti
Assignee: gordon → law
Component: Networking: Cache → XML
I suggest that bug 6119, bug 17889, bug 37023, and bug 40792 (and probably a few 
other bugs as well) all be made dependent on this bug; and that this bug be upped 
to major severity, since it involves loss of data.

Otherwise those other bugs are all going to run around doing various 
inappropriate things with the cache, when commands operating on the current file 
(save, print, view source, etc) shouldn't be dependent on the cache at all (e.g. 
if the sum total of files in the current Web page is larger than the disk cache 
size).

This bug really isn't much to do with XML. Mozilla should hold the source of
*all* files currently being used, whether they be XML, PNG, CSS, or whatever.

Changing component from XML to XPApps (as I think was intended) and nominating 
for nsbeta3 since I think this is a major problem - we need to use the cache for 
these things (and get things out of the cache even when expired in certain 
circumstances).
Component: XML → XP Apps
Keywords: nsbeta3
Nav triage team: Already works the way 4.x works.
Whiteboard: [nsbeta3-]
Removing nsbeta3- to trigger re-evaluation.

PDT: This causes silent data loss!

If you want to save the page currently being shown, but the page in question is 
no longer on the server, or is dynamically created, then Mozilla will silently 
lose the page, saving the 404 file not found page, or the different version in 
the case of dynamic content, without telling the user. 

This is also a problem with View Source, one that content developers find 
incredibly annoying with dynamic content. Say there is a bug on a dynamically
generated page. The user right clicks on the page to view the source, but
when the source comes up it is for a totally different page, since the server
has generated a whole new document.

There are many other examples of when this is critical (e.g. dialup connection
is dropped before user saves page, view source on a POST operation such as after
a money transaction over the net, saving NetCenter content that changes 
regularly, and so on and so forth).

IE does this correctly, so marking 4xp. Moving up to 'major' severity, since
this is a major loss of function. Is there a keyword for this kind of bug?
Marking 'correctness' unless someone can think of a more appropriate keyword...
Severity: enhancement → major
Keywords: 4xp, correctness
Whiteboard: [nsbeta3-]
[To reverse the old maxim: it's not a feature, it's a bug! Resummarizing.]
Hardware: PC → All
Summary: [RFE] Mozilla Needs to Hold Source of Current Page in Memory → Hold source of current page in memory
nav triage team: nsbeta3-, while we believe this to be important, we are 
concerned that this would be too difficult to do safely, this late in the game.
Whiteboard: [nsbeta3-]
Sanity check: this bug is nsbeta3-, but blocks bug 17889 which is nsbeta3+.
Removing nsbeta3- again for re-evaluation. We have a lot more information on
this bug now. Bugs 17889 and 6119 are closely related, and likely have the same
fix. 17889 is nsbeta3+, and I just nominated 6119 for + and dogfood. Unless
somebody can come up with a super-workaround for both bugs, this bug should
really be fixed.

Ian Hickson has excellent comments in bug 6119 describing the problem, and I
copied comments from a dup of 6119 which may help towards a fix.
Whiteboard: [nsbeta3-]
adding cc
See also bug 39957...

Okay, so I'm thinking about this now. I'm not sure what we currently do, but I
would imagine that we keep a DOM tree for our current document if it's HTML or
XML, and otherwise, we just display the document source. It would probably not
be a Good Thing to keep both a DOM and the source of a large document around at
once, but it might be the compromise we would have to make for beta3.

In the long run, what might be possible is to have our object model code for
XML/HTML store enough information to be reserialized exactly as it was in the
source, such as whitespace, capitalization, comments, entity substitutions, etc.
A lot of thought is being given to this stuff in the W3C's work on the infoset
(http://www.w3.org/TR/xml-infoset) and this sort of stuff has been mulled over
plenty in SGML's "groves" -- the issue of what data constitutes the "information
represented by the document" and what is also in the document source but is
considered extraneous... Anyway, the XML and HTML parsers could store this extra
information in the object model (and hopefully by this time we'll have a unified
object model behind all our DOMs), and thus avoid any duplication of
information.

That would be the intelligent solution to the view-source, save-as, and
send-page bugs (see bug 6119). The printing and changing charset bugs don't need
the exact source, so this isn't a direct solution to those, but the DOM does
need to get held in memory for them...

Now that I think about it, why should the printing and changing charset bugs
even exist, if the DOM is kept in memory already (which it must, right?) Does
the document actually get reloaded and reparsed before doing those things? That
would be really messed up.

I'm thinking that it might be two different underlying bugs: (1) we need to hold
the source for the view-source/save-as type bug, and (2) we need to make print,
change charset, etc. use the current DOM we have in memory.

Does this sound like a reasonable analysis?
nav triage team:
there are other ways to fix the specific user problems (like reading from the
cache) that is much less of a hit then this suggested fix.
nsbeta3-
M Future to consider later.
Whiteboard: [nsbeta3-]
Target Milestone: --- → Future
Johng, that is incorrect. You can't read from the cache for a page which Mozilla 
has been told not to cache, because it's not in the cache.

For example -- you want to save, or print, the order confirmation page for 
something you've bought over the Web, but the server (correctly) has told Mozilla 
not to cache that page for privacy reasons. So Mozilla goes to the cache, finds 
that the page isn't there, and then *reloads the page* ... so (unless the shop 
site is quite cleverly designed) you end up inadvertently doubling your order. 
Need I say, that is not desirable?

I'm not 100% positive it works that way.  I think most sites are set up so that 
an order triggers a redirect to the receipt page.  So a reload just reloads that 
receipt page.  Going *back* triggers a "re-post form data?" confirmation dialog.

But I'm out of my area of expertise, here.

Regardless, the bottom line is that there is no way for us (speaking for the 
"front end") to  tell Necko to do anything more than fetch it from the cache (if 
possible).  See 
http://lxr.mozilla.org/seamonkey/source/netwerk/base/public/nsIChannel.idl#198

There is no "reload from the secret stash that's not the cache."

I think, if the request is legitimate (which it very well might be), that it 
would have to be implemented, for the most part, in the http channel 
implementation, with a new load flag that we could then use to fetch the data in 
the various places where we want to reuse it (view source, save as, print, 
etc.).

So, I'm resetting the component.  At the least, Gagan will be able to comment on 
the subject with more authority than I possibly could.

Re-summarizing, too.
Assignee: law → neeti
Component: XP Apps → Networking: Cache
Summary: Hold source of current page in memory → Need means to reload current page without refetching from server
Not actually sure if this is a networking bug. I'm wondering what piece of code
actually hangs on to the document while it's being rendered, while the DOM is
being modified by scripts, etc.

Who owns the current document?... Please have a look at the attachment...

De-nominating for beta3, removing milestone, but marking blocker, since it
blocks two closely-related [nsbeta3+][dogfood+] bugs. This needs to be on
somebody's radar, I just don't know whose.
Severity: major → blocker
Keywords: nsbeta3
Summary: Need means to reload current page without refetching from server → Need means to reuse/reload current page without refetching from server
Whiteboard: [nsbeta3-]
Target Milestone: Future → ---
forgive me if im stating the obvious with any of this. even worse, im not quite
sure what DOM means, used in the context here. i'm taking it to mean the html
equvilent of java bytecode (optimized half-rendered page), so if i guessed wrong
there, i guess you should stop reading.

we need three source caches. the traditional disk and memory caches, plus one to
hold the source of the current page, regardless of the settings of disk/memory
caches, and weither or not the server told us to cache the current page.

seperately, there is the DOM cache (perhaps storing DOM for the last N pages
would be a neat cpu/memory trade, but lets not stray).

print/redraw and maybe character switching (17889) would ask for the DOM cache.
view source/send page/save would ask for the source cache, which would fetch
from the always-available current-page cache.

the other alternative, the reserialization dr explained, seems like a pain, and
error prone... maybe worth investigating in the future, i dont know.
jeremy: dom is the w3c standard object model interface for representing a
document.

> print/redraw and maybe character switching (17889) would ask for the DOM
> cache. view source/send page/save would ask for the source cache, which would
> fetch from the always-available current-page cache.

that's one way of looking at it, but you have the right idea: use the object
model where you don't need the source, to avoid reparsing; use the source where
you do need it.

> the other alternative, the reserialization dr explained, seems like a pain,
> and error prone... maybe worth investigating in the future, i dont know.

to your credit, it is definitely a pain if not well-thought-out beforehand.
there is talk, though, of redoing our dom code sometime in the future (because
we currently have separate code for html, xul, and other doms). this would be a
place to include the reserialization information for xml/html. but, like i said,
that would be for the future.

for the present, we just need to keep the source of the current document in
memory. perhaps if we're worried about bloat, we could choose only to keep it in
memory when it's the result of a form submission... something like that.
Bug 17889 (to do with changing encoding), nsbeta3+, means you need to retain 
source at the *byte* level ...
No longer blocks: 6119
For pages that dont give special no-cache settings, we behave the same as 4x

But if the page had headers that disabled caching, then we will hit this 
bug in 4.x too. Would be nice to fix this. Since we dont have time, 
we will fix this for later release.

The general idea of the fix is to keep all pages in cache (no-cache) and not 
get them from cache if they weren't supposed to be cached. Then add another 
reload policy that says get it from cache overriding these hints.

Removing 4xp
Keywords: 4xp
Target Milestone: --- → Future
No longer blocks: 39957
We don't behave the same as 4.x here -- bug 20843 is what is the same as 4.x,
that is, post results aren't cached, but in 4.x you at least were able to view
the source or save the current page without refetching. that has nothing to do
with the behavior of the cache.

Bill Law summarized the bug best by saying "There is no 'reload from the secret
stash that's not the cache'."

This bug probably shouldn't even be in the Cache component. Also, adding
dataloss (because it should have been there all along).
Keywords: dataloss
Is this the same as bug 50949?  The bugs that depend on each seem pretty 
similar, and that one's fixed :)
As of the 09-27-00 build (give or take a day or two), the Advanced Debug option
for "Enable Memory Cache" is gone, and documents do not seem to be getting
caching in memory.  Is this related to all this discussion of cache/no-cache
here (bug 40867)?

As I understand it, there are supposed to be two caches, disk and memory.  If a
document's HTTP header (set by the HTTP server) says "Don't cache", the document
should not be stored in the disk cache.  But shouldn't it still be stored in the
memory cache, for re-use by Save As, Print, etc.?

atovar: there (are/should be) three caches, disk, mem, and current. current
caches objects regardless of weither disk/mem are turned off, or if http sets
nocache.

I say are/should be, because i'm not sure of the status. if are is true, this
bug is half closed.
Per law@netscape.com, this bug now also tracks the usage for View Source, i.e.
"Make View source never load from the server". Thus nominating for mozilla0.9.
See recent comments in bug 6119 for rationales.
Keywords: mozilla0.9
Blocks: 56346
When I type the url, "about:cache", I never see anything listed under memory
cache, even when I have Debug / Enable Memory Cache set.

Does this accurately show that the memory cache is never used?

I was also thinking: Why is the memory cache disabled by default?  Is this
because we are relying on the operating system to provide sufficient in-memory
caching for objects stored to the disk cache?
Blocks: 55583
This bug is a *major* pain for Zope developers, since on server exceptions we 
embed a traceback in an HTML comment in the error page, so that it can be 
inspected with View Source.  Couldn't the original text simply be attached to 
the root node of the DOM as a read-only attribute?
Blocks: 6119
This seems to be an architectural problem, and a cascading one that's causing
many other bugs.  (With perhaps more bugs yet to be identified.)

My understanding (correct me if I'm wrong) is that all these various functions
(View Source, Save As, etc.) all ask for the page from the Necko library, which
tries to fetch it from the normal memory or disk cache if possible, loading it
from the network (or generating an error?) if it's not in the cache.  I assume
that the cache management subsystem is part of Necko, not external to it.  Is
this an accurate summary of how it works?  Have I made any errors?

Users and web developers alike have the same (reasonable) expectation -- that
what they see in the browser window is what they'll get if they view the source,
save it, etc.  While that's often true, clearly it sometimes isn't.  It can
cause data loss and confusion, impede debugging of web applications, etc.

I think the root cause of this problem is that these functions are asking Necko
for the information, when Necko's function is to fetch data from URLs of all
sorts.  The very concept of "loading" the data to be viewed or saved is really
nonsensical, because it's obviously ALREADY been loaded once!  Forcing that
"load" to come from the cache may improve the situation, but it doesn't solve
the problem at all.  It only masks the more obvious symptoms of the problem.

I hate to suggest architectural changes, because they can often be problematic,
needing to evaluate and fix any code that might contain implicit assumptions
based on the old architecture.  However, I believe the only solution that will
ever fix this problem for good will probably need to be architectural in nature.

Leaving Necko out of the picture for the moment, there should be an underlying
relationship between the data structures that reflects the way we want to use
them.  For functions like "View Source" and "Save As", we want to make sure that
we ALWAYS refer to the same source that is already being displayed in the window
where the functions were triggered.  This is a user requirement, and should be
reflected in the data structures for maximum consistency and correctness.

The simplest way to achieve this would be for the data structures associated
with the window in question to contain or directly point to the exact source
they are using.  I assume they must already contain the DOM generated from that
source, since that would be necessary to reflow the page if the window is
resized.  The simple solution would be to associate the original source with the
DOM, and access THAT via the window's data structures for functions like "View
Source" and "Save As", leaving Necko out of the picture entirely.

This would necessarily be more correct and more reliable than asking Necko to
fetch it from the cache.  Older windows with outdated content would work as
expected, despite the existence of updated content in the cache.  POST results
from form submission would work as expected.  Clearing the cache wouldn't
matter.  Things would work much more as the user expects them to, and this would
be a relatively small architectural change for a big improvement in correctness.
 The obvious downside of this simple solution is that it would use more memory;
many pages would be saved that would duplicate information already in the cache.

The better solution is probably a larger architectural change, separating the
cache manager from the Necko library.  After all, Necko depends on the caching
functionality, and it would be best for these original source copies to be
integrated cleanly with the existing cache -- but there's no good reason for
Necko to be in the middle of functions operating on previously-loaded data.  It
would be cleaner to make the cache manager independent of Necko, while keeping
Necko dependent on the cache manager.  These functions on previously-loaded data
could then depend on the independent cache manager without depending on Necko.
This architectural change would make it impossible for these functions to
unexpectedly reload the page from the network.

An independent cache manager would need to be more flexible than Necko needs.
(I don't know how flexible the current cache manager actually is; maybe it's
already more capable than I imagine.)  While Necko needs size-bounded in-memory
and on-disk LRU caches keyed on URL information, functions such as "View Source"
and "Save As" (AND Back/Forward history functions) need a reference-counted
and/or garbage-collected memory management system, bounded only by available
memory.  When the LRU cache Necko needs happens to point to identical data, the
cache manager could share the memory space.  However, if Necko needs to replace
its cache entry when content has been updated, or delete an entry, it must NOT
change or discard any data still being referenced from, say, a browser window,
if functions on previously-loaded data are to perform as expected.

I've dodged a sticky issue.  What should happen if the DOM is changed by
Javascript code between the time the page is loaded and when the "View Source"
or "Save As" function is called?  There's good arguments for modifying the
source to match the DOM changes, but this is clearly MUCH more difficult than
tracking a copy of the original source.  Even if this can be done, sometimes the
user is likely to want the original source despite the DOM changes, so it should
only be optional.  (I just filed bug 63892 about this, since it's really an
enhancement anyhow.)
No longer blocks: 6119
Keywords: nsbeta1
Blocks: 6119
cc to self
neeti, are you going to fix this, or should it be reassigned so some non-Necko
person?  It shouldn't be Futured without helpwanted for long.

/be
This is something I'd be happy to volunteer for, if nobody else is ready to work
on it.  However, I can't guarantee how much time I'll have available to work on
it, or how long it would take me.  I'm not familiar with the Necko code or the
current cache architecture, so I'd have to study it.  I also don't know all the
places it would need to be linked in.  (Some pointers would be helpful.)

My point is, I'm willing to work on this, but can't promise I'll get it done in
any reasonable timeframe, if at all.  If nobody else would be working on it
anyway, then there's nothing to lose.  If someone with more time available can
dedicate it to this, I can look for other things to work on.

Should I wait and see if anyone picks this up?  Should I start working towards
it just in case?  Should I not have said anything until I had a patch in hand?
Blocks: 57724
No longer blocks: 57724
Wouldn't it be nice if it were possible to generate the original source
accurately (including formatting) from the information in the dom?  If that were
true, then none of this would be an issue, we could just use the dom to
regenerate the source without having to keep an extra copy around.  Not throwing
away whitespace, and finding a way to represent whitespace in attributes, would
be a nice memory efficient solution to this problem, and would have the side
benefit that people would be much happier with composer since it wouldn't always
be reformatting their documents.
That seems like a possible solution, but it would require a considerable number
of extensions to the DOM (some HTML-only, some perhaps XML-only):

 * remembering the original case of normalized attribute and element names and
attribute values
 * remembering the whitespace in a tag:
     + after the element name
     + around the = separating attribute name from value
     + after each attribute
 * remembering the type of quotation marks around each attribute value
 * remembering the entire contents of any markup declarations (e.g., DOCTYPE)
 * remembering the delimeters of included marked sections
 * remembering the contents and delimeters of ignored marked sections
 * remembering which characters were originally entity references
 * remembering how every entity reference was terminated
 * properly remembering comment delimeters, since the W3C DOM Core doesn't fully
describe SGML comment structure
 * remember whatever badly formed HTML (and perhaps XML, too, for view-source
and save) is thrown at us
 * ...what did I forget?
This is an HTTP issue, not an HTML issue. Since right now things don't work 
right for *any* document type, it is premature to be discussing a complex 
optimization for one particular document type.

And if you ask me, reconstructing the *exact* source from the DOM in a way that 
works 100% of the time, and is more efficient than simply keeping a copy of the 
original source, sounds completely hopeless.
It's surely not more efficient in terms of CPU time to regenerate the original
source from the DOM, even if it's extended to represent insignificant (to the
DOM) aspects such as whitespace.  More importantly, it's a significant increase
in complexity that begs for subtle bugs where you may reconstruct the exact
source 99.9% of the time, but never 100% of the time.  This would really work
more cleanly and reliably at a lower level.

The best solution is probably to save the content, byte for byte.  Then it's
straightforward to "recreate" the original source, because it's not a complex
process of reconstruction from the DOM.  Also, saving a byte-for-byte copy
allows for changing the character set (bug 17889) or potentially reinterpreting
the content as a different MIME type entirely.  (Suppose the MIME type specified
by the server is wrong?  How about a function to reinterpret the data under a
user-specified MIME type?)

Also, if memory efficiency is a concern, and CPU time is considered a worthwhile
tradeoff, simply compressing the source in-memory with zlib is a viable option. 
The DOM could also be compressed if it's not actively in use (i.e. history)...
Regarding documents being reformatted by composer, a more straightforward
solution (especially if the byte-for-byte source is to be saved for this bug)
would be to associate parsed elements in the DOM with byte ranges in the saved
source.  This is probably much simpler than annotating the DOM to retain case
and whitespace information.  Wouldn't that be as effective?  (Modified elements
could replace byte ranges in place -- fancier versions could try to guess some
of the formatting conventions by heuristic tests on the surrounding source...)
IMO, viewing the source from the cache should be perfectly fine. If you need to 
debug, you should be able to control your server to ensure that the page can be 
cached or there should be a control in the browser to ignore "no-cache" HTTP 
headers. Please don't implement some drastic, complicated solution such as 
regenerating the source from the DOM or whatever. K.I.S.S.
I agree that reconstructing the source from the DOM is a drastic, complicated
solution.  However, I strongly disagree that simply reading from the cache is an
acceptable solution.  Not only do you have a problem with content that asks not
to be cached, but also with older content that has been replaced in the cache
with newer content (but may still be displayed in a window), form post results,
missing data from manually clearing the cache, etc.  The cache is inappropriate
for the purpose, when the true intent of the functions (in the user's mind) is
to operate on the EXACT data that they see in the window (or saw before).  This
is why I'm suggesting saving a byte-for-byte copy of the content, irrespective
of whether that content should normally be in the cache.  (Ideally, data thus
saved would share memory with duplicate cache data for efficiency reasons...)
To reiterate:

1. You can't fetch the source from the cache, since it may not be there, or may 
have been replaced with a newer copy.  I want the source for what I'm looking 
at, not what I'd get if I reloaded!
2. You can't reconstruct the source from the DOM, since the DOM may have 
changed, and it's terribly fragile to expect an accurate reconstruction even if 
it didn't change.

A copy of the source needs to be associated with the 
window/frame/layer/whatever.  It can be compressed.  It may be able to be 
shared with the cache, but that's tricky.  For now, why not just keep the 
compressed copy as an attribute of the appropriate DOM node, or history object, 
or whatever, and release it when the object it's attached to is released?
Speaking as someone who's already had to cancel a duplicate order caused by
trying to save an order confirmation, I'd say pulling the data from cache is not
sufficient.
You all have valid points. However, I think viewing the source is something that 
is only useful to developers, who will have the savvy to do what is necessary to 
look at the source (i.e. turn off the no-cache header in HTTP server or tell 
browser to ignore no-cache, don't reload page in another window, etc). 99% of 
ordinary users will never bother to view the source.

Saving the current page is a different activity. Ideally, the page save function 
will offer the ability to save in a variety of formats, including HTML. However, 
I think that the data to save can just be generated from the DOM object and 
there is no requirement that the HTML (if the HTML format is chosen) be anything 
like the original source code. I agree that "Save" should absolutely never cause 
another request to be sent.

In summary, I don't believe that "saving a page" is the same thing as 
viewing/saving the contents of the HTTP reply message that generated the page.
> Not only do you have a problem with content that asks not
> to be cached, but also with older content that has been replaced in the cache
> with newer content (but may still be displayed in a window), form post results,
> missing data from manually clearing the cache, etc.

A more flexible cache architecture could handle that (have a way of marking a
mem-cached page as "window still visible, so don't overwrite until window goes
away, but data not to be saved to disk cache" or some such).

> 2. You can't reconstruct the source from the DOM, since the DOM may have
> changed, and it's terribly fragile to expect an accurate reconstruction even  if
> it didn't change.

If the DOM has changed, then so has the display in the window; don't you want to
see the source for what you're really looking at, rather than what you loaded
some time ago?

> to associate parsed elements in the DOM with byte ranges in the saved source.

That would be very useful for composer.  The parser would have to do the
association, though; trying to reconstruct it later would be error prone and
probably CPU intensive.
cc'ing gordon who working on the new cache.
Nice summary, Evan.  Personally, I agree with everything you said in it.

Ken, bug 6119 was specific to viewing the source; this bug applies as much to
saving a page as to viewing the source.  You seem to be suggesting that saving a
page generated from the DOM would be sufficient.  As a user, I would NEVER want
the browser doing this.  Even ignoring the readability of the code, if the
browser is just saving what it *thinks* the source represents, it invites errors
to creep into saved pages that weren't in the original source.  A bug in the
HTML parser could cause a buggy page to be saved.  (What if it didn't render
properly and the user is saving the page in order to view it in another
browser?)  Unsupported HTML tags (or attributes) would be discarded.  Also,
Mozilla itself could fail to view the saved page as before, if there's a bug in
the generation code.

While regenerating the HTML from the DOM is possible, it's complex and risky,
and offers little advantage over the straightforward solution of saving the
original source in some fashion...
Akkana: See my comments (for this bug) of 2000-12-28 14:39 for some of my
earlier thoughts about how an independent cache manager could meet the needs of
Necko's LRU cache and this sort of caching of historical content simultaneously.

As for DOM changes, I don't think it's cut-and-dried.  Probably you'll want to
see the original source sometimes and the source as modified by DOM changes at
other times, depending on the circumstances.  As a user, I certainly wouldn't
want to lose the ability to save a byte-for-byte copy of the original source,
even if the modified DOM could be used to generate a more-current version.  Both
options would be nice, but I believe the original source is critical...

As far as the parser needing to associate the byte ranges within the DOM, that
was implied.  It would be senseless to try to figure that out afterwards; the
parser would definitely need to do that part of the job.

Neeti/Gordon: I guess I'm glad someone is going to be working on this, but I'm a
little disappointed at the same time.  I was hoping to take a stab at it; it
sounded like fun!  (But I wouldn't want it held up waiting on me, since I can't
guarantee I'd find the time -- I don't have the luxury of being paid to hack
Mozilla full-time...)

If a wild hare makes me play with it anyway, I'll let you know.  (But only if
there's something worth seeing!)
Actually, when I look at the source, 99% of the time I am interested in the 
source as it came from the server.  This is certainly the case when I try to 
save a page.  If the DOM has been modified by client-side scripts, I would want 
to be able to save the original source and scripts and not have extra HTML 
that's generated from the modifications to the DOM.

In an ideal world, we would have both options:
1) looking at the source of "what we see now".  This would need to be generated 
from the DOM.
2) looking at the original source.  This is most likely impossible to generate 
from the DOM because it does not necessarily correspond to the DOM (elements may 
have been added to the DOM through scripts).

It looks like the original purpose of this bug was to cover option #2.  Should 
we open a separate bug on option #1?
I wanted to disagree with KenW's comments.  I think that it is not a safe
assumption to make that a developer should be able to frob the cache directive
if they're debugging server-generated pages.  IMO, view-source should function
independently of any cache directives.

Furthermore, when I save a document, I want the original source in all cases
that I've encountered so far.  In my experience, I end up wanting to save a page
so that I can do a diff against an earlier saved page.  I think it's dangerous
to think of saving a page as a non-developer function.

Both viewing the source and saving the page should be lossless.


We have been working on some design changes to the cache that should make it 
easier to address these problems.  I hope to push a proposal out to mozilla.org 
in the next few days.

The decision to keep the source around for currently viewed pages doesn't belong 
to the cache, but some of the changes we want to make will make it easier for 
current pages to share data with the cache avoiding unnecessary duplication.

Adding self to CC.

*** Bug 67781 has been marked as a duplicate of this bug. ***
*** Bug 68677 has been marked as a duplicate of this bug. ***
I don't think 99% of the users never bother to view the source. Although HTML 
was not meant for such a purpose, there are many people out here that generate 
pages at hand, and thus want access to the source whenever requested. That 
includes Javascript that can modify the code -both unaltered and modified 
source are worth interest in that case, but the unaltered one is the more 
important.

When saving a page to the disk, the original page need to be saved. The changes 
in the DOM will operate when loading the page in a browser, executing whatever 
javascripts etc. there are in the page. If the settings of the browser have 
changed (disabling Javascript...), so will the displayed page. I consider this 
as a matter of consistency in the browser rendering.

A question I have is whether or not keep images, embedded objects of the page 
in memory or not. Suppose I want to print a page: using the pictures from the 
DOM would be OK, wouldn't it? There is no use in refetching them

cc'ing pavlov and saari for their view on images and printing.  Short answer: 
sometimes (often?) the images need to be redecoded at a different resolution for 
printing.

Yeah, pulling the unmodified data out of the disk cache and redecoding it when
we need to print would be ideal for images with greater than screen resolution dpi. 
jido@respublica.fr
don't think ... <people> never bother to <act>
is a double negative. While it is valid English, people tend to worry that the 
author is confused. the literal meaning is: believe <people> <act>.

Saving objects with the page is the subject of other bugs.

printing should use the current data.  WYSIWYG
Printing should produce a WYSIWYG replication of what you see on screen in the
highest possible resolution, and that will be bounded by the printing device or
resolution of the image, whichever is smaller.

We're not printing a 72 dpi image if we have a 300dpi copy sitting around.
so long as we know that the image @300dpi is related to the one @72 dpi and not 
something that could be entirely different.
Yes, both/all resolutions are derived from the same http get of the compressed data.
Blocks: 55055
Blocks: 68412
Gordon: do we have any progress on this bug? I'm currently working on a set of
webpage scripts that have form POSTs all over them and debugging them is almost
impossible due to this bug. It's driving me up the wall :(

I'm not sure if the problem I was encountering was related to this.
When I was using some sort of web account and making operations and submitting
forms with cookies, mozilla sometimes re-post my login information when the page
direct me back to the main page (which should be cached), and the server thought
I was attempting to multiple login and refused my request.

Cache bugs to Gordon
Assignee: neeti → gordon
Is the new Cache Manager going to fix this?

It'd be a real handy thing to have, as a developer.  e-commerce and other apps 
where reloading will totally screw you could use it, too.
The new cache itself won't fix this, but will enable the docShell to hold onto 
references to cache entries for the current page ensuring they won't get fetched 
from the server again.  I'm not sure if docShell will take advantage of this 
however.
Whiteboard: [cache]
Resetting milestone from "Future" because at the time it was set it just meant
"post netscape6 release", and we have milestones up to mozilla 1.2 by now.
Keywords: mozilla0.9.1
Target Milestone: Future → ---
Blocks: 74349
Depends on: 72519
Target Milestone: --- → mozilla0.9.1
Blocks: 64100
Any follow-up on this one now that the new cache is backing in the tree?
Keywords: mozilla0.9
From what I've learned about the new cache, we now have a way to keep the 
source for currently viewed pages in the cache using the hard reference 
capability.  Just keep the hard reference around.  This capability is is not 
used yet, however.
If I understand this correctly-- at this stage this is just a docshell issue.
->docshell owner then.
Assignee: gordon → adamlock
Component: Networking: Cache → Embedding: Docshell
QA Contact: tever → adamlock
I'm slightly worried about letting the cache handle source storage.  Consider 
this: user visits a dynamically generated page, and leaves the window open.  A 
little later, they visit the page again in a different window, getting 
different results.  Now the "same" URL is open in two different windows, with 
two different source texts.  Can the cache handle this correctly, or will both 
windows seem to have the source text that was loaded more recently?
The cache can handle this correctly *if* the docShell holds strong references 
(cache tokens) to the items in the cache that it used to render itself.  If 
docShell doesn't hold the strong references, viewing source on either window 
would use the latest version (possibly fetching a new third version from the 
net).

*** Bug 78740 has been marked as a duplicate of this bug. ***
Yes ... and it's much more efficient to hold it in the cache because then you 
have only one copy of the document ... why keep it around a 100K document in 
your current page structure for a long period of time if it's already sitting 
in the cache?  Here's the sequence of events you're talking about:

1. User surfs to http://www.yahoo.com.
- http://www.yahoo.com comes down from the network.  We grab a copy from the 
cache.
- We grab a hard reference to that copy immediately.
- We keep that hard reference around.  This means the cache agrees to never get 
rid of it.

2. The page expires from the cache.
- Since the entry has a hard reference, it stays around in memory but gets 
Doomed.  Thus it cannot be searched.
- Note that this is also what will happen if the page is set not to be cached 
at all.

3. The user surfs to http://www.yahoo.com in another browser.
- Since the entry is doomed, the search through the cache does not find it.
- http://www.yahoo.com comes down from the network.  We grab this new copy from 
the cache.
- We grab a hard reference to that copy immediately.
- We keep that hard reference around.  This means the cache agrees to never get 
rid of it.
- Both copies of the page are in memory now; each hard reference refers to its 
own copy.

4. User closes the first browser.
- The browser lets go of the first hard reference.
- The cache realizes that the entry is doomed and so immediately dumps the 
document from memory.

5. User surfs somewhere else in the second browser.
- The browser lets go of the second hard reference.
- The cache entry is not removed yet (because it's not ready to expire) but 
when it expires it will be removed normally now instead of being Doomed.
Exactly!

Gordon, can the cache token be passed around?  For example, view source loads in
a different docshell than the original document (it's in a different <browser/>
element...).  Is there a way to pass the cache token to the new docshell?  And
are such things scriptable?
Yeah, they could be passed around, but some recipients may not have enough 
context to be able to use them.  We would definitely want to enable their use by 
view-source.  The sooner we can work out the details of what's required the 
better.

I believe the http channel is scriptable (that's where you get and set cache 
tokens).  There may still be a bug open to implement the setting of cache tokens 
on http channels; Darin would know for sure.  We had been waiting to get an 
actual client for them first.  Take a look at nsICachingChannel.idl.

Cache tokens are just an opaque nsISupports that happen to contain an 
nsICacheEntryDescriptor, so a http channel knows how to use them to rehydrate 
itself.

this bug depends on a bug that is  marked future. I'm reflecting that in this
bug's milestone.
Target Milestone: mozilla0.9.1 → Future
Considering that this is a correctness/dataloss bug with 36 votes that has been
targetted to mozilla0.9.1, and that the reason given for futuring bug 72519 is
"there are no consumers for it", wouldn't it perhaps be better to ask whether
the futuring of bug 72519 could be re-evaluated than to just future this bug
which obviously a lot of people consider to be very important...

It's also not as if depending on a futured bug is an absolute impediment to this
being fixed; I notice that 3 of the bugs "blocked" by this bug, and one other
bug "blocked" by bug 72519, are already fixed anyway.
This no longer depends on bug 72519 anyway.  The function we need is 
GetCacheToken() and that is implemented now.  The bug covers both GetCacheToken
() and SetCacheToken(); it is the latter that is not yet working, and that is 
not necessary for this bug.

Everything necessary to fix this bug is now in Mozilla.  There is no need to 
change the milestone unless the developers cannot do it in time.
clobbering milestone and killing dependency.
No longer depends on: 72519
Target Milestone: Future → ---
Question: should the browser keep references to the current page's images 
also?  It seems like it should.  Also if the current page contains frames we 
need to keep around references to the frameset as well as every single frame.  
Otherwise this fix is not accomplishing everything it should--namely a perfect 
representation of the current page for Save, View Source and Print.
Setting cache tokens probably *is* necessary for this bug to be fixed reliably.  
Simply holding a cache token won't guarantee that what you request from HTTP is 
the same thing that you're holding the cache token for.

OK, my apologies.  Then there's something I don't understand.  When I read bug 
72519 I naturally assumed that GetCacheToken got you the 
nsICacheEntryDescriptor (hard reference) referring to the document.  Indeed, 
reading the code, that seems to be the case.

Hmm ... perhaps the hard reference is not always set up by the time you call 
GetCacheToken()?  Is SetCacheToken() supposed to place the document into the 
cache and set up the hard reference?  Or perhaps does http place the document 
into the cache automatically but SetCacheToken() has to be called to keep and 
hold a hard reference to the document?
GetCacheToken() will allow you to keep an entry in the cache, but only 
SetCacheToken() will allow you to recreate a channel to that entry.  Otherwise 
you have to create a channel with a URL which may or may not be what the cache 
token refers to (the entry has been doomed).

Oh, I see.  So what you're saying is we will do this:
* User types in URL
  - Browser creates HTTP channel
  - Browser has HTTP channel retrieve content from URL (possibly through cache)
  - Browser grabs hard reference from HTTP channel and keeps it around 
(GetCacheToken)
  - Browser destroys HTTP channel
* User hits View Source
  - View Source gets cache token from Browser
  - View Source creates HTTP channel
  - View Source has HTTP channel retrieve content using hard reference 
(SetCacheToken)
  - View Source destroys HTTP channel

Why not change the second part to this:
* User hits View source
  - View Source gets cache token from Browser
  - View Source grabs content directly from cache token

Then you don't need SetCacheToken().  HTTP channel seems like an unnecessary 
extra step in this case.  Maybe it's impossible to get a stream out of the 
cache, and that's why it's done this way?
I think this also causes us to suck on the IBENCH tests.
No that's bug 61363.  The IBENCH tests we're looking at aren't doing printing, 
saving, or view-source.  This bug is for operations on the currently displayed 
page, so they don't result in refetching from the net.  When the user wants to 
print, save, or view-source for the front-most window, they don't expect to get 
different results from the net.

Do you think we can get this resolved in time for mozilla0.9.1?

BTW, another situation when we should use the current version instead of
refetching is the "Send page" function (which does not work correctly even in
Netscape 4). With "Send page"not only you might end up loosing the data, you may
also end up e-mailing something very different from what you thought you are
e-mailing...
-> 0.9.3
Target Milestone: --- → mozilla0.9.3
*** Bug 79758 has been marked as a duplicate of this bug. ***
Whiteboard: [cache] → [Hixie-P1] (HTTP) [cache]
Blocks: 83792
bug 83792 - same cause, more unwanted effects.
Fix this NOW, please.

(Sorry for the spam)
No longer blocks: 83792
Blocks: 84106
Blocks: 85128
Blocks: 86261
Keywords: nsenterprise
a possible (minimal) solution:

toplevel page is loaded... before the necko channel goes away, QI to
nsICachingChannel.  then call nsICachingChannel::GetCacheKey.  store
the cache-key someplace.

when you want to reload the document, just create a new necko channel,
but before calling AsyncOpen, QI the new channel to nsICachingChannel,
and call SetCacheKey, passing the stored cache-key to the channel.
also, be sure to call SetLoadFlags(nsIRequest::LOAD_FROM_CACHE) on the
necko channel as well to prevent the usual cache validation.
the cache-key is used, for example, to distinguish the results from 
different POST requests on the same URL.

this solution does not guarantee that data will not be fetched from the
cache, but it does do the "right thing" regardless of whether the cache
is present or not, etc.  and, for file->SaveAs and view-source it will
almost always avoid refetching from the net.
Folks, could someone please step up and make a damn architectural decision here?
over a year of thinking about this issue in this bug alone and we don't seem
much closer to a solution. There's been a lot of good discusions and ideas
kicked around, and a number of possible solutions proposed, but we still don't
appear to have anyone setting a stake in the ground and saying "do this." (or
better yet a group of core moz folk that can agree on single solution.)

I'm currently loosing a $100 argument with buy.com over a duplicate order that
resulted from printing a copy of the order summary. At this point mozilla is
unusable for the average user's e-commerce (at least with sites as poorly
engineered as buy.com). I've had to resort to another browser for any form of
online purchase.
putterman: were you asking me about ecommerce and multiple orders?
Radha, what's the session history perspective on this?
PDT+ per selmer's request.
Whiteboard: [Hixie-P1] (HTTP) [cache] → [Hixie-P1] PDT+
Session History comes into picture only when you hit reload() which uses the
VALIDATE_ALWAYS loadflag.  Session History however stores the cachekey for the
current page in mOSHE. This cacheKey is passed to nsICachingChannel when the
loadtype is LOAD_HISTORY/RELOAD_NORMAL/RELOAD_CHARSET_CHANGE. If the solution to
this bug is to do a similar thing for view-source/printing, then I think the
current setup in nsDocShell::DoURILoad()  can be extended to meet the needs here. 
I'm not seeing any problems with printing POST results.  On both the trunk and
the 0.9.2 branch when printing the results of a form POST we *do not* reload
from the network (and do another POST)...

Are there any sites that exhibit this problem when printing?  (ie. reposting
when printing...)

If not, I suspect that this is only a SaveAs issue...  Because the printing code
deals with documents in a completely different way...

-- rick
The attached patch implements Darin's suggestion for passing in the cache key to
the request generated by a Save As... operation. The patch attempts to load out
of the cache. If there is post data and the stream isn't in the cache, it brings
up a dialog asking if the user wants to repost.

The Save As... menu item works as expected with the patch. I saw a problem with
the Save As... context menu item, but it seemed as if the JS in nsContextMenu.js
was passing a bogus false value for doNotValidate. Still investigating, but the
patch should be set to go in as-is pending r/sr's.
Rick, was that a recent change in printing? I know it used to go back to the
network (for get requests anyway) in pre 0.8 days... I had gotten burnt a few
times printing the results of some internal tools we use and getting hard copy
of the input forms instead.

with current trunk builds here's the matrix from a quick test I just did:
(pre vidur's patch which I just mid-air colided with... building with it now.)

               POST     GET
---------------------------
print           OK      OK
view source     OK      OK
save as        BAD!     OK

hmmm.... one in six... russian roulette anyone?

(p.s. after four months, mention of the state attourney general
last week seems to have finally solved the discusions with buy.com)
with that patch my test case is 100% OK.
Chris, how about "File -> Send page" (bug 86261)? I haven't checked it yet, but
I am afraid it will add a bullet or two to your "Russian roulette".
               POST    GET
---------------------------
print           OK      OK
view source     OK      OK
save as         OK      OK
send page      BAD!     OK



http://ntiaotiant2.ntia.doc.gov/test/mozillatest is a much better test case from
cweiss@iname.com on bug 55583.

summary for those not on that bug:

save as: "successfully" re-requests! (number incremented)  
view source: fails completely (server error, suspect GET instead of POST)
send page: fails completely (server error, ditto)
print: success :)


vidur/adam - any update to this PDT+ bug?  Pls update status whiteboard.
It seems from Chris Abbey's comments that the patch from Vidur works ok?   If
so, can this patch get r and sr and be checked in?  It looks like the patch will
fix Save As, but not Send Page.  Perhaps that is a separate bug.
Latest patch looks good to me so r=adamlock@netscape.com.

I think Send Page should be new bug.
Well, according to last Chris Abbey's test results, _all_ methods currently
re-request, so there is still a need for some universal method of avoiding
re-requesting. But once such method is implemented, making use of it becomes
separate bugs - bug 55583 for "view source", bug 84106 for "save as" and bug
86261 for "send page".

IMHO, what this all means is that the "save as" patch belongs to bug 84106 and
not to this bug.
URL: http://ntiaotiant2.ntia.doc.gov/test/...
lchaing: my append on 6 July was with the trivial test case I attached, the
append on 7 July however was with the more realistic testcase at the URL. (I'm
not sure the difference, aside from a cfm script vs a php script, but it clearly
demonstraits one). As a result I wouldn't hold much faith in the value of my
append from 6 July if I were anyone.... ;) 

Aleksy: as rick pointed out, and the URL proves, print works fine... but then
print is also a completely different beast in that it really doesn't *need* the
original html, the dom is fine.  Otherwise I agree with your take on this bug 100%.
The patch ensures that for Save As we first attempt to get the data from the
cache and, only if that isn't possible (e.g. someone explicitly clears the
cache), we put up a dialog asking if we should repost. The same pattern should
be used for the View Source and Send Page cases as well, though the fixes for
these two will require changing different code paths.

I got a verbal sr=rpotts@netscape.com, so I'm inclined to get this into the
trunk for some bake time.
Vidur, exactly, and this is why the patch belongs to bug 84106. What this bug is
about is creating a mechanism that would ensure that if you are viewing a page,
then you also have a cache takon that will ensure that the source will be in the
cache if you need it. This patch is doing the right thing (hopefully) and is
addressing a related issue (as covered by bug 84106), but it is not addressing
this bug.
Vidur, does that patch also fix "Save Frame As..."?  It also uses the savePage()
function, but I'm not sure the history entry you're getting has the post data
for a frame... (in fact, based on my testing it seems not to).
Also how about "View Frame Source"? I was having trouble with that not working
correctly for POSTed data with 0.9.2 today, which is what reminded me.
"View Frame Source" is what _I'm_ working on (bug 55583).  That was my
'testing'.  If there's a way to get the postData and cacheKey associated with a
document in a frame, I have yet to find it....  Hence my previous comment in
this bug.
Not sure if this is pertinent but thought someone should check. There have been
a number of comments posted about "postData" does this need to include/does
include the cookie data at time of posting?  Obviously if you were refetching
the page from the network that would then be important, but is it important for
re-obtaining the page from the cache?  or any other location?

For instance you go to the same page URL+Post data, but since the submission has
already been posted once, now the source could change based on it being the
second post.  So then seemingly identical "posts" could be different based on
cookies, or the current time, etc, but would both appear to be the same in the
cache, does the second post overwrite the first cache? should it?

Glad to see this bug finally getting some heavy attention....
docshell member mOSHE has the cachekey and postdata for the document. THis
applies for subframes too.
Radha: is that member somehow scriptable?  The problem is that we're working in
JS here...

Wiggins:  The important thing is the combination of cacheKey + postData.  The
behavior on load is then as follows:

if (cacheKey && postData) {
  // try to load from cache using cachekey.  If that fails pop up a warning
  // dialog before requesting from the network
} else if (cacheKey) {
  // load from cache but do not prompt before re-requesting
} else {
  // just load
}
mOSHE is a private member of nsDocShell, not available from JS. 
is all or parts of this on the trunk or branch?
what's left to do?  
can someone summarize and project an ETA?
Here's what needs to happen as far as I can tell:

1)  An API is needed to get from a docshell (any docshell, not just the root of
    the docshell tree which has the session history attached to it) the
    information needed to recreate it (load from history or reload from the web
    with the same post data).  At the moment this would be a cache key and a
    post data stream.  These could be gotten as themselves, in the form of an
    nsISHEntry, or whatever.
2)  A method needs to be added to the nsIWebNavigation interface that takes the
    information gotten in step #1 and loads a page using it.  This could be
    similar to the current nsIDocShell::loadURI or something along those lines.

Once that's done, this bug can probably be marked as fixed.  The dependent bugs
can then handle using these APIs to make save page, save frame, send page, view 
source, view frame source, and so on work correctly.

Ccing Rick and Jud who've been talking about this a bit lately, since they'd be
the ones who could provide an ETA (and corrections in case I've gotten something
wrong).
What's up with this bug?  No current status and time is almost gone!
At chofmann's request, I'm posting this summary of status. Currently, the
following work correctly if there's post data associated with a page:

- print page
- print frame
- save page

The following still don't work correctly if there's post data (I believe they
will always refetch with a HTTP GET):

- save frame
- view page source
- view frame source
- send page

For NS6.1 RTM we have the following options:
1) Disable the latter set of operations for pages that have post data. To do
this correctly, specifically to even be able to determine whether the document
in an individual frame has post data associated with it, we still need some
amount of infrastructure (as opposed to chrome JS) change. Our options include:
 a) Expose the currently private nsISHEntry member of a docshell.
 b) Expose a mechanism for getting information about post data at the
nsIWebNavigation level for both top-level content windows and individual frames.
Once one of these changes are made, we'd need to make JS changes to check for
the existence of post data before completing the corresponding operation.

2) Come up with a complete fix. Boris Zbarsky's description of such a fix (or at
least the infrastructure for it) is on the money. We'd have to:
 a) Expose a mechanism for getting the post data at the nsIWebNavigation level
(as in 1b).
 b) Allow for the post data to be passed in as an optional parameter to
nsIWebNavigation::LoadURI (bug 46870).
 c) Either expose the notion of a cache key in a similar manner as the post data
stream or hide the cache key completely. The latter option would require us to
create an internal mapping between URIs and cache keys (including URIs that are
not "officially" in the cache, such as those that have post data).
Once these changes are made, we'd also need to make JS changes to retrieve the
post data and pass it into the load methods used for the different operations.

3) Live with the status quo

Rick Potts is looking at the infrastructure changes for 1 and 2. He will first
work on the infrastructure changes needed for disabling (probably option 1b/2a).
He will then check in an existing patch for 2b (see bug 46870). The ETA for
these changes is 7/18.

At chofmann's request, I will open bugs for making the chrome JS changes
necessary to disable the features for pages with post data. If Rick can't get
the complete fix done by 7/18, chofmann will find owners for the chrome bugs.

By "parts working" I assume you mean there was a checkin on the branch.  If so,
I'm leaning toward option 3.
A few comments on what vidur wrote:

> 1) Disable the latter set of operations for pages that have post data.

This would be a branch-only thing, I presume?  On the trunk we would keep
working toward option #2?

> The latter option would require us to create an internal mapping between URIs
> and cache keys

This may not work so well for view source...  Consider the page at
http://www.mozilla.org.  To view its source we load the URL
view-source:http://www.mozilla.org.  The protocol handler for view-source:
creates an HTTP channel for http://www.mozilla.org and loads it.  _But_, we want
to use the source from the original URL.  So we want to take the cache key and
post data corresponding to the http: url and load the view-source: URL with those.

This does not mean that the cache key has to be exposed, it just means that
there needs to be a way to load a url of the form view-source:whatever using the
information for the "whatever" URL.
Assignee: adamlock → rpotts
>This would be a branch-only thing, I presume?

yes
 
> On the trunk we would keepworking toward option #2?

yes

The most troublesome remaing problem for most users would
be "send page",  I could see users completing an ecommerce
transaction and then attempting to send the page to themselves
as a confrimation of the transaction.

bug over to rpotts since he is working the infrastructure changes
that are now the focus of this bug.
In order to make this work correctly, 3 different API changes must be made:
1. Provide a 'postData' attribute on nsIWebNavigation to allow JS to access the
postData.
2. nsIWebNavigation::LoadURI(...) needs an additional 'postData' argument.
3. The underlying necko 'cache key' needs to be hidden from non-networking APIs.

Clearly, we can't make all of these changes on the branch :-)  #3 alone requires
a fair amount of work - and darin is currently gone :-(

So, here's my propsed plan...
-----------------------------
For the branch I can make API change #1 and expose a postData attribute to JS. 
This will be enough to allow all JS implementations to 'disable' if postData is
present.
Fortunately, this API change is *very* small and should not introduce
regressions...  it does not change any existing APIs and the only consumers will
be the new JS code written to disable the features.

For the trunk I can land API change #1 and #2 and start working on #3...  Once
all three changes have been made, we will be able to 'correctly' implement these
features in JS...

Does this sound reasonable to everyone?

-- rick
sounds like a good approach to me.
Okay, I'm an outsider here, but I'm slightly worried about the #3 approach.

Consider the following scenario. I'm assuming a page that simply provides an
incrementing counter every time it's hit, but the same argument applies to
almost any page that can give different results on successive hits with the same
querystring and/or post data - anything that, for example, is time-sensitive (eg
frequently updated news sites), relies on a backing database on the server which
changes state, is content-negotiated (if the user changed their language prefs)
or cookie based, etc etc etc.

User opens page in window 1. Page is fetched from server, returning "1". Cache
entry #1 is created, containing "1". User leaves this window open on this site
and goes to work in another window.

User later opens same page in window 2. Page is fetched from server again,
returning "2". Cache entry #2 is created, containing "2". Cache entry #2 has the
exact same URI, and also the exact same post data (if any) as #1.

User goes back to window 1 and selects "View source". If the "View source"
implementation only has the uri and post data to go on, it has no way to
distinguish between cache entries #1 and #2, but they contain different things!
Since #2 was created later it will probably get that, which is wrong.

Surely a better way to go here would be to expose the cache key on the document
directly (as an opaque pointer that you can't do anything with except load, of
course) and use this for "view source", "send page" etc. Relying on the URI plus
post data seems like a step backward.

Am I missing something?
I have the same concern as Stuart.  More specifically, hiding the cache key
should still somehow allow loading url "view-source:foo" and getting the source
of "foo" from the corresponding window from cache.
>Does this sound reasonable to everyone?
After the long wait, sounds like music to the ears :-) Although the reasonning 
so far throughout the bug has been that the cache key is "key" to fixing the bug 
as sballard has illustrated.
A second URL fetch performed with identical post data will not retrieve the same
cache entry unless a "cache key" or "cache token" (from the original channel)
was provided to the HTTP channel.  The original cache key can not be recreated
given an identical URL and identical POST data.

I'm not clear on why we want/need to hide the cache key.
One of the issues is that there is not a "single" cache key associated with the
document - correect?

there is a cache key per URI loaded for the document.  This means that to be
"correct" someone needs to hold onto *all* of the cache keys - including the
ones for linked CSS and JS...

Holding onto the cache key for the document URI will definately make things
"better" but i don't believe that it is sufficient to be "correct".

If these assumptions are correct, then I believe that someone has to hold onto
*all* of the cache keys...  And i'm not about to add an nsVoidArray argument
(for them) to LoadURI(...) :-)

I believe that we need to come up with a "correct" approach.

rick, 
when can you get:

   1. Provide a 'postData' attribute on nsIWebNavigation to 
   allow JS to access the postData.

on the branch?  lets make that happen and then we can take the pdt+ off
the this bug.  

vidur, 
can you add the bug numbers for the open bugs for making 
the chrome JS changes as a reference in this bug?
those also need to get pdt+




No issues with the branch which can have its quick fix -- but if it takes
adding a nsVoidArray argument for the correct fix later on the trunk, then there 
are 59 votes and XXX persons cc:ed here for that :-)
Can any netscape people comment on how 4.x handled this? (I never saw 4.x get it
wrong, although I can't say I tried very hard).

A single cache key would work fine for view source, at least.

Is there any way that the document itself can hold cache keys to the other
things that were loaded by it? That way everything gets cleaned up through
normal refcounting and we only need the single cache key to the document.

Perhaps the extra VoidArray should be added to the CacheKey object?
stop it... you all are hurting me :-) 

with nsVoidArray example rick meant to say that this is a bad idea... we clearly 
need to work a better solution than having to keep all the cache-keys (or any)

Lets stay focussed on resolving this for both short term and eventually with a 
better solution past this release. And rick: next time you suggest something 
like a nsVoidArray of cache keys (without sarcasm/kidding tags around it)... I 
am gonna come looking for you with my nerf gun :-)
gagan,

how many smiley faces do i need?  i always thought that one was enough :-)
but perhaps when the cache is involved i need a few more :-) :-) :-)

-- rick
once again i seem to have ignited a hot topic :-)

My main concern with exposing a cache key via the nsIWebNavigation API is that i
do not believe that it is sufficient to guarantee that *all* content will come
out of the cache.

I believe that we may need some other mechanism in order to "get it right" 100%
of the time.  So, rather than put the cache key there "for now" (and
realistically never revisit the issue until another PDT+ bug turns up) i think
that we should hold off and consider what the correct solution is - for the trunk.

I'll try to attach a patch for the branch (really soon) which exposes a
read-only attribute for the postData on nsIWebNavigation...   At least this will
allow us to *not* do the "wrong thing" :-)

Once the branch work has been done, i'll start looking into what the "right
thing" is for the trunk...

So, Gagan, get your nerf gun ready ;-)

I've just attached a patch which adds a read-only attribute to nsIWebNavigation
allowing access to the postData (if there is any...)

So with this patch, given an nsIDocShell you can QI to nsIWebNavigation and
check for postData.

When dealing with frames it is important to get the nsIDocShell associated with
the particular frame in order to insure that you are getting the right post data :-)

-- rick

As far as that goes...  we need to get frame docshells in response to context
menu actions.

At the moment when a context menu comes up we have the document involved
(this.target.parentDocument) and the window involved
(document.commandDispatcher.focusedWindow).

Neither of these allows easy access to the corresponding docshell.  One can walk
the docshell tree and for each docshell compare nsIDocShell.document to
this.target.parentDocument till a match is found (I have a function that does
this lying around).  Is there a better way?
from an nsIDOMDocument you can do the following to find the corrosponding
nsDocShell:

  QI(nsIDOMDocument) -> nsIDocument
  nsIDocument->GetScriptGlobalObject(...)
  nsIScriptGlobalObject->GetDocShell(...)

and you're there!!!

nsIDocument is not scriptable... I'm working in JS.
In terms of the question of how to fix it "right", my suggestion on 2000-12-28
was to make a separate cache manager instead of trying to adapt Necko to do it.
 This still seems cleaner to me; am I alone in thinking along these lines?
Stuart, as far as I remember 4x indeed did it right most of the time, but not
for the "send page".
Blocks: 91341
Blocks: 91342
The bugs to disable view source and send page on the branch are 91341 and 91342.
No longer blocks: 91341, 91342
The PDT+ on this bug on the branch is *just* for the patch necessary to make bug
91341 and 91342 possible (we've just PDT+ed those bugs.)  Can this be checked in
today?
Could someone review and superreview Rick's branch only patch? thanks!
Rick, we really need a way to get the docshell of a document from js. Do you
know of one?
r=bbaetz for the branch.

Having the input stream implement nsIRandomAccessStore doesn't help js, because
that interface isn't scriptable, and we need this for js. For the branch, blake
only needs to know if the data is post data or not, so thats OK. It needs work
for the trunk, and possibly a different solution.
The input stream is rewound at
http://lxr.mozilla.org/seamonkey/source/docshell/base/nsDocShell.cpp#4526 and
any loads that use the nsIWebNavigation APIs should be passing through there...

Code that creates channels directly instead of using the nsDocShell stuff will
have a C++ part that can rewind the stream.
Blake super-reviewed Rick Pott's branch only change and also checked it into the 
branch. PDT shd remove PDT+ from this as it is no longer a stopper for the 
branch.
Whiteboard: [Hixie-P1] PDT+ → [Hixie-P1] PDT+ branch fix is checked in.
hey blake,
jst and I talked about the best way to get an nsIWebNavigation from an
nsIDOMWindow...  and as you know, currently, there is no easy way.

The best solution we came up with was to an nsIWebNavigation as one of the
interfaces that you could get via GetInterface(...) this would require two small
changes in nsGlobalWindow.cpp

1. Add nsIWebNavigation to the GetInterface(...) implementation.

2. Change the ClassInfo flags so nsIInterfaceRequestor was visible to JS via
XPConnect.

Let me know if you think that these changes are necessary for the branch...  I
think that we will definately want to do this on the trunk... since we need a
way to map from an nsIDOMWindow to the embedding interfaces...

-- rick

Rick, thanks for the info. Last I heard was that we could live with it being a
problem in View Frame Source on the branch.

Lisa, does PDT want to pursue fixing it for view frame source too?

[removing PDT+; there's nothing that needs verification here anyways]
Whiteboard: [Hixie-P1] PDT+ branch fix is checked in. → [Hixie-P1] branch fix is checked in.
No way Netscape 4x did it right most of the time...
How many times did I curse Navigator trying to fetch the document again in offline mode for a printing or for a view source? And what about when View Source provided the source of the error page instead of the displayed one?
Maybe it got better with 4.75 or 4.76 but it was a very late effort.

Netscape 4.x doesn't do it right.  I just tested 4.76, and it's very broken. 
I'm sure 4.78 is much the same.

I've uploaded an attachment with the Perl code of the CGI that I used to test
with; it sends back the current date/time, the method used to call the CGI, and
a link to itself to trigger the GET method, and a submit button to itself to
trigger the POST method, so both can be tested.  If you wait at least 1 second
between tests, you can tell whether or not the CGI was reloaded.

Under 4.76 (Linux), here's what I found:

The GET method will perform the "View Source", "Print", "Send Page" and "Edit
Page" functions, but ALL of these will re-fetch the page and run the CGI again,
since the content has expired from the cache.  (If it's in the cache, it will
probably use that for some or all of these functions, but that's not correct
behavior either.)  All functions using the GET method will use the page, but
none of them use the copy being displayed in the window.

The POST method is even worse.  "Send Page" and "Edit Page" both reload the page
with a GET request (not POST), running the CGI again.  "Print" fails completely,
sending nothing to the "lpr" process, which returns the error "lpr: stdin: empty
input file".  "View Soruce" will bring up a window, but the source displayed is
that of the usual POST error message:

     <TITLE>Missing Post reply data</TITLE>
     <H1>Data Missing</H1>
     This document resulted from a POST operation and has expired from the
     cache.  If you wish you can repost the form data to recreate the
     document by pressing the <b>reload</b> button.

Basically, NONE of the functions in Netscape 4.x that should be using the
original source are correctly doing so.  When they manage to, it's only because
the content happens to be in the cache, and it could expire from the cache or be
replaced with a different version at any time.  For content that expires right
away, the ONLY thing that works on the original data is cut & paste with the
mouse into another window.  That's all.

Netscape 4.x doesn't even come close to getting it right.
As rpotts  explained earlier, I have attached a patch to nsGlobalWindow.cpp and
nsDOMclassInfo.cpp that exposes nsIInterfaceRequestor to JS. I need this
functionality for some other purposes too. 
Radha, don't expose new interfaces on the window object in JS unless it's
absolutely required, in this case it isn't. In stead, allow scripts to QI the
global object to this interface when this interface needs to be accessed. This
means that the window object in JS won't have a getInterface() method on it, in
stead you haveto do
window.QueryInterface(Components.interfaces.nsIInterfaceRequestor).getInterface(...),
this way we can support new functionality w/o polluting the global namespace on
web pages.

I'll attach a patch that does this part, your change to nsGlobalWindow.cpp is
fine, but don't check in the nsDOMClassInfo.cpp change in your patch. With my
patch (once tested n' all that) you'll have sr=jst
Re: jst's patch, I'm assuming that the new inclusion of
nsIXPCScriptable::DONT_ENUM_QUERYINTERFACE in the scriptable flags for Nodes,
Arrays, etc. is not going to break anyone.

sr=vidur
r=rpotts on jst's patch to nsGlobalWindow to expose nsIInterfaceRequestor via an
explicit QI(...)

and r=rpotts on redha's patch to add nsIWebNavigation to getInterface(...) once
the DOM_CLASSINFO_MAP_ENTRY(nsIInterfaceRequestor) is removed...
Vidur, the scriptable flags remain unchanged for Nodes, Arrays etc. (they
already have nsIXPCScriptable::DONT_ENUM_QUERYINTERFACE set), only the flags for
Window are changed.
The latest patch consolidates the previous 2 patches (patch ids 43271, 43347)
and a patch given by jband privately for a assertion problem. This patch
securely exposes nsGlobalwindow interfaces to JS.  will be checkin this to the
trunk. Reviews were given to individual patchlets.
I don't quite see how printing comes into this. Dosn't print work on the 
(currently displayed) DOM? IMHO it should, printing should be wysiwyg.
*** Bug 93157 has been marked as a duplicate of this bug. ***
Target Milestone: mozilla0.9.3 → mozilla0.9.4
*** Bug 93890 has been marked as a duplicate of this bug. ***
*** Bug 94417 has been marked as a duplicate of this bug. ***
What's the latest on this?  They want me to disable edit page/send page/view
source again for pages with postdata on the trunk unless this gets fixed the
proper way.
Depends on: 94205
Moving to nsenterprise+, adding nsBranch. I'm assuming the 0.9.2 branch fix 
hasn't been checked into the trunk as it wasn't the "complete, right fix."
Keywords: nsenterprise → nsenterprise+, nsBranch
Keywords: nsBranch → nsbranch
not happening for 0.9.4. some underlying work Rick is doing will get us closer
to this, but we're not there yet.
Target Milestone: mozilla0.9.4 → mozilla0.9.5
Blocks: 99194
Checkpoint for nsbranch . . .

Judson - Are we there yet on this one?
No longer blocks: 99194
Minusing this one per my email exchange with Judson. This looks like a good one
to get, but it is not gonna be fixed until later in TM0.9.5.
Keywords: nsbranch → nsbranch-
marking nsenterprise-.
Keywords: nsenterprise+ → nsenterprise-
*** Bug 96159 has been marked as a duplicate of this bug. ***
-> 0.9.6
Target Milestone: mozilla0.9.5 → mozilla0.9.6
Blocks: 104166
Blocks: 107067
Keywords: nsbranch-
Whiteboard: [Hixie-P1] branch fix is checked in. → [Hixie-P1] partial fix is checked into 0.9.2 branch
->0.9.7

bug #94205 will be landing soon - hope :-)
Target Milestone: mozilla0.9.6 → mozilla0.9.7
*** Bug 106931 has been marked as a duplicate of this bug. ***
Is there something corrupt in Comment #158 (from 7-20-01) that causes it not to
wrap in the Mozilla 0.9.6 window?  I don't know if this is a regression in
Mozilla (I doubt it) or something getting messed up in the buzilla database. 
Also, I apologize to anyone who thinks this is trivial; it is, but I'm hoping
it's easily fixed.

that was a bugzilla bug that was recently fixed i believe.
OK.  To summarize the current situation and get the ball rolling again.... At
the moment we can:

1)  Get the right docshell
2)  Get the postdata associated with the page
3)  Call loadURI with this info (bug 94025 has been fixed).

This is not sufficient for doing view source completely correctly, since there
could be multiple entries in the cache all with the same URI and same postdata
but different content.  It _is_ sufficient for fixing bug 64100.

Rick said:

> there is a cache key per URI loaded for the document.  This means that to be
> "correct" someone needs to hold onto *all* of the cache keys - including the
> ones for linked CSS and JS...

This is true in general.  In reality, however, we're dealing with the following
operations:

A)  save/save as.  This uses nsIWebBrowserPersist now anyway, so I am not sure
    this discussion applies anymore...  At the very least, this should be spun
    off into a separate bug on nsIWebBrowserPersist if it's still a problem.
    (bug 84106 should be retested).  Using the PERSIST_FLAGS_FROM_CACHE flag
    should have the desired effect here.

B)  Send page.  Is this going to send all the linked materials together with the 
    page?   (I sicerely hope this is not the case; munging urls in the source to 
    point them to the online content seems like a better solution, a la
    nsIWebBrowserPersist).  If it is _not_ sending the linked stuff, then we
    don't need a "whole page" solution.

C)  View (frame) source.  This could not care less about things linked from the
    page/frame. All it wants is the source for the page/frame itself as it came 
    off the wire.

D)  Changing character set.  This could actually benefit from the more gneral
    approach.

This seems to cover all the things mentioned in the bugs this blocks.

I propose:

I)  Bug 64100 should get reopened and fixed (this is not very difficult to do
    now, with the infrastructure we have in place).
II) Rick and company should decide whether we actually need a generic api for
    this (especially since intl seems to be opposed to the concept of fixing
    item D).

I'm getting very tempted to move view source into C++, from whence I could just
call nsIDocShell::LoadURI directly and not have all these problems.  The current
apis would allow this to work just fine.
*** Bug 118487 has been marked as a duplicate of this bug. ***
What about the milestone setting?  0.9.7 is long gone, are we going to have this
for 0.9.9 or 1.0??
*** Bug 55583 has been marked as a duplicate of this bug. ***
cc: self
I noted an interesting thing with the current implementation of view source.

It does not send cookies to the server?

I use cookies for session control. If you try to access the page
without cookie you get redirected to login page.

When I tried view source I got a blank page?
This should only be the case when you open without the session cookie?

I dont know if this is relevant or has any impact but I did not se any 
reference to this situation.

I do not have an test case at this moment but I can supply one if the need 
should arise.
Technically View Source should not be sending anything as is its dependency on
this bug.  View Source should only show what was received by Mozilla from the
server.  It should not cause a second request.
This might be redundant information, but I just can't pull myself to read all
the post. But it seem to me Internet explorer make heavy use of disk cache.
Efficient at it even.

When you view source and look at the file, its not a file in meory, but rather a
a physical copy. So can't we just stream the source to memory AND disk when we
loading a page, and view source just open the file at that particular time.
Correct me if I'm wrong, but Internet explorer's view source doesn't apply to
dynamically generated content. (So its not fetched from memory, but rather disk)

Someone know how opera handle this?
Mike: it is true that IE like mozilla will reuse content from the [disk or
memory] cache when it is available.  but, unlike mozilla, IE will generate HTML
when the [disk or memory] cache does not contain the requested document.

This bug is mostly solved already.  Mozilla now puts every downloaded document
into the cache, regardless of cache control headers (the headers are honored
when fetching the documents from the cache for normal page loads).

This does NOT guarantee that the document will exist in the cache when the user
views or saves the source of a document in a random browser window, but it does
pretty much guarantee that it'll work just after the document is loaded.

NOTE: i'm only talking about HTTP and HTTPS, not FILE or FTP.  though, i think
FTP could be modified to use the cache for downloaded material in a similar fashion.
Is http://bugzilla.mozilla.org/show_bug.cgi?id=120809 related?
An image gets refeched from the server every time, but definitely is
in the cache.
Blocks: 120809
darin: If I go to a page that is generated on the fly (e.g. slashdot), open it
in two windows (thus causing each window to have different content) and then
save both of these windows, will the saved copies be different?
you will save the most recently fetched version of slashdot.  is this a problem
we want to solve?  if so, then it means pinning documents in the cache, which
sort of goes against the idea of having a fixed sized cache.  i'm not saying
it's impossible... indeed, the cache provides facilities for pinning an entry in
the cache, but it certainly is not without its tradeoffs.
My opinion is that "yes, it is a problem we want to solve". I've got some CGI
scripts which generate oodles completely different types of content under
various circumstances. While debugging, I'll often look at the sources of
different outputs (using some non-Mozilla browser).

perhaps Yet Another Pref?

"Save raw HTML source in cache to avoid new requests when using the view-source
command."

Turned off by default, easy for web developer types to turn on.
Only folks who want the feature have to deal with the tradeoff.

-matt
Yes, it is the problem we want to solve, but No, it should not be a pref. This 
is what you would expect to happen, that the page you look at gets saved. If we 
don't pin it there is no way to be sure that that happens.

IMHO we also want to pin it for at least three other things; view-source, send-
page and history. Pinning the htmlsource should not be that expensive, how many 
html-pages does a "defaultinstall doing normal browsing" contain approx? I have 
171 files in my cache, though i guess some of them are images, i would happily 
waste 10-20 of that on pinned down pages
I might be naive, but when I say "save page as" or "send page", I mean "do it
with the stuff I am seeing on my screen right now", and it is very disturbing
for me to see a refetch when I do it.  Many pages are relatively small and
static, most servers are very responsive and many people have good connections,
but this is not always the case, so IMHO, treating a refetch as a Very Expensive
Operation and doing it only when absolutely necessary or when explicitly asked
for is The Right Thing.
As I understand it, we _do_ want to avoid having another refresh. Unfortunately,
that _doesn't_ necessarily mean "the contents on your screen right now."

Why? Because you can change the stuff on your screen. Fill in forms, execute
javascript, whatever. Things of that nature should, IMHO, not be reflected in
"view source," "save page as" or "send page". But what about changing the
character set? (that's bug 17889 btw) Changing the charset shouldn't cause a
page to reset.

If we do want to support a save/send page option which maintains the current
state of the page, it should be considered a different feature and not interfere
with the current effort to implement the current features. 
I don't think that saving the page EXACTLY like it looks now is what anyone
wants. The ideal situation would be to save the HTML document to the cache as
the server gave it to you, and then also save all the other files the server
sent over that the document needs to render correctly, such as StyleSheets,
images, Java and other ebedded things (Flash, etc)

That way, whenever you want to either see the source, or save the entire page
(HTML+dependencies) etc it is all already there, moz would just need to copy it
elsewhere. The only reason that moz needs to refetch anything from the server is
if the user hits Reload or the cache TTL expires. Or if the meta refresh is set,
of course.
-- #197 From Darin Fisher 2002-01-27 16:14 said: -------
> you will save the most recently fetched version of slashdot.  is this a problem
> we want to solve?

  YES.  View Source, Save As, Print -- they should ALL act on the presently
viewed page.
  In the case of view source, we should always view the html for the page that
is displayed in the window.  If this means that I have 10 browser windows open,
all which requested some http://something/time.cgi URL (which outputs the time
of day in html) at slightly different times, I should get 10 different view
sources.  Always.  This is vital to being able to develop content for Mozilla,
because all major server-side development these days is dynamic, and it is
*categorically impossible* to test or debug dynamic content with Mozilla if it
does a refetch when the user hits "view source."

  The same goes for Save As and Print from a user's perspective.  If I just hit
the "submit" button to place an order, I don't want that posted again if I want
to print the invoice page or save it to disk (which would place another order).
 And if I choose to place two different orders to the same site in two different
windows (sequentially), and then save/print each invoice after each has been
placed, it should "do the right thing" and save/print two different invoices.


------- #201 From Nick Lewycky 2002-01-27 18:14 said: -------
> As I understand it, we _do_ want to avoid having another refresh.
> Unfortunately,
> that _doesn't_ necessarily mean "the contents on your screen right now."
> 
> Why? Because you can change the stuff on your screen. Fill in forms, execute
> javascript, whatever. Things of that nature should, IMHO, not be reflected in
> "view source," "save page as" or "send page". But what about changing the
> character set? (that's bug 17889 btw) Changing the charset shouldn't cause a
> page to reset.

  Filling in forms doesn't change the source html.
  Executing javascript, in most cases, doesn't change the source html either
(document.write() is the only exception I can think of, and I believe that is a
microsftism), so the browser should just show the original html as sent by the
server for save/view source.

  But this does bring up an interesting print problem... Printing should, IMO,
show the current form contents, as well as any dhtml changes via javascript,
etc.  That information is not contained in the source html.
From previous discussion, I understood that Print worked with the DOM representation of the page anyway. So it's probably Ok already. Who wants to test printing a filled form?
Hey, what's wrong with the formatting of my previous comment?
Is it possible to edit a comment after posting it?
> Print

Not relevant.  Printing prints the DOM, not the source.  So you always print
exactly what you see.

All the concrete examples people have given so far of cases where they would
want to see separate source for the same url involve post data that was sent to
the URL.  In those cases, we _do_ cache each response separately.
The view source problem isn't limited to post data sent to the URL.  A
"timeofday.cgi" is a good example of a non-post request sent to a URL which
returns dynamic content, and the "view source" should be different for each
browser window/history/tab.

This isn't a contrived example.  JSPs work this way as well.  As do ASPs.  As do
PHPs.  etc.  URL accesses may cause the server to generate and send dynamic
content /without/ sending state information via (get/post/cookies/URLrewrite)
which must be cached for view source.
(I got a midair collision with somebody saying the same thing as me; this is in
response to comment #206)

Not at all; it applies to *any* site whose content might change over time. If I
visit slashdot.org at lunchtime, leave the window open, do some other browsing
in other windows, including visiting slashdot.org again later with new stories,
then go to view source of the original page, I expect to get the source of that
page, not the source of slashdot.org at some arbitrary later time that I
happened to view it.

From a user's perspective, this is basic functionality. If the page is sitting
right there in a window, why should it be impossible to view its source? I do
understand the implementation reasons why it's hard, but it's also hard to
implement a functional CSS engine and HTML4 viewer - that's not a reason to
choose not to do it, or we'd still be using MozillaClassic :)

It should *never* be possible to have a page in a window and not be able to view
source, or send or save the page, and get any result other than the exact
content of the window that is on your screen. Even if the page specifies
no-store. That's what users expect; it should be what we do. No exceptions.
i think one point needs to be made clear:

it is trivial (modulo some API issues) to fix mozilla to not hit the server for
view->source and file->saveas nearly 99% of the time (including POST results).
it is far more complex to ensure that this is correct 100% of the time, and it
is also far more complex to ensure that view->source and file->saveas correspond
to the page being viewed.

of course, i understand why solving this is important, and i would love to see
it solved.  i also think that we have to be careful when we go to solve this
problem, since there is a bloat factor to consider... even if it wouldn't be an
issue for the majority of users, we still need to think about people with
systems "circa" 266 Mhz.  moreover, some users disable their disk cache.  we
need a solution that works (or can be disabled) when there is no disk cache.
Blocks: 118487
Another thing I wanted to mention is "back" (the leftmost button in the toolbar).
moz treats "back" as a _refetch_ (unless it's an anchor in this page).
IMO this is wrong: "back" means "give me what I already saw", not "refetch".
Again, treating a refetch as a Very Expensive Operation and doing it only when
absolutely necessary or when explicitly asked for is The Right Thing.
sam: back currently loads the page from the cache.  it is true that if that page
is dynamically generated and has been more recently visited and hence modified
that you will not see what you originally saw.  fixing this would be difficult
and probably should not be done as it would result in some serious bloat,
especially for SSL pages which cannot be cached on disk for security reasons.
Does the cache not have a (timestamp visited), (timestamp last modified) for the
entry?
yes it does, but how would that help this situation?  sure it would allow you to
determine that the page in the cache is not the same page as you previously
visited, but then what?  you'd still be without the original page, so why not
just show what you've got in the cache.
What your looking as it the need to cache everything but (invalidate/show/hide)
the cache on specific requests (or specific components.)

It's agreed that Print, Save As, View Source and some other commands do not need
to reload the URL from the network.   One option to alleviate this, is to cache
all requests in both digested and raw formats regardless, but make them visible
only to specific services (Print, Save As, View Source, etc.) The "stealth
cached" items expire and are replaced/expired as a normally cached item (perhaps
with a rule always cache).  By using this method, you can have one efficient
cache for the system.

* Cache everything
* On cache request, determine visibility to decide if you should return item is
not in cache (even if it is) and must be reloaded from the network.
** If do not cache pref is set, then the item isn’t visible in the cache (to the
browser) and must be refetched.
** If Print, Save As, View Source, all cache is visible always.

This would involve some checking between components to see who is authorized to
view the stealth cache, but I think it would improve performance in addition to
eliminating double posts and gets.

These are just suggestions and probably nothing new, but they might help in the
discussion for a direction to go...
thomas: most of what you've described, mozilla already does (modulo some bugs).
 it achieves the effect using load flags.  there are three principle load flags:
LOAD_NORMAL, LOAD_FROM_CACHE, LOAD_BYPASS_CACHE.  clicking on a link or entering
an address in the URL bar sets the LOAD_NORMAL flag.  file->saveas,
view->source, file->print set LOAD_FROM_CACHE, and shift-reload sets
LOAD_BYPASS_CACHE.  i should really say that this is how mozilla is designed...
there are bugs and mozilla may not set the load flags correctly in all cases.

what mozilla does not do is "pin" entries in the cache.  this means that a page
you are viewing could be evicted from the cache if you spend sufficient time
browsing in another window.  as a result, file->saveas, etc. may require a
server hit.
if(cached(current_page)) {
&nbsp;&nbsp;&nbsp;display_source(cache(current_page));
} else {
&nbsp;&nbsp;&nbsp;echo "Sorry, the page in question was not found in the cache,
click here to re-retrieve this page";
}


If the a refetch is required, the user should be well aware of it (and mozilla
should ask permission), period.  Else I assume that's the source of the current
page.

This is just my take/solution on the whole thing, and what I expect should be
done, obviously the real code will probably be quite a bit more complex than my
lame psuedo code.  :)

Is it really that unacceptable to say "sorry I couldn't view the source, here's
why:"?
steve: you have described exactly what mozilla already tries to do (again,
modulo some bugs).
i think that a seperate bug should be filed to lock a window's history in the
disk cache, but i feel that full source versions of pages should not be stored
in memory. if someone decides to turn off disk cache it should be acceptable to
require a refetch from the server on view source.

what people are attempting to add into this bug would be best implemented by
changing the cache system--anything else is just a hack. i don't think anyone
wants to see the cache system extended this close to 1.0.

file a bug and dep it on this one. i see wins here if the cache is modified to
provide for a true history, especialy since the cache's traditional role is all
but obsolite due to dynamic pages and fast connections.

this really should be discussed in a news group.
darin@netscape.com 2002-01-30 11:54 wrote (#215):
> what mozilla does not do is "pin" entries in the
> cache.  this means that a page
> you are viewing could be evicted from the cache if
> you spend sufficient time
> browsing in another window.  as a result,
> file->saveas, etc. may require a
> server hit.

  The only problem with this, and I don't mean to harp on this point if it is
already abundantly clear, is that Mozilla's DOM is effectively a "memory cache",
and it does get out of sync with the real cache.

  If I can see the page in the mozilla window, it is only reasonable to expect
that I can view its source, save it to disk, or send it in email, WITHOUT
requiring the browser to get it from the server again.

  My humble picks from the best solutions yet suggested:

  1) Sync the caches.  Sync the caches.  Sync the caches.  Either (a) purge DOM
from memory when a page is purged from disk, or (b) pin entries in the cache. 
(a) is the same fix as popping up a "page is gone" requestor, but doesn't leave
the user scratching his head wondering why in the world Netscape's engineers
can't save the page to disk if they can render it right there on the screen,
because it will no longer render to screen.  If it's gone from the cache, it
can't be rendered without a request.

  or

  2) add yet another cache that mirrors the DOM, saving the pure HTML.  Better
yet, attach it somewhere as part of the DOM.  This eliminates the sync problem
because if Mozilla every goes to the cache, it will reconstruct the DOM from
(and with) the html.

  IMO, 1 is the best "fix it now!" solution and 2 is the best "real" fix, even
though it is either ugly or less memory efficient.  If I'm speaking out of my
****, forgive me.  Trying to help.
> purge DOM from memory when a page is purged from disk

You mean blank out the page the user is currently viewing?  The only place the
DOM is "cached" is in the actual view the user has.  As soon as you view
something else the DOM is destroyed....

> If it's gone from the cache, it can't be rendered without a request.

Correct.  The if you hit "back" to go back to a page that's gone from the cache
you will get a request (and an alert).  The problem is when the source has
already been rendered.

OK. And then we can come to conclusion that the only correct solution is to hold
a direct cache key to pinned cache entry for every dom which is held in mozilla.
But this was already suggested.

If i try to get back in history to other page and it's cached I load it and then
I get a new direct cache key. So there is no need to mess up with direct cache
keys and history. Certainly this behaviour is not perfect but it can't be
without other tradeoffs.

Note that there are other bugs where some people want to have cached the entire
DOM for purpose of page history. Then in this DOM history should be cached the
cache keys for sources.

Sorry for the unconstructive spam but...
If the only way to fix this is to modify the cache to allow "pinning" pages in 
the cache, we've had over 1.5 years to realize this and so I *do* think Mozilla 
1.0 should wait on it.

Also, at this point there have been SO MANY examples and counter-examples that 
I don't remember what works and what doesn't.  The basic idea seems to be: the 
original HTML for whatever is displayed (in a Mozilla window) should be 
accessible to various functions (Print, Save As, View Source, etc.) without a 
refetch.  Is this not correct, i.e. do these functions already work as I just 
described?

It seems that we are down to 2 areas of misbehavior:

1. GET requests that have been evicted from the cache will hit the server
*without* asking the user first.

2. POST requests do not work.

To address these problems i'll expose some kind of nsIWebPageDescriptor
interface that allows access to the following information:
  a) URI
  b) PostData
  c) Cache key

There will also be a LoadURI(nsIWebPageDescriptor) -- somewhere...

Additionally, i'll make sure that if the 'load-from-cache-only' load flag is set
on the view-source channel (which it should be) then the user is prompted BEFORE
a request is made to the server (in the case where the page has been evicted
from the cache...)

This is basically what boris proposed way back when in comment #186 ;-)

-- rick

If the majority of dynamic pages contain a "no-cache" meta tag, does the
"cache-pinning" solution mean that we're back to square 1 for developers wishing
to see their outputted page source, and for anyone wishing to save-as or send a
dynamic page (e.g., an invoice)?
Question: "Pinning" refers to preventing the expiration from cache of any 
element currently displayed in a browser window?  If not, what exactly does it 
mean?

Al: 'no-cache' documents are still put into the cache.  they are there for the
purposes of view->source, etc.  normal page loads of a 'no-cache' document will
bypass the cache.
How about no-store documents? Is there any way to ensure that those get pinned,
say, only in the memory cache - so that they can still have view-source done on
them, but will never ever enter the disk cache?
stuart: yes, even 'no-store' documents are cached (in the memory cache only) for
the purposes of view->source, etc.  this was only recently changed to be this
way... previously 'no-store' documents were not cached.
*** Bug 120457 has been marked as a duplicate of this bug. ***
darin: Would it be possible to associate a particular cache entry with each
document currently on the screen, and (through the magic of reference counting
or something along those lines) guarentee that that cache entry not be
overwritten so long as the relevant page is open? That would mean that the cache
could end up containing three or four copies of a page with the same URI, so
that view source would work (you would use these cache entries to load the page
in view source or save as); as soon as those windows got closed, the entries in
the cache would be deleted (or marked as out of date or whatever). In the real
world this would not cause much bloat (how often do you visit the same page in
multiple windows?) but it would solve the view-source-of-the-same-page-multiple-
times problem.
hixie: the cache already provides support for exactly what you're describing. 
we'd just need to hook up docshell and friends to utilize it.  there are issues,
of course, that need to be resolved... consider the hypothetical example of an
ever increasing number of windows (eg. you've just visited an evil website)...
at what point does the cache refuse to hold items in the cache?  does it ever
refuse?  does it honor or ignore its size limits?  it seems to me like we need a
solution that allows the cache to honor its size limits, otherwise we run the
risk of serious meltdown resulting from a malicious website.
Also, the cache would only hold multiple entries for the same URI if HTTP
thought they resulted in different documents.  Otherwise, the reference held by
each window points to the same cache entry.
I'd venture to suggest that if we refuse to hold the source in the cache, we
should refuse to open the window ( / display the page), period.

My suggestion would be to allow the cache to expand a limited amount over it's
official maximum size for this purpose (eg 10% more, with a lower limit to
handle the case where the cache is set to zero) and if loading a page would
break this rule, refuse to load the page.

The really difficult case would be what to do with a huge (bigger-than-cache)
file being displayed in a pre-existing window. I'm not sure what we should do
there - either truncate the page, or make an exception to the cache-size rules
for a single page that's bigger than the cache for the duration of time that
it's being displayed in a window. Either way, I don't think we should *ever*
break the contract that if we can display something in a window, we can provide
it's source on demand exactly as it was received from the server.
Darin:

A malicious website could exhaust the system memory just by launching a ton of
windows, but the problem would be escalated by the additional cache footprint if
pinned entries in the cached are never released as well.  I think the overhead
of the windows would have more of an impact.

FWIW, the cache should honor its constraints. I don't think the cache should
ever start refusing to add entries, but it definitely should start expiring
entries even pinned entries when the situation you described exists.

Could there be a warning or a notice to tell a user that they may want to
increase these options if the pinned cache size gets to close to the limits?  
Darin:

A malicious website could exhaust the system memory just by launching a ton of
windows, but the problem would be escalated by the additional cache footprint if
pinned entries in the cached are never released as well.  I think the overhead
of the windows would have more of an impact.

FWIW, the cache should honor its constraints. I don't think the cache should
ever start refusing to add entries, but it definitely should start expiring
entries even pinned entries when the situation you described exists.

Could there be a warning or a notice to tell a user that they may want to
increase these options if the pinned cache size gets to close to the limits?  
thomas: and if we're talking about the memory cache?  HTTPS only goes into the
memory cache... also, what if the user has disabled their disk cache?  then,
currently, only the memory cache is used.  meltdown is more serious and more
common if there are not hard limits on the size of the memory cache.
> My suggestion would be to allow the cache to expand a limited amount over it's
> official maximum size for this purpose

Currently-open document source (not DOM) should NOT count against cache. It's a
functionality issue, not performance. If it's not available, Save and view
source BREAK. Not performace "oh it's slow" break... dataloss "#!&%$" break.

Save and view source should work perfectly if the user has 0 mem and 0 disk
cache. Think lower-end machines with a network wide cache on the LAN. Why waste
seperate resources on the local machine. But certainly those users should still
expect save and view source (etc) to work.

Or think power users that know HTTP caching. Even they won't understand what a
DOM cache is, and why what they see they can't save.
> we should refuse to open the window ( / display the page) 

Let's not lose sight of things here.  This is a browser.  Its primary purpose is to display pages.  All else is secondary and, ultimately, unnecessary.  We should display the page no matter what.  If we can't show the source for a particular page we are displaying we should just say so ("The page has expired, ...."). 
I wouldn't go so far as to say non-browsing features are "unnecessary".  After
performing some sort of financial transaction via the web, saving the resulting
confirmation/tracking page can certainly be considered necessary.
Keywords: topembed
There has been a lot of discussion here about how to handle this bug.  I am not
sure that this is the appropriate place for discussion, but since everyone else
is doing it, I guess I will too.  :)

The end users expect to be able to be able to save, print, or view source of any
web page that they are looking at.  Any other behavior should be considered a flaw. 

In order to accomplish this, we need to be able access the source for every
document currently in any open window or tab. 

The consensus seems to be that the fix is to pin the source for each currently
open document in either memory cache or disk cache.  The debate seems to be
about how to handle the various scenarios presented by the fact that memory
cache and disk cache are both limited resources and too many open documents
being pinned can potentially cause Mozilla to deplete one or both of those
resources.

To address the potential issues, I propose the following (*see footnote) :

A) If disk cache is disabled :

1) The source for all documents currently in an active window or tab is pinned
in memory cache until memory cache is exhausted.

2) When memory cache is exhausted, prompt the user that memory cache is running
out and give 2 choices: a) close some open windows or tabs in order to free
memory cache, b) enable disk cache.

B) If disk cache is enabled :

1) Store and pin open https document source in memory cache.

2) Store and pin open non-https document source to disk cache.

3) If memory cache is exhausted by pinned https document source, prompt the user
that memory cache is running low and give 2 choices: a) close some open windows
or tabs to free memory cache, b) break security and enable temporaily enable
disk cache for https document source.

4) If the specified disk cache limit is reached, prompt the user that the disk
cache limit has been reached and give 3 choices: a) closing some of the open
windows or tabs, b) temporarily allow the disk cache limit to be exceeded, c)
increase disk cache limit.

C) Alternative :
1) Automatically store and pin non-https document source to disk even if disk
cache is disabled or the disk cache limit has been reached.  

2) If disk cache is disabled, these document source files can be deleted from
disk as soon as the document is no longer in an open window.  

3)If disk cache is enabled, these document source files just become unpinned
cache files and are handled accordingly by the disk cache.
 

This alternative method of addressing the issue can be a preference settting so
that users can avoid almost all of the prompts described in the previous method
(the exception being when memory cache becomes full of open https source). 

The only real danger of this alternative method is in the case of the malicious
website that opens an infinite number of windows and thus fills the user's hard
drive with pinned source files, but we can just file a different bug for that,
right?  :D



(*Note: I am by no means claiming that this solution is all my own original
idea. Obviously others have previously presented many of the aspects of this
solution; I am only trying to summarize and organize it all into one concise
description of a solution.  If anyone can think of a scenario that I have not
addressed, please let me know --preferably via email.)
I'm going to put my two cents in.  I'm not a mozilla developer but instead an
end user.  I develop web sites and thus have an understanding of what I expect
of mozilla based on this.  I don't understand a lot of the technical debate
regarding this bug and I don't have time too.  What I do know it what I expect
the browser to do.  Maybe by presenting my expectations you guys can figure out
how to 'make it so'

When I view the source of a web page, I expect the browser to show the source
for that page.  If I've got two or three different 'versions' of the page open,
then the source for each version should be correct for that version.  So if I'm
working on a CGI script and I make a change to the script (but haven't refreshed
the browser window yet) I expect to see the original scripts results, not the
changes I've just made.  NOTE: If I want an up to date view of the latest source
I will refresh the page before viewing the source. If I haven't reloaded the
page, then I want to see the source for the un-reloaded page, not the new source.

The same goes for printing, or any other aspect of the page.

As I said, I don't understand a lot of the technical aspects of caching and
what-not.  However I do expect the browser to act in a predictable way.  This
means that when i view source, or print, I get results based on the page as it
is, not as it may be now.  this means that if I have numerous versions of a page
 in various windows, and I have made changes to the underlying file, then each
page should be different and reflect this in the source, etc.

Hope this helps ;-]
Actually, now that I think about it a little more, this is what I think:

The "cached" copies of documents that are *currently* being displayed in
currently open browser windows (or tabs) should *not* be counted towards the
total size of the cache, whether it be memory or disk. In other words, the
maximum possible size of the cache should be the maximum size specified by the
user *plus* the size of all documents currently open in browser windows.

As soon as a window is closed, or the user browses to a different document
(causing a document to no longer be displayed in an open window), its size
should start counting towards the total cache size; at this point, therefore,
older entries should be evicted in order to bring the cache down within its size
limits.

This works out right if the cache is disabled, too: then the default cache size
is zero, so as soon as a document is no longer being viewed, it will become the
oldest document in the cache and be immediately evicted.

At first glance this seems like weird behavior (or at least it seemed weird to
me) due to the timing of when things would be evicted. But actually it's exactly
the behavior you would expect if you treat the "currently-viewed cache" as
completely distinct from the regular cache. As soon as a page is no longer
currently-viewed, it moves into the regular cache, and that has the same effect
as what happens today when a page is put into the regular cache.
> should *not* be counted towards the total size of the cache

The problem with this is that users limit cache sizes for a reason.  I have a
limited quota and I limit my cache such that cache will not push me over quota. 
If we go over that limit we better be doing it in some temp directory somewhere
and not in the profile....
FWIW, I agree that currently displayed elements should not count towards the
cache limits.  As far as most users are concerned, a cache is where *done* items
are stored for quick retrieval.  I also think that the common user would
*expect* a browser to fail gracefully and refuse to open new windows when, as
they see it, "their system's memory has been taxed to the limit."

In other words, don't be afraid to present the user with a refusal to perform. 
They are more likely to blame it on their computer, its lack of memory, or slow
processor, :-| than on mozilla itself. (And, imho, rightfully so!  It isn't
mozilla's responsibility to make sure there are enough resources to open yet
another item.  That duty belongs to the user or, more to the point, the user's
pocketbook.)
> I have a limited quota and I limit my cache such that

This is nothing to do with disk quota. What I (and about 3 other people now)
propose is keeping it in core. If it uses the normal memory cache mechanism,
fine. If it's seperate, whatever. Once the user leaves the page then Mozilla
will decide whether to keep it in memory or disk cache, or neither.

Users expect the more pages the have actively open, the more memory usage will
rise. This is how any other app works, cache or no cache.
just to bring this discussion back on track

 > consider the hypothetical example of an
 > ever increasing number of windows (eg. you've just visited an evil website)...

i just tried this out to see what currently happens in this situation, mozilla
crashed with 146 pages open (using 100% of cpu and all ram - 384meg). so
basicaly mozilla is going to crash or the system is going to become
<b>completely unusable long before the size of total pages open exceeds size of
cache</b>.

it would take 500 - 1000 pages to exceed the default disk cache size of 50000k
so this should never become an issue.

so all that needs to be done is to 

 > hook up docshell and friends to utilize it.


So I think that (per benm comments) we shouldn't try to put resource control 
into cache management routines (resources in the global meaning, that is,
utilization of machine's disk and CPU resources by Mozilla), and in this bug not
worry about this again.
A separate bug should be filed about implementing in the future some sort of
global resource control in Mozilla that would prevent /Java/Javascript/Flash
plugin/ any other plugin/ from executing DOS attacks that lock up client machine
(eg. by JavaScript opening too many windows, or injecting too much HTML code
into the document). This feature would impose dynamic limits on memory
allocation, cpu utilization, number of opened windows, etc. The limits would be
dependent on amount of recources available on the particular machine, would be
calculated dynamically and the user would be notified gracefully if the limits
were exceeded.
This issue is _separate_.
Please someone qualified (preferably someone who knows a proper component owner
for that :-)) file a separate bug.

The limits on cache size should only pertain to content that isn't displayed in
any window (that is, to content that is not pinned).
This may be difficult, but a browser isn't made in 6 days, especially one that
is implemented correctly :-)
benm, olo: we should not try to design a system that makes this worse.  i don't
understand why current problems w/ many windows is justification for not solving
this bug correctly.  we should care about enforcing cache limits... the cache
limits are a user preference.  you may argue that they are not your preference,
but then that is yet another preference.  IMO we should strive to come up with a
good compromise solution to this problem.  we shouldn't race to solve this bug
by pushing out any concern of cache limits.
darin: those of us proposing "ignoring" cache limits for currently-visible files
are not proposing that the cache limits be weakened, but rather that the
*meaning* of "cache limits" be redefined. When I wear my "developer" hat, I
understand the reason why you might want to consider a currently-being-viewed
page as part of the cache.

But when I put on my "user" hat, I don't think of it as part of the cache at
all: the cache is an *optional* feature, which can be disabled without hurting
browsing. It stores *recently accessed* pages, for ready availability if I
should view the same page *again*. From a user standpoint, even if not from an
implementation one, this is significantly different than storing the *currently
visible* page for ready access if I need to *do something else with that page,
such as see its source*.

So my position is that as a user, I would not expect the cache limits to include
a limit on the number of pages I can keep open at once without losing
functionality like "view source". When I, as a user, set a cache limit, I'm
making a tradeoff of memory/disk versus performance, not memory/disk versus
incorrectness.

The difficulty comes when the disk cache comes into play, and the user specifies
a location for that disk cache. By my theory, "currently viewed pages" should
not be stored in that location because they do not count towards disk cache
size, and the user might limit their cache size based on a known amount of free
space in the cache location. On the other hand, it is vital that moving items
from currently-viewed to cached (and back, if necessary) be an efficient
operation, which may not be the case between two different cache locations,
especially if they are on separate drives.

The only easy solution I can think of would be to ensure that currently viewed
documents only get stored in memory, not disk. But that adds to bloat. I guess
the UI could give a disclaimer that the cache directory will also be used to
store currently viewed pages and therefore will take more space than the limit
you give it?
sballard: i agree with your conclusion... clearly it is bad to store stuff in
the disk cache w/o counting toward the total size of the disk cache.  and
clearly storing in-use pages only in memory is bad too because of the bloat that
would incur.  and, i too agree that it would be very nice from the point of view
of the user/webdeveloper to be able to count on view->source and file->saveas
always giving me what i am looking at.  i'm not really arguing against that as a
goal.  instead, i'm just saying that we need to weigh the cost of doing so, and
consider that it might end up breaking down at some point.  that there might
need to be some threshold at which we simply cannot guarantee that the source
for an in-use page is accessible on the local system.  we should be able to set
that threshold high enough so that 99% of the users will never know it exists. 
i'm suggesting that a threshold exist and that it be configurable.
How about a pref that goes with the caching prefs, and looks like this:

"Allow [ 10]Mb of additional space in the disk cache for pages currently being
viewed (necessary for "view source" and "save page", among other things)."

Ideally, that number would default to something nice and high, but the option of
setting it to "unlimited" should also be available (I'd certainly choose that,
as a web developer). Perhaps:

      ( ) Unlimited
Allow                additional space in the ... (as above)
      (*) [ 10]Mb of

This wording would clearly indicate that this amount would be *additional* to
the amount already specified in the disk cache, so users would know to make sure
to allow enough extra space for it on disk.

If a user reduces that number, they can't complain if "view source" starts
working wrong. But if they specify "unlimited", it should be absolutely
impossible to get into the state where a currently-visible page cannot have its
source viewed or saved (correctly!).
i'm going to put on mpt's hat 'no more prefs'

you have 3 prefs (mem size, disk size, when to check for new objects), they 
should be enough.

I will ask one question: What do you expect mozilla to do when it encounters a 
single page that requires say 1gb of memory to render?

I have two versions of the page on my system, one with local images, and one 
with remote images.  I've loaded the page on things ranging from 24.0 dialup to 
a fast university net connection.  [My system has never had more than 1/2gb of 
physical ram]

If I were to load that page and leave some other windows open, should i expect 
mozilla to keep around the source and images for all my open windows?
timeless- Yes.


and give the user an enormous big red button that says 
"due to low system resources you will not be able to both load this entire page
and reliably use your other windows" 
[close other windows] [do my best to keep them all]
A couple questions for timeless -

After rendering those pages in a current version of Mozilla, does it do a
refetch of the source in order when you request to view source?

If you look at those pages in another browser (IE, Opera, whatever), are you
able to view source without a refetch?

And some comments for everyone -

The way I see it the primary issue of concern in this bug is about the text
source of the pages.  If Mozilla were to pin the text files in cache and not pin
any of the images, flash, sound, etc then that would resolve most of the
problems we currently face now and since the text source files are generally
fairly small, we are not likely to have the cache limit exceeded by pinned
source unless disk cache is disabled or set to a very low limit.

In regards to viewing or saving source, pinning only the text files would make
it work the way it should.

In regards to saving a whole page (text source, images, and all other
components), there could still be some problems with dynamic content if
everything did not get pinned, but we would not be any worse off than we are now
(in regards to saving the whole page correctly).

As for printing, I am not up to speed about how printing works now, does it
depend on refetching or cached source or is it entirely independant of this bug?
 I personally have not had any problems printing pages correctly lately, but I
can not say for sure if that is due to recent printing code changes or just good
luck.


It has been said before but I'll say it again:

Printing is NOT affected by this bug in ANY way. Printing works on the DOM and 
doesn't use the source. So printing never refetches from server even if the 
source of the page has disappeared from all caches. It also works perfectly if 
you have 10 pages from the same url open and they all look different.

If you don't like this behaviour then PLEASE don't argue about it here, this 
bug is spammy as is anyway. File a NEW bug and present your arguments THERE.
what are the chances of getting a fix for this before 099?
Keywords: nsbeta1
No longer blocks: 107067
> pin the text files in cache and not pin any of the images

How do you make this distinction?  What about SVG or other XML-based image
formats?  Those are definitely "images" and they definitely have a "source".  (I
can see not pinning things referenced by <img> tags or something, I guess).


Just something that should be kept in mind, there is such a thing as
dynamically-generated images -- think weather maps, for example.  So the "don't
pin images" idea will not be bulletproof.  Whether or not it's an acceptable
tradeoff I couldn't really say.
excuse me sticking my oar in as well, but at least one bug depending on this
assumes this will fix images, not just text files.

btw i'm not quite sure what's going on with the dependencies - bug 118487 is
dependent on this bug, but that is marked as a duplicate of 115174. shouldn't
someone fix it so the original is the dependent? bug 120809 is the other one
that wouldn't be fixed with a text-file-only pinning.
It's simple: pin the source text of the displayed URL.  If you open an SVG image
in its own window, then of course you pin its source.  On the other hand, if the
image is merely linked in the href of an HTML page, you don't pin it.  The same
applies to linked scripts and stylesheets.
> if the image is merely linked in the href of an HTML page, you don't pin it

This would definately be unacceptable. EVERY object on EVERY open page (windows
and tabs) needs to have its original data available in system memory. Anything
that can change by a DOM or what not would need a seperate copy that I presume
would be created on first-change.
_Why_ do you want every object to be immediately available?
I think it's enough to have every object that can be directly
saved/view-source'd (e.g. images, iframes, normal frames, the document itself)
available.

Things like .css or .js files seem to be unnecessary to me.
> Things like .css or .js files seem to be unnecessary to me.

"Save as..." -> "web page complete" needs them. We should be able to load a
complex page with all kinds of dynamic content and crazy nocache directives,
lose the network connection, and still be able to save the whole page.
This bug's summary is "need means to reuse/reload current page. . . " not
current image or current JavaScript or otherwise. In reality, everything on the
web is static, except for numerous HTML entities and some images. To fix this
bug, we have to focus on reusing the already downloaded HTML file. 

There is no way we can recreate an HTML file from the docshell exactly as it
was. For "view source" and other operations, however, that is the requirement. 

The simplest and most effective solution is to keep a copy of the current HTML
file in core. There would be no error messages, no cache keys, no pinning, and
no prefs. It would cost memory, but sometimes you have to bite the bullet. For
every open window that is displaying an HTML file (or files, if there are
frames), Mozilla should store the HTML file(s) in core. As a later enhancement,
we could gzip the HTML files so they will consume less RAM.

I assume that after Mozilla downloads an HTML file for display, it generates a
docshell, then it tosses the HTML file away. My suggestion is to not toss the
HTML file away until the window closes or the window displays some other page.

This will be hoggish when users start downloading 2GB HTML files every day, but
that hasn't happened yet to any degree of freqency. Should that become common,
we can address that problem at that time. Today, however, storing the HTML in
core makes sense. Take the worst case scenario, a power user with 20 Mozilla
windows open, each with 100k HTML files. That's only 2 Meg or so. As a power
user, he should expect to use a lot of memory. Most users have one or two
windows open, and most users download HTML files of less than 100k. Admittedly,
Slashdot is the exception. In this case, however, the exception proves the rule.

It would be reasonable to limit the size of any HTML file stored in core to an
arbitrary file size limit, such as 1 megabyte.

Then we can employ a simple formula such as: when view source (or related
operation) called, use HTML file from core. If not in core, use HTML file in
memory cache. If not in memory cache, use HTML file in disk cache. If not in
disk cache, re-download the file. When such operations ask for images,
JavaScript files, and stylesheets, just get them from cache or redownload them.
They're static anyway, so the only issue is performance.

If a 100 MB HTML file is generated dynamically, heaven help that server farm.

As for the possible SVG image problem, a separate bug should be filed for that.

Plain text files (TXT's in the Win32 world) should not be stored in core, since
they are going to be static files anyway.

As for dynamic images, we can't fix that. Remember, that printing is NOT
affected by this bug. See comment 255. 

One may wish to send an image via e-mail without redownloading the image. Most
dynamic images, however, will be "mapquest" maps (courtesy of Navigation
Technologies Inc., BTW), and will be the exact same when redownloaded. As for
weather maps, there is not any real problem. If a user downloads a weather map
at 12:00, and then waits 6 hours and finally sends the file by e-mail, and he
expects the sent map to be just as it was at 12:00, he has a problem that
Mozilla just can't solve. Most weather maps will be looked at and handled during
a few minutes time at most, making any change negligible. It is a safe
assumption that most users would want the most current map anyway. Finally, many
weather map sites force a redownload every so often, so there is little point in
trying to "fix" this.

Let us first get a mechanism for pinning the *html* page (or image or whatever 
is currently being viewed) before starting to talk about all linked resources. 

The scope of this bug is about creating a mechanism for pinning things, in 
particular the current page. Everything else is subject for another bug. Feel 
free to create a bug on "Make all linked resources to a page refetchable 
without refetching from server".

I won't put my thoughts on if i think that should be done in this bug since it 
would only encourage further discussions.


At least this is my oppinion, rpotts, darin, do you agree?
Why can't we add a new attribute to the DOM, say document.source, which would be
read-only attribute containing the HTML code of the corresponding page?
Um, why on earth would we need to expose the original (potentially hundreds of
kB) of source of a document *throught the DOM* as text?

PS. I haven't followed this discussion, so I could be missing context here...
jst, that was just an idea I had. It's not like we _need_ to do it.

I don't see, though, where the difference is between using the DOM to access it
or some other way.
Any change we make to the DOM can affect any webpage out there (by introducing
new names into the namespace in question), so to expose things throught the DOM
we need a really good reason to do so. And exposing huge strings to JS is in
general not a good idea since they end up living in the JS GC world and
potentially staying around in memory for a long time after being accessed.
People, the problem is not storing the data. The problem is storing the data 
_without adding too much bloat_. Just so that everybody understands, bloat in 
this context is memoryconsumption. So simply storing the data in the document  
is not an acceptable solution, since it will consume too much memory.

Remember that mozilla is already critisized for using much memory, and that is 
just for the features like CSS/DOM/XML/HTML/JS, iow things that many people 
think is more important then the features that are blocked by this bug.

I'm not saying that this is not important or that I think it should be solved. 
I'm just making sure you all understand what the problem is.

Using the cache system and pinning things in the cache saves us much memory, 
since most of the time we will get the pinned source for free; it already 
exists in the cache anyway. So what we need to solve is the remaining cases 
where the current cache system arn't solving the problem.

I.E. we need to give the cache the ability to guarantee that the source is 
available, as often as possible, preferably always. Please don't come with 
arguments like "anything less then 100% is acceptable", isn't it good if we can 
increase the number from the current, say 90%, to 99%? Sure, we should aim for 
100%, but if we don't get there right away it's not the end of the world. 
There's always tomorrow.
The advantages of hanging the source off of the DOM are:

1- it's the most natural place (from an internal perspective) for it to go,
since the DOM is what Mozilla keeps around to render/print the page, and
whenever the page is renderable/printable, it should be
view-sourceable/sendable/save-asable.

2- no ugly cache pinning/duplicating, etc problems

3- all the problems and nuances of pinning discussed in the last 50 or so
comments go away.  Viewable-but-not-saveasable goes away.  The 0-size cache
problem goes away.  Even the oh-gee-this-page-is-7-terabytes-what-should-we-do?
problem goes away because you simply run out of memory and go away or give an error.


The disadvantages:

1- this exposes the .source node to the user (through javascript or whatnot),
which really isn't a good idea or even beneficial to the user in really any way.

2- memory bloat.  I question this, however -- doesn't representing the document
as DOM take up more memory than representing it as plain html?  Even if the DOM
were a bit smaller than the html, we aren't talking about any orders of
magnitude, are we?
  It seems to me that if you want to cut back on Mozilla's bloat, there are a
thousand other things to do away with before you do away with "correct
functionality" and "the right solution."  I mean, for heaven's sake, purge all
the forward/backward pages and make me reload those (from the cache or the
server) before you make me reload the CURRENT page just to save it to disk.


  Can we somehow attach the source to the DOM internally without exposing it to
the user?  There is a 1-to-1 correspondence between pages that need the source
held and pages which have a current DOM in memory.  This is not true of any of
the cache pinning solutions.

  I really don't think it is unreasonable in any way to make the browser keep a
copy of the html of visible pages in memory -- isn't this how the most
rudimentary browser you could conceive of would do it anyway?  If we don't
attach it to the DOM, is there some other reasonable place in memory we can
stick the source, even if it means paging some of this out to disk once in a
while to appease the memory-bloat argument?  I'd hate to suggest the creation of
yet another cache, but it seems to me (a completely non partial observer :) that
page source needs to be held, and it seems equally obvious (from this week's
traffic on this bug) that the traditional cache is not the place to be doing it
unless you want to add all manner of kludges...



  I think I've now gone over my alloted 2 cents.

Yes, the in memory DOM representation is larger than the source of the document,
but whether or not it's on an order of magnitude larger is besode the point, the
point is that it's large enough to not want to keep it in memory. Either way,
keeping the source in memory, even if it is far smaller than the DOM
representation of the source is unnecessary bloat that we can not in any way
afford. We're way way too bloated as it is, if we'd start holding on to the
source of the document in memory too we'd need a really really good reason to do
that. I still haven't heard any reasons close to good enough to pay that cost
(not that I've listened that closely, but still...).
>Can we somehow attach the source to the DOM internally without exposing it to
>the user? 

Hm... from my understanding of DOM and IDL, this would be possible:
Just add this to nsIDOMDocumentInternal:
[noscript] nsAString getSource();
(or [noscript] string getSource(); or whatever an appropriate string type would be)

This would, as I understand, eliminate problems with GC, as the function would
only be called by C++ Code, which isn't GC'ed. Or is it?

Why does everyone seem to want to put everything in memory? The only thing that
_absolutely_ needs to be put in memory is encrypted files. Personally I dont
care if it takes 5 msec to see the source or 5 sec, just as long as I can
actually see it.
So why not just follow what others already have posted and dump everything we
get from the server in eg. /tmp, that includes images, html, xml etc. This
enables the view-source, save-as and other? features and makes them function
correctly.
This disk usage should not count against the setting for the cache, because it
is not cache. The cache setting is to specify just how much memory/disk you are
willing to spend on *caching* data! (I have never understood the memory cache of
mozilla/netscape, why is it needed? The OS just does fine caching the files
anyway, while being much better at handling the size of it...)

Various people suggested that this bug should refer to all accessible documents 
(including images and .js and .css files). However, we should think about the 
average use case. People who need save as/view source want it to save the 
correct HTML page they are looking at, without a refetch. When users use 
power-features like save entire page, they should expect a refetch of images or 
other non-critical data if it has been expired from cache. Also, they should 
expect a more updated version be saved, just as if they reopened the URL in a 
new window - You don't expect to see the exact same thing.
Therefore, I think only the currently-viewed HTML/xml/whatever source should be 
saved, and only if it's uncachable (if it's cachable, then we don't expect it to 
change, thus we may refetch). Maybe we should pin cachable documents to make 
them remain in cache.

Another option is that if a cachable page is removed from cache and the 
refetched for save/view source we should check that the newly fetched page is 
the same as the old one (via hashing or other means), and if not warn the user 
with a message such as:

"The page you are about to save/view may be different from the displayed page. 
Proceed with view/save?"

I suggest adding a pref such as:

Keep source of viewed pages:
( ) Keep the entire page (increases memory consumption).
(*) Keep only the main page and frames.
( ) Do not keep source (Save and View Source may not work as expected).

In my opinion, if a user views a very large static HTML file, it is very 
important to save it because the user will most likely want to save the file 
(think the Jargon file in one big HTML document).

In any case, if the fetch of original text is not applicable, I suggest a 
message similar to expired POST data:

"The page you are trying to view/save has expired from cache. Refetch?"

The only reason we care about bloat is that it has a negative impact on
performance (in virtual memory operating systems). If we keep the source file in
DOM, we would increase Mozilla's memory consumption by a few hundred kilobytes
on average. This would hurt performance. If we resolve this bug in a way that
results in heavy disk access by every browser window, as suggested by comment
274 and many others, however, then performance will take a huge hit. The first
option is better overall.

I'll be quiet after this message. I've made the case for keeping the source in
DOM. As suggested above, the potential security problem can be avoided. Finally,
after a couple more of the keyword: footprint bugs are fixed, the memory use
issue will balance out.
Responses to various people have said, and my last comment in this bug till
there is actual code involved:

> downloads an HTML file for display, it generates a docshell, then it tosses
> the HTML file away.

You make it sound like we have the HTML and then drop it.  We never have the
entire HTML file at once (except in the cache).  As packets come in necko sends
notifications to the parser which processes the new data and waits for more.  We
would need to keep reallocing the data as more came in (this would be quite slow). 

> Plain text files [...] are going to be static files anyway.

This is a completely bogus assumption...

> I mean, for heaven's sake, purge all the forward/backward pages and make me
> reload those (from the cache or the server) 

That's what we already do.  All that stuff is stored in the disk cache and if
it's expired from there we put up a dialog telling you so and saying that we
will be reloading from the server.  How often do you run across this dialog?

> isn't this how the most rudimentary browser you could conceive of would do it
> anyway?

The most rudimentary browser I can conceive of would do a lot of things that
scale poorly and are inefficient, including that one, yes.  It would also not
lay out most web pages out there...

At this point, the thing I suggest is that we wait for Rick's
nsIWebPageDescriptor work.  That should allow immediate improvements in View
Source and Save Page/Save Image that will solve the problems people are seeing
in most cases.  At that point we can see what issues still remain and how common
they are, and discuss ways to solve them.
1. Can someone confirm that "90% of the time" the source-HTML is not refetched 
when doing View Source/Save?

2. Am I correct in this summary of the competing PRO arguments?
- Pinning-the-cache is more "elegant" because it potentially
  does away with any memory bloat
- attaching the source-HTML to the DOM is simpler since
  there's already the one-to-one relationship with browser windows

3. What does the memory cache DO anyway?  I assumed the DOM for all open 
windows was stored separately since the memory footprint swells consistently as 
you open more windows.  Or, is the DOM part of the memory-cache but the memory-
cache is implemented per-window?

4. If we implement some form of cache-pinning, I would suggest a user-prompt 
like this,
 "A cached copy of this web page is no longer available.
  (You may need to increase the size of your disk/memory cache)
  Would you like to continue, by reloading the page from the web-site?"
  YES / NO

5. When you do a Save As in IE 5.5 it *reloads* most page elements, though I 
don't know about the HTML specifically.

Sorry for the additional spam, but I just did a quick test of the Save As.  
------------------
I had this bug's Bugzilla page open in both Mozilla 20020129 and in IE 5.5.  I
did a Save As from both.  Next, I opened a new IE window and submitted my
previous set of comments.  Then, after a minute-long pause for Bugzilla to
process it, I re-did the Save As from both original windows; my comments weren't
included.  So, in this simple test case, both Mozilla and IE continued to save
the displayed HTML rather than refetching.

Perhaps a new bug(s) needs to be created for the specific instances where
refetches occur?
I don't think we need a new bug thats what this one is for.  The easiest case is
just a simple form using the post method.  Type something in hit submit and then
view source, the page now shows the submitted text but when you view source you
still get the origional source of the form.  This is a huge issue since it make
mozilla hard to use for web developers, the people we want using mozilla to keep
the web from becoming an IE only wasteland.

Just goto http://joshuaeichorn.com/mozilla/view_source_example.php to see the
effect.

I think the majority of users would be happy if this case was fixed.
Did we max the Bugzilla db or something?  I'm seeing "only" the first 280
replies on the web-page now.
-------------------
Mark, Joshua, thank you for the clarifications.  Sorry, Joshua, but I got an
"unknown domain" error when I tried to use your test page
-------------------
There are way too many comments on this bug and I think it's because the title
makes it sound like a widespread problem.  Maybe it should be renamed, "Need
improved means to reuse current page without refetching (see blocked bugs)."

I also think we need to create more bugs (which will be marked as
blocked-by-40867) describing the remaining problem areas.  And, if we create two
new bugs for the two possible solutions (marked as *blocking* 40867), we could
then leave 40867 as the tracking bug.  Specifically, create the following new bugs:
- Need means to pin open-windows in cache
- Need means to attach source-HTML to DOM
- Need means to individually cache dynamic-URLs in multiple windows
- Need means to prevent expiration of cached copy of still-displayed windows
- Need means to "cache" (for re-use) pages marked no-cache (?)
- Need means to cache/re-use HTML forms without refetch (bug 68412 or 84106?)

I leave it for someone else (more knowledgeable than I) to do any of these
suggestions.

Finally, is there something specific about the HTTP info for
http://weather.yahoo.com/forecast/USCA0982_f.html that causes it to refetch on
Save As.  If so, this could become another blocked-bug.

I think breaking this into separate bugs is a good idea at this point (actually,
a whole lot sooner would have been better).

- Need means to pin open-windows in cache
- Need means to attach source-HTML to DOM
These are separate issues and deserve separate bugs to discuss their merits. 
"Need means to pin open-windows in cache" should probably be named:
- Open windows need to hold cache tokens for elements (or something like that).


- Need means to individually cache dynamic-URLs in multiple windows
- Need means to prevent expiration of cached copy of still-displayed windows
- Need means to "cache" (for re-use) pages marked no-cache (?)
These are not necessary.  The first two are already supported by the cache
service, and will be addressed if open windows hold cache tokens.  We are
already doing the third (though someone could open a bug to say we shouldn't).

- Need means to cache/re-use HTML forms without refetch (bug 68412 or 84106?)
I'm not quite sure what this means.  Data received from a server will be cached,
and reuse can be discussed in one of the bugs above.  If this refers to data
entered into a form by the user, then that would require a different bug,
because that data is not stored in the cache.
Keywords: nsbeta1 → nsbeta1+
Target Milestone: mozilla0.9.7 → mozilla0.9.9
Bug 115832 asks for page to be reused without refetching when doing File -> Edit
Page.
Blocks: 115832
Let me see if I've got this straight.

* This bug NEEDS to be fixed, ASAP.  E-commerce is unsafe while this bug is
outstanding; 'view source' works in a non-WYSIWYG, bandwidth-hogging, and
generally unacceptable manner; etc.  I personally will expect all the major
issues related to this to be fixed by Mozilla 1.0, or I'll abandon Mozilla in
disgust.

* This bug has been open for a year and a half, and there's been a lot of
confusion about its nature.

THE BUG IS:
"view source", "save", "save as", and "send page" should use the exact source
received from the server for the page currently being viewed in the current
window.  (Some people think "Back" should too, though this is debatable). (This
manifests itself in dozens of strange ways).


Early on, *two* fully functional solutions were proposed.
1) Store the raw text (usually HTML) of a document (page, frame, etc.) as 
a node in the DOM, and grab that.

People didn't like this because it added a bunch of text to the DOM, but it
would work.  No attempt has been made to implement it.

2) Store a 'hard reference', not URL-based, to the document's place in the cache
(it's stored in raw form in the cache) -- make sure that the cache doesn't wipe
out the document until the window containing it is gone by holding onto the hard
reference -- grab the document source using this.  (This is also known as the
'cache pinning' or 'cache keys' solution.)

This required an improvement in the cache to support this functionality.  These
changes to the cache are DONE: see comment #74.  So this solution is
HALF-IMPLEMENTED.

So now all that's needed is for the front end to actually use this.  Any page in
an open window needs a 'cache key' a.k.a. 'hard reference' which it holds on to.
 'View Source' and friends need to use this key to retrieve the source.

This has been waiting for nine months, and I can't figure out why.

1) Because nobody is willing to put the interface code in to hold a hard
reference for each document being displayed in a window?

2) Because nobody is willing to put the code in for the individual commands
(View Source, etc.) to retrieve their information using this hard reference?

3) Because people are too confused about the source of the bug to realize that
this is what needs to be done?

If number 3 is the problem, I hope I've solved it.  Number 2 is probably blocked
by number 1.  (Also number 2 deserves a separate bug for each command.)  What's
blocking number 1, which sounds like less than an hour's work for someone who
knows the code ?!?!

This bug deserves priority P1, for obvious reasons.
4) Because there's still back end work required to actually *use* cache keys.

I believe there is back end work being done behind the scenes.  I don't remember
where Boris said it, but he is waiting on someone else so he can do the front
end work.   (IIRC)
Nathan: I think you are referring to rpotts' comment #223 and bzbarsky's comment
#277.
Currently it doesn't appear there's a bug filed on this, I could only see
nsIWebPageDescriptor mentioned by Boris Zbarsky in bug 99642, so it really looks
like Rick Potts works on this thing behind-the-scenes now.
We have to wait.
Keywords: topembed → topembed+
Target Milestone: mozilla0.9.9 → mozilla1.0
I like the speed of the rendering engine, I like the way mozilla looks....
But I'm really getting angry when I want to do a simple view-page-source  and gets
unexpected results! 
I lived with the bug for almost one year, expecting it to be fixed from version
to version.
Hell, no - almost 2 years (by reading these posts) and it's still there! I
always had to keep a
copy of netscape around and I can't say that I'm happy with this.

I read many of the notes here, let me phrase some of my thoughts, in no
particular order.
- I never understood why a view-page/print/whatever would have to go to reread
it from network. 
  The content might change at any time no matter what headers or url or get/post
data says.
  As somebody already said - even if the content changes and I open the same
link in another window,
  I expect that each window will give me at print/save/view-source the correct
information, i.e.
  the one they built their information on.

- I don't get the memory consumption argument in this thread...
  I want from a program to behave correctly and not to eat too many resources.
In that order!
  So, I would have expected to be given a correct information when doing a
view-source. Only if
  that worked I would mind about the memory consumption. If it doesn't work... I
consider it a bug
  and the whole program is already compromised, i.e. I wouldn't trust it anymore
(since it's not
  doing things right).

- I don't get the reason why not to keep all windows _original_
html/js/images/etc bytes that are
used to render a page? And I'm meaning not _only_ the HTML files. You'll never
know when somebody
needs to save some javascript, some images or some flash file...
Why not do it ? Memory consumption ? Wow - that's a good one. I really think
that HTML files (most
often content type) will have about 1 order of magnitude compared to its DOM
representation.
Not to mention compared to all the other structures needed to keep information
regarding the
window state, XML/HTML/DOM representation, XUL, theme and many other such things.

- Somebody said that after opening more than 100 windows the system crashed
while the browser's
cache of 50Mb wasn't yet filled. Wow - that enforces my thought about the fact
that not the original
content retreived from the web would be the memory hog... So, then, again - why
not keep it as is
and make the browser behave as it should?
  Of course I also can't imagine somebody with more than 20-30 windows opened :-)

- I don't mind where you keep the original bytes - be it cache, separate memory
space or file
system. Of course I think the memory is the normal place to use. I do care
though that it should
be kept! I would accept that the browser tells me that it
doesn't have more memory to open new windows than to give unexpected content
when doing a
preview/save/view-source. Of course, this would probably happen when reaching
that >100
open windows case :-)

- I wouldn't count the original bytes as belonging to the cache (even if it's
stored somehow in
there - pinpointed entries?). Some of the posters keep telling that the cache
size limitations
(memory cache size, file system cache size) _must_ be respected and that it
can't keep all the
original bytes. Ok - if you insist: what is this
good since I still can't predict or limit the amount the whole browser process
uses? What good to
know that the cache is limited to xx Mb and you have very strict rules to obey
this (while not
having the expected behaviour) and still - after opening 2-3 windows the process
already has 30 MB
in size and is increasing very much with each new opened window?

- I don't want to switch to another browser, but if mozilla will keep behaving
wrong....
- As a memory optimisation, you can always compare each new content to the ones
that could be the
same (same URL for example) and keep only one reference-counted memory block. Of
course, the cache
will also use probably the last-modified version of these URLs.
And whenever a window gets closed, decrement the reference count of all objects
it used.

In fewer words - I want more this wrong behaviour fixed than mind about memory
consumption and get a bad program.

Sorry if this was a little bit too long - but I'm a bit angry on this behaviour.

Here's some initial work to add the infrastructure necessary for new windows to
leverage cached content...  More work is needed, but this gives an idea of the
direction...

-- rick
Rick, that looks pretty good at first glance.  The only two issues I see on the
view source front are: not handling "view frame source" (need to pass the page
descriptor to openDialog() in nsContextMenu.js inside viewFrameSource()) and the
code in viewsource.js that does argument handling.  typeof(null) == "object",
unfortunately.  It should be fine to just assume the second arg to be the
charset (if not null) and the third arg to be the descriptor (if not null).

On a non-viewsource topic, the persistence object could use a page descriptor as
well to save pages that are the result of post requests (right now it sets the
post data stream, but not the cache key on the channel).  It looks like you're
trying to keep the descriptors as opaque as possible, so maybe it would make
sense to have a "open channel using this descriptor" global helper that would
open a channel, set the post data stream, cache key, whatever flags are needed
to make sure the channel reads from cache, etc.  This would encapsulate
knowledge of what these descriptors actually are in just two places (with some
bending over, one _could_ try to use this helper in docshell, but that seems
uncalled-for to me).

Thanks for doing this,
Boris
hi boris,

this patch smacks nsContextMenu.js a bit so that view-frame-source works....

i wish i could have avoided introducing the BrowserViewFrameSource() function
and instead just called BrowserViewSourceOfWindow(...) passing in the 'focused'
window.  but i had *no* idea how to do this :-(

i also added a try-block to the argument parsing in viewsource.js to deal with
the situation where 'null' is passed.  i believe it should just ignore the
'null' and keep looking for more args.

let me know if you think this patch is the right direction... if so, i'll clean
up the patch and land it...

thanks,
-- rick

I am adding my vote to the 174 of them that already exist, to show my annoyance
of the bug, especially while troubleshooting a web script.  However, from the
very recent patches that I notice, it looks like this might be near fixed.

I appreciate the hard work of the coders, and hope to see the results in the
next version of Mozilla.  Thanks, guys!
That looks reasonable...

I have to admit I'm still a little confused as to why the code in viewsource.js
can't just assume that the second arg (if non-null) is the charset and the third
(if non-null) is the descriptor.  That lets people add more args as needed and
not have them accidentally used as descriptors or charsets (if a future fourth
arg happens to be a string starting with "charset=") or the like...
hey boris,

you're absolutely right!!  i think it must have been paranoia (or lack of sleep)
on my part -- it's getting so i can't tell the difference :-)

i'll fix up viewsource.js to assume the following (possibly null) arguments:
  [0] - url
  [1] - charset
  [2] - page cookie

thanks,
-- rick
Comment on attachment 73999 [details] [diff] [review]
initial patch allowing view-source to use cached content...

Minor problem: nsIWebPageDescriptor.idl uses the wrong (old) license.
dude this bug sucks my ass. your fixing of this bug would please me greatly and
I will be more than happy to send you beer if it is fixed.
Blocks: 132638
No longer blocks: 132638
Many, many kudos to Rick.

(You're about to fix a bug with 188 (the third-most) votes; blocking eight bugs
including three major; a correctness, data loss *and* performance issue; a
Netscape 4 parity issue (despite lack of tag); a problem which prevents
architecture stabilization and dates back several years!  Your solution will
also, it seems, fix 55583, a bug with 240 (the second-most) votes and 93 (the
most) duplicates!)

If you can get this fixed by Mozilla 1.0, you will be my hero.

Is there anything we the observers can do to help make sure this actually gets
in by 1.0?  It doesn't seem to have any of the markers indicating that
mozilla.org considers it a must-have-for-1.0 (although it obviously is a
must-have to a lot of web developers, including me).


        Attachment #73999 -
        Attachment is obsolete: true

        Attachment #74439 -
        Attachment is obsolete: true

    
    

    
  
r=bzbarsky on the xpfe changes.

The docshell changes look good to me too, with one possible caveat... could it 
become an issue that we have two distinct session history entries floating 
about that hold the same post data stream?  Would it ever happen that we'd try 
to rewind the stream and then read it on two different threads or something 
evil like that (causing the stream to be rewound on one thread while it's being 
read from on another)?  I don't recall ever running into that problem when I 
played with a similar approach that cloned the session history entry a while 
back, but....

    
    

    
  
- In nsDocShell::LoadPage():

+        nsCString spec, newSpec;

Use nsAutoCString, or whatever it's called nowadays.

+        newSpec.Append(NS_LITERAL_CSTRING("view-source:"));
+        newSpec.Append(spec);

do newSpec.Append(NS_LITERAL_CSTRING(...) + spec); to avoid double append and
potentially double reallocs in the string code.

- In nsSHEntry::Clone():

+    rv = dest->QueryInterface(NS_GET_IID(nsISHEntry), (void**) aResult);
+    NS_RELEASE(dest);

Why not simply:

  *aResult = dest;

I don't see the need for the QI call here, the compiler should do the right cast
with the above and it's less code, and one fewer AddRef/Release.

Other than that the changes look good to me, sr=jst

    
    

    
  
hey johnny,

thanks for the comments -- i'll fix up the patch...

The extra QI/Release in Clone() is purely habit :-)  It came from when we used
to hand-roll factory functions !!  Since the [out] result was nsISupports the QI
was necessary and we used to have a 'rule' about using this form :-)

But you're absolutely right, when assigning a known class into a correctly typed
result, the compiler will do the right thing...

Hey, old habits die hard ;-)

-- rick

    
    

    
  
cc'ing darin, who can comment on the postdata issue further.

The session history changes look good to me. Make sure that subframe navigation
continues to work fine. About  cloning postdata, I don't think it will be a 
issue, because, nsDocShell::GetCurrentDescriptor() primarily looks to send mOSHE
as the page descriptor. mOSHE is either the entry for the current page, if the
current page is done loading and docshell is in a stable state or  entry for the
previous page if the user has just started loading a page and necko has not yet
started the data transfer. mOSHE will be null only when session history is
disabled, in which case mLSHE will also be null and GetCurrentDescriptor() will
return error.

A sticky situation could be, when docshell is loading a page with postdata and
has just set the mOSHE in Embed(), (ie., data transfer and consumption has
started) but network is having trouble completing the postdata submission to the
server, (thereby possibly in the middle of reading the postdata stream) *and*
the user does a view-source at this time, which will rewind the postdata stream.
  But I think while reading from a stream, necko maintains offset values which
will enable it to read from where it left last time. So, this shouldn't be an
issue, but network experts can comment further.

    
    

    
  
What are the footprint impacts of the current patch?  What if a large number of
windows/tabs are open, or very large flat documents?  If there are impacts, how
can they be removed by people (embedders, etc) who don't need this?  (No view
source/etc available).

This pagedescriptor - are we "pinning" the source in the cache as was mentioned?
 Which cache, memory or disk?  What if we have no disk cache or it's turned off?


    
    

    
  
The current implementation incurs NO extra bloat... It merely allows the
existing caching mechanisms to work...

In the future we can deal with the cases where the cached content is not
available.  However, I believe that this new API is sufficient to deal with more
complex 'pinning' strategies...

Right now, I think it's important to leverage the current 'pinning' strategy. 
Especially since this should deal with 99.9% of the common cases (for
view-source at least).

-- rick


    
    

    
  
Comment on attachment 76086 [details] [diff] [review]
New patch that addresses the previous comments...

a=asa (on behalf of drivers) for checkin to the 1.0 trunk

        Attachment #76086 -
        Flags: approval+

    
    

    
  
>In the future we can deal with the cases where the cached content is not
>available.  

If my reading of comment #74 is correct, the current 'pinning' implementation in
the cache ensures that the cached content is always available (provided Mozilla
doesn't crash or run out of memory) for an open window or a frame in an open
window.  Even for pages which are 'not cached' (such pages will only be
accessible through hard references, not through the regular cache mechanism). 

Since 'view source' as currently designed can only be applied to pages in an
open window, this should deal with *100%* of the cases for 'view source'.

    
    

    
  
Am I correct in assuming that this will mean hitting the back button will ALWAYS
pull the cached copy?  While pages that have "pragma: no cache" should not have
a physical cache for good reasons, it still should be at least cached in the
memory, so that when you hit the back button, you're not reloading the damn page
again.  IE has the same problem.

This has major problems with CGI scripts with FORMs that disappear when you hit
the back button.  All of the contents disappear, and you have to start over
again.  Is this another bug, or directly linked to this one?

    
    

    
  
Brendan: You're probably thinking of bug 112564.

    
    

    
  
Patch checked in!!!


    
    

    
  
*sniff*

rpotts, please tell me that's not an april fool's joke

    
    

    
  
It's not.

    
    

    
  
The test case now works.  Congratulations on the good work.

However, the problem isn't solved yet. *sigh*

Go to a page which changes often, like 
http://gcc.gnu.org/ml/gcc-patches/2002-04/

Leave that window around for a while.  Open the same page in another window. 
See, it's changed.  Try 'view source' in the first window.  You get the source
for the newer version.

Obviously the 'cache key' currently used is *not* exactly what we want.
(Trying to figure out the maze of interfaces in Mozilla is still very difficult
for me, so I thought it was.)  Apparently what we want is now called a 'cache
token', and is *still* not fully implemented.  Please add dependency on bug 72519.



    
    

    
  
Nathanael: reusing cache tokens was not part of the design of view-source, etc.
for a good reason.  it impairs the cache's ability to manager resources and to
make room for newer content.  that said, i can see the benefits of holding cache
tokens for HTML documents, and since there aren't very many at one time, we
probably could get away with pinning these in the cache.  however, i think
that's a second-level enhancement to view-source.  the solution thus far ensures
that we do the right thing on pages that are freshly navigated to, which IMO
covers the majority of the view-source cases.

    
    

    
  
yeah, it is a second-level enhancement.  I still think it's important but what's
left isn't really a 'must be fixed by 1.0' kind of thing.

    
    

    
  
I'm sorry to spam a load of people with this message but I've been keeping an
eye on this bug ever since I started using Mozilla as it irks me so much. What
I've never fully understood through this whole period is /why/ Moz seems to have
such difficulty in doing View Source when others like NS4.7x, IE6 handle it
fine. Is there some fundamental flaw in the way Moz is/was written from the
beginning that stops VS working like it does in the other browsers?
I'd be grateful for an explanation (preferably through my EMail so this bug
isn't tangled up) anyone can provide. If anyone can come up with a good reason
as to why VS was never implemented and catered for from the beginning? Other
than "Because" and "Most people don’t use it", I mean. 


    
    

    
  
Because ;-)

Marking fixed since it is.
Status: NEW → RESOLVED
Closed: 22 years ago
Resolution: --- → FIXED

    
    

    
  
Please open a new bug to deal with the case that's still broken, if you plan on
marking this one fixed.

(I'm referring to the situation where the same page is open in multiple windows
with different contents; only one cache entry is preserved for the content,
instead of one for each different set of content). Otherwise we still *don't*
have a way to "reuse/reload current page without refetching from server" for all
cases, so this bug is NOT fixed.

    
    

    
  
I would actually like to understand this bug better myself.  A technical
explination of the issues would help... how does Netscape 4.x/IE 6 deal with the
view-source issue?  Why doesn't Mozilla use the same style?  What are the
advantages to Mozilla's method?

And I have to agree that I don't think this is completely fixed, although it is
1000 times better than what it was!  Good work!

    
    

    
  
I just checked and 4.x *does* get the remaining case right: I viewed the top
story on slashdot.org and then opened the same page in a new window. I refreshed
the new window until the number of posts for "-1" in the "select threshold"
dropdown list went up, without touching the original window. Then I did "View
source" in the original window and looked at the bit of the source that
identifies the threshold dropdown. It indicated the original number, as it
should have.

(Oh, and I made sure I got to the story page just by clicking a link, so there
was no postdata)

I haven't tested, but from comments here Mozilla would have given the value from
the other window. The fact that NS4.x gets this right is an indication that the
tradeoffs described in this bug *aren't* necessary - although I didn't test with
cache disabled or set too small to hold a slashdot page, so I can't say for sure
what happens in that pathological case for 4.x.

caillon, you marked this fixed, do you have an opinion on whether this bug
should be kept open for the remaining cases or closed in favor of a new bug for
the case where the same source is in two windows? How about a new bug for the
pathological case (cache too small to hold all open windows) too?

    
    

    
  
Please don't re-use this bug for new issues, it's big enough as it is...

    
    

    
  
Thank you again for a fix to this bug.  Remaining issue spun off to bug 136633

    
    

    
  
verifying, Man is it good to se this in!
Status: RESOLVED → VERIFIED

    
  
Blocks: majorbugs

    
  
No longer blocks: majorbugs

    
    

    
  
Why is this bug marked as VERIFIED FIXED in 2002 when Firefox 3.x still has this same issue?  I still am unable to save, view source, etc. pages, images and others.  It reloads them from the network every time.

KenW: "I think that the data to save can just be generated from the DOM object and there is no requirement that the HTML (if the HTML format is chosen) be
anything like the original source code."

I disagree.  There is a very strong requirement that the source saved be exactly, byte-for-byte what the server provided.  These are the fundamental and essential functions of an HTTP browser: download, view, and save a copy of a web page.  If you are not saving the original web page, then the browser has failed in its most basic functioning.

    
    

    
  
Bug 288462 provides the full summary of all related issues, complete with RFC violations.  Should be linked to this one.

    
    

    
  
Here I was getting totally crazy why some $_POST (php) output I printed in <!-- --> didn't show up in view source...

Really, if I want to get the source of a clean request to a URL, then I'll use wget!

Not verified!

Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.6) Gecko/2009020410 Fedora/3.0.6-1.fc10 Firefox/3.0.6

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: