What you call a 'formatting tag' is called an 'inline element'. What you call a 'division tag' is called a 'block element'. The only problem is that there is a general rule that inline elements may not contain block elements. So Mozilla has to do all kinds of acrobatics when it comes across this case -- for example, <B> <H1> </H1> </B> -- to try to make it work. The W3C suggest the use of stylesheets for formatting instead of formatting tags. See http://www.w3.org/TR/REC-CSS1 for more details.
Could you give me a head start on where to look in the code if I wanted to try and fix this myself? It's rather counterintuitive behavior, even if the spec does say that. Even a general module would be useful.
Sure. I believe the code to change is in .../mozilla/htmlparser/src/nsElementTable.cpp You may wish to see bug 47827 where a similar bug (but this time where we got the standards wrong) was recently fixed. Note, though, that unless you have a very strong reason to fix this (e.g., a top100 site breaks on it) then there are many more somewhat more important bugs that you could help fix...