See bug 538323 comment #29.
For smooth transitions between Images with different sizes without synchronizing with the main thread, we should give ImageLayer a size (updated on the main thread, of course) and have ImageLayer letterbox Images to that size. Then we can rely on that for all letterboxing and get rid of the letterboxing in nsVideoFrame (which currently uses transforms).
This isn't a priority right now since we don't support chained Oggs or any other format that supports variable frame sizes.
Chris Pearce added sizes to ImageLayers shortly after this bug was filed: http://hg.mozilla.org/mozilla-central/rev/3742af50864f