1446039 - Randomize the Page Cache

Status: NEW

(Core :: Memory Allocator, enhancement, P3)

Description

[Tom Ritter [:tjr]]

When a page (or sequence of pages) is no longer needed we may cache it for reuse. Adding a non-deterministic order to reusing these cached pages can prevent an attacker predicting the location of a recycled cached page. This might make it more difficult to deterministicly lay out memory before exploiting an overflow. 

Mozjemalloc will cache unused chunks for recycling via chunk_record and chunk_recycle. As the cache is maintained via an ordered red black tree based on the chunk size, it may be unfeasible to randomize this ordering without introducing an O(n) search over the tree. Under the existing operation of chunk_recycle, if the discovered node that holds the chunk to be recycled had its neighbors checked for an equal sized chunk, one of the two nodes found could randomly be selected. This could continue for all nodes of the same size.

Comment 1

[Mike Hommey [:glandium]]

Chunk recycling is not the main way pages are reused. That also happens when runs are allocated/deallocated. That is also tracked with a RB-tree.

Those RB-trees are ordered by size and address, and only size actually matters. We could maybe randomize addresses in the RB-tree compare function...