528050 - Improve shmem protection scheme

Status: RESOLVED WONTFIX

(Core :: IPC, defect)

Description

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Currently it looks like

  |[ (magic byes) (Header) (empty) ]|  // front sentinel, no access rights
  |[ user area ...                 ]|
  ...
  |[ last page ]  (empty)           |
  |[ (empty)                       ]|  // back sentinel, no access rights

where | ... | is a memory page, and [ ... ] blocks are "important" space.  Notice that we potentially leave some "unimportant" (empty) space at the end of the user's area.  The user could write into this without penalty, even though that's a buffer overflow.

I think a better scheme would be to fill the (empty) bytes in the front sentinel with 0x5 bytes, and fill the (empty) bytes in the last user page and back sentinel with 0xa (arbitrary choices of 0x5/0xa, could be the same).  When mapping/unmapping/sending/receiving a page, Shmem code would verify the magic bytes, header, and 0x5 and 0xa bytes.  This can potentially catch the aforementioned buffer overflow, and additionally threadsafety errors like fiddling with the shmem segment on another thread while the main thread is mapping/unmapping/verifying.