532965 - Our Ts cold startup on the Mac is not cold enough

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[Joel Reymont (:joelr)]

purge on Mac OSX does not purge the pages of the code mmap-ed into memory by the dynamic linker. Purging the code is what we care about and there 3 ways of accomplishing it

1) Requesting 2x physical memory from the virtual memory pager and then touching every page, e.g.

#!/usr/bin/env python

import mmap

SIZE = 8589934592L # 8Gb

map = mmap.mmap(-1, SIZE)
map[::4096] = '\x01' * (SIZE // 4096)

or 

#include <sys/mman.h>
#include <fcntl.h>
#include <stdio.h>

static long SIZE = 8589934592L;

int main(int argc, char**argv)
{
  char *result = mmap(0, SIZE, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0); 
  if (result != MAP_FAILED)
  {
    unsigned long i;
    
    for (i = 0; i < SIZE; i += 4096)
      result[i] = 1;
    return 0;
  }
  return 1;
}

or 

2) Rebooting and waiting about a minute for the kernel to stop streaming whatever it streams from disk, or

3) Unmounting the file system where our code lives, which clears the Unified Buffer Cache (UBC).

so

#1 takes 5-10 minutes for 8Gb of memory on my laptop (4Gb physical)

#2 may not be an option

#3 is the option I recommend

I have a USB HD plugged in and a shell script that takes care of mounting and unmounting 

#!/bin/sh

FF=/tmp/firefox

mkdir $FF > /dev/null 2>&1
umount $FF > /dev/null 2>&1
mount_hfs /dev/disk1s2 $FF

$FF/Minefield$*.app/Contents/MacOS/firefox-bin -no-remote -foreground file://$FF/startup.html#`python -c 'import time; print int(time.time() * 1000);'`

As suggested by Taras, we can also mount and unmount a file system. This can be a partition (results most truly matching real life) or a disk image. 

I'm not sure what kind of an overhead going to a disk image on a real file system adds. It's certainly a level of indirection which may skew results not in our favor but it does not require re-partitioning the disks in our test machines.

Comment 1

[Joel Reymont (:joelr)]

Here's a simpler and more secure way to unmount and remount the file system.

#!/bin/sh

FF=/Volumes/Fujitsu80Gb

diskutil unmount force $FF
diskutil mount /dev/disk1s2

$FF/Minefield$*.app/Contents/MacOS/firefox-bin -no-remote -foreground -P clean file://$FF/startup.html#`python -c 'import time; print int(time.time() * 1000);'`

Comment 2

[Dietrich Ayala (:dietrich)]

In your case the app is on a USB HD and I'm presuming that's what you're un/mounting here?

I don't see how that can work on the talos boxes, where we don't have any unmountable storage, and the code is not on there anyways.

I'll dig around for another way to clear UBC.

Comment 3

[Joel Reymont (:joelr)]

As Taras suggested, create a disk image with Firefox and mount and unmount that.

You can also create a separate file system on the talos boxes and I would prefer that option since running from a disk image will likely slow things down a bit. 

Creating a separate file system on the talos boxes will require formatting the drives to create a separate partition, though.

Comment 4

[Joel Reymont (:joelr)]

Dietrich,

You can also clear the UBC using the approach in my original post. Just allocate twice the physical memory then touch every page. Use my C or Python code. Takes about 5 minutes for 4Gb of physical memory.

Comment 5

[Geoff Brown [:gbrown]]

Mass closing bugs with no activity in 2+ years. If this bug is important to you, please re-open.