681680 - Change Tinderbox from locking the entire directory to per-tree locking

Status: RESOLVED WONTFIX

(Webtools Graveyard :: Tinderbox, enhancement)

Description

[:glob ✱]

<justdave> it wouldn't take much to implement
<justdave> we're talking about the mail handling.
<justdave> the script that sendmail shells out to drops all the mail into a queue directory
<justdave> tinderbox has a cron job that fires once per minute and starts processing that queue
<justdave> the way it works now is when the cron job fires, it sets a lock on the entire queue, and then runs until the queue is empty
<justdave> then it rebuilds the cache files for all of the trees that got touched by an incoming mail during that pass.
<justdave> the envisioned idea to make that faster is to lock individual queue items instead of the entire queue
<justdave> so if the cron job fired off again and the previous one hadn't exited yet, instead of just bailing it would grab an unhandled queue item and keep going
<justdave> and removing the cache updating from the mail handling thread altogether
<justdave> and make that a separate cron job
<justdave> have the mail handling just drop a flag file somewhere that says that tree needs updating
<justdave> processbuild.pl is the queue handler cron job
<justdave> although we probably want to keep track of how many runners are going and put some sane limit on it so we don't overload the cpu
<justdave> handlemail.pl is the one sendmail pipes into
<justdave> and that's a pretty simple "generate a unique filename and dump it into the queue directory"

Comment 1

[:glob ✱]

here's what i'm thinking on how to attack this; feedback appreciated :)

update processbuild:
  - add a "tree" switch, to limit processing to the specified tree
  	- when set, the lock filename will include the tree's name
  - add a "no_mail" switch, to skip processing mail
  - add a "no_build" switch, to skip building static pages
  - add a "build_static" switch, to build static pages for the specified tree

create a new script, called from cron instead of processbuild:

  worker_count = 25

  read pending mails
    extract tree from mail
      if (a lock file for this tree exists)
        worker_count--
      else if (mail is older than the oldest mail we've seen for the tree)
        update the oldest mail timestamp

  sort trees by mail timestamp

  for 1..worker_count
    pop tree from tree list
    fork
      run processbuild -tree $tree -no_build
      touch build.$tree file

  foreach build.$tree files
    if (unlink build.$tree)
      fork
        run processbuild -tree $tree -no_mail -build_static

Comment 2

[:glob ✱]

hrm, the build_static switch is not required; we just need -tree, -no_mail and -no_build.

Comment 3

[:glob ✱]

bug 659724 comment 30 indicates this is no longer required.