ISO software for CPU allocation over networked workstations

[berriz -8 at 8- chasma.harvard.edu (Gabriel Berriz)]

 Dear netters:
 At the risk of getting myself summarily booted out of here for gross
 inappropriateness, I would like to ask the readership a question about
 CPU allocation software.
 Presently, the "CPU allocation problem" is an endless source of
 conflict among lab members.  Partly due to due to historical reasons
 and partly due to the KrazyKwilt heterogeneity of our present
 hardware, we have not being able to agree on a CPU allocation policy
 that is efficient, productive, and fair to all users.  (Our jobs range
 from minor interactive ones such as word processing to background
 multi-CPU-day simulations).  We are now in the process of overhauling
 our lab's computer system, and the question of CPU allocation keeps
 rearing its ugly head.  If the choice hardware configuration has any
 bearing on the kinds of CPU allocation policies that can be easily and
 effectively implemented, we want to make sure we choose the one with
 the least nightmarish options.
 One possibility we are considering for the new system is simply a
 cluster of reasonably powerful (e.g. Pentium Pro), at roughly one
 machine per lab member.  To avoid wasting CPU time, however, we would
 like to set it up as follows: to give everyone an adequate baseline
 level of computational power, each individual would have top priority
 on his/her assigned machine (i.e. his/her jobs would be "native" in
 this machine), BUT any other member of the group would still be
 allowed to run low-priority "foreign" jobs on it as well.  Each
 machine would be set up such that its system would allocate little or
 no CPU to all foreign jobs whenever at least one native job was
 running.
 This setup, however, is not yet ideal, since it leaves the problem of
 assigning foreign jobs to the users, which means that it would be
 likely that machines would go idle, or that the foreign job load over
 the entire cluster would become unbalanced.  Our quixotic dream,
 therefore, is to find job management software that would periodically
 monitor CPU usage over all our networked workstations, and *move
 around* foreign jobs as necessary to avoid idle CPU's and load
 imbalances.  Is this even technically possible?  Does software to do
 this exist?  If so, for which hardware platforms is it available?
 Another problem with this idea is the same problem with lab benches:
 the lab's size fluctuates, so times will certainly come when we have
 more people than machines.  As with physical benches, I suppose that
 this will require some "doubling up"...
 If anyone has experience with or general comments and criticism about
 such a set up, I would be very grateful to hear about it.
 Thank you very much in advance for all your help.  I'll summarize the
 responses (I'll try at least).
 Gabriel Berriz
 berriz -8 at 8- chasma.harvard.edu
 Department of Chemistry and Chemical Biology
 Harvard University