Package schrodinger :: Package job :: Module queue
[hide private]
[frames] | no frames]

Module queue

A class for running multiple jobs under job control.

This module provides an extended version of the class JobDJ from the schrodinger.job.jobdj module.

The name JobDJ refers to an old implementation of "Job-control for Distributed-Job drivers," and the module name "queue" reflects the fact that added jobs are queued until ready to run.

Like the original JobDJ, its main purpose is to provide for job distribution on multiple hosts. It will keep the number of active jobs on each host at or under the number of processors specified. It can also restart jobs (see documentation on max_retries in the JobDJ.__init__ method).

Additional functionality that is provided in this implementation includes the following:

  1. The ability to create dependencies between jobs so that JobDJ can avoid starting a job until its prerequisites are met. See the BaseJob.addPrereq method.
  2. The ability to run jobs locally (i.e. not under job control). These jobs can also have dependencies.

Step by step instructions for basic use:

  1. Create a JobDJ instance. You can optionally specify the hosts to run the jobs on and the maximum number of jobs for each host by passing a list (host, max_jobs) tuples. By default, the host list is read from the SCHRODINGER_NODEFILE, which is created from the -HOST host list by jlaunch. For example:
       job_dj = queue.JobDJ( [('localhost',1), ('cluster',20)] )
  2. Add commands to the JobDJ instance by using the JobDJ.addJob method. The addJob method takes either a command (as a list of strings) for a job control job, or a BaseJob instance. For example:
       job_dj.addJob(["impact", "input1.inp"])
       job_dj.addJob(["impact", "input2.inp"])
  3. Run all jobs with the method. This is simple:

    Alternatively, you can use the updatedJobs generator:

       for job in job_dj.updatedJobs():

    This yields a job every time its status changes. One reason to do things this way is to allow restarting from a pickle. (A JobDJ instance that is restored from a pickle can be started with the run method or updatedJobs generator.)

Because an added job can be a BaseJob instance, additional functionality can be gained by subclassing. For example, the JobDJ instance that a job has been added to can be retrieved with the getJobDJ method and new jobs can be added dynamically by a job at any point during its normal lifecycle.

Copyright Schrodinger, LLC. All rights reserved.

Classes [hide private]
A base job class for jobs run under JobDJ.
This is a base class that indicates a job has special requirements (e.g.
This class should be used as a mixin on a job type that requires jobs to be run on localhost.
A job for running an external process.
This class is for jobs that don't run under job control yet can have significant computational requirements in the python part of the JobDJ process.
This class defines a job control job to be run under JobDJ.
A node for the LinkedList class, holding a value, and a reference to the previous and next node in the list.
A doubly linked list, providing constant time addition, size, and truth checks.
A LinkedList subclass that tracks running jobs and keeps a tally of jobs running on each machine.
A class to allow specification of a function to be invoked when the final job of a group completes successfully.
Class for running commands/jobs in parallel under jobcontrol.
Functions [hide private]
backup_file(log_file, copy)
Backs up the file log_file copying it to *.1, *.2, etc.
Return True if specified host is a queue
get_command(base_command, host='localhost', procs=1, local=False)
Given a base command and additional launching specifications, return a modified command that is ready to pass to jobcontrol.launch_job.
_wrap_job_finalizer(function, run_dir=None)
Wrap a function that takes a single job as an argument so that it runs in a separate directory.
add_multi_job_finalizer(function, jobs, run_dir=None)
Create a finalizer function that will be called when all jobs in the jobs iterator are complete.
Variables [hide private]
  _version = '$Revision: 1.8 $'
  WAITING = 'waiting'
  ACTIVE = 'active'
  FAILED = 'failed'
  FAILED_RETRYABLE = 'failed but retryable'
  DONE = 'done'
  NOLIMIT = -1
  _status_messages = {}
  logger = log.get_output_logger("output.schrodinger.job.queue")
  syslog = log.get_logger("schrodinger.job.queue")
  _host_is_queue_dict = {}
  __package__ = 'schrodinger.job'
Function Details [hide private]

_wrap_job_finalizer(function, run_dir=None)


Wrap a function that takes a single job as an argument so that it runs in a separate directory. If run_dir is not specified, run it in the working directory of the job.