h1. MongoQueue
MongoQueue is a PHP queue that allows for moving tasks and jobs into an asynchronous process for completion in the background. The queue is managed by "Mongo":http://pecl.php.net/package/mongo
MongoQueue is an extraction from online classifieds site "Oodle":http://www.oodle.com. Oodle uses MongoQueue to background common tasks in order to keep page response times low.
h2. Requirements
- mongod 1.5.4+
- PHP 5.3+
- Mongo PECL (PHP mongo driver)
h2. Installation
Extract the source files into a directory in your PHP library path. Because the queue is backed by Mongo, there is no need to migrate or set up a table for MongoQueue. It will use the @mongo_queue@ collection in a database of your choice.
h3. Usage
Before any MongoQueue actions are taken you'll need to initialize the MongoQueue class with the proper settings:
$mongo = new Mongo('mongodb://host:port', array('connect' => false));
MongoQueue::$connection = $mongo;
MongoQueue::$database = 'my_database';
Jobs are PHP objects that extend from @MongoJob@. Object states are not serialized so jobs must be defined as @public static function@.
require_once('MongoQueue/lib/MongoJob.php');
class QueueTracer
extends MongoJob
{
public static $context;
public static function trace()
{
echo "Tracer hit\n";
}
}
MongoQueue::push('QueueTracer', 'trace', array(), time() + 500); # run QueueTracer 500 seconds later
You can also take advantage of @MongoJob@'s built in @later()@ method which automatically delays a method for asynchronous running:
QueueTracer::later(500)->trace();
You can also batch calls which have the same parameter so that they'll only be run once by workers. The run time will be decided by the first job inserted and the subsequent jobs will not have their run times obeyed.
QueueTracer::batchLater(500)->trace();
MongoQueue also supports prioritization. Priorities are ascending and @null@ (the default) is considered to have the lowerst priority.
QueueTracer::later(500, false, /*priority:*/ 1)->trace();
h3. Running the jobs
Jobs are run via @MongoQueue::run()@. Before running job, you'll need to set two extra MongoQueue initializers: @environment@ and, optionally, @context@. MongoQueue currently does not include a daemon runner, but here is an example runner:
$mongo = new Mongo('mongodb://host:port', array('connect' => false));
MongoQueue::$connection = $mongo;
MongoQueue::$database = 'my_database';
MongoQueue::$environment = 'classes/jobs';
MongoQueue::$context = array('context' => array('foo', 'bar'));
if (MongoQueue::run())
echo("Found a job to run");
else
echo("Nothing to run");
@$environment@ is the path under which jobs are automatically loaded by the runner. The convention is that if your job class is called @FooBar@ then the file that is loaded is @$environment/FooBar.php@
Any keys set in @$context@ are available as static variables inside the Job class. (e.g. 'context' in the @QueueTracer@ example)
h4. Priorities
By default, MongoQueue will use priorites (which are all @null@ by default). If you want to run the next job regardless of priority, you can pass @false@ as the third param to @run@. e.g:
MongoQueue::run(null, null, false)
h3. Administration
h4. Indexing/Performance
You'll want to add indices to MongoQueue to ensure high performance. You'll need at least the following index for basic operation:
db.mongo_queue.ensureIndex({locked: 1, when: 1, priority: 1});
If you plan to use the batching feature you want an index on the type of jobs already in the queue:
db.mongo_queue.ensureIndex({object_class: 1, object_method: 1, parameters: 1, locked: 1});
h4. Sharding
MongoQueue comes with built-in support for manual sharding. Note that due to the key restriction on updates for auto-sharding, the ability to scale with only auto-sharding is limited.
If you initialize @MongoQueue::$connection@ to an array instead of a single Mongo connection object, MongoQueue will insert or pull jobs at random from the set of connections. This way, you can have dedicated workers per physical machine and insert in a distributed way (or vice versa). For example:
MongoQueue::$connection = array($mongo1, $mongo2, $mongo3, $mongo4);
MongoQueue will send batchable jobs to the same connection so that batched jobs with identical parameters will not be distribued across multiple mongo instances.
h4. Monitoring
All jobs are stored as Mongo documents in the mongo_queue collection. @MongoQueue@ does not come with any built in administrative scripts, but you can view the queue via your mongo console. @locked_at@ will tell you whether or not a job is currently being processed.
