|
NAMEDirectory::Queue - object oriented interface to a directory based queueSYNOPSISuse Directory::Queue; # # sample producer # $dirq = Directory::Queue->new(path => "/tmp/test"); foreach $count (1 .. 100) { $name = $dirq->add(... some data ...); printf("# added element %d as %s\n", $count, $name); } # # sample consumer (one pass only) # $dirq = Directory::Queue->new(path => "/tmp/test"); for ($name = $dirq->first(); $name; $name = $dirq->next()) { next unless $dirq->lock($name); printf("# reading element %s\n", $name); $data = $dirq->get($name); # one could use $dirq->unlock($name) to only browse the queue... $dirq->remove($name); } DESCRIPTIONThe goal of this module is to offer a queue system using the underlying filesystem for storage, security and to prevent race conditions via atomic operations. It focuses on simplicity, robustness and scalability.This module allows multiple concurrent readers and writers to interact with the same queue. A Python implementation of the same algorithm is available at <https://github.com/cern-mig/python-dirq>, a Java implementation at <https://github.com/cern-mig/java-dirq> and a C implementation at <https://github.com/cern-mig/c-dirq> so readers and writers can be written in different programming languages. There is no knowledge of priority within a queue. If multiple priorities are needed, multiple queues should be used. TERMINOLOGYAn element is something that contains one or more pieces of data. With Directory::Queue::Simple queues, an element can only contain one binary string. With Directory::Queue::Normal queues, more complex data schemas can be used.A queue is a "best effort" FIFO (First In - First Out) collection of elements. It is very hard to guarantee pure FIFO behavior with multiple writers using the same queue. Consider for instance:
Who should be first in the queue, Writer1 or Writer2? For simplicity, this implementation provides only "best effort" FIFO, i.e. there is a very high probability that elements are processed in FIFO order but this is not guaranteed. This is achieved by using a high-resolution timer and having elements sorted by the time their final directory gets created. QUEUE TYPESDifferent queue types are supported. More detailed information can be found in the modules implementing these types:
Compared to Directory::Queue::Normal, Directory::Queue::Simple:
Directory::Queue::Null is special: it is a kind of black hole with the same API as the other directory queues. LOCKINGAdding an element is not a problem because the add() method is atomic.In order to support multiple reader processes interacting with the same queue, advisory locking is used. Processes should first lock an element before working with it. In fact, the get() and remove() methods report a fatal error if they are called on unlocked elements. If the process that created the lock dies without unlocking the element, we end up with a staled lock. The purge() method can be used to remove these staled locks. An element can basically be in only one of two states: locked or unlocked. A newly created element is unlocked as a writer usually does not need to do anything more with it. Iterators return all the elements, regardless of their states. There is no method to get an element state as this information is usually useless since it may change at any time. Instead, programs should directly try to lock elements to make sure they are indeed locked. CONSTRUCTORThe new() method of this module can be used to create a Directory::Queue object that will later be used to interact with the queue. It can have a "type" attribute specifying the queue type to use. If not specified, the type defaults to "Simple".This method is however only a wrapper around the constructor of the underlying module implementing the functionality. So: $dirq = Directory::Queue->new(type => Foo, ... options ...); is identical to: $dirq = Directory::Queue::Foo->new(... options ...); INHERITANCERegardless of how the directory queue object is created, it inherits from the "Directory::Queue" class. You can therefore test if an object is a directory queue (of any kind) by using:if ($object->isa("Directory::Queue")) ... BASE METHODSHere are the methods available in the base class and inherited by all directory queue implementations:
SECURITYThere are no specific security mechanisms in this module.The elements are stored as plain files and directories. The filesystem security features (owner, group, permissions, ACLs...) should be used to adequately protect the data. By default, the process' umask is respected. See the class constructor documentation if you want an other behavior. If multiple readers and writers with different uids are expected, the easiest solution is to have all the files and directories inside the toplevel directory world-writable (i.e. umask=0). Then, the permissions of the toplevel directory itself (e.g. group-writable) are enough to control who can access the queue. SEE ALSODirectory::Queue::Normal, Directory::Queue::Null, Directory::Queue::Set, Directory::Queue::Simple.AUTHORLionel Cons <http://cern.ch/lionel.cons>Copyright (C) CERN 2010-2021
Visit the GSP FreeBSD Man Page Interface. |