Daemon class

The Daemon class provides a facility for issuing Model 204 commands from inside a User Language request, simulating the calling of programs as if they were subroutines.

Daemon objects are much like an object-oriented version of the Sirius $COMMxx $functions ($CommBg, $Command, $CommndL), which use Sirius sdaemons to run a command on another thread on behalf of the issuing user. Daemon objects offer a number of advantages over $COMMxx functions and over non-daemon program execution:

• They can execute multiple calls per sdaemon login/logout sequence.
• From within a User Language program, you can issue commands that ordinarily cannot be executed within that program, for example, a REORG (for which you typically have to exit the program and execute another procedure).
• From within a User Language program, you can call a legacy application that is already packaged as a program as if it were a subroutine. Ordinarily, such a program is equivalent to a command, and it cannot be called as a subroutine. You have to switch to it and set up complex mechanisms to pass control between the two routines.
• You can reduce the size of your programs, gaining relief from server size limitations. Although a network service like a socket might also provide benefits comparable to those listed above, a Daemon object offers simplicity, both in the ability to implement and in the ease of adoption. Security-wise, there is no doubt which user is issuing the command, what subsystem that user is running, what files the user has open, what privileges the user has, etc.
• Data structures are more easily passed between a master and an sdaemon than over a communications connection, where every data structure has to be serialized by the sender and deserialized by the receiver.

The Daemon API is implemented as a set of functions, subroutines, and properties, documented as the Daemon methods.

Working with Daemon objects

In this Wiki, the terms "daemon" or "daemon thread" are often used to mean "the sdaemon thread that is activated or involved when you employ or operate on a Daemon object." The user or thread that invokes a daemon is the "master" or "master thread." If a daemon itself invokes another daemon, both daemons have the same master, and the first daemon is the "parent" of the second. For more information about the master/parent distinction, see the MasterNumber property.

A daemon thread runs a command on behalf of the issuing user, and the thread is only available to that master user. The thread does not, for example, maintain some context and accept requests from different users. A daemon thread operates (only) synchronously: processing on the issuing thread pauses to wait for the daemon thread to run the command(s) it receives. To pass a command to a daemon, you use the Run function with the command as argument. Two examples follow for the %foo Daemon object:

%foo:Run('OPEN FILE MYPROC') %list = %foo:Run('I REPORT')

To have a daemon thread run multiple commands, you can specify multiple Run method calls, or you can define the commands as items in a Stringlist, and then specify the Stringlist as the argument in the Run method. The Stringlist is fed to the daemon as terminal input and executed item by item until that input runs out. For example, if the commands from the examples above were defined in that order as the items in Stringlist %testproc, the following statement would include the procedure REPORT from the MYPROC file:

%foo:run(%testproc)

You use a Stringlist argument in this way to pass a User Language request to a daemon. Note: If the code in a request you pass to a daemon in a Stringlist depends on being mixed case, precede your Run method call with a statement like the following, so the daemon thread will preserve mixed case:

%foo:run('*LOWER')

Although a daemon thread is request-specific, Daemon objects do have persistence. The daemon thread waits for work as long as the Daemon object exists. Since the Daemon object can be Global or Session as well as not, its span of existence can be arbitrarily long or short, depending on how it is being used.

By default, a daemon thread does not share record locks or transactions with its master thread, and Commits and Backouts apply only to the thread that issues them. However, you can easily create daemons that share the update unit with the issuing thread. When such a daemon ("Transactional daemons", below) commits or backs out a transaction, the issuing thread's updates are committed or backed out as well.

If a daemon thread does a user restart, the request is cancelled, but the calling thread is not also restarted. If a calling thread is logged off or restarted, its associated daemon threads are deleted. A daemon is not automatically restarted if the master restarts. The daemon does a user restart if it is processing a User Language request that calls for terminal input, and no Run method input follows the end of request.

Deleting a Daemon

You can use the system Discard method to explicitly delete a Daemon. In addition to deleting the object, Discard causes the daemon thread to log off. If the daemon is not in a state to notice the Discard and shut down gracefully, it is bumped with a Model 204 user restart. This makes Discard instantaneous from the issuing thread level and as fast as possible from the daemon thread level.

As soon as the Discard returns, the count of daemons used by that master thread is reduced by one, even if the daemon hangs for some reason. The MAXDAEM system parameter stipulates the maximum number of daemon threads per master thread.

Besides explicit Discards, a daemon is also deleted if its master thread is logged off or restarted. In such a case, the daemon is not automatically restarted if the master restarts.

A Daemon example

In the following example, the Run method is called three times for the %speed daemon. In the first two calls, the daemon executes a command that has no output object. In the final call, it executes a request and returns an XmlDoc. Note the use of text to %list, which captures to a Stringlist object the program submitted to the daemon:

b %speed is object daemon %list is object stringList %silly is object xmlDoc %speed = new %speed:run('V UTABLE'):print %speed:run('*LOWER'):print %list = new text to %list b %doc is object xmlDoc  %(daemon):getInputObject(%doc) %doc:selectSingleNode('/outer/inner'):addAttribute('foo','bar')  %(daemon):returnObject(%doc) end end text %silly = new %silly:loadXml('<outer><inner><innest>huh?</innest></inner></outer>') %speed:run(%list, %silly, %silly) %silly:print end

The example result follows:

HDRCTL X'01' HEADER CONTROL LVTBL 3000 LENGTH OF VTBL LNTBL 150 LENGTH OF NTBL . . . . . . . . . FIXSIZE 20392 FIXED SERVER SIZE REQUIREMENT SERVACTV 150416 CURRENT SIZE OF THIS SERVER <outer> <inner foo="bar"> <innest>huh?</innest> </inner> </outer>

Daemon record locking

Prior to Sirius Mods version 6.8, sdaemons running on behalf of other threads, via the $COMMxx functions or via Daemon objects, can suffer record locking conflicts with other threads in the same thread family, because they are separate update units. Two threads are in the same family if one is a synchronous child daemon of the other via a $COMMxx function or via a Daemon object. In addition, all families with common threads are considered to be a single family. So, if thread A is a synchronous parent of thread B, which is a synchronous parent of thread C, threads A, B, and C are all considered part of the same family. Furthermore, if in this example, thread B has two other synchronous children (via Daemon objects), threads D and E, then threads A, B, C, D, and E are all considered part of the same family.

As of version 6.8, only one condition remains under which two threads in the same family can suffer a record locking conflict: if they both try to update the same record, which would require both threads to have an exclusive pending update lock on the record being updated. However, even this possibility of a record locking conflict is eliminated for transactional daemons.

Sirius Mods version 7.0 introduced support for asynchronous and independent running of daemon requests. A daemon running asynchronously or independently is, at that point, no longer part of the family for the purposes of record locking. That is, an independently or asynchronously running daemon, or any of its daemons, cannot hold record locks that would conflict with its original master or any other daemons in the master's family.

Effectively, while a daemon is running independently or asynchronously, the master's family is split from the daemon's family. For example, thread A has daemon threads B, C, and D, and D has daemon threads E and F; if D runs asynchronously, then while it is running asynchronously, the family A, B, and C is considered separate from D, E, and F as far as record locking goes. Since an asynchronous request will (usually) eventually go back to waiting for its parent, at that point the families are rejoined. Since an independent request never rejoins its parent, once a family is split because a daemon is running independently, it is never rejoined. Since a daemon thread (and its daemons) that is running asynchronously or independently leaves its family at the moment it starts running asynchronously or independently, it is possible for the RunAsynchronously or RunIndependently methods to fail because they would result in a record locking conflict betwee the newly split families.

Transactional daemons

The New constructor for Daemon objects has a Boolean, named parameter called Transactional:

%pantalaimon is object daemon ... %pantalaimon = new(transactional=true)

If Transactional is set to True, the master and daemon thread will share a single updating transaction. This means that if updates are performed by both the master and daemon thread, these updates are all committed or all backed out when a commit or backout is performed on either the daemon or master thread. If the master thread of a transactional daemon closes a file that was open in update mode, the transaction is committed; if the transactional daemon closes the file, the transaction is unaffected. If a transactional daemon thread is bumped, its master is also bumped.

If a thread has multiple transactional daemons, then all those daemons share a single updating transaction with the master thread. In addition, if a transactional daemon thread itself has transactional daemon threads, those threads also share a single updating transaction with the ultimate master thread. Effectively, all of a thread's transactional daemons, and all those transactional daemon's transactional daemons, and so on, make up a transactional family that shares a single updating transaction. The processing of updating transactions on a transactional daemon thread differs from other threads in these ways:

• Transactional daemon threads never do an implied commit. Other threads typically do implied commits at end of request, unless inside a non-autocommit subsystem. In addition, certain operations, such as a procedure update, cause an implied commit on non-transactional daemon threads.

  Of course, an explicit Commit or Backout statement run on a transactional daemon thread results in the transaction being committed or backed out, including those updates performed on the master thread or some other thread in the transactional family.

• As an outgrowth of the previous item, transactional daemon threads can perform procedure updates in the middle of an updating transaction, as well as other file maintenance operations, such as field definitions.
• Transactional daemon threads cannot perform User Language updates against files that their ultimate master thread does not have open in update mode. The ultimate master thread is the thread's master or the master's master, if that thread is, itself, a transactional daemon, and so on.

  The actual commit or backout is always performed on the highest level master thread in a transactional family, so that thread must have update privileges for all files in an updating transaction. One benefit of this approach is that it's possible for a daemon thread to perform some updates, go away (log off), and for the update to still not be committed.

  As noted in "Daemon record locking", transactional daemons cannot suffer record locking conflicts with other threads in the family, and they can even update the same records as other threads in the transactional family.

Janus Web API methods

Under Model 204 7.8 and later, if the WEBOPT X'10' bit is set, transactional daemons can access web data via $web methods as long as the ultimate master is a thread created to process a Janus Web request. The $Web_Api function is available so a thread can determine if it can access web data with $web functions, whether it is the master web thread or a descendant transactional daemon.

Asynchronous and Independent daemons

RunAsynchronously and RunIndependently are daemon-spawning methods that return immediately, before their daemon has completed processing its input command:

%pantalaimon is object daemon ... %pantalaimon = new ... %pantalaimon:runAsynchronously(%commands) ... %pantalaimon:runIndependently(%commands)

The ContinueAsynchronously and ContinueIndependently methods, discussed below in "ReturnToMaster in Daemons", invoke asynchronous and independent processing in already-created daemons.

Under RunAsynchronously

With RunAsynchronously, when the daemon commands are all processed, the daemon goes back to waiting for the master to tell it what to do next. To do this, the master thread has to make sure the daemon thread has completed processing by issuing the WaitAsynchronous method against the Daemon object:

%pantalaimon is object daemon ... %pantalaimon:runAsynchronously(%commands) ... %pantalaimon:waitAsynchronous

Even if a daemon thread running asynchronously has completed processing its commands, a WaitAsynchronous method must be issued against the Daemon object before further commands can be sent to the daemon. Most methods applied to a Daemon object that is running asynchronously result in request cancellation. Once the daemon that had been running asynchronously has been resynched with the master thread via WaitAsynchronous, the master thread can treat it just as any other daemon thread and issue further Open, Run, RunAsynchronous, RunIndependently, or other methods against the daemon.

Under RunIndependently

With RunIndependently, the daemon logs off when the daemon commands are all processed. This means that a daemon thread for which RunIndependently is issued can no longer be controlled by its master thread (as the method name would suggest). In fact, upon return from the RunIndependently method, the Daemon object is discarded, so any further methods invoked against it would result in request cancellation.

RunIndependently is roughly analogous to the $CommBg function with some important differences:

• The RunIndependently method allows interaction with the daemon before it does its independent processing.
• RunIndependently operates on a Daemon object that already has a daemon thread associated with it. $CommBg obtains a daemon thread as part of its processing and, if one is not available or the maximum independent/background daemons are already running, the request is enqueued.

That RunIndependently already has a thread has positive and negative aspects: If the master thread wants to be sure that the daemon thread is actually running, RunIndependently has the preferred behavior. If the master thread doesn't care whether the request runs immediately (and in fact, would prefer that the request be run later if/when it is convenient), then $CommBg's behavior is preferable. Note, however, that there is no guarantee that a request enqueued by $CommBg will ever be run — it is quite possible that the request remains enqueued until the Online is brought down.

Despite their similarities, $CommBg and RunIndependently use different terminology for the daemon threads. $CommBg refers to the threads running "in the background," while RunIndependently refers to them running "independently."

As alluded to above, there is a limit to the number of background or independent requests that may be running at a time. When this limit is exceeded for $CommBg requests (or when there are simply no daemon threads available to run the requests), the requests are enqueued, until a daemon thread is available to run the request and the limit for background/independent requests would not be exceeded. RunIndependently, on the other hand, causes a request cancellation if the limit for background/independent requests is to be exceeded. RunIndependently cannot fail, because there are no daemon threads available since the daemon thread is assigned to the object at the time it is instantiated (with the New method), so RunIndependently does not have to allocate a daemon thread for the request.

Thread limits

Before Sirius Mods 7.0, the limit for background/independent requests was one-half of, or ten fewer than, the number of daemon threads, whichever was greater. That is, if there were 10 daemon threads defined in an Online, no more than 5 of them could be running $CommBg requests. If there were 40 daemon threads defined, no more than 30 could be running $CommBg requests. Under Sirius Mods 7.0 and later, the system parameter MAXBG indicates the maximum number of simultaneously running background/independent requests that are allowed at a time. For backward compatibility with previous Sirius Mods versions, the default value for MAXBG is one-half of, or ten fewer than, the number of daemon threads defined to the Online, whichever is greater.

Daemon threads running asynchronously, as well as threads that are about to run independently (but aren't yet), are counted against the master thread's MAXDAEM limit. When a thread switches to running independently, it is no longer counted against the master's MAXDAEM limit.

When to run asynchronous daemons

RunAsynchronously can be used to allow a thread to do parts of large, complex tasks in parallel. The master thread could be doing some of the work at the same time a daemon thread is asynchronously doing some of it. In fact, multiple daemon threads can be running asynchronously at the same time for a given master thread. When the master thread is done with its work, it can then do a WaitAsynchronous for each daemon running asynchronously until all the work is completed.

If the work being run in parallel is very CPU intensive, there is not likely to be any benefit in running work asynchronously unless the Online is using multiple tasks (MP/204 authorized and AMPSUBS > 0). On the other hand, if the work being performed asynchronously is I/O intensive, a significant reduction in the time required to perform the work can be achieved compared to doing all the I/O serially in the master thread. Work involving network delays is an ideal candidate for running asynchronously, especially if multiple such delays are present in a batch of work.

ReturnToMaster in Daemons

As of M204 Version 7.5 (or Sirius Mods 8.1, it is possible for a User Language request to return to the master thread without terminating the User Language program. This is done with the ReturnToMaster method. The master thread can then tell the daemon to continue processing with the Continue method (or the related ContinueAsynchronously and ContinueIndependently methods). Objects can be passed back and forth between the daemon and master in this exchange, allowing the master and daemon threads to communicate with each other while both are running User Language requests.

In the following example: a daemon thread passes an XmlDoc indicating its current status, waits for the master thread to tell it to continue, and then extracts the next task from an XmlDoc sent by the master thread:

%doc is object xmlDoc ... %doc = new %doc:addElement('status', %status) %(daemon):returnObject(%doc) %(daemon):returnToMaster %(daemon):getInputObject(%doc) %nextTask = %doc:value('nextTask')

And here is the master thread code that receives the status XmlDoc and then passes the next task XmlDoc back to this daemon thread:

%daemon is object daemon %doc is object xmlDoc ... %daemon:run(%command, output=%doc) %status = %doc:value('status') ... determine next task %doc = new %doc:addElement('nextTask', %nextTask) %daemon:continue(input=%doc)

The DaemonState enumeration

The DaemonState enumeration describes the current state of a daemon thread associated with a Daemon object. A newly created Daemon object starts in the Waiting state and then changes state as certain methods are performed on it. Certain methods are restricted to Daemon objects in a particular subset of states.

The values of the DaemonState enumeration are:

AsynchronousFinished	The master thread has invoked a RunAsynchronously or ContinueAsynchronously method, and the daemon thread has completed the processing for that request. The master thread should invoke the WaitAsynchronous method to return the daemon to a Waiting or WaitingForContinue state.
Lost	The daemon thread has been lost (restarted or BUMPed). This state is only possible on a daemon object if a daemon processing a RunAsynchronously request is BUMPed or restarted.
RunningAsynchronously	The master thread has invoked a RunAsynchronously or ContinueAsynchronously method and the daemon thread has not yet completed the processing for that request.
Waiting	The daemon thread is at command level and is waiting for a request from the master.
WaitingForContinue	The daemon thread has invoked the ReturnToMaster method and is waiting for a request from the master.

List of Daemon methods

The List of Daemon methods shows all the class methods, with a brief description of each.