10.2.2 Considerations for Extended Duration Processing Sequences
Many LANSA Composer users implement solutions involving Processing Sequences that run for an extended duration – perhaps continuously.
Frequently such solutions are designed to monitor some source of in-coming data, such as new files arriving in a specified directory, and initiate further processing when an item is detected.
This style of Processing Sequence is very different to one that has a known and finite unit of work to perform, performs it and ends. Consequently, some special considerations apply to the design and implementation of this type of Processing Sequence.
Below we have listed some suggestions that you may wish to consider when implementing such solutions. Refer to the following headings:
The supplied example Processing Sequence EXAMPLE_WATCH01 is an example of an extended duration processing sequence and implements some of the suggestions in this article. You may wish to examine and run EXAMPLE_WATCH01 to further investigate the techniques suggested. Refer to 10.2.1 Example Processing Sequences for more information.
You may wish to reduce the logging level for your extended duration Processing Sequences from the values you have specified in System Settings. In selected Processing Sequences you can override the logging values specified in System Settings by changing the values of the and built-in variables. The processing sequence EXAMPLE_WATCH01, for example, contains the following ASSIGN directives near the beginning of its processing:
Reducing the logging level in this way will reduce the logging burden of your extended duration Processing Sequence without affecting the logging behavior of other Processing Sequences that you run.
If your System Settings specify logging level and is also selected, then an extended duration Processing Sequence that uses an activity that in turn uses the LANSA Integrator JSM may take a very long time to end after it has completed its normal work. This is usually due to retrieving and logging the large LANSA Integrator trace data that has been generated. You can avoid this by implementing either or both of the logging overrides shown using the built-in variables.
Allow your processing sequence to frequently enter an idle or wait state
It is usually a characteristic of extended duration processes that they spend a great deal of their time waiting for something to happen – for a file to arrive, for example, or for a message to be received in a message queue.
It is important that your implementation should allow the process to enter an idle or wait state and then to stay in that state for as long as is practical. This will reduce the impact the process has on the system and on other jobs running on the system.
In many cases, activities used in such Processing Sequences have a parameter that specifies how long the activity should wait for the item that it is receiving. Some examples include:
- The DTAWAIT parameter of the DTAQ_RECEIVE activity;
- The WAITTIME parameter of the MSG_RECEIVE activity;
- The MSGWAIT parameter of the MSGQ_RECEIVE activity;
- The WAIT parameter of the WATCH_DIRECTORY activity;
You should make sure that you set those parameters appropriately. If you choose a value that is too short or, even worse, choose to specify a zero wait, then you may find that your Processing Sequence expends a lot of system resources doing very little but looping endlessly. Using a sensible wait value will allow the Processing Sequence to frequently enter an idle/wait state when it has no work to do.
Note that, at the other extreme, specifying an indefinite wait is not always a good idea either. Refer to Check for controlled end to find out one reason why.
If your extended duration Processing Sequence uses an activity, such as MAIL_RECEIVE or FTP_INBOUND for example, that does not have a built-in "wait" function and you are using that activity to "poll" for in-coming data, then you should consider combining the activity in a loop with the SLEEP activity to create a similar effect to, for example, the WAIT parameter of the WATCH_DIRECTORY activity.
Delegate transaction processing
Frequently extended duration Processing Sequences are designed to monitor some source of in-coming data, such as new files arriving in a specified directory, and initiate further processing when an item is detected.
It is usually good practice to separate the former (the waiting and discovery) from the latter (the processing). The less work that your extended duration Processing Sequence does, the better. As soon as it "discovers" a new item to process, it should usually initiate another process to do the processing, while it stays busy with waiting for further new items.
This approach has the additional benefit that the processing of any individual transaction does not create a bottleneck that prevents or delays the receipt and processing of further transactions.
You can use the COMPOSER_RUN activity in your extended duration Processing Sequence to initiate the secondary Processing Sequence through the LANSA Composer request server. Remember to specify 'NO' for the SYNCHRONOUS parameter to make sure that the first Processing Sequence does not have to wait for the secondary Processing Sequence to complete.
Frequently, it is the very nature of an extended duration Processing Sequence that its task is infinite, or at least indefinite. That is, it will continue monitoring for whatever it is that it monitors throughout the business day and perhaps the night and weekends too.
Activities such as WATCH_DIRECTORY are specifically designed for this indefinite processing. This activity will continue iterating until or unless your solution ends the loop explicitly or until the job, subsystem or system end.
Alternatively, if your extended duration Processing Sequence uses an activity, such as MAIL_RECEIVE or FTP_INBOUND in a loop to monitor for new items, you have probably designed the loop to be indefinite using a LOOP, WHILE or UNTIL directive.
In either event, it is good practice to implement the solution such that a controlled end for it can be externally triggered. In most cases, this can be done by checking and acting upon the value of the built-in variable at an appropriate point in your solution. The processing sequence EXAMPLE_WATCH01, for example, implements such a provision using the condition specified on a LEAVE directive:
This will allow you to end the processing Sequence run using the window in the LANSA Composer client or using the . The window, for example, provides an toolbar button or the command on the menu – these are shown only when the Processing Sequence run is .
If controlled end for a Processing Sequence run is requested in this way, then the condition shown above will be satisfied. In the case of EXAMPLE_WATCH01, this will result in the main WATCH_DIRECTORY iterator loop ending and the Processing Sequence will then end too.
On IBM i servers, the test of the *shutdown built-in variable will also detect when a controlled end has been initiated for the IBM i job, subsystem or system, using the ENDJOB, ENDSBS or PWRDWNSYS commands or equivalents.
By placing a test for controlled shutdown in an appropriate place in your extended duration Processing Sequence in this way, you can help to ensure that, when the Processing Sequence is shutdown, it is done in a controlled fashion and at an appropriate transaction boundary, thereby protecting the data integrity of your solution and your application.
Periodically restart for manageability and performance
As already discussed, it is frequently in the very nature of an extended duration Processing Sequence that its task is infinite, or at least indefinite. While that may be so, there are several very good reasons that you might not want the task to be served with a single Processing Sequence run that runs over many hours or even days. These reasons include:
- The Processing Sequence Log will become exceptionally large, which will make it slow to load and exceptionally difficult to analyse in the event that you have to perform troubleshooting;
- Over an extended time, and for various reasons, the system resources consumed by the job containing the Processing Sequence run may gradually increase, possibly impacting performance on the system as a whole;
It is usually good practice to limit the lifetime of a single Processing Sequence run, allow it to end and begin another to resume and continue its processing. Doing so is likely to offer benefits with respect both to the manageability and the performance of the solution.
There is more than one way to accomplish this. The Processing Sequence EXAMPLE_WATCH01 implements the following sequence of directives at an appropriate point inside its main WATCH_DIRECTORY iterator loop for this purpose:
The COMPOSER_RUN activity simply submits the same Processing Sequence again through the LANSA Composer Request Server, and then the Processing Sequence ends. In this instance, this occurs after 999 iterations (the value of the variable &ITERATIONS_MAX is set to 999 earlier in the Processing Sequence).
This results in multiple Processing Sequence runs, each covering a specific time period, which makes it much easier to examine the logs and diagnose issues, when necessary.