WAIT_FILESREADY

LANSA Composer

WAIT_FILESREADY

This activity is intended for use in processing sequences that identify and process newly-discovered files.  For example, a processing sequence that uses the WATCH_DIRECTORY activity may need to also use this activity.

Such applications may not be able to safely assume that the file just discovered is immediately available for further processing.  On the contrary, the detection of a new file will frequently precede the file's availability for further processing – often because the application that created the file is still writing its contents.

The WAIT_FILESREADY activity provides services designed to allow LANSA Composer solutions to address these issues and prevent contention conditions that may otherwise arise.  These services include:

  • Providing a cache of files awaiting "ready" status;
  • Permitting the solution to add newly-discovered files to the cache;
  • Performing periodic checks on the status and selected metrics of each cached file to determine its "ready" status;
  • Notifying the solution when a cached file is considered "ready" for further processing.
  • The file cache that the WAIT_FILESREADY activity provides is "global" to the Processing Sequence.  In other words, multiple instances of the WAIT_FILESREADY activity in one processing sequence will refer to the same cache.

 

This is a specialized and advanced activity that behaves in a different way to many other LANSA Composer activities that you may be more used to.  You should ensure you thoroughly understand the activity before designing your solution to use it.

This activity will usually appear in some sort of loop in a processing sequence – for example inside the block associated with an instance of the WATCH_DIRECTORY activity:

  • When executed inside the loop, the processing sequence usually wants to add a newly-discovered file to the cache provided by the WAIT_FILESREADY activity. 
  • It will usually request that the activity also return a file that is considered "ready" for further processing, if there is one.
  • Both these operations can be done in one call to WAIT_FILESREADY when you use the default *PUTGET value for the OPERATION parameter.

 

It is vital to understand that in that one execution of the WAIT_FILESREADY activity like this, the file path added to the cache and the file path retrieved from the cache (if any) will NEVER refer to the same file.  Furthermore, there will ALWAYS be a delay between adding a file to the cache and a later execution of WAIT_FILESREADY reporting that it has reached "ready" state – indeed this delay is a part of the very intention of this activity.

If you have understood this, then you will most likely also appreciate that at any point in time there may well be one or more files in the cache that have not yet been assessed as "ready" and that await further processing.  Understanding this is very important, because when your processing sequence completes the loop that contains the WAIT_FILESREADY activity, it has not necessarily processed all the files that were added to the cache.

Once you have understood these two things, then you are a well on the path to being able to implement the WAIT_FILESREADY activity successfully in your solution.

You can refer to the EXAMPLE_WATCH01 Processing Sequence for an example of using the WAIT_FILESREADY activity.

Note: This activity requires that the LANSA Integrator JSM is executing on a JVM at Java 7 or above.

 

Note: the file status and metrics checked by the WAIT_FILESREADY activity to determine "ready" state will be sufficient for many solutions.  However some applications may call for checks that are not performed by WAIT_FILESREADY.  You should make sure you understand the characteristics of the environment in which you propose to use WAIT_FILESREADY to make sure that the activity is suitable for your purpose.

 

Restarting the WAIT_FILESREADY activity

The WAIT_FILESREADY activity supports processing sequence restart if an error occurs while it is active.  However, the contents of the files cache are not automatically restored.  Consequently, it is possible that some previously cached files may have been lost when the activity restarts.  Your solution may need to take account of this.  Often, when used with the WATCH_DIRECTORY iterator activity with *YES specified for the EXISTINGCONTENTS parameter, for example, the loss of the cache will not be significant, and no special action will be required.

 

INPUT Parameters:

OPERATION: Optional

This parameter specifies the action(s) that the activity is to perform.  The default value used is *PUTGET.

You may specify one of the following:

  *PUT: adds the file specified by the FILEPATHPUT parameter to the cache, using the values specified by the INTERVALS, READYREPEATS and EVALUATE parameters as the values that will apply to determing the "ready" status of this file.

  *GET: retrieves the next available file from the files cache that has reached "ready" status (if there is one), removes it from the cache and puts its file path into the FILEPATHGET output parameter.

  *PUTGET: combines the functions of both the *PUT and *GET operation

  *GETNOTREADY: retrieves a file from the files cache without evaluating its "ready" status (if there is one), removes it from the cache and puts its file path into the FILEPATHGET output parameter.  When this operation is used, the files are retrieved in FIFO order.  You might use this operation in a loop (while COUNTWAITING is not zero, for example) when a processing sequence is ending to take some action of your choosing for any files that remain in the files "ready" cache.

  *LOG: writes a processing sequence log entry for each file remaining in the cache.  The log entry identifies the file and a value that indicates the last reason that the file was evaluated to be not "ready".  This operation is intended for use during the design phase of your solution – it should not normally remain in a completed solution.

FILEPATHPUT: Optional

If *PUT or *PUTGET is specified for the OPERATION parameter, this parameter must specify the full path to the file that is to be added to the files cache.

INTERVALS: Optional

If *PUT or *PUTGET is specified for the OPERATION parameter, this parameter may specify the intervals (in seconds) at which the file's "ready" status will be evaluated.  In other words, the file's status will not be evaluated until at least this number of seconds has elapsed since the file was added or since the last status check was performed.

If you do not specify a value, the default value used is 40 (seconds).

READYREPEATS: Optional

If *PUT or *PUTGET is specified for the OPERATION parameter, this parameter may specify the number of consecutive times the file must be assessed as "ready" before it is finally released from the cache in response to a *GET or *PUTGET operation.

If you do not specify a value, the default value used is 3 (three).  Note that, when used with the default value of 40 for the INTERVALS parameter, this will result in at least two minutes elapsed before the activity will consider the file as having reached "ready" status, even if no changes to the file occurred after it was added to the files cache.

EVALUATE: Optional

If *PUT or *PUTGET is specified for the OPERATION parameter, this parameter specifies the criteria or file attributes that are considered when subsequently evaluating the file's "ready" status.  You may specify one or more of the following values (each value should be separated by at least one space):

  *ZEROLEN:  this test stipulates that a zero-length file will NOT be considered to be "ready".

  *SIZE: this test requires that the file's size must not have changed between consecutive status checks in order for the file to be eligible for "ready" state.

  *LASTCHANGED:  this test requires that the file's last-modified date and time must not have changed between consecutive status checks in order for the file to be eligible for "ready" state.

  *READABLE:  this test checks that a file exists and that the job has appropriate privileges that would allow it to open the file for reading.

  *LOCKABLE:  this test attempts to acquire an exclusive lock on the file (the lock is immediately released, if obtained).  If the lock cannot be acquired the file will NOT be considered to be "ready".

The default value includes all the criteria listed above.  All the specified criteria must be satisfied for the file to be considered "ready" and therefore eligible to be released from the cache on a *GET or *PUTGET operation.

NB: You must specify quote marks around the list of value(s) to distinguish them from built-in variable names.

OUTPUT Parameters:

RESULTPUT:

If *PUT or *PUTGET is specified for the OPERATION parameter, this output parameter will contain the result of the *PUT operation.  It will contain one of the following values:

  OK:  the file was successfully added to the files cache.

  ER:  an error occurred while attempting to add the file to the files cache.

Note that the error condition does not result in the activity ending with an error status – in other words, a CATCH processing sequence directive will not be fired for this condition.  You need to test the value of the RESULTPUT operation if you wish to specifically handle the 'ER' case.

RESULTGET:

If *GET, *PUTGET or *GETNOTREADY is specified for the OPERATION parameter, this output parameter will contain the result of the *GET or *GETNOTREADY operation.  It will contain one of the following values:

  OK:  a file in the cache was evaluated as "ready" (*GET and *PUTGET operations only), removed from the cache and its file path was put into the FILEPATHGET output parameter.

  EMPTY:  no file was retrieved from the cache because the files cache is presently empty.

  NOTREADY:  no file was retrieved from the cache because no files in the cache were evaluated as "ready" (*GET and *PUTGET operations only).

Your processing sequence should, at the minimum, test for the 'OK' result code and treat it as the trigger for further processing of the file that was retrieved.

FILEPATHGET:

If *GET, *PUTGET or *GETNOTREADY is specified for the OPERATION parameter and if the RESULTGET output parameter contains 'OK', then this output parameter will contain the file path of the file that was retrieved from the cache.

REASONNOTREADY:

If *GETNOTREADY is specified for the OPERATION parameter and if the RESULTGET output parameter contains 'OK', then this output parameter will contain a value that indicates the reason that the file was last evaluated to be not "ready".  The reason codes can include *NOTEVALUATED. *ZEROLEN, *SIZE, *LASTCHANGED, *READABLE, *LOCKABLE and *READYREPEATS.

COUNTWAITING:

Irrespective of the requested operation or its result, this output parameter will contain a count of the number of files remaing in the files cache.