11.27 Estimating Application Performance by Running an Application
A common problem when developing applications for the IBM i is trying to estimate how they will actually execute on a "busy" or "loaded" production system.
If you test a function and "guess" that it seems to run okay, then you run the following risks:
- You may have run it many times before you do the test. This may mean that parts of the program or the other objects it accesses are still resident in the computer's main memory, thus reducing the time it takes for the program to be accessed.
For example, take the following simple program which does no initial processing, just puts up a screen panel:
REQUEST FIELDS(#PRODNO)
You may test this program and find that the panel appears in 1 second on average. However, you may find that if you sign onto the computer at the start of the day and then attempt to access the program initially it takes 4 seconds.
If you then repeatedly access the program from the menu, you find the access time gets less and less. The faster you access the program and return to the menu and (re)access the program the lower the access time seems to get. In fact you might find that the access time drops down to an average of 1/4 a second or less.
You might also notice that if you stop accessing the program for 5 minutes, and have a coffee break, your next initial access climbs back to 3 seconds or so.
How can this be, when you know that the program is doing exactly the same amount of work every time it is accessed (assuming that it is not an *HEAVYUSAGE program) ?
The reason is simple. When you access and re-access the program very quickly, the program and the objects that it accesses tend to remain in the computer's memory. However, if it is the very first access they are not in memory. Likewise, if you take a long break between interactions, it is likely that the objects are "rolled out" of memory back onto disk. If you wish to re-access them, they must be "rolled in" to memory again, causing you to wait while this is done.
This phenomena is not new, and applies to many "virtual memory" operating systems such as the IBM i, but you should be aware of its existence when you are testing your applications.
The faster you work, the quicker the system will respond. The slower you work, the slower the system will respond. This effect is very pronounced in systems that have very long "think times" such as complex database inquiries, and least pronounced in system that have very short "think times" such as high speed data entry systems.
You should allow for this in any testing. If your system has a long "think time", you should emulate this in testing, or you may be using the operating system in a way that does not truly reflect the way that users of your system will.
- You may be on a totally different type of machine or using a very different type of environment. If you do your test on an IBM i 270 at 9.00pm at night, when everybody else has gone home, and the actual target machine is a very busy 170, then the results you obtain will be totally misleading.
- The system may not be busy enough. When the IBM i begins to become loaded up with work, its performance characteristics change.
The transaction throughput curves for most IBM i rise slowly until a certain load is reached. At this point the curve rises almost vertically, so that almost no more work is being done, but the response times degrade very rapidly.
If you are using a system that is operating near to this "knee of the curve" you will find that a relatively small increase in the machine load will dramatically degrade the response time of all online users and batch jobs.
Hopefully all of these points combined are enough to convince you that attempting to estimate how an application will perform in a production system is not a simple task and it should not be just doing by some simple testing and then concluding that "it seems to run okay".