Leading Mortgage Company Improves Mainframe Response 60 Percent Without Buying Hardware
Industry: Mortgage
Location: North America
Founded in 1948, North American Mortgage Company focuses completely on home mortgage lending and is the nation's ninth largest residential lender. At nearly every North American Mortgage Company office, all loan processing, underwriting, and funding are done locally. This means that the company’s customers deal with mortgage veterans who are empowered to act on their behalf immediately.
The Challenge
Mainframe performance is critical to North American Mortgage because during peak
periods in the refinancing cycle the company faces the challenge of processing over
900,000 online transactions per day in a single CICS region. While hardware changes
have helped, even greater improvements have come from optimizing VSAM buffers, local
shared resource (LSR) pools, data and index buffers, control intervals, and other
parameters to reduce physical I/Os.
North American Mortgage is pushing IBM’s VSE/ESA 2.3 operating environment to the limit by running an average of between 700,000 and 900,000 online transactions per day and close to 1,000,000 transactions per day during peaks. The company also sees a significant monthly processing cycle as volume is relatively light during the early part of the month but builds to a much higher volume in the last week as sales reps rush to complete jobs by month’s end. The company uses the Mortgage Loan Closing System originally developed by Symbolics, but which has since gone through several hands and is now being supported and upgraded by the company’s own staff. The fact that North American Mortgage’s business ranges from Hawaii to the East Coast means that online processing must be available from 4 a.m. to 9 p.m. PST leaving only a seven-hour batch window.
About eight months ago,” said Ron Griffiths, vice president of technology services, “we were unhappy with both our online response time and our batch processing performance.” The company upgraded the processor of its 9672 X17 mainframe from a 105 MIPS two-processor arrangement to a single UNI processor with 178 MIPS. The processor upgrade reduced average response time to 0.3 seconds but Griffiths felt that further improvements were needed to increase user satisfaction. The company also faced problems with excessive batch run time during peak processing periods.
“When volume was heavy, we were unable to complete processing during the nightly batch window with the result that the operating reports our managers relied upon for decision-making were a day late,” Griffiths said. He gave consultant Walt Kondraftieff the assignment of tuning the company’s operating environment to provide further reductions in online response and batch processing time.
The Solution
Kondratieff said that selecting the right monitoring tools was critical to an efficient
tuning process. “Tuning the CICS/VSE environment is a time-consuming and difficult task,”
he said. “A monitoring tool that forces you to go through a lot of hoops to get the
information you need can really slow down the process. We selected
ASG-TMON™ for CICS
and ASG-TMON for VSE because these tools provide an interface that allows you to quickly
identify problems that are slowing system performance and easily zero in on the root cause.”
Kondratieff said that he has discovered that minimizing physical I/O is generally the key to CICS/VSE performance optimization. He uses a technique called step analysis which involves looking one by one at each step of various programs to determine statistics such as number of physical I/Os and files that are utilized and even down to the level of which extent in these files is being hit most frequently. The I/O analysis screen in TMON for CICS identifies how much I/O is being performed on each flat file and also identifies what proportion of those requests are filled from buffers versus those that require DASD access. Often he discovers that a few particular files make up the majority of the I/Os and focuses his effort in that direction. He arranged the TMON monitor to sort the files in order of physical I/O frequency.
“At a glance,” Kondratieff said, “I can see which VSAM files and extents are generating the most physical I/O and make appropriate adjustments to the buffer space. My usual goal is to achieve a 10 percent or lower ratio of physical I/Os to total I/O requests.”
One of his first steps is usually to tune the LSR pools. With 15 pools available, he starts by isolating the heaviest usage files and dedicating a pool to each one of them. Kondratieff also increases index buffers for these high usage files. He also considers how each file was defined, in particular the size of the control interval, which is the unit of transfer between memory and DASD. While CI splits are not a major concern because the company has a fast processor, CA splits have a significant impact on response time. This occurs when a physical I/O is spread over multiple DASD tracks. “No one method works to reduce physical I/O on every file,” Kondratieff said.
At any given moment, a certain range of loan numbers is active while there are also a large number of older loans that are kept on DASD for reference purposes that are only infrequently accessed and will eventually be purged and archived on tape. The result is that the files that are receiving the most hits are continually changing as new records are created and older records become inactive. TMON for CICS makes it possible to track control interval and free space allocation so that new space and buffers can be allocated to files as needed for new records and removed from older files that are used only for reference purposes. Kondratieff continually adjusts the buffer space allotments so that the most active files receive the majority of the buffer space.
Measurable Results
TMON for VSE has provided special help in reducing batch processing times by providing a
console log that gives detailed information on previous runs. Kondratieff uses this tool
to quickly scan jobs the previous night’s processing, determining the overall time taken
by each job and drill down for more detailed information such as the average amount of
time in seconds to provide I/O to a particular step on jobs that he feels bear further
investigation. For example, he used this method recently to identify a job step that
was doing hundreds of thousands of physical I/Os. “It stood out like flashing neon on
the monitor screen,” he said. “I saw that there was no buffering at all on this particular
file and, even worse, that it was configured to transfer records with a typical size of
700K in 38 byte chunks.” With the problem identified, Kontratieff was able to quickly fix
it by adjusting VSAM buffers and control intervals.
“During the past eight months we have cut the average response time from 0.3 second to 0.12 second without any hardware changes. We have also reduced the batch-processing window to the point that we have plenty of slack to handle unusually heavy days within the normal processing windows. By consistently utilizing the procedures outlined here, we have continually improved both our online response time and reduced the length of the batch processing window,” said Griffiths.
