I’m very glad to see we had the capability to identify
a problem of this nature, that we did in fact identify the problem ourselves
and took immediate action to not only correct and resolve it but put procedures
in place to prevent its recurrence and advised the board in a timely manner.
Joel
From: Shelley,
Barry, N1VXY [mailto:bshelley@arrl.org]
Sent: Tuesday, June 10, 2008 9:44
AM
To: arrl-odv
Subject: [arrl-odv:16820] Re:RE:
QST Delivery Problems
Bob, et al:
I
apologize if the answer is a bit long-winded but it is important to understand
something of the computer configuration to understand the answer.
The process
of creating what is referred to as the “main run” (the monthly list
of addresses to which QST is sent) uses one of the several servers in the
Siebel system configuration called the “report server”. This
particular server contains a copy of the database as of the close of business
the prior day. It is this server which is used for almost all our reports as
the process of culling through all the information for certain queries (i.e.
reports) on the production server would slow in-day response times and
processing significantly.
The main run
process is run against the report server and during this a work file is created
which is accessed automatically and sent to the printer for use in creating the
mailing labels. Through an operator error, the main run process for the June
issue was run against the live server, creating the appropriate work file, but
on the production server. The automated process then drew the work file
from the report server, as usual, but, since that file had not been updated, it
was the prior month’s file and hence the error. The error was discovered
when they were processing the main run for the July issue.
Since the
error was discovered, we have modified the main run processing code to include
steps whereby the computer operator must verify and confirm that the correct
information is being used to create the file on the proper server or the
program will not continue. In addition, we are creating checklists to be used
which identify the critical steps in the workflow.
The
configuration noted above has been in place since the installation of the
Seibel system and is part of the appropriate back-up procedures currently
established. I’m sure everyone understands that keeping the entire
database on only one server would be completely inappropriate. In addition,
limiting access to the production server is a necessary security precaution.
The direct
cost of this situation is around $4,000 in postage. There was some additional
time spent by staff to identify and resolve the situation. We have not
quantified it at the present time but I would estimate that it involved about
15-20 hours of time in total.
73,
Barry J. Shelley, N1VXY
Chief Financial Officer
ARRL, Inc.--The National Association for Amateur Radio
Telephone: (860) 594-0212
E-mail: bshelley@arrl.org
www.arrl.org
From: Bob
Vallio [mailto:rbvallio@gmail.com]
Sent: Mon 6/9/2008 7:37 PM
To: arrl-odv
Subject: [arrl-odv:16819] Re: QST
Delivery Problems
"standard monthly process
failed when the wrong server was accessed for the run and that the possibility
of this problem re-occurring has now been removed."
The question for me, that begs an answer, is how did this
"possibility" occur? The damage to our value in our member's
eyes is very difficult to measure. I'm certain that there will be some
number of members who are very displeased at this outcome. Additionally,
what did this "possibility" cost us in materials, man-hours, and postage?
I'll be the first to say that my IT knowledge is not great, but why would files
reside on more than one server; and, those servers be independently accessible
for files used for as important a purpose as assuring that our members receive
our flagship publication?
Bob Vallio -- W6RGG