I'm very glad to see we had the capability to identify a problem of this nature, that we did in fact identify the problem ourselves and took immediate action to not only correct and resolve it but put procedures in place to prevent its recurrence and advised the board in a timely manner. Joel _____ From: Shelley, Barry, N1VXY [mailto:bshelley@arrl.org] Sent: Tuesday, June 10, 2008 9:44 AM To: arrl-odv Subject: [arrl-odv:16820] Re:RE: QST Delivery Problems Bob, et al: I apologize if the answer is a bit long-winded but it is important to understand something of the computer configuration to understand the answer. The process of creating what is referred to as the "main run" (the monthly list of addresses to which QST is sent) uses one of the several servers in the Siebel system configuration called the "report server". This particular server contains a copy of the database as of the close of business the prior day. It is this server which is used for almost all our reports as the process of culling through all the information for certain queries (i.e. reports) on the production server would slow in-day response times and processing significantly. The main run process is run against the report server and during this a work file is created which is accessed automatically and sent to the printer for use in creating the mailing labels. Through an operator error, the main run process for the June issue was run against the live server, creating the appropriate work file, but on the production server. The automated process then drew the work file from the report server, as usual, but, since that file had not been updated, it was the prior month's file and hence the error. The error was discovered when they were processing the main run for the July issue. Since the error was discovered, we have modified the main run processing code to include steps whereby the computer operator must verify and confirm that the correct information is being used to create the file on the proper server or the program will not continue. In addition, we are creating checklists to be used which identify the critical steps in the workflow. The configuration noted above has been in place since the installation of the Seibel system and is part of the appropriate back-up procedures currently established. I'm sure everyone understands that keeping the entire database on only one server would be completely inappropriate. In addition, limiting access to the production server is a necessary security precaution. The direct cost of this situation is around $4,000 in postage. There was some additional time spent by staff to identify and resolve the situation. We have not quantified it at the present time but I would estimate that it involved about 15-20 hours of time in total. 73, Barry J. Shelley, N1VXY Chief Financial Officer ARRL, Inc.--The National Association for Amateur Radio 225 Main St. Newington, CT 06111 Telephone: (860) 594-0212 E-mail: <mailto:bshelley@arrl.org> bshelley@arrl.org <http://mymail.arrl.org/exchweb/bin/redir.asp?URL=http://www.arrl.org> www.arrl.org _____ From: Bob Vallio [mailto:rbvallio@gmail.com] Sent: Mon 6/9/2008 7:37 PM To: arrl-odv Subject: [arrl-odv:16819] Re: QST Delivery Problems "standard monthly process failed when the wrong server was accessed for the run and that the possibility of this problem re-occurring has now been removed." The question for me, that begs an answer, is how did this "possibility" occur? The damage to our value in our member's eyes is very difficult to measure. I'm certain that there will be some number of members who are very displeased at this outcome. Additionally, what did this "possibility" cost us in materials, man-hours, and postage? I'll be the first to say that my IT knowledge is not great, but why would files reside on more than one server; and, those servers be independently accessible for files used for as important a purpose as assuring that our members receive our flagship publication? Bob Vallio -- W6RGG