Tom, Please define "duplicates" in relation to your message -- e.g. duplicate of a Q previously entered via LOTW, duplicate of Q previously entered by paper QSL or either/both? Tnx, Jim Weaver, K8JE Director, Great Lakes Division 5065 Bethany Rd. Mason, OH 45040 Tel. 513-459-1661; e-mail K8JE@arrl.org ARRL: The reason Amateur Radio Is Members: The reason ARRL is -----Original Message----- From: arrl-odv-bounces@reflector.arrl.org [mailto:arrl-odv-bounces@reflector.arrl.org] On Behalf Of Tom Frenaye Sent: 27 December, 2012 12:11 PM To: arrl-odv Subject: [arrl-odv:21347] Re: LoTW At 11:06 AM 12/27/2012, Kermit Carlson wrote:
I do realize that the hardware upgrade will help the throughput, but using these numbers - 2250 files with 300 QSO (approximately) each in 24 hours is the same as a average processing time of 128 milliseconds per QSO record, which would seem high.....hopefully the hardware will provide an improved situation.
I'm told that 80% of the QSOs being processed are duplicates, so there is a lot of extra work being done to deal with dupes. Mike Keane says that about 800k QSO records are being processed daily. That translates to almost 25m records a month, but December will end with the number of QSO records added to the database at only about 7.2m for the month. This is primarily because many people are submitting logs that includes new and previous QSOs, just to be sure... (or submitting logs a second or third time) It reminds me of the "insurance QSOs" that people make with DXpeditions. Good for the individual, bad for everyone else as the throughput and efficiency suffers. I know there is a lot of thinking going on by IT about how to minimize the impact of the duplicates, and how to educate users so the impact is less. Mike Keane's analysis was that the primary way to improve/reduce the time it took to process one record was speeding up time it took to write a new record to the database. That time is constrained by the magnetic disk drives we are using. Upgrading to solid state disk drives will provide a significant increase in speed. (He said roughly 25x faster - >100k IOPS vs 4k IOPS - input/output operations per second - see http://en.wikipedia.org/wiki/IOPS ) The hardware upgrade should provide a fairly immediate impact, but the impact of duplicate records will need work in the future. They're working hard to get the new drives on line as soon as possible. Barry - correct me if I'm wrong ... Tom ===== e-mail: k1ki@arrl.org ARRL New England Division Director http://www.arrl.org/ Tom Frenaye, K1KI, P O Box J, West Suffield CT 06093 Phone: 860-668-5444 _______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org http://reflector.arrl.org/mailman/listinfo/arrl-odv ----- No virus found in this message. Checked by AVG - www.avg.com Version: 2013.0.2805 / Virus Database: 2637/5999 - Release Date: 12/31/12