[arrl-odv:21347] Re: LoTW
At 11:06 AM 12/27/2012, Kermit Carlson wrote:
I do realize that the hardware upgrade will help the throughput, but using these numbers - 2250 files with 300 QSO (approximately) each in 24 hours is the same as a average processing time of 128 milliseconds per QSO record, which would seem high.....hopefully the hardware will provide an improved situation.
I'm told that 80% of the QSOs being processed are duplicates, so there is a lot of extra work being done to deal with dupes. Mike Keane says that about 800k QSO records are being processed daily. That translates to almost 25m records a month, but December will end with the number of QSO records added to the database at only about 7.2m for the month. This is primarily because many people are submitting logs that includes new and previous QSOs, just to be sure... (or submitting logs a second or third time) It reminds me of the "insurance QSOs" that people make with DXpeditions. Good for the individual, bad for everyone else as the throughput and efficiency suffers. I know there is a lot of thinking going on by IT about how to minimize the impact of the duplicates, and how to educate users so the impact is less. Mike Keane's analysis was that the primary way to improve/reduce the time it took to process one record was speeding up time it took to write a new record to the database. That time is constrained by the magnetic disk drives we are using. Upgrading to solid state disk drives will provide a significant increase in speed. (He said roughly 25x faster - >100k IOPS vs 4k IOPS - input/output operations per second - see http://en.wikipedia.org/wiki/IOPS ) The hardware upgrade should provide a fairly immediate impact, but the impact of duplicate records will need work in the future. They're working hard to get the new drives on line as soon as possible. Barry - correct me if I'm wrong ... Tom ===== e-mail: k1ki@arrl.org ARRL New England Division Director http://www.arrl.org/ Tom Frenaye, K1KI, P O Box J, West Suffield CT 06093 Phone: 860-668-5444
Tom said:
The hardware upgrade should provide a fairly immediate impact, but the impact of duplicate records will need work in the future. They're working hard to get the new drives on line as soon as possible.
Barry - correct me if I'm wrong ...
Barry has some meetings this afternoon, but I can provide an update. Tom, you're right. The hardware arrived, is installed, and appears to be meeting expectations. There's more work and testing to be done before it's ready for "production" but so far, so good. As I understand it, files that are exact duplicates aren't much of a problem. Files that consist mostly, but not entirely, of duplicate QSOs are more of a problem. This occurs, for example, when someone uploads their entire log every time rather than just the new QSOs, or when someone makes corrections to a contest log and sends the whole thing again rather than just the corrected QSOs. The question of how best to deal with duplicates will be addressed after the new hardware is on line. 73, Dave Sumner, K1ZZ P.S. Until this morning I had only been in the office for three hours since December 13, the result of a combination of travel, illness and holiday. I am taking Friday and Monday off and plan to be in the office full time beginning on January 2. If you have sent me an email and are waiting for a reply, please be patient -- my queue is not as large as LoTW's but unlike LoTW I am not working through it on a strict first-in/first-out basis.
Tom, Please define "duplicates" in relation to your message -- e.g. duplicate of a Q previously entered via LOTW, duplicate of Q previously entered by paper QSL or either/both? Tnx, Jim Weaver, K8JE Director, Great Lakes Division 5065 Bethany Rd. Mason, OH 45040 Tel. 513-459-1661; e-mail K8JE@arrl.org ARRL: The reason Amateur Radio Is Members: The reason ARRL is -----Original Message----- From: arrl-odv-bounces@reflector.arrl.org [mailto:arrl-odv-bounces@reflector.arrl.org] On Behalf Of Tom Frenaye Sent: 27 December, 2012 12:11 PM To: arrl-odv Subject: [arrl-odv:21347] Re: LoTW At 11:06 AM 12/27/2012, Kermit Carlson wrote:
I do realize that the hardware upgrade will help the throughput, but using these numbers - 2250 files with 300 QSO (approximately) each in 24 hours is the same as a average processing time of 128 milliseconds per QSO record, which would seem high.....hopefully the hardware will provide an improved situation.
I'm told that 80% of the QSOs being processed are duplicates, so there is a lot of extra work being done to deal with dupes. Mike Keane says that about 800k QSO records are being processed daily. That translates to almost 25m records a month, but December will end with the number of QSO records added to the database at only about 7.2m for the month. This is primarily because many people are submitting logs that includes new and previous QSOs, just to be sure... (or submitting logs a second or third time) It reminds me of the "insurance QSOs" that people make with DXpeditions. Good for the individual, bad for everyone else as the throughput and efficiency suffers. I know there is a lot of thinking going on by IT about how to minimize the impact of the duplicates, and how to educate users so the impact is less. Mike Keane's analysis was that the primary way to improve/reduce the time it took to process one record was speeding up time it took to write a new record to the database. That time is constrained by the magnetic disk drives we are using. Upgrading to solid state disk drives will provide a significant increase in speed. (He said roughly 25x faster - >100k IOPS vs 4k IOPS - input/output operations per second - see http://en.wikipedia.org/wiki/IOPS ) The hardware upgrade should provide a fairly immediate impact, but the impact of duplicate records will need work in the future. They're working hard to get the new drives on line as soon as possible. Barry - correct me if I'm wrong ... Tom ===== e-mail: k1ki@arrl.org ARRL New England Division Director http://www.arrl.org/ Tom Frenaye, K1KI, P O Box J, West Suffield CT 06093 Phone: 860-668-5444 _______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org http://reflector.arrl.org/mailman/listinfo/arrl-odv ----- No virus found in this message. Checked by AVG - www.avg.com Version: 2013.0.2805 / Virus Database: 2637/5999 - Release Date: 12/31/12
At 01:27 PM 12/31/2012, Jim Weaver K8JE wrote:
Please define "duplicates" in relation to your message -- e.g. duplicate of a Q previously entered via LOTW, duplicate of Q previously entered by paper QSL or either/both?
Hi Jim - These are duplicates of previously submitted electronic QSOs on LoTW. The usual problem is someone who just sends his whole log each time. It can also be someone who makes a correction in his log (one QSO) and then sends it in again. This is not a duplicate in the way you might think of it in a contest log. These are repeats of the same QSO at the same date/time in a second LoTW file. They do have easy ways to handle files that are exact copies of previous files. It's the ones that are just a little different that require the extra work. For example, a second file may just have one QSO record repeated, so one record will be added to LoTW but all of the other records have to be checked. This may be better explained in person than by me in e-mail.... hope I haven't caused more confusion. Tom
-----Original Message----- From: arrl-odv-bounces@reflector.arrl.org [mailto:arrl-odv-bounces@reflector.arrl.org] On Behalf Of Tom Frenaye Sent: 27 December, 2012 12:11 PM To: arrl-odv Subject: [arrl-odv:21347] Re: LoTW
At 11:06 AM 12/27/2012, Kermit Carlson wrote:
I do realize that the hardware upgrade will help the throughput, but using these numbers - 2250 files with 300 QSO (approximately) each in 24 hours is the same as a average processing time of 128 milliseconds per QSO record, which would seem high.....hopefully the hardware will provide an improved situation.
I'm told that 80% of the QSOs being processed are duplicates, so there is a lot of extra work being done to deal with dupes. Mike Keane says that about 800k QSO records are being processed daily. That translates to almost 25m records a month, but December will end with the number of QSO records added to the database at only about 7.2m for the month.
This is primarily because many people are submitting logs that includes new and previous QSOs, just to be sure... (or submitting logs a second or third time)
It reminds me of the "insurance QSOs" that people make with DXpeditions. Good for the individual, bad for everyone else as the throughput and efficiency suffers.
I know there is a lot of thinking going on by IT about how to minimize the impact of the duplicates, and how to educate users so the impact is less.
Mike Keane's analysis was that the primary way to improve/reduce the time it took to process one record was speeding up time it took to write a new record to the database. That time is constrained by the magnetic disk drives we are using. Upgrading to solid state disk drives will provide a significant increase in speed. (He said roughly 25x faster - >100k IOPS vs 4k IOPS - input/output operations per second - see http://en.wikipedia.org/wiki/IOPS )
The hardware upgrade should provide a fairly immediate impact, but the impact of duplicate records will need work in the future. They're working hard to get the new drives on line as soon as possible.
Barry - correct me if I'm wrong ...
Tom
===== e-mail: k1ki@arrl.org ARRL New England Division Director http://www.arrl.org/ Tom Frenaye, K1KI, P O Box J, West Suffield CT 06093 Phone: 860-668-5444
_______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org http://reflector.arrl.org/mailman/listinfo/arrl-odv
----- No virus found in this message. Checked by AVG - www.avg.com Version: 2013.0.2805 / Virus Database: 2637/5999 - Release Date: 12/31/12
_______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org http://reflector.arrl.org/mailman/listinfo/arrl-odv
===== e-mail: k1ki@arrl.org ARRL New England Division Director http://www.arrl.org/ Tom Frenaye, K1KI, P O Box J, West Suffield CT 06093 Phone: 860-668-5444
Tnx, Tom. FB. HNY to all. Jim Weaver, K8JE Director, Great Lakes Division 5065 Bethany Rd. Mason, OH 45040 Tel. 513-459-1661; e-mail K8JE@arrl.org ARRL: The reason Amateur Radio Is Members: The reason ARRL is -----Original Message----- From: arrl-odv-bounces@reflector.arrl.org [mailto:arrl-odv-bounces@reflector.arrl.org] On Behalf Of Tom Frenaye Sent: 31 December, 2012 1:48 PM To: 'arrl-odv' Subject: [arrl-odv:21368] Re: LoTW At 01:27 PM 12/31/2012, Jim Weaver K8JE wrote:
Please define "duplicates" in relation to your message -- e.g. duplicate of a Q previously entered via LOTW, duplicate of Q previously entered by paper QSL or either/both?
Hi Jim - These are duplicates of previously submitted electronic QSOs on LoTW. The usual problem is someone who just sends his whole log each time. It can also be someone who makes a correction in his log (one QSO) and then sends it in again. This is not a duplicate in the way you might think of it in a contest log. These are repeats of the same QSO at the same date/time in a second LoTW file. They do have easy ways to handle files that are exact copies of previous files. It's the ones that are just a little different that require the extra work. For example, a second file may just have one QSO record repeated, so one record will be added to LoTW but all of the other records have to be checked. This may be better explained in person than by me in e-mail.... hope I haven't caused more confusion. Tom
-----Original Message----- From: arrl-odv-bounces@reflector.arrl.org [mailto:arrl-odv-bounces@reflector.arrl.org] On Behalf Of Tom Frenaye Sent: 27 December, 2012 12:11 PM To: arrl-odv Subject: [arrl-odv:21347] Re: LoTW
At 11:06 AM 12/27/2012, Kermit Carlson wrote:
I do realize that the hardware upgrade will help the throughput, but using these numbers - 2250 files with 300 QSO (approximately) each in 24 hours is the same as a average processing time of 128 milliseconds per QSO record, which would seem high.....hopefully the hardware will provide an improved situation.
I'm told that 80% of the QSOs being processed are duplicates, so there is a lot of extra work being done to deal with dupes. Mike Keane says that about 800k QSO records are being processed daily. That translates to almost 25m records a month, but December will end with the number of QSO records added to the database at only about 7.2m for the month.
This is primarily because many people are submitting logs that includes new and previous QSOs, just to be sure... (or submitting logs a second or third time)
It reminds me of the "insurance QSOs" that people make with DXpeditions. Good for the individual, bad for everyone else as the throughput and efficiency suffers.
I know there is a lot of thinking going on by IT about how to minimize the impact of the duplicates, and how to educate users so the impact is less.
Mike Keane's analysis was that the primary way to improve/reduce the time it took to process one record was speeding up time it took to write a new record to the database. That time is constrained by the magnetic disk drives we are using. Upgrading to solid state disk drives will provide a significant increase in speed. (He said roughly 25x faster - >100k IOPS vs 4k IOPS - input/output operations per second - see http://en.wikipedia.org/wiki/IOPS )
The hardware upgrade should provide a fairly immediate impact, but the impact of duplicate records will need work in the future. They're working hard to get the new drives on line as soon as possible.
Barry - correct me if I'm wrong ...
Tom
===== e-mail: k1ki@arrl.org ARRL New England Division Director http://www.arrl.org/ Tom Frenaye, K1KI, P O Box J, West Suffield CT 06093 Phone: 860-668-5444
_______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org http://reflector.arrl.org/mailman/listinfo/arrl-odv
----- No virus found in this message. Checked by AVG - www.avg.com Version: 2013.0.2805 / Virus Database: 2637/5999 - Release Date: 12/31/12
_______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org http://reflector.arrl.org/mailman/listinfo/arrl-odv
===== e-mail: k1ki@arrl.org ARRL New England Division Director http://www.arrl.org/ Tom Frenaye, K1KI, P O Box J, West Suffield CT 06093 Phone: 860-668-5444 _______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org http://reflector.arrl.org/mailman/listinfo/arrl-odv ----- No virus found in this message. Checked by AVG - www.avg.com Version: 2013.0.2805 / Virus Database: 2637/5999 - Release Date: 12/31/12
participants (3)
-
Jim Weaver K8JE -
Sumner, Dave, K1ZZ -
Tom Frenaye