December 2012 - arrl-odv - reflector.arrl.org

[arrl-odv:21371] Reducing duplicate submissions to LotW
by G Widin 31 Dec '12

31 Dec '12

It occurs to me that we could treat files with duplicates differently from others. If we screened a file for dupes and found some number of duplicated QSOs, then we could defer processing of that file for some period of time. It's pretty hard to have zero dupes for logs like this, but if we put some tolerance on it, we could probably capture all the egregious violations and allow through those logs that, for example, were submitted on a day and then had additional QSOs on the same date (since date is a usual way to select which QSOs to upload). There may be other ways, but I daresay that putting logs with high duplicates on a "slow track" would result in an instant reduction in such submissions. Of course, better feedback to a user that his log has been received but not processed would also help close the loop, which is another substantial part of improvements we could make. I've suggested this before but haven't heard any recent discussion of it. On to New Orleans and further discussions! 73, Greg, K0GW On Mon, Dec 31, 2012 at 12:47 PM, Tom Frenaye <frenaye(a)pcnet.com> wrote: > At 01:27 PM 12/31/2012, Jim Weaver K8JE wrote: > > >Please define "duplicates" in relation to your message -- e.g. duplicate > of > >a Q previously entered via LOTW, duplicate of Q previously entered by > paper > >QSL or either/both? > > Hi Jim - > > These are duplicates of previously submitted electronic QSOs on LoTW. The > usual problem is someone who just sends his whole log each time. > It can also be someone who makes a correction in his log (one QSO) and > then sends it in again. > > This is not a duplicate in the way you might think of it in a contest log. > These are repeats of the same QSO at the same date/time in a second LoTW > file. > > They do have easy ways to handle files that are exact copies of previous > files. It's the ones that are just a little different that require the > extra work. > For example, a second file may just have one QSO record repeated, so one > record will be added to LoTW but all of the other records have to be > checked. > > This may be better explained in person than by me in e-mail.... hope I > haven't caused more confusion. > > Tom > > > > > >-----Original Message----- > >From: arrl-odv-bounces(a)reflector.arrl.org > >[mailto:arrl-odv-bounces@reflector.arrl.org] On Behalf Of Tom Frenaye > >Sent: 27 December, 2012 12:11 PM > >To: arrl-odv > >Subject: [arrl-odv:21347] Re: LoTW > > > >At 11:06 AM 12/27/2012, Kermit Carlson wrote: > > > >> I do realize that the hardware upgrade will > >>help the throughput, but using these numbers - > >>2250 files with 300 QSO (approximately) each > >>in 24 hours is the same as a average processing > >>time of 128 milliseconds per QSO record, which > >>would seem high.....hopefully the hardware will > >>provide an improved situation. > > > > > >I'm told that 80% of the QSOs being processed are duplicates, so there is > a > >lot of extra work being done to deal with dupes. Mike Keane says that > >about 800k QSO records are being processed daily. That translates to > almost > >25m records a month, but December will end with the number of QSO records > >added to the database at only about 7.2m for the month. > > > >This is primarily because many people are submitting logs that includes > new > >and previous QSOs, just to be sure... (or submitting logs a second or > third > >time) > > > >It reminds me of the "insurance QSOs" that people make with DXpeditions. > >Good for the individual, bad for everyone else as the throughput and > >efficiency suffers. > > > >I know there is a lot of thinking going on by IT about how to minimize the > >impact of the duplicates, and how to educate users so the impact is less. > > > >Mike Keane's analysis was that the primary way to improve/reduce the time > it > >took to process one record was speeding up time it took to write a new > >record to the database. That time is constrained by the magnetic disk > >drives we are using. Upgrading to solid state disk drives will provide a > >significant increase in speed. (He said roughly 25x faster - >100k IOPS > vs > >4k IOPS - input/output operations per second - see > >http://en.wikipedia.org/wiki/IOPS ) > > > >The hardware upgrade should provide a fairly immediate impact, but the > >impact of duplicate records will need work in the future. They're > working > >hard to get the new drives on line as soon as possible. > > > >Barry - correct me if I'm wrong ... > > > > Tom > > > > > >===== > >e-mail: k1ki(a)arrl.org ARRL New England Division Director > >http://www.arrl.org/ > >Tom Frenaye, K1KI, P O Box J, West Suffield CT 06093 Phone: 860-668-5444 > > > > > >_______________________________________________ > >arrl-odv mailing list > >arrl-odv(a)reflector.arrl.org > >http://reflector.arrl.org/mailman/listinfo/arrl-odv > > > > > >----- > >No virus found in this message. > >Checked by AVG - www.avg.com > >Version: 2013.0.2805 / Virus Database: 2637/5999 - Release Date: 12/31/12 > > > >_______________________________________________ > >arrl-odv mailing list > >arrl-odv(a)reflector.arrl.org > >http://reflector.arrl.org/mailman/listinfo/arrl-odv > > ===== > e-mail: k1ki(a)arrl.org ARRL New England Division Director > http://www.arrl.org/ > Tom Frenaye, K1KI, P O Box J, West Suffield CT 06093 Phone: 860-668-5444 > > > _______________________________________________ > arrl-odv mailing list > arrl-odv(a)reflector.arrl.org > http://reflector.arrl.org/mailman/listinfo/arrl-odv > -- 73, Greg Widin, K0GW ARRL Dakota Division Director ARRL--of, by and for the Radio Amateur

2 1

[arrl-odv:21370] Fwd: An analysis of LotW data
by G Widin 31 Dec '12

31 Dec '12

All, Below is an analysis of LotW's recent statistics compiled by K0RC in Dakota Division. Bob is a thoughtful person who is well versed in computer matters, so his views are worth listening to. I have already responded to the CC list in his message, but I thought the Board might be interested in his analysis of the data. His message was accompanied by the long thread that preceded it, and I have trimmed that, feeling that ODV probably didn't need to plow through all of this. I might mention that Bob is the only I've seen to send messages simultaneously to 3 generations of Dakota Directors. One important reply to Bob's original note came back from Dave, AA6YQ that he felt that this was too early a time to establish any sort of steering committee. I have sufficient respect for Dave's technical judgment to say that we should accept that advice for now. We have important things to discuss about LotW in New Orleans. I'm looking forward to those discussions. 73, Greg ---------- Forwarded message ---------- From: Robert Chudek - K0RC <robert.chudek(a)gmail.com> Date: Sun, Dec 30, 2012 at 5:47 PM Subject: Re: [MWA] LOTW To: G Widin <gpwidin(a)comcast.net> Cc: tod(a)k0to.us, k0qb(a)arrl.net, "aa6yq(a)ambersoft.com" <aa6yq(a)ambersoft.com>, Rick Murphy/K1MU <k1mu-lotw(a)rickmurphy.net> Hello *Greg*, Tod, and Jay... This is a follow-up message to recap the LoTW situation since my last message 6 weeks ago. You probably know a new ARRL webpage was added about 3 weeks ago. It displays the current status of the input queue to the LoTW system. It also shows the previous 11-hours of information. Here's the link to that resource: http://www.arrl.org/logbook-queue-status I have been gathering those reports into Excel and have generated a variety of graphs which show the trends. I attached a copy of this 3-page PDF as a reference. In general, the graphs visually show the 'non-progress' LoTW is making with the backlog of user logs. The two most revealing charts are those on the top of page 2 and page 3. The first one shows the number of logs *waiting to be processed*. You will see the system makes temporary gains in working down the 31,000+ log backlog but the overall trend continues to climb. The second graph shows *how far behind LoTW has fallen in processing logs*. Earlier today it achieved a new milestone of 12 days, compared to 7 days almost three weeks ago. That means logs submitted 12 days ago are now just being processed. When you couple that with the increased number of logs received during the past 12 days, the delay will continue to get worse. One aspect of this growing delay is that when users don't see results of their uploads in a timely manner (subjective, but a day or two?) they think their log was "lost". They subsequently reload their log a second, or third, or more times. This is only one aspect of other issues created by the user community, albeit unintentionally. There is great LoTW community anticipation that the new SSD subsystem will quickly eliminate this processing backlog. We also know that the ARRL has engaged Dave Bernstein AA6YQ and Rick Murphy K1MU to revisit the tQSL and tCERT 'front end' that feeds the LoTW system. This is great news, and I can't think of anyone more qualified to review and update this part of the system. There is 'hear say' that the ARRL has no plan to review the entire LoTW architecture at this time. To me this represents a false sense of security in that the new SSD storage subsystem is a long-term fix. Hopefully this rumor is false. *I want to propose the ARRL create a task-force* to leverage their past 10-years of LoTW experience, review current IT technologies and architectures, and begin the process of designing a modern "*LoTW-2*" for the future. Just like the original design team, a review and update to the system could take a year or more. We also know, the IT industry and technology have made tremendous strides during the last decade. Many of these improvements might well be applied to LoTW and reduce the total cost of ownership. I want to encourage the ARRL to make a statement regarding their decisions regarding the LoTW system. I would expect to see this sometime after the first Board of Directors meeting in 2013. In my view, LoTW creates an image similar to the ARRL website and it should be elevated to a higher level of management and development. Thank you for your time and support! Please feel free to forward my messages to anyone you feel should see them. Respectfully submitted, Bob Chudek - KØRC

1 0

[arrl-odv:21347] Re: LoTW
by Tom Frenaye 31 Dec '12

31 Dec '12

At 11:06 AM 12/27/2012, Kermit Carlson wrote: > I do realize that the hardware upgrade will >help the throughput, but using these numbers - >2250 files with 300 QSO (approximately) each >in 24 hours is the same as a average processing >time of 128 milliseconds per QSO record, which >would seem high.....hopefully the hardware will >provide an improved situation. I'm told that 80% of the QSOs being processed are duplicates, so there is a lot of extra work being done to deal with dupes. Mike Keane says that about 800k QSO records are being processed daily. That translates to almost 25m records a month, but December will end with the number of QSO records added to the database at only about 7.2m for the month. This is primarily because many people are submitting logs that includes new and previous QSOs, just to be sure... (or submitting logs a second or third time) It reminds me of the "insurance QSOs" that people make with DXpeditions. Good for the individual, bad for everyone else as the throughput and efficiency suffers. I know there is a lot of thinking going on by IT about how to minimize the impact of the duplicates, and how to educate users so the impact is less. Mike Keane's analysis was that the primary way to improve/reduce the time it took to process one record was speeding up time it took to write a new record to the database. That time is constrained by the magnetic disk drives we are using. Upgrading to solid state disk drives will provide a significant increase in speed. (He said roughly 25x faster - >100k IOPS vs 4k IOPS - input/output operations per second - see http://en.wikipedia.org/wiki/IOPS ) The hardware upgrade should provide a fairly immediate impact, but the impact of duplicate records will need work in the future. They're working hard to get the new drives on line as soon as possible. Barry - correct me if I'm wrong ... Tom ===== e-mail: k1ki(a)arrl.org ARRL New England Division Director http://www.arrl.org/ Tom Frenaye, K1KI, P O Box J, West Suffield CT 06093 Phone: 860-668-5444

3 4

[arrl-odv:21354] HAPPY NEW YEAR 2013
by Joyce Birmingham 31 Dec '12

31 Dec '12

Hello everyone! I would like to extend to you all my very best wishes for a Happy and Healthy 2013! Also, I will say Goodbye to you, as Hudson Division Director and thank you all for your hard work and support! I wish you the best at the January 2013 ARRL Annual Board Meeting in New Orleans! God speed! 73, Joyce KA2ANF Joyce Birmingham, KA2ANF 235 Van Emburgh Avenue Ridgewood, NJ 07450-2918 ka2anf(a)arrl.net

9 8

[arrl-odv:21356] Draft Board Meeting Agenda
by Sumner, Dave, K1ZZ 31 Dec '12

31 Dec '12

Officers, Directors and Vice Directors: Attached is the draft agenda for our meeting in New Orleans. 73, David Sumner, K1ZZ Secretary

2 2

[arrl-odv:21363] HAPPY NEW YEAR 2013
by James F. Boehner MD 31 Dec '12

31 Dec '12

Joyce, I'm happy to say that I knew you as an Assistant Director (when you attended my SM training session), a Vice Director and as a Director of the ARRL. Thank you so much for your service to amateur radio and your friendship. I am sure we will meet further down the log.... It has been a pleasure serving with you over this last decade. Take care, and hope to see you in Dayton! '73 de JIM N2ZZ ARRL Vice Director Roanoke Division From: arrl-odv-bounces(a)reflector.arrl.org [mailto:arrl-odv-bounces@reflector.arrl.org] On Behalf Of Joyce Birmingham Sent: Sunday, December 30, 2012 5:20 PM To: Arrl-Odv Cc: Joyce A. Birmingham Subject: [arrl-odv:21354] HAPPY NEW YEAR 2013 Hello everyone! I would like to extend to you all my very best wishes for a Happy and Healthy 2013! Also, I will say Goodbye to you, as Hudson Division Director and thank you all for your hard work and support! I wish you the best at the January 2013 ARRL Annual Board Meeting in New Orleans! God speed! 73, Joyce KA2ANF Joyce Birmingham, KA2ANF 235 Van Emburgh Avenue Ridgewood, NJ 07450-2918 ka2anf(a)arrl.net

1 0

[arrl-odv:21353] Pleasant surprise
by Marty Woll 30 Dec '12

30 Dec '12

Happy New Year, all. On opening an envelope recently received from HQ I found a certificate for last year's 10-Meter Contest. I was not expecting one, but the real surprise was how great it looked. The graphics are excellent, right down to the OWA yagi shown on the tower. Whoever developed this one did a really nice job! Dave and Harold, please pass along my compliments. 73, Marty N6VI

1 0

[arrl-odv:21351] LoTW
by James F. Boehner MD 28 Dec '12

28 Dec '12

Dave, Thanks for the clarification. So, it would follow that if someone sends an adif file of their entire log rather than new/corrected contacts, the system gets bogged down. When I go to the LOTW pages, there appears to be no request to submit partial logs, particularly on the "upload file" page. From trusted QSL, it is an option, although there is no encouragement to do so. This may be a concept that most members do not understand. I know I sent full logs in the past. Perhaps an education piece on the LOTW website or incorporated into Trusted QSL could supplement the current efforts of hardware upgrades. Those of the board family that are receiving complaints could certainly ask the individuals how they are submitting logs, full or partial. It is a sign of the times. In the past, we would normally wait numerous years for gathering QSL's and submitting them to the ARRL for credit, generally associated with a significant expense (postage, green stamps/IRC's, SASE's, Registered mail, etc.). Now there is just a minimal fee (LOTW), and credit is expected immediately. Whew. I really do not understand the magnitude of the outcry when the LOTW system is not operating perfectly. It should be common knowledge that computer systems, no matter how well designed, can fail. With the volume that system handles, it will take some processing time. I know the data can be sent via e-mail or the website. I do not know if that data is directly imported to LOTW, or requires human intervention to do so. If a staff member has to input the data file, eliminating that step could be another opportunity to speed up processing. After hearing the complaints after the last downtime, I attempted to join the ARRL-LOTW yahoogroup-my membership was denied... I think the ARRL's response was timely and appropriate, responding with a short term and long term plan. This is the information that I give members. I am interested as to how well these SSHD's work with LOTW. My son put one in his computer and a full Windows 7 boot-up takes all of 35 seconds! '73 de JIM N2ZZ -----Original Message----- From: arrl-odv-bounces(a)reflector.arrl.org [mailto:arrl-odv-bounces@reflector.arrl.org] On Behalf Of Sumner, Dave, K1ZZ Sent: Thursday, December 27, 2012 3:55 PM To: arrl-odv Subject: [arrl-odv:21350] Re: LoTW Tom said: >The hardware upgrade should provide a fairly immediate impact, but the impact of duplicate records will need work in the future. They're working hard to get the new drives on line as soon as possible. >Barry - correct me if I'm wrong ... Barry has some meetings this afternoon, but I can provide an update. Tom, you're right. The hardware arrived, is installed, and appears to be meeting expectations. There's more work and testing to be done before it's ready for "production" but so far, so good. As I understand it, files that are exact duplicates aren't much of a problem. Files that consist mostly, but not entirely, of duplicate QSOs are more of a problem. This occurs, for example, when someone uploads their entire log every time rather than just the new QSOs, or when someone makes corrections to a contest log and sends the whole thing again rather than just the corrected QSOs. The question of how best to deal with duplicates will be addressed after the new hardware is on line. 73, Dave Sumner, K1ZZ P.S. Until this morning I had only been in the office for three hours since December 13, the result of a combination of travel, illness and holiday. I am taking Friday and Monday off and plan to be in the office full time beginning on January 2. If you have sent me an email and are waiting for a reply, please be patient -- my queue is not as large as LoTW's but unlike LoTW I am not working through it on a strict first-in/first-out basis. _______________________________________________ arrl-odv mailing list arrl-odv(a)reflector.arrl.org http://reflector.arrl.org/mailman/listinfo/arrl-odv

2 1

[arrl-odv:21349] Re: Brennan's
by Tom Frenaye 27 Dec '12

27 Dec '12

At 01:11 PM 12/27/2012, Dr. David Woolweaver wrote: >Breakfast at Brennan's has been for many years a New Orleans favorite. Actors, royalty, politicians and even a few honest people have been fortunate to enjoyed the unique cuisine. >For the ARRL Board, I have been able to obtain a reservation at 9:30 AM Sunday, January 20. It is not too late to join the group. Please let me know ASAP, however. The on-line menu looks nice, not sure yet what our plans are - we're going to stay an extra two days to explore the neighborhood (+/- 200 miles). How are the prices? When I see a menu with no prices I start to worry. I don't want to see an eHam article about lavish ARRL Board spending for breakfast (or dinners). Looking at a few web reviews for Brennans, the breakfast and other meal) prices are out of my range for a Board (member paid) meal. It's been a few years since we've met outside of Windsor CT (Houston-2008 and Ft Worth-2002) did not provide the same level of culinary temptations... Tom ===== e-mail: k1ki(a)arrl.org ARRL New England Division Director http://www.arrl.org/ Tom Frenaye, K1KI, P O Box J, West Suffield CT 06093 Phone: 860-668-5444

1 0

[arrl-odv:21348] Brennan's
by Dr. David Woolweaver 27 Dec '12

27 Dec '12

Breakfast at Brennan's has been for many years a New Orleans favorite. Actors, royalty, politicians and even a few honest people have been fortunate to enjoyed the unique cuisine. For the ARRL Board, I have been able to obtain a reservation at 9:30 AM Sunday, January 20. It is not too late to join the group. Please let me know ASAP, however. Thanks de David

1 0