David; "In fact, LoTW continues to process uploaded logs at about 40 QSOs per second. We did an upgrade recently that caused that performance to tick up slightly. Once the other upgrade is done, perhaps later this week, we will examine how the impact of a significant increase in horsepower affects performance. While this is going on, we are also doing all of the foundational work to move LoTW into the cloud at AWS" I received yet another e-mail from a member today about the rants over on the LoTW Facebook page today, and what he perceives as our inaction to squelch the flames. People are wondering IF we are doing anything to improve things on LoTW in the near term until a new software package is completed. Members acknowledge the fact that the Board approved the formation of a new committee to look into a new version of LoTW, but that doesn't tell members what we are doing NOW to keep LoTW going in the near term. My thought at present is that a putting a message out there just like what you state above will do a lot to squelch the flames. The following is NOT a criticism, but not being involved with this other than dealing with complaints I need to ask: I'm an RF guy, not a computer guy, but this begs the basic question: At the root is this a hardware problem, or a software problem? Is it because LoTW really requires a complete re-write of the software to support multi-core processing power, and it was originally written to support a 8088 with a math co-processor, such that an improvement in hardware power will not make the software run any faster? Or, are we hardware limited due to using old technology platter type hard drives in the servers rather than newer solid state drives and lightning fast multi-core processors? Is it I/O driver limited, where it's a throughput problem at the front end, not related to the processor or memory speeds or capability? The bottom line: Other than telling members "live with it", I think perhaps we need to send a better message. The line I absolutely hated to hear at my last job: "It is what it is", which often came from the engineering department when a new product didn't meet customer expectations. It was even worse when the CEO echoed it. Again, just my unsolicited two watts... 73; Mike W7VO
On 04/10/2023 1:48 PM Minster, David NA2AA (CEO) <dminster@arrl.org> wrote:
I want to make a short comment on this latest rant from Dave Bernstein.
He is wrong and his information is based on an incomplete picture of LoTW today.
In fact, LoTW continues to process uploaded logs at about 40 QSOs per second. We did an upgrade recently that caused that performance to tick up slightly. Once the other upgrade is done, perhaps later this week, we will examine how the impact of a significant increase in horsepower affects performance. While this is going on, we are also doing all of the foundational work to move LoTW into the cloud at AWS.
Dave is looking exclusively at uploaded logs being processed. He has completely ignored the download requests from LoTW by users. Increasingly, software authors are using LoTW as a way to supplement logbook information on the local computer by downloading it from LoTW. These requests that are processed at the same priority as uploaded logs cause processing gaps in handling uploaded logs so the metrics appear worse than they are. No one is lying. No one is hiding information. And no, we are NOT giving Dave access to anything internally at ARRL.
Dave’s arrogant, angry, confrontational, and “know it all” approach – as demonstrated yet again by his message below – shows that he thinks he knows everything that is going on and knows how LoTW needs to be dealt with now and in the future. And in terms of the messaging Bob Naumann has been providing? It is truthful, timely, and clear. That Dave chooses to attack the facts that don’t fit into his narrative is yet another indication that he has problems – and that he is wrong.
Dave is a retired programmer with generation++ old programming skills who writes single user software that he gives away. I cannot imagine how his ego would survive finding that no one would actually pay for it.
His message, again largely based on opinion and not fact, is the kind of garbage he puts out on our LoTW groups.io forum that causes people to unsubscribe and move on. Bob Naumann can provide you with the details if you too are not a believer.
I reiterate: it is not in our best interest to engage with Dave. He is uninformed. He is opinionated. He is a know it all. He will not collaborate with anyone. We’ve turned the page on Dave, and do not intend to engage with him.
Thanks.
David
=======================================================
From: Dave AA6YQ <aa6yq@ambersoft.com mailto:aa6yq@ambersoft.com > To: w5ov@arrl.org mailto:w5ov@arrl.org Cc: K5UR@arrl.org mailto:K5UR@arrl.org ; k1twf@arrl.org mailto:k1twf@arrl.org ; W6RGG@arrl.org mailto:W6RGG@arrl.org ; k3rf@arrl.org mailto:k3rf@arrl.org ; k9la@arrl.org mailto:k9la@arrl.org ; ac0w@arrl.org mailto:ac0w@arrl.org ; k5uz@arrl.org mailto:k5uz@arrl.org ; wa8efk@arrl.org mailto:wa8efk@arrl.org ; np4h@arrl.org mailto:np4h@arrl.org ; k0aiz@arrl.org mailto:k0aiz@arrl.org ; ab1oc@arrl.org mailto:ab1oc@arrl.org ; w7vo@arrl.org mailto:w7vo@arrl.org ; k6wx@arrl.org mailto:k6wx@arrl.org ; n2zz@arrl.org mailto:n2zz@arrl.org ; k0rm@arrl.org mailto:k0rm@arrl.org ; n4mb@arrl.org mailto:n4mb@arrl.org ; n6aa@arrl.org mailto:n6aa@arrl.org ; n5aus@n5aus.com mailto:n5aus@n5aus.com ; Dave AA6YQ <aa6yq@ambersoft.com mailto:aa6yq@ambersoft.com > Sent: Fri, Apr 7, 2023 8:32 pm Subject: "WHEREAS, Logbook of the World (LoTW) has become one of the most popular and relied upon..."
Since without warning or explanation, you have prevented me from posting on the ARRL-LoTW forum, we'll communicate directly.
From 2017 through December 2022, LoTW consistently processed submitted QSOs at the rate of 40 per second while responding to
download requests and award queries; surges of submissions after a popular weekend contest were processed within one day.
In January 2023, this aggregate performance suddenly dropped to less than 10 QSOs per second. Your immediate reaction was to blame
this on logging applications that can be configured to automatically submit QSOs to LoTW as they are logged - a convenience for
users. You provided no evidence or analysis supporting this claim:
"Many FT8 users upload after each QSO. This practice is totally unnecessary and the overhead of processing thousands of one QSO
files is enormous. A daily upload, or maybe, every 10 to 25 QSOs, would be better and would greatly reduce processing overhead."
https://groups.arrl.org/g/ARRL-LoTW/message/36950 https://groups.arrl.org/g/ARRL-LoTW/message/36950
The LoTW Server upgrade project, completed in 2017, had doubled LoTW's QSO processing performance, and made it straightforward to
accomplish further increases when needed by replacing LoTW's database engine with a higher-performance alternative, of which several
are available. At that time, ARRL management indicated that LoTW could accommodate single-QSO uploads. Most logging applications now
provide that capability. When I pointed that out, you eventually responded:
"Those developers who made that statement years ago never anticipated what we have going on today especially with FT8 and thousands
of upload files per hour.
They were wrong. Repeat: They were wrong!!
No dancing around it, they should not have made such a foolish statement.
Oh, and none of them are here any more. And, the people saddled with dealing with this mess now, were not part of that group of bad
decision makers."
(your emphasis)
https://groups.arrl.org/g/ARRL-LoTW/message/37981 https://groups.arrl.org/g/ARRL-LoTW/message/37981
While it is certainly more efficient for LoTW to process one batch of 10 QSOs than 10 batches of one QSO, there are three things
wrong with your response:
1. Without any subsequent performance improvement, LoTW performance consistently exceeded user expectations through December 2022,
as summarized above. Then from one month to the next, its performance fell by a factor of four. You've presented no evidence showing
that the number of single-QSO uploads increased dramatically enough from December to January to cause that large a sudden decrease
in performance.
2. "The developers who made that statement years ago" did anticipate that the load on the LoTW Server would increase - due to
support for additional awards, improved usability, and the increased participation that these improvements would drive. That's why
the LoTW Server upgrade project made it straightforward to further increase performance with a higher-performance database engine -
an opportunity that continues to exist.
3. Even if your assessment was correct, which it demonstrably is not, to denigrate ex-employees in a public forum is outrageously
inappropriate. Have the ARRL's attorneys run out of work? Any software development candidate your organization seeks to hire will
review your public posts and - on encountering this one - run the other way.
Today, you announced "We have been having underlying server problems as reported for several weeks". (my emphasis).
https://groups.arrl.org/g/ARRL-LoTW/message/38252 https://groups.arrl.org/g/ARRL-LoTW/message/38252
Then you proceeded to list 9 "culprits", with no data or analysis in support of your claims. You did provide statistics showing
increases in DXCC award processing for each month of this year, but failed to include the same statistic for December 2022 - making
it impossible to determine whether DXCC award processing was a significant contributor to the 4X degradation in aggregate
performance from December to January. This is "Performance Analysis 101".
Among the culprits, you list
"As reported here, downloads also use processing power, and take away cycles from uploads"
During the first week of January, users attempting to submit QSOs to LoTW were greeted with "Internal Server Error" messages, and
LoTW's QSO processing performance dropped to zero. Your team spent a week discovering that a defect in the popular Grid Tracker
application was continuously invoking the LoTW function that client applications can use to download confirmations. The Grid Tracker
developers promptly corrected their defect, and you reported that you modified LoTW to protect itself against a repeat performance
from Grid Tracker.
As I and others pointed out, LoTW remains vulnerable to any other application that unintentionally hammers the LoTW Server by
continuously invoking that LotW function; a popular application with tens of thousands of users could easily bring LoTW to its
knees, as Grid Tracker inadvertently did. I suggested that you augment LoTW with a simple protective mechanism ("Client Keys") that
would enable you and your staff to immediately identify any application responsible for a torrent of function invocations, and to
immediately terminate the processing of those invocations by that application. Club Log has long protected itself with this
mechanism. You declined:
"These sorts of things will apply to the new LoTW when we get there. We will not be making the types of changes required to support
things like Client Keys in the current system. It's just not going to happen despite the frequent suggestions made here."
https://groups.arrl.org/g/ARRL-LoTW/message/37946 https://groups.arrl.org/g/ARRL-LoTW/message/37946
Leaving LoTW in this vulnerable state for years while "Project X" is staffed, designed, implemented, tested, and documented is an
extraordinarily bad decision. We now know that at least two smartphone-hosted applications can be configured to continuously invoke
LoTW's "report confirmations" function, which could generate tens of thousands of download requests each day - and could already be
doing so. One of them is "open source", meaning that it can be modified to invoke this function as frequently as its user desires.
Frankly, "It's just not going to happen" is a poor justification for not taking the few days of development time required to
eliminate this serious vulnerability.
Earlier today, you posted the message "LoTW Emergency Downtime - Today April 7", stating
"At about 1900 UTC +/- we will be bringing the entire LoTW system down for maintenance. We have IT staff ready to begin to attempt
to correct the underlying problems that we have been enduring as of the last few weeks."
https://groups.arrl.org/g/ARRL-LoTW/message/38268 https://groups.arrl.org/g/ARRL-LoTW/message/38268
LoTW's performance has been severely degraded for the past 3 months, Bob.
Soon after, you posted
"We have been running now a little over 1 hour with our new platform for one of the servers in the LoTW system. We had hoped to move
another, but there are DNS Issues and other networking concerns that need to be addressed. Even so, it appears that we have been
able to make a huge improvement in performance. We are now almost 24 hours behind but making up more than one hour, per hour. This
was not possible before. Thanks to our non-ham IT staff for determining how to make this possible."
https://groups.arrl.org/g/ARRL-LoTW/message/38277 https://groups.arrl.org/g/ARRL-LoTW/message/38277
If it was possible to do this today, it was possible to do it back in January.
I sincerely hope that whatever changes were made today will restore LoTW to the performance it provided from 2017 through December
2022. Without root cause analysis, however, we won't know for sure until the next popular contest.
de AA6YQ
- recipient, ARRL 2008 Technical Innovation Award - "one of the earliest logging-program authors to integrate functions for ARRL's
Logbook of the World"
https://ema.arrl.org/2008/08/02/aa6yq-wins-arrl-technical-innovation-award/ https://ema.arrl.org/2008/08/02/aa6yq-wins-arrl-technical-innovation-award/
- author, "Using Logbook of the World"
https://lotw.arrl.org/lotw-help/ https://lotw.arrl.org/lotw-help/
- recipient, ARRL 2020 President's Award - "in recognition of "exemplary, outstanding, and continuing service" to ARRL and its
members as part of the ARRL Logbook of The World team."
http://www.arrl.org/news/arrl-board-grants-awards-and-recognitions http://www.arrl.org/news/arrl-board-grants-awards-and-recognitions
_______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org https://reflector.arrl.org/mailman/listinfo/arrl-odv