Mickey is correct.

The root cause is still a mystery to us. There are a ridiculous number of people requesting downloads of their confirmations. Either a website or a new release of software is killing us. We are being careful about broadcasting our concerns as we don’t want a run on the server.

The uploads are a single threaded process, one log at a time. The downloads, which include basic user queries, are in parallel and are unrestricted. Most times, basically ALL the time, this is not an issue. The past few days we’ve been blasted by download requests. One European user made over 1800 different download requests of his confirmations. I suspect something like HRD must have new functionality that is using the LoTW API, blissfully ignorant that their users are now killing the system.

It is all hands on deck right now. And again, we are being very careful about the messaging until we’ve figured this out.

David

From: arrl-odv <arrl-odv-bounces@reflector.arrl.org> On Behalf Of Baker, Mickey, N4MB (Dir, SE)
Sent: Wednesday, January 4, 2023 2:09 PM
To: Jairam, Ria, N2RJ (Dir, HD) <n2rj@arrl.org>; arrl-odv@reflector.arrl.org
Subject: [arrl-odv:34446] Re: Recent LoTW outage

The problem is software performance due to a design that was never intended for this workload.

Software performance is the problem – Michael Keane wrote that the use of 256bit encryption keys impbedded in the database field causes LoTW to store 10x as much data as needed. The security model ignores improements over the last 20+ years.

Workload is the root cause.

Mickey Baker, N4MB

Director, Southeastern Division

ARRL, The National Association for Amateur Radio®

Phone (561) 320-2775

Email: n4mb@arrl.org

ARRL Customer Service: (860) 594-0355

From: arrl-odv <arrl-odv-bounces@reflector.arrl.org> on behalf of Jairam, Ria, N2RJ (Dir, HD) <n2rj@arrl.org>
Date: Wednesday, January 4, 2023 at 1:01 PM
To: arrl-odv@reflector.arrl.org <arrl-odv@reflector.arrl.org>
Subject: [arrl-odv:34445] Recent LoTW outage

Members are asking me about it. I would appreciate if there was something I could tell them as to root cause.

I would also like to know the details.

Thanks

Ria

N2RJ