[arrl-odv:31422] LoTW Status
As you've seen & heard, LoTW is struggling under the weight of the CQWW logs -and- people pounding on the system to see if their logs were processed, re-uploading the same file, and checking to see what affect their weekend effort had on their DXCC totals. Here's the headlines: * We are using MaxDB (MySQL on steroids) for the database - Great for hobby use, not for enterprise use * We are not at the latest version * The database is configured for a maximum of 256 concurrent processes - which is THE bottleneck * Although that parameter could be increased (it was once 1024), downing and restarting the database is a unique and unknown process, only known to Michael, who is not responding to email * The database is at 94% utilization, which is a bad thing and is also likely causing paging to occur, slowing down the database * We've had new servers (faster with more storage) since this summer, but LoTW was never migrated * The system is written as a single queue single contact process that checks each line THEN checks it against EVERY award that user is configured for - which is unnecessarily SLOW * It is not likely we would be able, as architected, to gain any benefit from forcing into the cloud - we could get onto bigger machines, but we already have them! * Project X is exactly what our LoTW programmer (Doug) believes needs to happen: 3 tier design (database, logs & awards engines, UI) * Once the system is architected for Cloud, the UI and Log Processing can each run efficiently using independent servers, while the awards engines can auto-scale additional servers as needed * Project X will be sent to ODV ahead of the January board meeting as a proposed project with next steps I have asked Greg Widin what the best course of action is in updating members. David
Thank you David. This aligns with my suspicions. Glad to hear it confirmed, and that we have a plan to fix it. Looking forward to seeing the full report in January for Project X. :) 73 Ria. N2RJ On Tue, Dec 1, 2020 at 5:20 PM Minster, David NA2AA (CEO) <dminster@arrl.org> wrote:
As you’ve seen & heard, LoTW is struggling under the weight of the CQWW logs -and- people pounding on the system to see if their logs were processed, re-uploading the same file, and checking to see what affect their weekend effort had on their DXCC totals.
Here’s the headlines:
We are using MaxDB (MySQL on steroids) for the database - Great for hobby use, not for enterprise use We are not at the latest version The database is configured for a maximum of 256 concurrent processes – which is THE bottleneck Although that parameter could be increased (it was once 1024), downing and restarting the database is a unique and unknown process, only known to Michael, who is not responding to email The database is at 94% utilization, which is a bad thing and is also likely causing paging to occur, slowing down the database We’ve had new servers (faster with more storage) since this summer, but LoTW was never migrated The system is written as a single queue single contact process that checks each line THEN checks it against EVERY award that user is configured for – which is unnecessarily SLOW It is not likely we would be able, as architected, to gain any benefit from forcing into the cloud – we could get onto bigger machines, but we already have them! Project X is exactly what our LoTW programmer (Doug) believes needs to happen: 3 tier design (database, logs & awards engines, UI) Once the system is architected for Cloud, the UI and Log Processing can each run efficiently using independent servers, while the awards engines can auto-scale additional servers as needed Project X will be sent to ODV ahead of the January board meeting as a proposed project with next steps
I have asked Greg Widin what the best course of action is in updating members.
David
_______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org https://reflector.arrl.org/mailman/listinfo/arrl-odv
This is a bit scary.. "unique and unknown process, only known to Michael, who is not responding to email" Mark, HDX On Tue, Dec 1, 2020 at 2:20 PM Minster, David NA2AA (CEO) <dminster@arrl.org> wrote:
As you’ve seen & heard, LoTW is struggling under the weight of the CQWW logs -and- people pounding on the system to see if their logs were processed, re-uploading the same file, and checking to see what affect their weekend effort had on their DXCC totals.
Here’s the headlines:
- We are using MaxDB (MySQL on steroids) for the database - Great for hobby use, not for enterprise use - We are not at the latest version - The database is configured for a maximum of 256 concurrent processes – which is THE bottleneck - Although that parameter could be increased (it was once 1024), downing and restarting the database is a unique and unknown process, only known to Michael, who is not responding to email - The database is at 94% utilization, which is a bad thing and is also likely causing paging to occur, slowing down the database - We’ve had new servers (faster with more storage) since this summer, but LoTW was never migrated - The system is written as a single queue single contact process that checks each line THEN checks it against EVERY award that user is configured for – which is unnecessarily SLOW - It is not likely we would be able, as architected, to gain any benefit from forcing into the cloud – we could get onto bigger machines, but we already have them! - Project X is exactly what our LoTW programmer (Doug) believes needs to happen: 3 tier design (database, logs & awards engines, UI) - Once the system is architected for Cloud, the UI and Log Processing can each run efficiently using independent servers, while the awards engines can auto-scale additional servers as needed - Project X will be sent to ODV ahead of the January board meeting as a proposed project with next steps
I have asked Greg Widin what the best course of action is in updating members.
David _______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org https://reflector.arrl.org/mailman/listinfo/arrl-odv
Doing a little research, MaxDB is not really a MySQL variant at all. It looks like there was just some co-marketing. The one in use is probably based on SAP DB. https://en.wikipedia.org/wiki/MaxDB <https://en.wikipedia.org/wiki/MaxDB> Still, the points stand. There are many good high performance alternatives, especially if you relax ACID <https://en.wikipedia.org/wiki/ACID> constraints. I discussed with Michael more than a year ago whether ACID was required for LOTW. He admitted, and I concur, that it is not.
On Dec 1, 2020, at 2:20 PM, Minster, David NA2AA (CEO) <dminster@arrl.org> wrote:
As you’ve seen & heard, LoTW is struggling under the weight of the CQWW logs -and- people pounding on the system to see if their logs were processed, re-uploading the same file, and checking to see what affect their weekend effort had on their DXCC totals.
Here’s the headlines:
We are using MaxDB (MySQL on steroids) for the database - Great for hobby use, not for enterprise use We are not at the latest version The database is configured for a maximum of 256 concurrent processes – which is THE bottleneck Although that parameter could be increased (it was once 1024), downing and restarting the database is a unique and unknown process, only known to Michael, who is not responding to email The database is at 94% utilization, which is a bad thing and is also likely causing paging to occur, slowing down the database We’ve had new servers (faster with more storage) since this summer, but LoTW was never migrated The system is written as a single queue single contact process that checks each line THEN checks it against EVERY award that user is configured for – which is unnecessarily SLOW It is not likely we would be able, as architected, to gain any benefit from forcing into the cloud – we could get onto bigger machines, but we already have them! Project X is exactly what our LoTW programmer (Doug) believes needs to happen: 3 tier design (database, logs & awards engines, UI) Once the system is architected for Cloud, the UI and Log Processing can each run efficiently using independent servers, while the awards engines can auto-scale additional servers as needed Project X will be sent to ODV ahead of the January board meeting as a proposed project with next steps
I have asked Greg Widin what the best course of action is in updating members.
David _______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org <mailto:arrl-odv@reflector.arrl.org> https://reflector.arrl.org/mailman/listinfo/arrl-odv <https://reflector.arrl.org/mailman/listinfo/arrl-odv> -Kristen (K6WX)
"Your eyes ... it's a day's work just looking into them" Laurie Anderson (--... ...-- -.. . -.- -.... .-- -..-)
participants (4)
-
Kristen McIntyre -
Mark J Tharp -
Minster, David NA2AA (CEO) -
rjairam＠gmail.com