John: We've already begun developing a list of possible actions/recommendations that we want to consider. That is on the list. Our first priority is insuring we've eliminated the threat and any lingering vulnerabilities. Then we'll tackle the list going forward. 73, Barry, N1VXY Sent from my Verizon Wireless 4G LTE DROID "Stratton, John, N5AUS" <jrs@hamradio.us.com> wrote: Barry Thank you for keeping the Board up to date. Have we considered bringing in outside security experts to review our implementation of security protocols and to make recommendations for improvements or additional methods to harden our data structure and servers? 73 ----------------------------------------------------- John Robert Stratton N5AUS Office telephone: 512-445-6262 Cell: 512-426-2028 PO Box 2232 Austin, Texas 78768-2232 ----------------------------------------------------- On 10/3/14 4:19 PM, Shelley, Barry, N1VXY wrote: ARRL Board of Directors: This is a confidential notification to the Board that we have identified an unauthorized breach of the ARRL computer systems. We are currently working through all the appropriate steps as described below. In addition, we are in the process of making the necessary notification to law enforcement agencies. Situation: Early yesterday, we became aware of files on one of our many servers that were “unusual” and the IT Dept., in their investigation, determined that there had been a breach of one of our oldest servers (P1K), a proxy server for the web site. While uncertain of the exact vulnerability, the recently identified “Shellshock” vulnerability for older Linux systems may have been a factor in this attack. We were able to track IP address activity to Eastern Europe and identify the activities from those sites. We immediately shut down access to P1K from outside the building. This has impacted some web services such as the On-line DXCC application and legacy connections to LoTW. On-going Activities: In these situations the general steps to follow are: 1. Lock systems down. 2. Identify what, if anything was accessed. 3. Notify law enforcement. 4. Decisions on notification. 5. Take corrective actions. At this moment we still have prohibited access to the P1K server from outside the building. We are in the process of bringing up another server to support those functions that were supported by the old server but with all the current patches, updates applied. Because of the age of the old server, we were not able to apply all the current updates, etc. Projections for the new proxy server coming on-line are early next week. We will give priority to the applications with a higher level of activity like LoTW and On-line DXCC. Please note, it is only the legacy applications that use the P1K proxy to access the LoTW server. In addition, IT is going through all the code for the database servers and programs, eliminating any potential vulnerabilities. The IT Department has been reviewing the log files in detail and, to this point it would appear that the database the hackers had most activity with was the On-line DXCC database. They did have access, however, to names, addresses and e-mail addresses from our main date base. Current web site passwords are encrypted but passwords used on the former website (prior to 2010) were not encrypted and archival copies of those passwords stored on HQ servers may have been compromised. A limited test of old passwords indicates that possibly 10% of the old passwords may still be in use. We have not seen any unusual activity on the LoTW server. Because of the configuration, the hackers were not able to get information from the ARRL web site itself. Hackers could have accessed name, address, phone numbers and e-mail addresses. We do not keep credit card or social security information in our databases so this is not information that could be accessed. We are still working on reviewing all the logs and activity in the various databases to identify what information, if any, might have been breached and transferred. This effort will be on-going. We are in process of notifying the appropriate law enforcement agencies, including Federal, but quite honestly, I don’t expect much attention from them. More important would be our obligation to notify individuals of a possible breach of their information. State laws vary so it will be complicated. I’ve been given the name of a law firm with an excellent reputation in these matters and plan to contact them as soon as possible. A quick reading of Connecticut statutes doesn’t indicate a requirement to notify in this situation because of the nature of the information potentially compromised, but there are 49 other states with different regulations and we have members in all of them. I’m sure there will be many questions but please understand that the situation is very fluid. I will provide you with updates as we move forward. _______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org<mailto:arrl-odv@reflector.arrl.org> http://reflector.arrl.org/mailman/listinfo/arrl-odv

TU Barry, This has become a problem. Our agency's servers were hacked a few years ago by Al Queida! The NSA, DOD, CIA and every other three letter agency was looking at out system! It was quite an experience. 73 David A. Norris, K5UZ Director Delta Division Sent from my iPhone On Oct 3, 2014, at 6:11 PM, Doug Rehman <doug@k4ac.com> wrote:

I will remind my fellow Board members of my proposal for creating an IT Strategic Planning Committee. Had this Committee already existed and been properly constituted, this breach could have been averted.

This is the second time in the exceedingly short time I have been on the Board that a system has been found with outdated software. (The email reflectors were running a years old version of the software.) Having anything in a production IT environment that is not running the most current software is absolutely unacceptable. This is basic computer security 101 and we have failed twice this year that I know of.

I ask the members of the Executive Committee to take up my proposal at the upcoming meeting and put it out to the Board for an electronic vote. We cannot wait for the next crisis—we need to start getting ahead of problems instead of trying to play clean up.

Doug K4AC (and Valencia College Professor in the Digital Forensics and Cyber Security degree program…)

From: arrl-odv [mailto:arrl-odv-bounces@reflector.arrl.org] On Behalf Of Shelley, Barry, N1VXY Sent: Friday, October 3, 2014 6:24 PM To: Stratton, John, N5AUS Cc: arrl-odv Subject: [arrl-odv:23378] Re: CONFIDENTIAL -- Security Breach - Outside Help?

John:

We've already begun developing a list of possible actions/recommendations that we want to consider. That is on the list.

Our first priority is insuring we've eliminated the threat and any lingering vulnerabilities. Then we'll tackle the list going forward.

73, Barry, N1VXY

Sent from my Verizon Wireless 4G LTE DROID

"Stratton, John, N5AUS" <jrs@hamradio.us.com> wrote:

Barry

Thank you for keeping the Board up to date.

Have we considered bringing in outside security experts to review our implementation of security protocols and to make recommendations for improvements or additional methods to harden our data structure and servers?

73

----------------------------------------------------- John Robert Stratton

N5AUS

Office telephone: 512-445-6262 Cell: 512-426-2028 PO Box 2232 Austin, Texas 78768-2232

-----------------------------------------------------

On 10/3/14 4:19 PM, Shelley, Barry, N1VXY wrote: ARRL Board of Directors:

This is a confidential notification to the Board that we have identified an unauthorized breach of the ARRL computer systems. We are currently working through all the appropriate steps as described below. In addition, we are in the process of making the necessary notification to law enforcement agencies.

Situation:

Early yesterday, we became aware of files on one of our many servers that were “unusual” and the IT Dept., in their investigation, determined that there had been a breach of one of our oldest servers (P1K), a proxy server for the web site. While uncertain of the exact vulnerability, the recently identified “Shellshock” vulnerability for older Linux systems may have been a factor in this attack. We were able to track IP address activity to Eastern Europe and identify the activities from those sites. We immediately shut down access to P1K from outside the building. This has impacted some web services such as the On-line DXCC application and legacy connections to LoTW.

On-going Activities:

In these situations the general steps to follow are:

1. Lock systems down. 2. Identify what, if anything was accessed. 3. Notify law enforcement. 4. Decisions on notification. 5. Take corrective actions.

At this moment we still have prohibited access to the P1K server from outside the building. We are in the process of bringing up another server to support those functions that were supported by the old server but with all the current patches, updates applied. Because of the age of the old server, we were not able to apply all the current updates, etc. Projections for the new proxy server coming on-line are early next week. We will give priority to the applications with a higher level of activity like LoTW and On-line DXCC. Please note, it is only the legacy applications that use the P1K proxy to access the LoTW server.

In addition, IT is going through all the code for the database servers and programs, eliminating any potential vulnerabilities.

The IT Department has been reviewing the log files in detail and, to this point it would appear that the database the hackers had most activity with was the On-line DXCC database. They did have access, however, to names, addresses and e-mail addresses from our main date base. Current web site passwords are encrypted but passwords used on the former website (prior to 2010) were not encrypted and archival copies of those passwords stored on HQ servers may have been compromised. A limited test of old passwords indicates that possibly 10% of the old passwords may still be in use. We have not seen any unusual activity on the LoTW server. Because of the configuration, the hackers were not able to get information from the ARRL web site itself. Hackers could have accessed name, address, phone numbers and e-mail addresses.

We do not keep credit card or social security information in our databases so this is not information that could be accessed.

We are still working on reviewing all the logs and activity in the various databases to identify what information, if any, might have been breached and transferred. This effort will be on-going.

We are in process of notifying the appropriate law enforcement agencies, including Federal, but quite honestly, I don’t expect much attention from them.

More important would be our obligation to notify individuals of a possible breach of their information. State laws vary so it will be complicated. I’ve been given the name of a law firm with an excellent reputation in these matters and plan to contact them as soon as possible. A quick reading of Connecticut statutes doesn’t indicate a requirement to notify in this situation because of the nature of the information potentially compromised, but there are 49 other states with different regulations and we have members in all of them.

I’m sure there will be many questions but please understand that the situation is very fluid. I will provide you with updates as we move forward.

_______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org http://reflector.arrl.org/mailman/listinfo/arrl-odv

_______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org http://reflector.arrl.org/mailman/listinfo/arrl-odv

I’m hoping that the following sentence from Barry’s original email is incomplete, “We immediately shut down access to P1K from outside the building.“ An infected computer should be immediately 100% disconnected from anything else. Disconnecting it from the outside world doesn’t prevent it from running automated routines to search out other vulnerable computers it can connect to and infecting them. If we are going to have a discussion about why this breach occurred, what could have been done to prevent it, and how to address systemic IT issues—especially ones related to security, let’s lay the credentials of everyone who wishes to provide guidance on the table so their actual level of real world knowledge can be assessed. I have been investigating cybercrimes for two decades both as a state law enforcement agent and in the private sector. I served as the President of the Florida Association of Computer Crime Investigators for a decade. I have testified in Congress and taught at the FBI Academy. The first federal wiretaps on computer accounts were predicated upon probable cause generated in my online investigations. In addition to teaching in the Digital Forensics and Cyber Security program at Valencia College, I am the Chairman of the College’s Computer Engineering Technology Industry Advisory Board. (By the way, Valencia was named the best community college in the nation last year and we are designated by the National Security Agency and the Department of Homeland Security as a National Center of Academic Excellence in Information Assurance.) I have been accepted by state, federal, and military courts in Florida, Tennessee, New York, Ohio, Oklahoma, Pennsylvania, and Texas as an expert in digital forensics, computers, networks, and data center practices. My expert testimony has covered malware, network intrusion, and a wide variety of other computer topics. In 1999 I authored a manual for drafting computer policies that addressed topics including data and information security, electronic security, and physical security. It was used by hundreds of organizations around the world including the US Military, various federal agencies, city governments, banks, insurance companies, and schools. For the past 12 or so years, I maintained a server in my office running Microsoft Exchange and web sites. I have now migrated those applications to the cloud. I have an Oracle database server in my office that is used in my forensic network. I ran an SQL server on the forensic network for a couple years for analyzing millions of log file records in a multi-million dollar network intrusion case. I’ve been running a Linux firewall for the Exchange/web server for about the last decade, including intrusion detection/prevention software. All of these servers were built and maintained solely by me, from putting together the hardware components to installing and configuring the software. I conduct digital forensic investigations to find evidence and determine how that evidence came to exist. This process involves examining everything from cell phones to personal computers to servers to routers. Understanding network topology and functionality is a requirement to conduct my investigations. My clients include Fortune 500 corporations, the US Military, and federal prosecutors. Sorry for the long summary, but I believe it is important for you to know my background beyond being a retired cop and owner of a pack/ship store when assessing my comments and opinions on IT/cybersecurity. I am certainly very interested in details of the IT/cybersecurity backgrounds of other members of the Board that I’m not aware of—especially ones that offer opinions on what is acceptable practices. Knee jerk about the League’s IT problems, REALLY??? It is quite clear that the oversight that A&F is supposed to be providing for IT is not working. It is unacceptable that an ancient server was allowed to be in the production environment where it could be used as a launching point for forays into other servers. Clearly a better oversight and planning method is called for. The first step in developing a strategic plan is to take inventory of what already exists. Any competent inventory would have identified the antiquated server and called for its immediate replacement. Anyone who actually understands cybersecurity knows that outdated servers are especially vulnerable to attack and compromise. They are sitting there available 24/7 for hackers to attack. Regardless of whether the attack vector was a just recently discovered vulnerability or one that has been known for a long time, acceptable practices for network security do not allow for any outdated server. Another issue that would have been examined by the proposed committee would be what, if any, Intrusion Detection/Prevention System (IDS) is in use, how frequently are its rules updated, and how frequently are its logs reviewed. The open source SNORT IDS had rules in place at least as early as 9/26/2014. If we actually have an IDS and its rules are being properly updated, this means that our network was compromised more than a week ago and the compromise went undetected for that entire period. If we aren’t running an IDS, that is inexcusable. The level of importance of the email reflector is irrelevant. The take away from the situation was that we were running software that was years out of date. This means that there is apparently no program in place to ensure that software on our servers is regularly updated. Someone that actually understands cybersecurity is well aware that flaws in older applications, even seemingly unimportant ones, are frequently the attack vector exploited by hackers to gain deeper access into a server and ultimately the network. By the way, the reflector software, Mailman, is free so any argument about the cost of implementing the latest version is moot. Surely someone could have found the time over the course of years to update it at least once. This is a failure of policy—a policy that all server software will be patched/updated when the patches/updates become available. This security breach was foreseeable as will be the next one if we don’t address this now. We can continue to play cleanup after the mess is made by not dealing with creating the IT Strategic Planning Committee until at least January or we can get on the road to discovering and eliminating the next embarrassment by having the EC take up my proposal and submit a Motion to the entire Board. There’s an old saying: if you always do what you always did, you’ll always get what you always got. Applying that to this situation we got hacked and unless we change the way we do IT, we’ll get hacked again. If you have read this far, thank you. Doug K4AC From: gpwidin1@gmail.com [mailto:gpwidin1@gmail.com] On Behalf Of G Widin Sent: Friday, October 3, 2014 10:57 PM To: Doug Rehman Cc: Shelley, Barry, N1VXY; Stratton, John, N5AUS; arrl-odv Subject: Re: [arrl-odv:23380] Re: CONFIDENTIAL -- Security Breach - Outside Help? While I agree that the security breach reported by Barry is serious, it is evidently receiving urgent attention from ARRL's IT Department. The steps reported by Barry are appropriate, and I believe our staff is capable of handling the situation if we stay out of their way. I don't believe for one minute that an IT strategic planning committee would have prevented this situation. Unless such a committee were to examine all of ARRL's extensive and diverse applications in detail, it would not have found the specific vulnerabililty that was exploited in this incident. The A&F Committee is responsible for the strategic review of ARRL's IT resources, and will report at the next Board meeting in response to Director Rehman's proposal of a separate, specific committee to concern itself solely with IT strategy. But a separate committee on IT would not have found and targeted this specific situation unless it was operating at a level that is far from strategic. ALL computers run software that is not the latest version. This is inherent in the evolution of computing and software techniques. Until comparatively recently, all PCs were running DOS under the hood--did that make everything unacceptable? Running a years-old version of an e-mail reflector doesn't automatically create a problem. Such choices are sometimes necessary in an organization that does not have unlimited funds. The Board should note that despite this, the most critical data, such as credit card and social security numbers, had been given appropriate special treatment that made them inaccessible to the hackers. The Board referred the proposal for a strategic IT committee to A&F in July, and A&F will duly report its response in January. At this time, I think a knee-jerk response to the current breach is only likely to confuse the situation and impede our best response to it. I do not think a response by the EC is needed or appropriate at this time. 73, Greg, K0GW On Fri, Oct 3, 2014 at 6:11 PM, Doug Rehman <doug@k4ac.com> wrote: I will remind my fellow Board members of my proposal for creating an IT Strategic Planning Committee. Had this Committee already existed and been properly constituted, this breach could have been averted. This is the second time in the exceedingly short time I have been on the Board that a system has been found with outdated software. (The email reflectors were running a years old version of the software.) Having anything in a production IT environment that is not running the most current software is absolutely unacceptable. This is basic computer security 101 and we have failed twice this year that I know of. I ask the members of the Executive Committee to take up my proposal at the upcoming meeting and put it out to the Board for an electronic vote. We cannot wait for the next crisis—we need to start getting ahead of problems instead of trying to play clean up. Doug K4AC (and Valencia College Professor in the Digital Forensics and Cyber Security degree program…) From: arrl-odv [mailto:arrl-odv-bounces@reflector.arrl.org] On Behalf Of Shelley, Barry, N1VXY Sent: Friday, October 3, 2014 6:24 PM To: Stratton, John, N5AUS Cc: arrl-odv Subject: [arrl-odv:23378] Re: CONFIDENTIAL -- Security Breach - Outside Help? John: We've already begun developing a list of possible actions/recommendations that we want to consider. That is on the list. Our first priority is insuring we've eliminated the threat and any lingering vulnerabilities. Then we'll tackle the list going forward. 73, Barry, N1VXY Sent from my Verizon Wireless 4G LTE DROID "Stratton, John, N5AUS" <jrs@hamradio.us.com> wrote: Barry Thank you for keeping the Board up to date. Have we considered bringing in outside security experts to review our implementation of security protocols and to make recommendations for improvements or additional methods to harden our data structure and servers? 73 ----------------------------------------------------- John Robert Stratton N5AUS Office telephone: 512-445-6262 Cell: 512-426-2028 PO Box 2232 Austin, Texas 78768-2232 ----------------------------------------------------- On 10/3/14 4:19 PM, Shelley, Barry, N1VXY wrote: ARRL Board of Directors: This is a confidential notification to the Board that we have identified an unauthorized breach of the ARRL computer systems. We are currently working through all the appropriate steps as described below. In addition, we are in the process of making the necessary notification to law enforcement agencies. Situation: Early yesterday, we became aware of files on one of our many servers that were “unusual” and the IT Dept., in their investigation, determined that there had been a breach of one of our oldest servers (P1K), a proxy server for the web site. While uncertain of the exact vulnerability, the recently identified “Shellshock” vulnerability for older Linux systems may have been a factor in this attack. We were able to track IP address activity to Eastern Europe and identify the activities from those sites. We immediately shut down access to P1K from outside the building. This has impacted some web services such as the On-line DXCC application and legacy connections to LoTW. On-going Activities: In these situations the general steps to follow are: 1. Lock systems down. 2. Identify what, if anything was accessed. 3. Notify law enforcement. 4. Decisions on notification. 5. Take corrective actions. At this moment we still have prohibited access to the P1K server from outside the building. We are in the process of bringing up another server to support those functions that were supported by the old server but with all the current patches, updates applied. Because of the age of the old server, we were not able to apply all the current updates, etc. Projections for the new proxy server coming on-line are early next week. We will give priority to the applications with a higher level of activity like LoTW and On-line DXCC. Please note, it is only the legacy applications that use the P1K proxy to access the LoTW server. In addition, IT is going through all the code for the database servers and programs, eliminating any potential vulnerabilities. The IT Department has been reviewing the log files in detail and, to this point it would appear that the database the hackers had most activity with was the On-line DXCC database. They did have access, however, to names, addresses and e-mail addresses from our main date base. Current web site passwords are encrypted but passwords used on the former website (prior to 2010) were not encrypted and archival copies of those passwords stored on HQ servers may have been compromised. A limited test of old passwords indicates that possibly 10% of the old passwords may still be in use. We have not seen any unusual activity on the LoTW server. Because of the configuration, the hackers were not able to get information from the ARRL web site itself. Hackers could have accessed name, address, phone numbers and e-mail addresses. We do not keep credit card or social security information in our databases so this is not information that could be accessed. We are still working on reviewing all the logs and activity in the various databases to identify what information, if any, might have been breached and transferred. This effort will be on-going. We are in process of notifying the appropriate law enforcement agencies, including Federal, but quite honestly, I don’t expect much attention from them. More important would be our obligation to notify individuals of a possible breach of their information. State laws vary so it will be complicated. I’ve been given the name of a law firm with an excellent reputation in these matters and plan to contact them as soon as possible. A quick reading of Connecticut statutes doesn’t indicate a requirement to notify in this situation because of the nature of the information potentially compromised, but there are 49 other states with different regulations and we have members in all of them. I’m sure there will be many questions but please understand that the situation is very fluid. I will provide you with updates as we move forward. _______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org http://reflector.arrl.org/mailman/listinfo/arrl-odv _______________________________________________ arrl-odv mailing list arrl-odv@reflector.arrl.org http://reflector.arrl.org/mailman/listinfo/arrl-odv