Thanks for the update Barry. Please keep us informed if the cause is determined. 73, Doug K4AC From: Shelley, Barry, N1VXY [mailto:bshelley@arrl.org] Sent: Monday, July 6, 2015 6:20 PM To: Rehman, Doug, K4AC; Sumner, Dave, K1ZZ; Vallio, Bob, W6RGG Cc: arrl-odv; Keane, Michael, K1MK Subject: RE: [arrl-odv:24397] Re: Where has arrl.org gone??? Doug, et al. : I apologize for the delay in responding but I was out of the office today due to an unavoidable circumstance. The following was provided primarily by Michael Keane, K1MK, ARRL IT Manager. We know there was some kind of event on Rackspace's internal network that glitched off the remotely mounted disks on our cloud servers. The cloud servers remotely mount directories from our (leased) database server; these are the directories which contain the web site content. The fact that the mount went missing is what caused the cloud servers to fall back to displaying their default web page rather than our website. It was not traffic related, there was only minimal activity on the site when the connection was broken. All four cloud servers lost their connection at basically the same time, so this wasn't the kind of cascading failure that we've seen in the past when there were congestion and capacity issues. In the four+ years that we've had this configuration at Rackspace this is the first instance where this -- losing the NFS mounts between the cloud servers and the db server -- occurred. The ticket has not been closed yet so we're still waiting to learn the source of the event. 73, Barry J. Shelley, N1VXY Chief Financial Officer ARRL, Inc. _____ From: Rehman, Doug, K4AC Sent: Monday, July 06, 2015 10:12 AM To: Shelley, Barry, N1VXY; Sumner, Dave, K1ZZ; Vallio, Bob, W6RGG Cc: arrl-odv Subject: RE: [arrl-odv:24397] Re: Where has arrl.org gone??? Was the cause ever determined? Thanks, Doug K4AC From: arrl-odv [mailto:arrl-odv-bounces@reflector.arrl.org] On Behalf Of Shelley, Barry, N1VXY Sent: Wednesday, July 1, 2015 9:12 AM To: Sumner, Dave, K1ZZ; Vallio, Bob, W6RGG Cc: arrl-odv Subject: [arrl-odv:24397] Re: Where has arrl.org gone??? Bob et al. : The IT Department has been working since early this morning and has the web site back up at this point. There is still testing on-going to identify the root cause. 73, Barry J. Shelley, N1VXY Chief Financial Officer ARRL, Inc. The National Association for Amateur Radio (860) 594-0212 www.arrl.org <http://www.arrl.org> From: arrl-odv [mailto:arrl-odv-bounces@reflector.arrl.org] On Behalf Of Sumner, Dave, K1ZZ Sent: Wednesday, July 01, 2015 5:56 AM To: Vallio, Bob, W6RGG Cc: arrl-odv Subject: [arrl-odv:24396] Re: Where has arrl.org gone??? Bob, it is not quite 6 AM here. I'm sure there will be an explanation in due course later today. Dave Sent from my Verizon Wireless 4G LTE DROID "Vallio, Bob, W6RGG" <rbvallio@gmail.com <mailto:rbvallio@gmail.com> > wrote: This is the screen I get when I try to log on. Does anyone care? W6RGG Apache 2 Test Page powered by CentOS This page is used to test the proper operation of the Apache HTTP server after it has been installed. If you can read this page it means that the Apache HTTP server installed at this site is working properly. _____ If you are a member of the general public: The fact that you are seeing this page indicates that the website you just visited is either experiencing problems or is undergoing routine maintenance. If you would like to let the administrators of this website know that you've seen this page instead of the page you expected, you should send them e-mail. In general, mail sent to the name "webmaster" and directed to the website's domain should reach the appropriate person. For example, if you experienced problems while visiting www.example.com <http://www.example.com> , you should send e-mail to "webmaster@example.com <mailto:webmaster@example.com> ". If you are the website administrator: You may now add content to the directory /var/www/html/. Note that until you do so, people visiting your website will see this page and not your content. To prevent this page from ever being used, follow the instructions in the file/etc/httpd/conf.d/welcome.conf. You are free to use the images below on Apache and CentOS Linux powered HTTP servers. Thanks for using Apache and CentOS! <http://httpd.apache.org/> <http://www.centos.org/> About CentOS: The Community ENTerprise Operating System (CentOS) Linux is a community-supported enterprise distribution derived from sources freely provided to the public by Red Hat. As such, CentOS Linux aims to be functionally compatible with Red Hat Enterprise Linux. The CentOS Project is the organization that builds CentOS. We mainly change packages to remove upstream vendor branding and artwork. For information on CentOS please visit the CentOS website <http://www.centos.org/> . Note: CentOS is an Operating System and it is used to power this website; however, the webserver is owned by the domain owner and not the CentOS Project. If you have issues with the content of this site, contact the owner of the domain, not the CentOS Project. Unless this server is on the CentOS.org domain, the CentOS Project doesn't have anything to do with the content on this webserver or any e-mails that directed you to this site. For example, if this website is www.example.com <http://www.example.com> , you would find the owner of the example.com <http://example.com> domain at the following WHOIS server: http://www.internic.net/whois.html