After a signal disruption affecting more than 10,000 ADS radio transceivers resulting from what ADS called a “bad packet” of data from AES technology, the two companies have taken steps to correct the issue and ensure the network does not experience further disruptions.
The disruption, which affected ADS customers in parts of Illinois and Wisconsin, required ADS to make brief site visits to replace the radio transceivers. “The radio outage really pushed our company and its employees to the limit,” said Nick Bonifas, corporate counsel, Alarm Detection Systems Inc. “In about six weeks, we performed over 10,500 site visits just to restore monitoring for our customers. The only way you accomplish something like that is to have employees step up and contribute in ways that seemed unimaginable two months ago. I cannot thank our employees enough.”
According to an AES statement by President and COO Bill Kieckhafer released to its dealers (only one of which — ADS — was affected), “The outage was entirely limited to that one network (and the 465.9875 MHz frequency specifically) and was caused by factors wholly external to that particular network.” Kieckhafer further stated the event was unique and “highly unlikely to ever be repeated,” but emphasized it is “moving proactively to ensure that the AES-IntelliNet network remains the industry standard for security and reliability.”
AES stated a French satellite that transmits on the same frequency as its customers’ AES-IntelliNet network flooded the earth’s surface in the Illinois area with un-decodable data packets. This alone, AES said would not have caused the disruption; however, “a simultaneous incident of severe, ground-based radio frequency interference (RFI) in the same general area of Illinois” occurred as well, Kieckhafer reported.
“We therefore have concluded,” Kieckhafer continued, “that the simultaneous combination of the French satellite transmitting over the same frequency that also was flooded with data from the independent ground level RFI incident provided too much corrupted data too quickly to be filtered out by the first generation AES-IntelliNet units remaining in service in that specific network. The corrupted data then propagated throughout the entire network, but only that network.”
AES maintains that neither incident would have been enough to cause the outage, and emphasized that “in more than two decades of designing, building and supporting the AES-IntelliNet system for customers, this was the first incident of its kind.”
Kieckhafer said that AES responded immediately after discovering the issue. “We basically shut down our engineering to look at this,” Kieckhafer said. “Because it was such a unique event, it took us a little longer. After a while we were able to determine [the cause] through a process of elimination.”
Kieckhafer described the process AES engineers used to discover the cause of the issue, saying, “We were unable to generate this mis-formed packet just using our equipment; that’s where we realized there had to be some outside source, and that’s when we ultimately found it was over-the-air corruption. We are working on a number of solutions that we are testing in our labs now that could minimize it even further, even though it happened only once in 22 years.”
In regard to AES’ efforts to support ADS following the incident, Kieckhafer said, “We had engineers on the ground at ADS two days after it happened. I was made aware of it on Monday, and our engineering team was already working on it. Once ADS established a plan of action, we have supported them by sending them significant equipment, including flying equipment in from Asia to support their efforts.”
Kieckhafer said the company is continuing to develop technology to ensure such an event never occurs again, despite its recurring being already highly unlikely.
