Why Your Data Center Needs Sequence of Events Recorders

Data centers have a critical mission within every corporation simply because so many people; vendors, customers, employees, and other stakeholders rely on IT and web services to communicate and interact. So, when a data center goes down due to a power interruption, everything just stops…the server that powers e-commerce, cloud storage, email servers, VOIP, streaming services and a host of other services all fail.

Sequence of Events Recorders (SERs) provide the insight to help restore full operation quicker if a power outage does occur and can also help prevent a repeat of the same type of interruption. The SER, just like an airliner’s black box, stores precisely what happened that led to the disruption and thereby points to its root cause. With SERs in place, there’s no reason to wonder what happened when power was interrupted or why.

Data center managers rely on protective relays, UPS’s, backup generators and a host of other devices to create a system designed to prevent power disruptions. They may also use (PLCs), Programmable Logic Controllers to help monitor and control equipment in the electrical system.

Some of these devices have precision time stamped data but are limited only to what they can see and cannot extend this capability throughout the power distribution system. Without a well distributed precision timed monitoring system, it is difficult to pinpoint when a failure occurred and what chain of events caused it. Sequence of Event Recorders, or SERs on the other hand, quietly monitor your entire electrical system and can capture thousands of data points that make forensic and root cause analysis easy. In short, sequence of events recorders are a crucial asset to any significant data center.

SERs can identify potential problems before they lead to a power outage— and they answer the question “what happened and when?” to minimize downtime if an outage does occur.

What Do Sequence of Events Recorders Do?

SERs monitor the many intelligent electrical devices (IEDs) that comprise the overall electrical power system. Examples of IEDs and how SERs interact with them include…

• Uninterruptible Power Supplies (UPS). Every UPS has a performance spec that defines how long it takes to activate when external power fails. The SER can capture and report the actual time a given UPS takes to switch on. If activation time falls out of spec, the SER gathers that information, which can help pinpoint the UPS’s failure and how it contributed to a power outage.

• Automatic Transfer Switches (ATS). An ATS is an intelligent power switching device controlled by logic circuitry. When the usual power source fails, the ATS switches the load to another power source such as a standby generator. These switches can fail altogether, but more commonly, they may not act quickly enough to initiate backup power. The SERs monitor the switching time of the ATS to help determine if it still meets requirements or if maintenance may be required.

• Power Distribution Units (PDUs), as their name indicates, deliver AC power from an uninterruptible power supply (UPS), a generator, or utility power source to servers, networking hardware, telecom equipment, and other devices. The SER monitors the status of the PDU and records any changes from Normal to Bypass or Transfer status during routine maintenance or a power event.  This helps to rule out the PDU as a point failure during a disruption in power.

• Protective Relays detect the presence of a power event and operate circuit breakers that isolate the faulty system before the event causes damage. By monitoring the status of the relay along with the corresponding breaker, the SER shows the elapsed time between the relay command and the breaker response. This information is useful in verifying the equipment works as designed.

SERs synchronize the clocks of all connected IEDs in the entire system. SERs then record state changes (a device going from “on” to “off,” for instance) and store them in non-volatile memory. Without an accurate and consistent time stamp on every event the SER records, it’s impossible to determine what happened first, second, and so forth — making the root cause analysis difficult at best..

To accurately monitor power transfers, analyze breaker functioning and reliability, track power loss events and reconstruct ‘what happened and when,’ SERs provide simple but precise time synchronization and status monitoring for the entire power distribution system.

Enter Time Protocols

Personal computers, servers, smartphones, and other electronics display the correct time because they access the Internet to get the real-time from a Network Time Protocol (NTP) server. NTP generally provides accurate time to within a second. However, when applied to IEDs in a complex electrical system, the accuracy can be anywhere between 10 and 250 ms depending on the device. One of the many benefits of using an SER, is that it can provide timestamp accuracy of 1 millisecond. This is crucial in an environment where status changes can occur in less than a quarter cycle. 

SERs use Precision Time Protocol (PTP) to reach 1 millisecond accuracy over an Ethernet network. Due to this greater accuracy, PTP is generally the preferred time protocol over NTP, especially in data centers’ highly complex electrical power systems.

Some IEDs are not designed to work with PTP, but that’s not a problem. Cyber Sciences SERs can accommodate legacy devices that do not yet support PTP. In that case a CyTime SER (used as a PTP slave) can serve as a “time-sync hub” to output any of the following legacy protocols: IRIG-B, DCF77, ASCII, or 1per10.

Having one authoritative time source for time-stamped data recorded by the SERs makes it possible to see the exact order in which events leading up to a power interruption occurred. Considering that a single SER can monitor up to 32 IEDs, larger data centers may require many SERs to monitor an array of IEDs distributed throughout a complex power management system, you can see that both the SERs accurate event timestamping and time synchronization are essential.

SERs Deliver These Important Benefits

SERs synchronized throughout the electrical system give facility engineers and technicians
the information needed to…

• Gather more data than metering alone can provide.

• Reconstruct power outage events by conducting root-cause analysis.

• Improve reliability of the overall electrical power system.

• Avoid downtime and reduce the duration of outage events saving both time and money.
  

SERs also turn into actionable information. They…

• Provide a common time reference across an entire electrical system.

• Timestamp events to 1-millisecond resolution.

• Simplify analysis of circuit breaker operation by recording “delta time” that shows whether a breaker operates within specs.

• Provide time synchronization to other electrical distribution devices.

• Enable power system analytics and root cause identification

Even More!

Additionally, Cyber Science’s CyTime Event Manager — a supplementary device that organizes data from all SERs on the network — provides seamless system-level status monitoring, data aggregation, and reporting. There’s no longer any reason to log into an individual SER to determine the status of monitored inputs.

With the Event manager you can view the events of all system SERs in one view. You can also filter this information for faster analysis. The Event Manager also helps to keep the firmware of all downstream SERs up to date. Need to quickly change the setup and configuration of one or more SERs? With configuration templates, the Event Manager makes these changes easy and repeatable.

View the Cyber Sciences EPMS-High Def Data Sheet

https://landing.cyber-sciences.com/en-us/download-epms-hi-def-data-sheet-0

Sources

RFC-5905 “Network Time Protocol Version 4: Protocol and Algorithms Specification” http://www.ntp.org

IEEE Std 1588™-2008, IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems https://www.cyber-sciences.com/wp-content/uploads/2019/01/DataSheet_EPMS-HD.pdf

Back