How It Works: Power Monitoring in Selectel Data Centers
In your opinion, what do these three events have in common: A social network (VK) user posts a new photo; someone uses Assist to make an online purchase with a bank card; and car is brought to CarPrice and appears in the auction stream.
And did each case end with the expected outcome?
The answer is like this: all of these actions were possible because—and we’re really skipping around here—all of the servers hosting these applications are hosted in Selectel data centers with a steady and reliable stream of power, which is constantly monitored. This may sound a bit primitive, but the foundation of any IT infrastructure is its power infrastructure. We’ve already talked about the power supply in Selectel data centers in our article Powering a Data Center, so today, we’re going to shed some light on the subject of monitoring.
Monitoring against the Second Law of Thermodynamics
It’s no exaggeration that the power infrastructure is where all the beauty, or maybe it’d be better to say ugliness, of the second law of thermodynamics manifests itself. “If energy is not spent maintaining order, every system would fall into chaos.” Not allowing chaos into the data center power supply is the most important task, and it’s entrusted to many multilevel power monitoring systems and the Selectel technical team.
Let’s look at how electricity from the city grid passes through all the power supply levels of Selectel’s Tsvetochnaya 2 data center (pictured above) and eventually makes it to every server’s power supply and hard drive. And of course, we’ll talk about which parameters are measured at each level and stage, since you can’t manage something you don’t measure.
Why did we decide to discuss power monitoring? We understand that clients are interested in the abstract and massive systems in our data center infrastructure, as well as the continuous operation of this equipment. In turn, for Selectel, as a commercial organization, the most important business criteria is client satisfaction: your capital helps us grow, while your feedback and ideas help us improve the quality of our services.
A Tour of the Monitoring System
We doubt anyone’s really interested in reading some dry extracts from our power monitoring instructions and regulations, right? So instead, let’s take a quick tour!
Selectel owns and operates six data centers. In today’s article, we’ll be looking at the power supply for the Tsvetochnaya 2 data center in St. Petersburg. The power supply and monitoring thereof is implemented similarly in the other five data centers: Tsvetochnaya 1 in St. Petersburg; Dubrovka 1, 2, and 3; and Berzarina 1 in Moscow.
Monitoring begins as soon as power enters the data center. Tsvetochnaya 2 has two independent power feeds. Each feed brings 6kV to the data centers on Tsvetochnaya street (or 10-20kV in the case of our other data centers). For the data center to properly use this power, the current is lowered to 380V in two PS 6-10/0.38kV transformer substations (TS1 and TS2) located on the premises of the data center. Current and frequency are measured as power enters and leaves these substations. The temperature in the substation is also measured to ensure it stays within the allotted range.
From the transformer substations, external feeds enter two groups of uninterruptible power supplies (UPS), linked to a block (bank) of 12V batteries. The incoming current and inverter, condenser, and fan status are all monitored for each UPS, as is the overall UPS status. We also measure and track the current and frequency leaving our UPS clusters.
The battery bank consists of 12V lead batteries, which are similar to car batteries and interconnected in a huge array. By the way, do you remember that a while back there was talk on the Internet that Tesla batteries were supposedly made of hundreds of AA batteries? That’s how data center backup power is provided: over a hundred everyday 12V batteries. The temperature and current of battery banks is measured in real time. If any one 12 V battery malfunctions, Selectel engineers will quickly be notified of the problem and replace it.
If both TS1 and TS2 substations are completely cut off from the city feeds, the battery banks can fully power the data center (under full workload) for 10 minutes. During this time, our diesel generators are started, which then take on the full workload. In addition to the current exiting the generators, many other parameters are monitored. These includes engine rotation, temperature and oil pressure, coolant temperature, fuel supply, active and reserve power, and total engine hours.
The generators are kept in a state of preheat; oil and coolant temperatures are maintained at +40C. This lets us easily start the generators in cold weather and ensures they can warm up and take on the full workload in just one minute.
Power from the UPS then enters distribution boards in each machine room and is sent to racks with servers, storage systems, and network equipment. Each rack receives two independent 220V AC feeds. This lets us install servers and storage systems with two power supplies at 2N redundancy. Power entering the rack is measured, and current and power consumption is monitored. This is how Selectel engineers keep track of the power consumption for each rack and the data center overall.
If IT equipment with one one power supply (if a client has that type of equipment, for example) is mounted in a rack, then we can optionally use a rack-mounted ATS (automatic transfer switch). If one rack input ever loses power, the ATS immediately transfers the workload to the other. This happens so fast that the equipment attached to it don’t experience any disturbance. The ATS is also monitored, and Selectel engineers receive reliable information about the power supply for each rack.
We’ve followed the entire electrical route of the data center, starting from the city grid and ending at a single rack in the server room. Next on our tour, we’ll visit the data center dispatch center, where the infrastructure is managed and monitored, and where we’ll meet some Selectel employees who perform around-the-clock monitoring of our power supply.
Our Heroes: The Engineering and Technical Department
Interchangeability, mutual support, and mentorship—these are the principles our Engineering and Technical Department (ETD) work by. The department employs over 40 professionals who are spread across three shifts in different data centers. When necessary, employees can assist one another, assuming the task of monitoring power metrics and ventilation remotely and giving advice to one another.
Next, we have two videos (with English subtitles) that show us just how the ETD team works. In our first video, Tatyana Svirko, the head of ETD, explains how monitoring is implemented in our data centers.
We’ll now complete our tour with engineer Dmitry Kostryukov, who shows us how information on infrastructure metrics are gathered in different areas in Tsvetochnaya 2.
This is how we thoroughly check and monitor every element of our infrastructure in Selectel data centers. We hope this look inside helps put our clients at ease, knowing their equipment is safe and power guaranteed.