Powering a Data Center
Just over two years have passed, or rather flown by, since the grand opening of the Tsvetochnaya 2 data center. In our article covering the event, we not only shared photos from the festivities, but briefly looked at the data center’s infrastructure. We soon started thinking that it’d be a great idea to publish a series of articles that take a more in-depth look at these systems. Well, we’ve finally managed to take a break from our day-to-day and put our plan into action.
At the heart of it, the operations of any data center rest on three pillars: power, cooling, and security. Without a doubt, the key system here is power.
Uninterruptible Foundation
Tsvetochnaya 2 is designed to handle 2500 kVA of power consumption and meets international Tier III standards. Accordingly, all power and cooling systems are 2N redundant, giving significant advantages to Colocation clients and the data center itself; we can perform maintenance and replace malfunctioning parts without equipment or the data center itself losing power.
The data center is powered by two independent municipal feeds, which lead into two company-owned 10(6)/0.4 transformer substations. The substations are located within the complex, which is also Selectel property. We bring up ownership because this directly affects clients, especially in terms of stability: in Russia, a fairly large number of data centers are located on rented property, so their business and prices are directly influenced by the volatility of the leasing market and even their landlord’s mood.
In the substations, 6 kV of municipal current is lowered to 380 V (400V), and then distributed to the main distribution board with protective automation.
Since this energy can be plagued by various kinds of interference (surges and dips, high-frequency noise, etc.), it could pose a potential threat to sensitive servers and communications equipment, not to mention HVAC systems and data center automation.
This is why the 380 V (400 V) bus ducts lead to two groups of industrial uninterruptible power supplies (UPS), each housing a 6×400 kVA module with double conversion functionality. Here, the alternating current is converted to a direct current, and then again to 380 V (400 V) AC power, but now with perfect sine waves.
In addition to removing interference, the UPS serve another important role: if the municipal feeds “go dark”, their batteries power the data center while the workload is transferred to emergency power sources — diesel generators. To do this, the UPS are combined in a bidirectional redundant matrix of 130 12 V battery cells. These batteries can power the full workload of the data center for 10 minutes. If external power is cut, automatic load transfer automation launches three 1800 kVA generators in a separate location, which can power the data center for a nearly endless period of time.
Diesel generators are N+1 redundant, or in other words, Tsvetochnaya has the two generators it requires to run normally, plus a backup. A 48-hour fuel supply is kept on hand and contracts are in place with two fuel suppliers for urgent deliveries in the event of an extended power outage.
It takes an average of 3 minutes to launch the diesel generators and for them to take over the workload. The UPS cells supply the power for that time and have additional charge just in case. To ensure the generators can start, even in the winter, the generators’ coolant is constantly heated, and this processes can be controlled on site or from a remote monitoring system.
The power feeds and diesel generators are connected to one another through an automatic transfer switch (ATS), which leads to two input distribution switchboards (2x). Power is sent from the two boards to two UPS groups respectively. From each UPS group, power is sent to a UPS output switchboard, which directly connects to the rack power distribution cabinets (circuit breakers) in the machine rooms. These cabinets distribute a 380 V (400 V) current along 230 V buses to power computers and network equipment. UPS provide the 400/230 V European standard.
Power is fed to each machine room by four separate bus ducts from two independent sources. Each duct follows a different path to avoid simultaneous mechanical issues. This is how we achieve 2N power redundancy in our data center.
Precise Power Consumption Reporting
Each breaker contains a smart device that keeps track of power consumption in each rack and records these indicators to a log file. Even though power consumption doesn’t affect Colocation service costs, this information is still very important for data center personnel; unusually high or low consumption indicates a possible error and requires immediate attention.
Power is sent from breakers to racks along two separate cables (and with an ATS upon request). Servers with two power supplies are connected to PDUs that are powered by different feeds, and equipment with one supply to the ATS (optional).
Maintenance and Monitoring – The Key to Reliable Power
Equipment requires constant monitoring and regular maintenance. Detailed information regarding the status of the data center’s electrical system, including the generators, is gathered and displayed on large screens in the Monitoring Center. On-duty engineers also have tablets, which notify them if anything diverts from the norm.
Pursuant to regulations, on-duty engineers and electricians make constant rounds of the machine rooms and areas where electrical (power supply) equipment is held. Their tablets include an inspection plan with a description of each task to be performed at each point.
When an employee completes an inspection, they make a note on their tablet, and when they finish making their rounds, an inspection report is automatically generated and saved in the data center’s archive. To an outsider, this would be like a pilot reading through his pre-flight checklist, even when they already know it by heart.
Electrical equipment also undergoes regular maintenance. It’s first looked at while turned on with a thermal imager to check heating contacts. It is then turned off and all of its contacts are scanned. At the same time, cabinets and other dust-collectors are cleaned. It’s worth mentioning that shoe covers must be worn at all times in these areas, and some rooms even require dust-proof overalls.
Welcome to Your Tour of the Data Center
This sophisticated electrical complex provides power to Selectel clients’ servers and network equipment. Tours of Tsvetochnaya 1 and 2 are sometimes given as part of our open seminars and events here. During these tours, clients can see our power supply, air condition and security systems in action.