Tier 1-4: Increasing secure 24/7 operation in data centres

Certified power quality monitoring and fire protection enhancement

Preamble

The volume of digital data continues to increase rapidly and steadily. Not least caused by cryptocurrencies, such as Bitcoin (XBT), Ether (ETH), Litecoin (LTC), etc. or, for example, by the blockchain process, a quasi-type database that works on various networked servers. The increasing demand for data exchange also increases the need for data centres, which are planned, built and maintained on a large scale globally. However, data centres in the electrical energy context are subject to complex challenges that can influence (legally) secure 24/ operation.

The basic requirements

When planning the energy supply of a data centre, many demands must be taken into account:

• Secure location in terms of energy supply and environmental conditions
• High energy efficiency to minimise operating costs
• Maximum availability (Uptime Insitute Tier 1-4) through use of redundancies, e.g. UPS, generators
• High security (fire protection, access, defence against cyber attacks)
• System stability and reliability of the operating resources used
• Scalability

The problem description

Various studies show that power quality problems cost billions each year. Already in 2007, the study “Pan European LPQI Power Quality Survey 2007” shows that the damage amounts to 157 billion euros/anno. In the process, the challenges for all continue to grow. Especially for the data centres. The reasons for this are as follows:

(1) Mains interference level

• Strong increase in non-linear loads (LED lighting, computers, charging devices, frequency converters, etc.) that generate harmonics
• Increase in decentralised feed-ins (e.g. wind power, PV systems), which lead to instabilities in voltage maintenance

Illustration: Dr. Jan Meyer, TU Dresden on the necessity of grid quality monitoring due to grid pollution of non-linear consumers and decentralised feed-ins.

(2) Effect of the interference levels:

• Newer equipment (e.g. servers, controllers, lifts, fire alarm systems, etc.) are more sensitive to noise levels and can fail.
• Entire components can be destroyed
• Lifecycle are reduced
• Business interruptions cost a lot of money
• Perplexity in the event of system malfunctions, as causes are often not recognisable
• Measuring equipment used cannot sufficiently detect faults
• No data recording, as the meter supply and communication is also disturbed
• Ultimately, a root cause analysis requires expensive specialist personnel with experience.

The solution

As described at the beginning, power quality problems can lead to disruptions and outages, which are always associated with effort and costs. Particularly in the data centre segment, where failures and possible damage are actually to be avoided through many investment-intensive redundancies (e.g. UPS, generators, multiple feeds), interference levels represent a non-negligible risk.

Ideally, all the equipment used meets the standards with regard to mains feedback and immunity to interference, and trouble-free operation is thus probable. However, under unfavourable conditions, e.g. caused by many loads of the same type, asymmetrical mains load, etc., significant level exceedances may occur. In order to be able to assess and limit the risks, permanent power quality monitoring is essential. Depending on the structure and extent of the data centre, monitoring at different points within the energy supply makes sense:

• At the grid operator’s feed-in point, the so-called point of coupling (PCC).
• In all protected service areas
• At the feed-in point of emergency power systems

Figure: Principle of recording and evaluating power quality measurement data

In addition to the evaluations described above, the recorded power quality data also allow existing or emerging problems to be detected at an early stage before they lead to damage. For the conformity assessment, the recorded statistics are compared with normative limit values. For data centres, these are:

• EN 50160 (characteristics of voltage in public LV, MV, HV supply networks), which normally serves as the basis for the contract with the energy supplier.
• IEC 61000-2-4 (compatibility levels in industrial installations), in particular class 1 (protected supplies)

The above standards provide guidelines on how the network should behave at the observed point during normal operation. This does not cover those exceptional situations that may lead to temporarily restricting the supply of energy. Although such faults, such as voltage dips or failures, must be recorded, their number is not limited for the purpose of meeting the standard. Es ist die Aufgabe der USV oder Netzersatzanlagen, solche Versorgungseinschränkungen zu überbrücken. However, this bridging is limited to the most important resources, so that other components may experience functional restrictions.

It is therefore essential that the operating personnel are informed promptly about the occurrence of events according to IEC 61000-4-30. This is done, for example, via an automated e-mail message to the expert persons. For the exchange of power quality data it makes sense to use a standardised format, e.g. PQDIF (Power Quality Data Interchange Format) according to IEEE 1159.3. Consequently, the choice of power quality data analysis software is not limited to proprietary manufacturer systems.

RCM as fire protection

To avoid uncontrolled interruptions of operation, no devices for residual current monitoring with direct tripping (RCDs) are used in data centres. Instead, it is mandatory to permanently monitor residual currents. RCM (Residual Current Monitoring) is used for this purpose, which serves not only essential personal protection but also plant and fire protection. Furthermore, changes in the fault currents allow insulation deterioration to be detected at an early stage and measures can be initiated in good time. Errors that occur in the TN-S system (e.g. impermissible or additional PE-N connections) can also be detected early and thus corrected.

Figure: Permanent residual current monitoring with warning and alarm thresholds

Correct measurement data through metrological traceability

An old locksmith’s saying goes: “Centimetre is a watchmaker’s measure”. In other words, “If you measure, you measure crap.” In this way, technicians and economists are aware of the well-known but nevertheless real phrase and take care of the corresponding measurement methods. And although the requirements for a power quality device are precisely defined in terms of measurement methods (IEC 61000-4-30), device characteristics (IEC 62586-1) and testing of compliance with the standards (IEC 62586-2), there are still differences between the manufacturers. In particular, providers often cannot prove why their analyser meets the specifications, i.e. measures correctly. Proof of a truly correct measurement is only possible via an independent certification body, in the best case by a metrological institute. Non-certified test bodies or even self-declarations by manufacturers cannot replace metrological certificates and should therefore also be viewed critically. Especially when it comes to sensitive areas, such as data centres, which are associated with high costs and risks.

The benefits of metrologically certified power quality monitoring

The main benefit of professional and permanent power quality monitoring is an increase in data centre availability. In this context, power quality defines itself as an essential component of the quality of supply and naturally also applies to many other sensitive areas outside of data centres (e.g. in hospitals, in sensitive industrial plants, in transport infrastructure such as airports, public building complexes such as shopping centres, etc.). The benefit comes from analysing the recorded long-term information by observing the changes and discovering correlations. It is not only compliance with the contractual feed-in guidelines that plays a role here. Additional and relevant knowledge can also be derived from the following processes:

• Comparison of normal operation to UPS or emergency power operation
• Evaluations of harmonics and their influence on the equipment
• Assessment of the change in power quality over a longer period of time
• Change in power quality after changes in installation
• Changes in power quality after switching on or off equipment
• Evaluation of the voltage events according to duration and residual voltage (ITIC curve) and their effect on the service life of the equipment

A further and specific benefit is based on a permanent RCM. With a correctly executed and permanent monitoring of a residual current, the periodically recurring as well as manual testing of the insulation strength can possibly be omitted. This means that it is not necessary to shut down the system during the test (= increase in availability). The enormous testing effort with the associated costs of time and personnel is eliminated.

Conclusion

The right insights from a metrologically certified monitoring of power quality incl. RCM, lead to sustainable investment protection, cost reduction during operation, maximisation of data availability and ultimately the important satisfaction of all stakeholders involved. This includes customers, employees, energy suppliers, operators, investors, service staff, politicians, associations, etc. Finally, it also helps to reduce CO2 emissions by enabling a more efficient and secure operation of the data centre.

Decarbonisation of the internet as an outlook

If you look at the development of global data volumes, you will see that the challenges for planners and operators will have to become greater. In China alone, the current >500,000 existing data centres are to be expanded to 1,000,000 by 2023 (increasing 21% per year). In addition to the whole issue of power quality, the question of how to regulate the growing energy demand according to PUE (Power User Effectiveness) will be asked more and more specifically, as the energy infrastructure as well as the required building space could reach their limits. According to American scientist Jonathan Koo-mey, data centres already account for about 1.1 – 1.5% of global electricity consumption. In the Frankfurt conurbation alone, data centres already account for more than 20% of total electricity consumption.

“Server performance versus electrical power” – what specific contribution can be made to PUE here?

For this purpose, it is advisable to monitor the energy used per data volume and adjusted to the operating point of the system and to transfer it directly into billing models for the supply as well as on the customer side of the data exchange. Quasi a real data billing with the actual energy demand per data unit. In this way, a real “data consumption model” would define the energy price and possibly raise awareness among data providers and users. Data could be used more sparingly due to the actual energy costs generated.

Technically, integrated energy monitoring would be conceivable for this purpose, based on a comprehensible reference model (definition of the measurement standard), which in turn monitors individual servers, racks or similar data equipment and measures the actual energy used per data volume at the operating point and thus provides valid billing. Furthermore, it should be considered whether cybercriminal attacks on the power supply in data centres or other sensitive areas cannot be additionally prevented by qualified and permanent monitoring of the power quality. This is virtually a redundancy to the existing monitoring facilities, which are already established today via software solutions, but are subject to enormous dynamics. The aim is to find out whether changes in power quality can be linked to cyber attacks on the servers as well as on the infrastructure of a data centre and thus to prevent attacks at an early stage. In both cases, server power versus electrical power (PUE) and the additional protection against cyber attacks by means of power quality analysis, the reference references (definition of the measurement standard) will be decisive.