Excessive heat is expected in the coming days, with outside temperatures reaching over +100F in heat. The National Weather Services has issued a heat warning due to this. While we have made several changes to prepare for the coming heat, it may have a small impact on our services. https://www.weather.gov/lsx/excessiveheatbriefing
Mitigation:
To help mitigate heating issues, we have obtained 4 5-ton and 2 3-ton spot cooling systems to supplement our cooling. We have also optimized airflow throughout our data center to move heat out quickly and cool air in. We have also implemented several monitoring systems to deal with potential issues proactively.
Resolution:
To resolve potential heat-related issues, All clients are advised to hold off on any planned work that may require higher-than-normal loads on their devices until the heat advisory has been dismissed. We also ask that all clients use their best judgment when implementing "Power saving" methods on their servers, which help lower the overall heat a given device produces.
The 06/21/2024 cooling disparity has been corrected, and monitoring over the weekend has confirmed that the fix has kept temperatures stable. The issue was identified as a faulty temperature sensor in the Glycol loop, causing the system to detect a lower-than-actual temperature in the loop of -16f, which was not the case. Due to this, it turned the chiller system off, thinking it was too cold to run.
We have identified the cooling disparity and are working with the related parties to resolve it. You should begin to see things stabilize shortly.
We are currently investigating the system cooling at this time.
