Dependency Prediction of Long-Time Resource Uses in HPC Environment

Abstract

High-Performance computing provides a new infrastructure for scientific calculation and its simulation. However, unbalanced load distribution among the processors causes a decreased performance in such computations, and creates a massive requirement of computing resource allocation, that requires an increased simulation. Therefore long-range resource utilization prediction becomes essential to achieve optimal performance in an HPC environment. This paper introduces a novel ensemble technique, which includes two algorithms, the Feature-based capability prediction algorithm(FBCA), and the Accuracy and Relative Runtime Error Prediction Algorithm (ARRE). A three-level architectural framework (the simulation environment, resource prediction, and resource queue) has also been proposed and tested on Phold and SoS. The proposed framework can deal with the requirements of computing and simulations. The FBCA algorithm reduces the redundancy between available features, and the ARRE algorithm ensures our ensemble technique's effectiveness. We have compared the performance of the proposed schemes with other existing methods such as the Regressive Approach, Linear Regression and Random Forest, and found that our proposed algorithm achieves better accuracy from 8% to 18%.