Multilevel Structure Parallel IO Algorithm Based on HDF5

A multi-level parallel IO (Input/Output) scheme based on Hierarchical Data Format (HDF5) was proposed for large-scale data input and output applications. The parallel IO scheme was divided into two layers: Inter-node IO data was taken as unit, intra-node IO data was allowed to work cooperatively or independently. According to the internal working mode of nodes, a multi-level parallel IO algorithm and a multi-level sentinel parallel IO algorithm were proposed respectively, which could effectively improve IO efficiency and avoid redundancy of output files. Considering the two typical application scenarios of heterogeneous computing and pure CPU computing, multi-group experiments with a maximum of 4096 cores and 256G data were carried out on Shuguang platform and Intel platform, respectively. The results showed that the IO efficiency of multi-level parallel IO algorithm was increased by 1.97~25.87 times. The IO efficiency of multi-level sentinel parallel IO algorithm was increased by 6.53~9.36 times, and the number of output files was reduced to 1/4 and 1/32 of the number of parallel IO algorithms.