Improvement of Local Scheduling Algorithm Policy in Hadoop Cluster Environment

Abstract

The nature of the scheduling strategy of the local scheduling algorithm is to improve the data locality. However, due to the different completion time of the Map task, the waiting phenomenon of Reduce task affects the completion average time of the job, the completion time of the job is increased, and then the performance parameters of the system are not good. In this thesis, we propose to integrate the preemptive scheduling based on the local requirement of the original algorithm. Based on the above scheduling strategy, this thesis designs the qualitative scheduling of integrated preemptive strategy. In order to validate the improved algorithm, the local scheduling algorithm and the integrated preemptive local scheduling algorithm are compared by experiments. Experimental results show that, on the same data, the average completion time of the integrated preemptive local scheduling algorithm is significantly reduced.