TY - GEN
T1 - Graceful performance degradation in Apache Storm
AU - HoseinyFarahabady, Mohammad Reza
AU - Taheri, Javid
AU - Zomaya, Albert Y.
AU - Tari, Zahir
PY - 2021/2/21
Y1 - 2021/2/21
N2 - The concept of stream data processing is becoming challenging in most business sectors where try to improve their operational efficiency by deriving valuable information from unstructured, yet, contentiously generated high volume raw data in an expected time spans. A modern streamlined data processing platform is required to execute analytical pipelines over a continues flow of data-items that might arrive in a high rate. In most cases, the platform is also expected to dynamically adapt to dynamic characteristics of the incoming traffic rates and the ever-changing condition of underlying computational resources while fulfill the tight latency constraints imposed by the end-users. Apache Storm has emerged as an important open source technology for performing stream processing with very tight latency constraints over a cluster of computing nodes. To increase the overall resource utilization, however, the service provider might be tempted to use a consolidation strategy to pack as many applications as possible in a (cloud-centric) cluster with limited number of working nodes. However, collocated applications can negatively compete with each other, for obtaining the resource capacity in a shared platform that, in turn, the result may lead to a severe performance degradation among all running applications. The main objective of this work is to develop an elastic solution in a modern stream processing ecosystem, for addressing the shared resource contention problem among collocated applications. We propose a mechanism, based on design principles of Model Predictive Control theory, for coping with the extreme conditions in which the collocated analytical applications have different quality of service (QoS) levels while the shared-resource interference is considered as a key performance limiting parameter. Experimental results confirm that the proposed controller can successfully enhance the p -99 latency of high priority applications by 67%, compared to the default round robin resource allocation strategy in Storm, during the high traffic load, while maintaining the requested quality of service levels.
AB - The concept of stream data processing is becoming challenging in most business sectors where try to improve their operational efficiency by deriving valuable information from unstructured, yet, contentiously generated high volume raw data in an expected time spans. A modern streamlined data processing platform is required to execute analytical pipelines over a continues flow of data-items that might arrive in a high rate. In most cases, the platform is also expected to dynamically adapt to dynamic characteristics of the incoming traffic rates and the ever-changing condition of underlying computational resources while fulfill the tight latency constraints imposed by the end-users. Apache Storm has emerged as an important open source technology for performing stream processing with very tight latency constraints over a cluster of computing nodes. To increase the overall resource utilization, however, the service provider might be tempted to use a consolidation strategy to pack as many applications as possible in a (cloud-centric) cluster with limited number of working nodes. However, collocated applications can negatively compete with each other, for obtaining the resource capacity in a shared platform that, in turn, the result may lead to a severe performance degradation among all running applications. The main objective of this work is to develop an elastic solution in a modern stream processing ecosystem, for addressing the shared resource contention problem among collocated applications. We propose a mechanism, based on design principles of Model Predictive Control theory, for coping with the extreme conditions in which the collocated analytical applications have different quality of service (QoS) levels while the shared-resource interference is considered as a key performance limiting parameter. Experimental results confirm that the proposed controller can successfully enhance the p -99 latency of high priority applications by 67%, compared to the default round robin resource allocation strategy in Storm, during the high traffic load, while maintaining the requested quality of service levels.
KW - Apache storm streaming processing platform
KW - Elastic resource controller
KW - Performance modeling of computer system
KW - Quality of Services (QoS)
U2 - 10.1007/978-3-030-69244-5_35
DO - 10.1007/978-3-030-69244-5_35
M3 - Conference contribution
AN - SCOPUS:85104421760
SN - 9783030692438
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 389
EP - 400
BT - Parallel and distributed computing, applications and technologies: 21st International Conference, PDCAT 2020, proceedings
A2 - Zhang, Yong
A2 - Xu, Yicheng
A2 - Tian, Hui
PB - Springer Cham
T2 - 21st International Conference on Parallel and Distributed Computing, Applications, and Technologies 2020
Y2 - 28 December 2020 through 30 December 2020
ER -