Abstract
A significant portion of research work in the past decade has been devoted on developing resource allocation and task scheduling solutions for large-scale data processing platforms. Such algorithms are designed to facilitate deployment of data analytic applications across either conventional cluster computing systems or modern virtualized data-centers. The main reason for such a huge research effort stems from the fact that even a slight improvement in the performance of such platforms can bring a considerable monetary savings for vendors, especially for modern data processing engines that are designed solely to perform high throughput or/and low-latency computations over massive-scale batch or streaming data. A challenging question to be yet answered in such a context is to design an effective resource allocation solution that can prevent low resource utilization while meeting the enforced performance level (such as 99-th latency percentile) in circumstances where contention among applications to obtain the capacity of shared resources is a non negligible performance-limiting parameter. This paper proposes a resource controller system, called QSpark, to cope with the problem of (i) low performance (i.e., resource utilization in the batch mode and p-99 response time in the streaming mode), and (ii) the shared resource interference among collocated applications in a multi-tenancy modern Spark platform. The proposed solution leverages a set of controlling mechanisms for dynamic partitioning of the allocation of computing resources, in a way that it can fulfill the QoS re-quirements of latency-critical data processing applications, while enhancing the throughput for all working nodes without reaching their saturation points. Through extensive experiments in our in-house Spark cluster, we compared the achieved performance of proposed solution against the default Spark resource allocation policy for a variety of Machine Learning (ML), Artificial Intelligence (AI), and Deep Learning (DL) applications. Experimental results show the effectiveness of the proposed solution by reducing the p-99 latency of high priority applications by 32 % during the burst traffic periods (for both batch and stream modes), while it can enhance the QoS satisfaction level by 65 % for applications with the highest priority (compared with the results of default Spark resource allocation strategy).
| Original language | English |
|---|---|
| Title of host publication | 2021 IEEE 20th International Symposium on Network Computing and Applications (NCA): Proceedings |
| Editors | Mauro Andreolini, Mirco Marchetti, Dimiter R. Avresky |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| ISBN (Electronic) | 9781665495509 |
| ISBN (Print) | 9781665495516 |
| DOIs | |
| Publication status | Published - 31 Jan 2022 |
| Externally published | Yes |
| Event | 20th IEEE International Symposium on Network Computing and Applications, NCA 2021 - Boston, United States Duration: 23 Nov 2021 → 26 Nov 2021 |
Publication series
| Name | IEEE International Symposium on Network Computing and Applications (NCA): Proceedings |
|---|---|
| ISSN (Print) | 2643-7910 |
| ISSN (Electronic) | 2643-7929 |
Conference
| Conference | 20th IEEE International Symposium on Network Computing and Applications, NCA 2021 |
|---|---|
| Country/Territory | United States |
| City | Boston |
| Period | 23/11/2021 → 26/11/2021 |
Bibliographical note
Publisher Copyright:© 2021 IEEE.
Keywords
- Dynamic resource allocation
- Iterative Computation
- Quality of Service (QoS)
- Streaming Big Data Processing
ASJC Scopus subject areas
- Computer Networks and Communications
- Computer Science Applications
- Hardware and Architecture
- Information Systems and Management
- Safety, Risk, Reliability and Quality
Fingerprint
Dive into the research topics of 'QSpark: distributed execution of batch streaming analytics in spark platform'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver