Skip to main navigation Skip to search Skip to main content

QSpark: distributed execution of batch streaming analytics in spark platform

  • M. Reza Hoseinyfarahabady
  • , Javid Taheri
  • , Albert Y. Zomaya
  • , Zahir Tari

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

A significant portion of research work in the past decade has been devoted on developing resource allocation and task scheduling solutions for large-scale data processing platforms. Such algorithms are designed to facilitate deployment of data analytic applications across either conventional cluster computing systems or modern virtualized data-centers. The main reason for such a huge research effort stems from the fact that even a slight improvement in the performance of such platforms can bring a considerable monetary savings for vendors, especially for modern data processing engines that are designed solely to perform high throughput or/and low-latency computations over massive-scale batch or streaming data. A challenging question to be yet answered in such a context is to design an effective resource allocation solution that can prevent low resource utilization while meeting the enforced performance level (such as 99-th latency percentile) in circumstances where contention among applications to obtain the capacity of shared resources is a non negligible performance-limiting parameter. This paper proposes a resource controller system, called QSpark, to cope with the problem of (i) low performance (i.e., resource utilization in the batch mode and p-99 response time in the streaming mode), and (ii) the shared resource interference among collocated applications in a multi-tenancy modern Spark platform. The proposed solution leverages a set of controlling mechanisms for dynamic partitioning of the allocation of computing resources, in a way that it can fulfill the QoS re-quirements of latency-critical data processing applications, while enhancing the throughput for all working nodes without reaching their saturation points. Through extensive experiments in our in-house Spark cluster, we compared the achieved performance of proposed solution against the default Spark resource allocation policy for a variety of Machine Learning (ML), Artificial Intelligence (AI), and Deep Learning (DL) applications. Experimental results show the effectiveness of the proposed solution by reducing the p-99 latency of high priority applications by 32 % during the burst traffic periods (for both batch and stream modes), while it can enhance the QoS satisfaction level by 65 % for applications with the highest priority (compared with the results of default Spark resource allocation strategy).

Original languageEnglish
Title of host publication2021 IEEE 20th International Symposium on Network Computing and Applications (NCA): Proceedings
EditorsMauro Andreolini, Mirco Marchetti, Dimiter R. Avresky
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781665495509
ISBN (Print)9781665495516
DOIs
Publication statusPublished - 31 Jan 2022
Externally publishedYes
Event20th IEEE International Symposium on Network Computing and Applications, NCA 2021 - Boston, United States
Duration: 23 Nov 2021 → 26 Nov 2021

Publication series

NameIEEE International Symposium on Network Computing and Applications (NCA): Proceedings
ISSN (Print)2643-7910
ISSN (Electronic)2643-7929

Conference

Conference20th IEEE International Symposium on Network Computing and Applications, NCA 2021
Country/TerritoryUnited States
CityBoston
Period23/11/2021 → 26/11/2021

Bibliographical note

Publisher Copyright:
© 2021 IEEE.

Keywords

  • Dynamic resource allocation
  • Iterative Computation
  • Quality of Service (QoS)
  • Streaming Big Data Processing

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Information Systems and Management
  • Safety, Risk, Reliability and Quality

Fingerprint

Dive into the research topics of 'QSpark: distributed execution of batch streaming analytics in spark platform'. Together they form a unique fingerprint.

Cite this