A Scalable and Composable Map-Reduce System

Mahwish Arif; Hans Vandierendonck; Dimitrios S. Nikolopoulos; Bronis R. de Supinski

doi:10.1109/BigData.2016.7840854

A Scalable and Composable Map-Reduce System

Mahwish Arif, Hans Vandierendonck, Dimitrios S. Nikolopoulos, Bronis R. de Supinski

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Citations (Scopus)

680 Downloads (Pure)

Abstract

This paper presents a novel map-reduce runtime system that is designed for scalability and for composition with other parallel software. We use a modified programming interface that expresses reduction operations over data containers as opposed to key-value pairs. This design choice admits higher efficiency as the programmer can select appropriate data structures. Our runtime targets shared memory systems, which are increasingly capable of performing data analytics on terabyte-sized data sets stored in-memory. Our map-reduce runtime is built over the Cilk programming language and outperforms Phoenix++, by 1.5x–4x for 5 out of 7 map-reduce benchmarks on 48 threads. These results arise from a combination of factors: (i) the reduction of framework overheads, including the elimination of repeated (de-)serialization of key-value pairs; (ii) the use of more appropriate intermediate data structures that reductions over containers support.

Original language	English
Title of host publication	Proceedings of 2016 IEEE International Conference on Big Data (Big Data)
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	2233-2242
Number of pages	10
DOIs	https://doi.org/10.1109/BigData.2016.7840854
Publication status	Published - 06 Feb 2017
Event	3rd Workshop on Advances in Software and Hardware for Big Data to Knowledge Discovery (ASH) - Washington, United States Duration: 05 Dec 2016 → 08 Dec 2016 http://www.cecsresearch.org/ASH/

Conference

Conference	3rd Workshop on Advances in Software and Hardware for Big Data to Knowledge Discovery (ASH)
Country/Territory	United States
City	Washington
Period	05/12/2016 → 08/12/2016
Internet address	http://www.cecsresearch.org/ASH/

Access to Document

10.1109/BigData.2016.7840854Licence: Unspecified

A Scalable and Composable Map-Reduce System
© 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Accepted author manuscript, 855 KBLicence: Unspecified

R1451CSC: Hybrid Static/Dynamic Scheduling for Task Dataflow Parallel Programs
Vandierendonck, H. (PI)
28/07/2014 → 02/03/2017
Project: Research
R6438CSC: An Adaptive, highly Scalable Analytics Platform
Vandierendonck, H. (PI), Nikolopoulos, D. (CoI) & Robinson, P. (CoI)
21/03/2014 → 28/02/2017
Project: Research

Cite this

@inproceedings{8923549cac474f1187d7fe8c39e6e99c,

title = "A Scalable and Composable Map-Reduce System",

abstract = "This paper presents a novel map-reduce runtime system that is designed for scalability and for composition with other parallel software. We use a modified programming interface that expresses reduction operations over data containers as opposed to key-value pairs. This design choice admits higher efficiency as the programmer can select appropriate data structures. Our runtime targets shared memory systems, which are increasingly capable of performing data analytics on terabyte-sized data sets stored in-memory. Our map-reduce runtime is built over the Cilk programming language and outperforms Phoenix++, by 1.5x–4x for 5 out of 7 map-reduce benchmarks on 48 threads. These results arise from a combination of factors: (i) the reduction of framework overheads, including the elimination of repeated (de-)serialization of key-value pairs; (ii) the use of more appropriate intermediate data structures that reductions over containers support.",

author = "Mahwish Arif and Hans Vandierendonck and Nikolopoulos, {Dimitrios S.} and {de Supinski}, {Bronis R.}",

year = "2017",

month = feb,

day = "6",

doi = "10.1109/BigData.2016.7840854",

language = "English",

pages = "2233--2242",

booktitle = "Proceedings of 2016 IEEE International Conference on Big Data (Big Data)",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

address = "United States",

note = "3rd Workshop on Advances in Software and Hardware for Big Data to Knowledge Discovery (ASH) ; Conference date: 05-12-2016 Through 08-12-2016",

url = "http://www.cecsresearch.org/ASH/",

}

Arif, M, Vandierendonck, H, Nikolopoulos, DS & de Supinski, BR 2017, A Scalable and Composable Map-Reduce System. in Proceedings of 2016 IEEE International Conference on Big Data (Big Data). Institute of Electrical and Electronics Engineers Inc., pp. 2233-2242, 3rd Workshop on Advances in Software and Hardware for Big Data to Knowledge Discovery (ASH), Washington, United States, 05/12/2016. https://doi.org/10.1109/BigData.2016.7840854

TY - GEN

T1 - A Scalable and Composable Map-Reduce System

AU - Arif, Mahwish

AU - Vandierendonck, Hans

AU - Nikolopoulos, Dimitrios S.

AU - de Supinski, Bronis R.

PY - 2017/2/6

Y1 - 2017/2/6

N2 - This paper presents a novel map-reduce runtime system that is designed for scalability and for composition with other parallel software. We use a modified programming interface that expresses reduction operations over data containers as opposed to key-value pairs. This design choice admits higher efficiency as the programmer can select appropriate data structures. Our runtime targets shared memory systems, which are increasingly capable of performing data analytics on terabyte-sized data sets stored in-memory. Our map-reduce runtime is built over the Cilk programming language and outperforms Phoenix++, by 1.5x–4x for 5 out of 7 map-reduce benchmarks on 48 threads. These results arise from a combination of factors: (i) the reduction of framework overheads, including the elimination of repeated (de-)serialization of key-value pairs; (ii) the use of more appropriate intermediate data structures that reductions over containers support.

AB - This paper presents a novel map-reduce runtime system that is designed for scalability and for composition with other parallel software. We use a modified programming interface that expresses reduction operations over data containers as opposed to key-value pairs. This design choice admits higher efficiency as the programmer can select appropriate data structures. Our runtime targets shared memory systems, which are increasingly capable of performing data analytics on terabyte-sized data sets stored in-memory. Our map-reduce runtime is built over the Cilk programming language and outperforms Phoenix++, by 1.5x–4x for 5 out of 7 map-reduce benchmarks on 48 threads. These results arise from a combination of factors: (i) the reduction of framework overheads, including the elimination of repeated (de-)serialization of key-value pairs; (ii) the use of more appropriate intermediate data structures that reductions over containers support.

U2 - 10.1109/BigData.2016.7840854

DO - 10.1109/BigData.2016.7840854

M3 - Conference contribution

SP - 2233

EP - 2242

BT - Proceedings of 2016 IEEE International Conference on Big Data (Big Data)

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 3rd Workshop on Advances in Software and Hardware for Big Data to Knowledge Discovery (ASH)

Y2 - 5 December 2016 through 8 December 2016

ER -

A Scalable and Composable Map-Reduce System

Abstract

Conference

Access to Document

Fingerprint

Projects

R1451CSC: Hybrid Static/Dynamic Scheduling for Task Dataflow Parallel Programs

R6438CSC: An Adaptive, highly Scalable Analytics Platform

Cite this