HPLT's second data release

Nikolay Arefyev; Mikko Aulamo; Marta Bañón; Laurie Burchell; Pinzhen Chen; Mariia Fedorova; Ona de Gibert; Liane Guillou; Barry Haddow; Jan Hajič; Jindřich Helcl; Erik Henriksson; Andrey Kutuzov; Veronika Laippala; Bhavitvya Malik; Farrokh Mehryary; Vladislav Mikhailov; Amanda Myntti; Dayyán O'Brien; Stephan Oepen; Sampo Pyysalo; Gema Ramírez-Sánchez; David Samuel; Pavel Stepachev; Jörg Tiedemann; Dušan Variš; Jaume Zaragoza-Bernabeu

HPLT's second data release

Nikolay Arefyev
, Mikko Aulamo
, Marta Bañón
, Laurie Burchell
, Pinzhen Chen
, Mariia Fedorova
, Ona de Gibert
, Liane Guillou
, Barry Haddow
, Jan Hajič
, Jindřich Helcl
, Erik Henriksson
, Andrey Kutuzov
, Veronika Laippala
, Bhavitvya Malik
, Farrokh Mehryary
, Vladislav Mikhailov
, Amanda Myntti
, Dayyán O'Brien
, Stephan Oepen

Sampo Pyysalo, Gema Ramírez-Sánchez, David Samuel, Pavel Stepachev, Jörg Tiedemann, Dušan Variš, Jaume Zaragoza-Bernabeu

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Downloads (Pure)

Abstract

We describe the progress of the High Performance Language Technologies (HPLT) project, a 3-year EU-funded project that started in September 2022. We focus on the up-to-date results on the release of free text datasets derived from web crawls, one of the central objectives of the project. The second release used a revised processing pipeline, and an enlarged set of input crawls. From 4.5 petabytes of web crawls we extracted 7.6T tokens of monolingual text in 193 languages, plus 380 million parallel sentences in 51 language pairs. We also release MultiHPLT, a cross-combination of the parallel data, which produces 1,275 pairs, as well as releasing the containing documents for all parallel sentences in order to enable research in document-level MT. We report changes in the pipeline, analysis and evaluation results for the second parallel data release based on machine translation systems. All datasets are released under a permissive CC0 licence.

Original language	English
Title of host publication	Proceedings of Machine Translation Summit XX: Volume 2
Editors	Pierrette Bouillon, Johanna Gerlach, Sabrina Girletti, Lise Volkart, Raphael Rubino, Rico Sennrich, Samuel Läubli, Martin Volk, Miquel Esplà-Gomis, Vincent Vandeghinste, Helena Moniz, Sara Szoc
Place of Publication	Geneva, Switzerland
Publisher	European Association for Machine Translation (EAMT)
Pages	101-102
Number of pages	2
ISBN (Print)	9782970189718
Publication status	Published - 01 Jun 2025
Externally published	Yes
Event	Machine Translation Summit XX - Geneva, Switzerland Duration: 23 Jun 2025 → 27 Jun 2025

Conference

Conference	Machine Translation Summit XX
Country/Territory	Switzerland
City	Geneva
Period	23/06/2025 → 27/06/2025

Access to Document

HPLT's second data release
Copyright 2025 The Authors. This is an open access article published under a Creative Commons Attribution-NonCommercial-NoDerivs License (https://creativecommons.org/licenses/by-nc-nd/4.0/), which permits distribution and reproduction for non-commercial purposes, provided the author and source are cited.
Final published version, 86.3 KBLicence: CC BY-NC-ND

https://aclanthology.org/2025.mtsummit-2.21/Licence: CC BY-NC-ND

Cite this

Arefyev, N., Aulamo, M., Bañón, M., Burchell, L., Chen, P., Fedorova, M., de Gibert, O., Guillou, L., Haddow, B., Hajič, J., Helcl, J., Henriksson, E., Kutuzov, A., Laippala, V., Malik, B., Mehryary, F., Mikhailov, V., Myntti, A., O'Brien, D., ... Zaragoza-Bernabeu, J. (2025). HPLT's second data release. In P. Bouillon, J. Gerlach, S. Girletti, L. Volkart, R. Rubino, R. Sennrich, S. Läubli, M. Volk, M. Esplà-Gomis, V. Vandeghinste, H. Moniz, & S. Szoc (Eds.), Proceedings of Machine Translation Summit XX: Volume 2 (pp. 101-102). European Association for Machine Translation (EAMT). https://aclanthology.org/2025.mtsummit-2.21/

Arefyev, Nikolay ; Aulamo, Mikko ; Bañón, Marta et al. / HPLT's second data release. Proceedings of Machine Translation Summit XX: Volume 2. editor / Pierrette Bouillon ; Johanna Gerlach ; Sabrina Girletti ; Lise Volkart ; Raphael Rubino ; Rico Sennrich ; Samuel Läubli ; Martin Volk ; Miquel Esplà-Gomis ; Vincent Vandeghinste ; Helena Moniz ; Sara Szoc. Geneva, Switzerland : European Association for Machine Translation (EAMT), 2025. pp. 101-102

@inproceedings{053c245bf312441b9b840f908e5346e1,

title = "HPLT's second data release",

abstract = "We describe the progress of the High Performance Language Technologies (HPLT) project, a 3-year EU-funded project that started in September 2022. We focus on the up-to-date results on the release of free text datasets derived from web crawls, one of the central objectives of the project. The second release used a revised processing pipeline, and an enlarged set of input crawls. From 4.5 petabytes of web crawls we extracted 7.6T tokens of monolingual text in 193 languages, plus 380 million parallel sentences in 51 language pairs. We also release MultiHPLT, a cross-combination of the parallel data, which produces 1,275 pairs, as well as releasing the containing documents for all parallel sentences in order to enable research in document-level MT. We report changes in the pipeline, analysis and evaluation results for the second parallel data release based on machine translation systems. All datasets are released under a permissive CC0 licence. ",

author = "Nikolay Arefyev and Mikko Aulamo and Marta Ba{\~n}{\'o}n and Laurie Burchell and Pinzhen Chen and Mariia Fedorova and \{de Gibert\}, Ona and Liane Guillou and Barry Haddow and Jan Haji{\v c} and Jind{\v r}ich Helcl and Erik Henriksson and Andrey Kutuzov and Veronika Laippala and Bhavitvya Malik and Farrokh Mehryary and Vladislav Mikhailov and Amanda Myntti and Dayy{\'a}n O'Brien and Stephan Oepen and Sampo Pyysalo and Gema Ram{\'i}rez-S{\'a}nchez and David Samuel and Pavel Stepachev and J{\"o}rg Tiedemann and Du{\v s}an Vari{\v s} and Jaume Zaragoza-Bernabeu",

year = "2025",

month = jun,

day = "1",

language = "English",

isbn = "9782970189718",

pages = "101--102",

editor = "Pierrette Bouillon and Johanna Gerlach and Sabrina Girletti and Lise Volkart and Raphael Rubino and Rico Sennrich and Samuel L{\"a}ubli and Martin Volk and Miquel Espl{\`a}-Gomis and Vincent Vandeghinste and Helena Moniz and Sara Szoc",

booktitle = "Proceedings of Machine Translation Summit XX: Volume 2",

publisher = "European Association for Machine Translation (EAMT)",

note = "Machine Translation Summit XX ; Conference date: 23-06-2025 Through 27-06-2025",

}

Arefyev, N, Aulamo, M, Bañón, M, Burchell, L, Chen, P, Fedorova, M, de Gibert, O, Guillou, L, Haddow, B, Hajič, J, Helcl, J, Henriksson, E, Kutuzov, A, Laippala, V, Malik, B, Mehryary, F, Mikhailov, V, Myntti, A, O'Brien, D, Oepen, S, Pyysalo, S, Ramírez-Sánchez, G, Samuel, D, Stepachev, P, Tiedemann, J, Variš, D & Zaragoza-Bernabeu, J 2025, HPLT's second data release. in P Bouillon, J Gerlach, S Girletti, L Volkart, R Rubino, R Sennrich, S Läubli, M Volk, M Esplà-Gomis, V Vandeghinste, H Moniz & S Szoc (eds), Proceedings of Machine Translation Summit XX: Volume 2. European Association for Machine Translation (EAMT), Geneva, Switzerland, pp. 101-102, Machine Translation Summit XX, Geneva, Switzerland, 23/06/2025. <https://aclanthology.org/2025.mtsummit-2.21/>

HPLT's second data release. / Arefyev, Nikolay; Aulamo, Mikko; Bañón, Marta et al.
Proceedings of Machine Translation Summit XX: Volume 2. ed. / Pierrette Bouillon; Johanna Gerlach; Sabrina Girletti; Lise Volkart; Raphael Rubino; Rico Sennrich; Samuel Läubli; Martin Volk; Miquel Esplà-Gomis; Vincent Vandeghinste; Helena Moniz; Sara Szoc. Geneva, Switzerland: European Association for Machine Translation (EAMT), 2025. p. 101-102.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - HPLT's second data release

AU - Arefyev, Nikolay

AU - Aulamo, Mikko

AU - Bañón, Marta

AU - Burchell, Laurie

AU - Chen, Pinzhen

AU - Fedorova, Mariia

AU - de Gibert, Ona

AU - Guillou, Liane

AU - Haddow, Barry

AU - Hajič, Jan

AU - Helcl, Jindřich

AU - Henriksson, Erik

AU - Kutuzov, Andrey

AU - Laippala, Veronika

AU - Malik, Bhavitvya

AU - Mehryary, Farrokh

AU - Mikhailov, Vladislav

AU - Myntti, Amanda

AU - O'Brien, Dayyán

AU - Oepen, Stephan

AU - Pyysalo, Sampo

AU - Ramírez-Sánchez, Gema

AU - Samuel, David

AU - Stepachev, Pavel

AU - Tiedemann, Jörg

AU - Variš, Dušan

AU - Zaragoza-Bernabeu, Jaume

PY - 2025/6/1

Y1 - 2025/6/1

N2 - We describe the progress of the High Performance Language Technologies (HPLT) project, a 3-year EU-funded project that started in September 2022. We focus on the up-to-date results on the release of free text datasets derived from web crawls, one of the central objectives of the project. The second release used a revised processing pipeline, and an enlarged set of input crawls. From 4.5 petabytes of web crawls we extracted 7.6T tokens of monolingual text in 193 languages, plus 380 million parallel sentences in 51 language pairs. We also release MultiHPLT, a cross-combination of the parallel data, which produces 1,275 pairs, as well as releasing the containing documents for all parallel sentences in order to enable research in document-level MT. We report changes in the pipeline, analysis and evaluation results for the second parallel data release based on machine translation systems. All datasets are released under a permissive CC0 licence.

AB - We describe the progress of the High Performance Language Technologies (HPLT) project, a 3-year EU-funded project that started in September 2022. We focus on the up-to-date results on the release of free text datasets derived from web crawls, one of the central objectives of the project. The second release used a revised processing pipeline, and an enlarged set of input crawls. From 4.5 petabytes of web crawls we extracted 7.6T tokens of monolingual text in 193 languages, plus 380 million parallel sentences in 51 language pairs. We also release MultiHPLT, a cross-combination of the parallel data, which produces 1,275 pairs, as well as releasing the containing documents for all parallel sentences in order to enable research in document-level MT. We report changes in the pipeline, analysis and evaluation results for the second parallel data release based on machine translation systems. All datasets are released under a permissive CC0 licence.

M3 - Conference contribution

SN - 9782970189718

SP - 101

EP - 102

BT - Proceedings of Machine Translation Summit XX: Volume 2

A2 - Bouillon, Pierrette

A2 - Gerlach, Johanna

A2 - Girletti, Sabrina

A2 - Volkart, Lise

A2 - Rubino, Raphael

A2 - Sennrich, Rico

A2 - Läubli, Samuel

A2 - Volk, Martin

A2 - Esplà-Gomis, Miquel

A2 - Vandeghinste, Vincent

A2 - Moniz, Helena

A2 - Szoc, Sara

PB - European Association for Machine Translation (EAMT)

CY - Geneva, Switzerland

T2 - Machine Translation Summit XX

Y2 - 23 June 2025 through 27 June 2025

ER -

Arefyev N, Aulamo M, Bañón M, Burchell L, Chen P, Fedorova M et al. HPLT's second data release. In Bouillon P, Gerlach J, Girletti S, Volkart L, Rubino R, Sennrich R, Läubli S, Volk M, Esplà-Gomis M, Vandeghinste V, Moniz H, Szoc S, editors, Proceedings of Machine Translation Summit XX: Volume 2. Geneva, Switzerland: European Association for Machine Translation (EAMT). 2025. p. 101-102

HPLT's second data release

Abstract

Conference

Access to Document

Fingerprint

Cite this