Validation of data for use in civil infrastructure big data applications

Connor O'Higgins*, Connor Kent, David Hester, Su Taylor

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Downloads (Pure)

Abstract

Big data applications are becoming more popular across many different fields and civil engineering is no different. The benefits of big data lie in its potential to provide valuable insights into various large datasets. Big data applications can identify patterns and trends that were previously unknown, which can help them make informed decisions and develop effective strategies. In the case of civil engineering, this could be taking large datasets that have been produced in relation to pieces of infrastructure and using them to create more efficient management strategies. One of the issues with using big data is that if the input dataset is flawed then the output resulting from any big data analysis will be compromised. Therefore, data validation is used, which is the process of ensuring that data is accurate, complete, and consistent. The consequence of not undertaking data validation may be inaccurate or inconsistent data which can lead to incorrect insights and decisions in big data applications. This paper explores the necessity of validating data before it is used in a big data application. It outlines some of the different methods used for validating data and provides an overview of the potential issues that may arise from data validation errors. A case study is then presented showing the process of validation on data collected from four bridges and provides recommendations for implementing data validation as part of a larger big data workflow. The results of the case study show that validation of data is an important step in the big data process both for confidence in the outputs and to make big data applications more useful and more common in the civil engineering field. The discussion and presented case study in this paper highlight the necessity of validating data. It has shown some of the potential issues that may arise from not undertaking data validation.

Original languageEnglish
Title of host publicationProceedings of the 14th International Workshop on Structural Health Monitoring 2023: designing SHM for sustainability, maintainability, and reliability
EditorsSaman Farhangdoust, Alfredo Guemes, Fu-Kuo Chang
PublisherDEStech Publications Inc.
Number of pages9
ISBN (Electronic)9781605956930
DOIs
Publication statusPublished - 14 Sept 2023
Event14th International Workshop on Structural Health Monitoring 2023 - Stanford, United States
Duration: 12 Sept 202314 Sept 2023

Conference

Conference14th International Workshop on Structural Health Monitoring 2023
Abbreviated titleIWSHM 2023
Country/TerritoryUnited States
CityStanford
Period12/09/202314/09/2023

Fingerprint

Dive into the research topics of 'Validation of data for use in civil infrastructure big data applications'. Together they form a unique fingerprint.

Cite this