Securing AI-based Security Systems

Together with the innovations in systems and services based on advances in artificial intelligence (AI), the vulnerabilities associated with its increasing use in a broad range of areas have been reported.Of these vulnerabilities, embedded bias or algorithmic discrimination is well recognised, such as racial and gender biases in algorithmic tools used for recruitment decision-making, criminal risk assessment, healthcare resource allocation, etc. To address the issue of embedded bias,steps have been proposed such as identifying the algorithms used,understanding the target of the solution (e.g. considering the diversity and representativeness of end users and/or subjects in the data),assessing performance toward that goal (e.g. testing for specific target groups or cases of problematic use), retraining based on the performance assessment and introducing oversight bodies.Beyond bias, AI systems are also recognised to suffer from brittleness(the inability to generalise or adapt to conditions outside a narrow set of assumptions), catastrophic forgetting (when a model has to process new data and can no longer classify the old data), and lack of explainability (the absence of details and reasons given by a model to make its functioning clear or easy to understand). This GCSP Strategic Security Analysis paper addresses the question of AI robustness when AI techniques and models are adopted in security systems. Robustness refers to the reliable operation of a system across a range of conditions (including attacks). Firstly, the distinction between AI and machine learning (ML) is highlighted, with reference to the Artificial Intelligence and UK National Security report.5 Whereas “general AI” refers to machine intelligence with the agency, reasoning and adaptability of a human brain, “narrow AI” refers to machine intelligence trained to perform narrowly defined cognitive tasks, such as playing chess, driving a car or translating documents. This paper addresses “narrow AI”, for which the terms AI and ML are used interchangeably.