Metamorphic Testing for Investigation of Context Recognition from Smart Home Voice Commands

Abstract

The integration of voice control into the Internet of Things (IoT) ecosystem marks a significant advancement, enabling automation through smart assistants such as Amazon Alexa, Google Assistant, and Apple Siri. These assistants rely on neural networks and machine learning models to interpret and execute commands, necessitating high robustness and reliability to avoid errors. However, their unclear internal decision-making mechanism poses challenges in distinguishing between the correct and incorrect behavior of the model, known as the 'oracle problem', complicating testing and verification. Metamorphic testing has emerged as a solution, effectively evaluating and refining machine learning models. In this paper, we present a metamorphic testing approach consisting of 10 metamorphic relations to verify and validate a novel context recognition model developed for voice commands and day-to-day conversations with smart assistants. The proposed approach involves a systematic design and application of metamorphic relations, generating test cases based on those relations, and assessing the model's ability to withstand the variations in learning parameters and input data. Experimental results demonstrate that metamorphic testing effectively identifies flaws in the context recognition model, providing a pathway to enhance its performance and reliability.

Department(s)

Computer Science

Document Type

Conference Proceeding

DOI

10.1109/CIoT63799.2024.10757078

Keywords

Metamorphic relation, neural network, oracle problem, validation, verification

Publication Date

1-1-2024

Journal Title

7th Conference on Cloud and Internet of Things Ciot 2024

Link to Full Text

Share

COinS