Date of Graduation
Fall 2025
Degree
Master of Science in Computer Science
Department
Computer Science
Committee Chair
Mohammed Y. Belkhouche
Abstract
The constant evolution of malware presents a critical challenge to today's interconnected world. It poses an increasing threat on different scales, spanning from individuals, organizations to critical infrastructures such as government’s security. Hackers continuously develop new techniques to evade detection methods. When confronted with the high volume and variety of malware, conventional approaches tend to struggle to perform in robust, accurate and timely manner. This thesis explores the application of deep learning methods to improve malware detection and classification techniques. By analyzing API call sequences, the proposed approach leverages Autoencoders to compress high-dimensional malware data into more optimized representations that captures the most discriminative features. Classification models then use these encoded representations to differentiate between benign and malicious software. The suggested framework includes Variational Autoencoders (VAEs), specifically used to enhance detection accuracy, generalize to unseen malware families, and effectively manage complex, high-dimensional data. The results show that autoencoder-based dimensionality reduction enhances detection accuracy, reduces computational costs, and provides a robust latent representation that supports effective classification.
Keywords
malware detection, cybersecurity, deep learning, autoencoders, variational autoencoder (VAE), dimensionality reduction, API call analysis, classification, high-dimensional data
Subject Categories
Computer Sciences
Copyright
© Selma Bouraoui
Recommended Citation
Bouraoui, Selma, "Balancing Performance and Efficiency: An Autoencoder Approach to API-Based Malware Detection" (2025). Graduate Theses/Dissertations. 4119.
https://bearworks.missouristate.edu/theses/4119
