Detecting Deceptive Design Patterns in Mobile Apps
Detecting Deceptive Design Patterns in Mobile Apps
Noam Sheena, M.Sc. student at School of Industrial & Intelligent Systems Engineering
Advisor: Prof. Eran Toch
Abstract:
We all encounter deceptive design patterns daily. These are user interfaces that aim to manipulate users into taking actions they would not otherwise choose, such as a pop-up banner that highlights the option that benefits the service provider. These practices have prompted increased scrutiny from users and regulators; however, automatically identifying and analyzing deceptive patterns at scale presents a significant obstacle to mitigating their harm systematically. The constantly evolving taxonomy of deceptive patterns makes it challenging to keep up and develop new detection methods. Our method leverages visual language models to analyze application screenshots and extract the visual features of UI elements. This allows us to build a foundational set of visual features capable of adapting to and identifying emerging deceptive patterns, facilitating scalable automated detection. We present an evaluation based on a dataset of 104 Android applications, comprising 2,484 annotated screenshots, compared with manual identification by crowd workers (n = 500). We demonstrate that visual language models can assist users in identifying specific deceptive patterns and achieve accuracy comparable to that of human annotators when following the same labeling instructions. Using the same dataset, we incorporate additional structural information and apply a machine learning method for automatic detection. Our model achieves high performance in identifying deceptive design patterns (F1 = 0.933). These results are competitive with state-of-the-art approaches and highlight the effectiveness of combining visual and structural UI features for scalable detection of deceptive patterns in mobile applications.
Bio:
Noam Sheena is an M.Sc. student at the School of Industrial Engineering & Intelligent Systems Engineering, where he researches deceptive design patterns at the IWIT Lab under the supervision of Prof. Eran Toch. He holds a B.Sc. degree in Industrial Engineering and Management from Tel Aviv University.
