GUI Element Identification with Semantic Mapping
Logacheva, Evanfiya (2023)
Logacheva, Evanfiya
2023
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi-fe2023050240408
https://urn.fi/URN:NBN:fi-fe2023050240408
Tiivistelmä
User Interface test automation faces significant obstacles due to test failures connected to application changes. Additionally, current User Interface testing methods are not context aware and usage-based, which makes exploring web application functionality challenging. Robots used for crawling web application interfaces are slow and do not reflect human interaction with them. Semantic mapping
(semantic matching) has been proposed as a method for reusing existing tests between web applications in the same domain to mitigate issues with testing speed and context awareness. This thesis explores semantic mapping for robust User Interface element identification that could alleviate the issue with test failures upon application changes.
Semantic mapping uses textual cues of User Interface elements neighboring testable features to identify similar features in other applications of the same domain. This work argues that the same technique can be applied to various versions of the same web application. Existing tools leverage text attributes of features' neighbors based on the hierarchy and position of an element, while this study applies semi-supervised learning methods to extract relevant text from elements surrounding features. It uses state-of-the-art pre-trained language models for embedding textual cues. To find similar features, it uses cosine similarity between sentences as a measure of semantic similarity.
This implementation of semantic matching has demonstrated promising results for User Interface element identification between two versions of the same web application.
(semantic matching) has been proposed as a method for reusing existing tests between web applications in the same domain to mitigate issues with testing speed and context awareness. This thesis explores semantic mapping for robust User Interface element identification that could alleviate the issue with test failures upon application changes.
Semantic mapping uses textual cues of User Interface elements neighboring testable features to identify similar features in other applications of the same domain. This work argues that the same technique can be applied to various versions of the same web application. Existing tools leverage text attributes of features' neighbors based on the hierarchy and position of an element, while this study applies semi-supervised learning methods to extract relevant text from elements surrounding features. It uses state-of-the-art pre-trained language models for embedding textual cues. To find similar features, it uses cosine similarity between sentences as a measure of semantic similarity.
This implementation of semantic matching has demonstrated promising results for User Interface element identification between two versions of the same web application.