Objektu atpazīšanas modeļu lietošana mājaslapu kvalitātes kontrolei un kļūdu identificēšanai

Abstract

Mākslīgā intelekta un datorredzes tehnoloģiju integrācija strauji attīstās visdažādākajās nozarēs, sākot ar veselības aprūpi, kur tiek identificēti rentgena attēlu raksturlielumi, līdz pat autotransporta drošības sistēmām, kur tiek uzlabota satiksmes plūsmu analīze. Neskatoties uz to, to ieviešana programmatūras testēšanas instrumentos un uzņēmumu izstrādātajās lietojumprogrammās vēl aizvien ir ierobežota. Šveices telekomunikāciju uzņēmums Swisscom ir viens no datu avotiem šim pētījumam. Sarunas ar uzņēmuma darbiniekiem atklāj, ka bieži kvalitātes kontroles darbi, kurus varētu veikt automātiski, joprojām tiek izpildīti manuāli. Izteiksmīgs piemērs ir lietotāja saskarnes prasību pārbaude, kur nepilnības aplikācijās vai tīmekļa vietnēs var novest pie kvalitātes pasliktināšanās. Lai uzturētu aplikācijas kvalitāti ir svarīgi šādas problēmas atpazīt agri un tās izlabot. Šis pētijums pārbauda ‘You Only Look Once’ v7 (YOLOv7) objektu atpazīšanas modeļa lietojamību, lai automatizētu trīs visbiežāk sastopamo izkārtojuma kļūdu identifikāciju, ko nevar veikt ar tradicionāliem testēšanas ietvariem. Apmācītie modeļi sasniedza zemāku vidējo precizitāti nekā gaidīts - (mAP@0.5) 0.19 svarīgākajai kļūdu kategorijai. Rezultāti joprojām uzrāda ievērojamu potenciālu šīs tehnoloģijas izmantošanai programmatūras testēšanā un kvalitātes kontrolē. Tika secināts, ka YOLOv7 un līdzīgus datorredzes modeļus var pielietot automatizētai kļūdu atpazīšanai, bet ar noteiktiem precizitātes ierobežojumiem. Ieteikums turpmākiem pētījumiem ir veikt padziļinātāku analīzi, izmantojot plašāku un kvalitatīvāku datu kopu, vai mainīt piemēroto datorredzes metodi, lai iegūtu padziļinātāku izpratni par efektīvākajiem kļūdu atpazīšanas veidiem. Atslēgvārdi: datorredze, kvalitāte, pielietojums, kļūdu, precizitāte

In the past 20 years software test automation has increased significantly with frontend testing frameworks like Selenium or Cypress. However, these frameworks cannot detect all problems in a highly complex application. A small study in Swisscom, a telecommunications company providing a part of the dataset for this study, indicated that many quality control tests are still done manually. This kind of process can be slow and cause quality issues in the application as often some time passes before developers receive news of the presence of the issue. The main objective is to test whether object detection models like You Only Look Once v7 (YOLOv7) can automate the detection of content or alignment issues in the application that cannot be detected with traditional testing frameworks. Currently there are not many studies done researching the application of computer vision in UI testing, thus, this study also aims to fill a research gap. Three different datasets were created each representing a category of commonly found layout/content issues in web applications - platform indicated, symmetry/alignment issues and general chaotic layouts. Three different YOLOv7 model variations were trained and tested on each of these datasets. The accuracy data was gathered to compare the performance and accuracy of the model variations as well as to analyze the feasibility of this kind of application. The results showed that the models were less accurate than expected. Detection of symmetry/alignment issues was most reliable, with a consistent accuracy and a mean Average Precision (mAP@0.5) of 0.19. The platform-specific dataset had a high, but variable accuracy, suggesting potential dataset issues. Testing on chaotic layouts performed poorly, indicating problems with object definition and category classification. The narrow margin in rejecting the alternative hypothesis for symmetry/alignment suggests that slight dataset improvements could enhance model accuracy proving that the use case requires more research. The study indicated that computer vision tools can be used in software testing even with small datasets. The trained models and datasets can be used for further research or to further testing in real time applications. The automated detection of general issues offers many potential benefits, for example, the integration with CI/CD pipelines among others. Further integrations with large language models may allow for the automated patching of such issues, not just detection. The major limitations of the study are associated with the datasets created. They were comparatively small and had arguable issues with the labelling and examples given as shown by the highly variable results for 2 of the datasets used. It could also be argued the datasets did not fully represent the environment of detection. These problems are mainly associated with the lack of an existing dataset due to a research gap and the limited amount of time to develop one. The results showed limited success in detection, suggesting further exploration of computer vision in software testing, like instance segmentation, is promising. Moreover, defining an error category was one of the most difficult tasks in the study, thus, more research on labelling methods and categorization of UI issues would also help further the study. Finally, a larger and more refined dataset would help accelerate the development of solutions for all researchers and developers. Keywords: dataset, model, computer vision, user interface (UI), training, accuracy