# Never-Ending Learning of User Interfaces

**URL:** <https://forum.devtalk.com/t/never-ending-learning-of-user-interfaces/122058>\
**Category:** In The News\
**Tags:** learning, in-the-news\
**Created:** [27 August 2023 01:00 UTC](https://forum.devtalk.com/t/never-ending-learning-of-user-interfaces/122058 "2023-08-27T01:00:38Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![CommunityNews](https://forum.devtalk.com/user_avatar/forum.devtalk.com/communitynews/32/1113_2.png) [@CommunityNews](https://forum.devtalk.com/u/CommunityNews)\
**Post date:** [27 August 2023 01:00 UTC](https://forum.devtalk.com/t/never-ending-learning-of-user-interfaces/122058/1 "2023-08-27T01:00:38Z")

</div>

> Never-ending Learning of User Interfaces.  
> Machine learning models have been trained to predict semantic information  
> about user interfaces (UIs) to make apps more accessible, easier to test, and  
> to automate. Currently, most models rely on datasets that are collected and  
> labeled by human crowd-workers, a process that is costly and surprisingly  
> error-prone for certain tasks. For example, it is possible to guess if a UI  
> element is “tappable” from a screenshot (i.e., based on visual signifiers) or  
> from potentially unreliable metadata (e.g., a view hierarchy), but one way to  
> know for certain is to programmatically tap the UI element and observe the  
> effects. We built the Never-ending UI Learner, an app crawler that  
> automatically installs real apps from a mobile app store and crawls them to  
> discover new and challenging training examples to learn from. The Never-ending  
> UI Learner has crawled for more than 5,000 device-hours, performing over half a  
> million actions on 6,000 apps to train three computer vision models for i)  
> tappability prediction, ii) draggability prediction, and iii) screen  
> similarity.

Read in full here:

> **[Never-ending Learning of User Interfaces](https://arxiv.org/abs/2308.08726)**
>
> Machine learning models have been trained to predict semantic information about user interfaces (UIs) to make apps more accessible, easier to test, and to automate. Currently, most models rely on datasets that are collected and labeled by human...

_This thread was posted by one of our members via one of our news source trackers._