Understanding a Learning Ecosystem: Topic Models of App Description and Reviews

Understanding a Learning Ecosystem: Topic Models of App Store Descriptions and User Reviews

Abstract: Educational apps provide a vast opportunity for individuals to learn a variety of topics. By some estimates, there are more than several hundred thousand education applications on the iOS and Android platforms. These applications allow individuals to learn in a portable and personalizable format. However, the market for these apps is treacherous for consumers. Individuals have relatively little information about an app until they have purchased it (Hirsh-Pasek et al., 2015). This information asymmetry is costly and time consuming for the users. Moreover, many of these apps make unsubstantiated and undocumented claims with regards to their efficacy and pedagogical value (Hulleman, Burke, May, Daniel, & Charania, 2017).

One area where users are provided information with regards to an app’s quality and efficacy is user reviews. While not a fully representative sample of the population, these reviews are often the most easily accessible source of information for users. Moreover, app developers focus intently on responding to comments and critiques that occur within this space, suggesting that content of reviews influence consumer and producer action (Liang, Li, Yang, & Wang, 2015)(Licorish, Tony, Savarimuthu, & Keertipati, 2017).

To better understand the nature of the educational application ecosystem, we randomly sampled 48,000 apps from the Amazon app store and the Itunes app store. Specifically, we subsampled from apps that have the category tag of education. To date, there has been relatively little study of this particular part of the app ecosystem. This dataset will be shared with the general public upon release of the paper.

We adopted the methodology of (Fu et al., 2013) and perform topic modelling of both the developer description of the product as well as user reviews to understand the mix of apps on each of these platforms. This first set of analyses will provide an understanding of the relative distribution of the app store ecosystems (e.g. math apps, language apps, programming apps, etc.) We also perform the same analysis with user reviews to understand their composition e.g.(attractiveness, stability, accuracy, compatibility, connectivity, cost, device, picture, media, spam). The goal of these analyses is to understand the sources of complaints and praise within an app.

The second set of analysis in this paper examines how these topic models performs in predicting individual user reviews relative to a bag of words baseline and other types of machine learning models. In particular, we pay attention to two aspects of mobiles app that are of importance to children. We analyze whether or not reviews that mention advertisements have negative valence relative to apps that do not have content regarding advertisements(Meyer et al., 2019). We also evaluate the role of in-app purchases to understand whether or not these features have negative valence.

Finally, we perform an analysis where we treat every review associated with a particular app as a single document. We then use this information to predict the global features of the application (price, ratings, downloads). These analyses will help inform whether or not app reviews contain information beyond their individual rating. It will also help us understand what features usually come in positively reviewed apps as opposed to negatively reviewed apps.

The ultimate goal of these analyses is to provide an easier and more transparent means of communicating app quality as well as surfacing features that may best inform a consumer’s decisions. A secondary goal is to more accurately describe the app store marketplace such that educational technologists can more readily interpret and understand consumer feedback.

References

Fu, B., Lin, J., Li, L., Faloutsos, C., Hong, J., & Sadeh, N. (2013). Why People Hate Your App-Making Sense of User Feedback in a Mobile App Store. Retrieved from http://cmuchimps.org/uploads/publication/paper/3/why_people_hate_your_app_making_sense_of_user_feedback_in_a_mobile_app_store.pdf

Hirsh-Pasek, K., Zosh, J. M., Golinkoff, R. M., Gray, J. H., Robb, M. B., & Kaufman, J. (2015). Putting Education in “Educational” Apps: Lessons From the Science of Learning. Psychological Science in the Public Interest, Supplement. https://doi.org/10.1177/1529100615569721

Hulleman, C. S., Burke, R. A., May, M., Daniel, D. B., & Charania, M. (2017). Merit or Marketing ?: Evidence and Quality of Efficacy Research in Educational Technology Companies. Retrieved from http://symposium.curry.virginia.edu/wp-content/uploads/2017/06/WG-D-Evidence-and-Quality-of-Efficacy-Research-in-Educational-Technology-Companies_FINAL.pdf

Liang, T.-P., Li, X., Yang, C.-T., & Wang, M. (2015). What in Consumer Reviews Affects the Sales of Mobile Apps: A Multifacet Sentiment Analysis Approach. International Journal of Electronic Commerce, 20(2), 236–260. https://doi.org/10.1080/10864415.2016.1087823

Licorish, S. A., Tony, B., Savarimuthu, R., & Keertipati, S. (2017). Attributes that Predict which Features to Fix: Lessons for App Store Mining. https://doi.org/10.1145/3084226.3084246

Meyer, M., Adkins, V., Yuan, N., Weeks, H. M., Chang, Y.-J., & Radesky, J. (2019). Advertising in Young Childrenʼs Apps. Journal of Developmental & Behavioral Pediatrics, 40(1), 32–39. https://doi.org/10.1097/DBP.0000000000000622