Journal of Obesity & Metabolic Syndrome

Search

Article

J Obes Metab Syndr 2019; 28(3): 194-202

Published online September 30, 2019 https://doi.org/10.7570/jomes.2019.28.3.194

Copyright © Korean Society for the Study of Obesity.

Development and Usability of a Life-Logging Behavior Monitoring Application for Obese Patients

Ju Yeon Lee1, Ju Young Kim2,* , Seung Ju You3, You Soo Kim3, Hye Yeon Koo4, Jeong Hyun Kim5, Sohye Kim6, Jung Ha Park2, Jong Soo Han7, Siye Kil2, Hyerim Kim2, Ye Seul Yang1, Kyung Min Lee1

1Department of Family Medicine, Seoul National University Hospital, Seoul; 2Department of Family Medicine, Seoul National University Bundang Hospital, Seongnam; 3YSBio Co., Ltd., Seoul; 4Health Promotion Center, Gachon University Gil Medical Center, Incheon; 5Department of Psychiatry, 6Nutrition Care Services, and 7Health Promotion Center, Seoul National University Bundang Hospital, Seongnam, Korea

Correspondence to:
Ju Young Kim
https://orcid.org/0000-0001-6018-3337
Department of Family Medicine, Seoul National University Bundang Hospital, 82 Gumi-ro 173beon-gil, Bundang-gu, Seongnam 13620, Korea
Tel: +82-31-787-7796 Fax: +82-31-787-4078 E-mail: kkamduri@snubh.org

Received: June 24, 2019; Reviewed : July 23, 2019; Accepted: August 30, 2019

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Background

Obesity is a global epidemic. Behavior change monitoring using a smartphone application (app) can support weight management in obese patients. These apps must undergo usability testing, which is an important step in mobile healthcare app development. The current study aimed to develop a mobile app for behavioral monitoring and to test its usability including technical effectiveness, user efficiency, and user satisfaction for obese adults.

Methods

Development of the Dr. Youth app components included information on behavioral monitoring indicators and their obesity subtypes. The usability of the app was tested with 50 obese adults in a university hospital. Participants were asked to complete eight tasks for evaluating the technical effectiveness of the app. The time to complete each task was measured to test user efficiency. To explore user satisfaction, each participant completed the System Usability Scale (SUS). Descriptive statistics were used to examine the mean user efficiency and SUS scores.

Results

Fifty adults (14 men and 36 women, aged 20–59 years) who are obese (body mass index ≥25 kg/m2) were recruited. The mean age of participants was 42.6 years (standard deviation [SD], 10.8 years); their mean body mass index was 29.6 kg/m2 (SD, 5.7 kg/m2). The tasks were completed with a 99% success rate. The overall mean SUS score was 76.65 (SD, 15.43).

Conclusion

The Dr. Youth app shows acceptable technical effectiveness, user efficiency, and user satisfaction. Future study is warranted to establish the app’s clinical efficacy.

Keywords: Health behavior, Mobile applications, Obesity, Patient satisfaction

The prevalence of obesity has increased worldwide and is now considered a global epidemic. Globally, the number of obese people is 650 million1 and the rate of increase of obesity in Korea is similar to the global increase.2 Obesity is an important risk factor for chronic diseases such as arthritis, diabetes, hypertension, dyslipidemia, cardiovascular disease, and cancer.3 The socioeconomic costs of obesity in Korean adults are substantial.4 In order to control the socioeconomic burden attributable to obesity, effective strategies for managing obesity should be established and implemented.4

Since approaches targeting behavioral changes are important in obesity management, cognitive behavioral interventions, and lifestyle improvement programs are needed to meet users’ emotional needs. Moreover, a recent study of adult weight management has suggested that mobile health interventions may be effective.5 Because of their widespread use and accessibility, smartphone applications (apps) offer a substantial opportunity to support health behavior changes and weight management. Obese patients can manage their obesity efficiently with a smartphone app by managing behaviors, setting goals, and tracking information.

Usability describes a system’s capacity to allow users to carry out their tasks safely, effectively, efficiently, and enjoyably.5 Usability testing is an important step towards refining the development of a mobile healthcare app that can be used in the treatment of obesity. Users’ experiences with mobile apps vary depending on the type of device used; users often report difficulty using mobile apps. It is vital that the end user is considered throughout app development (particularly where the app is to be used in clinical cohorts) and that testing for both technical and clinical effectiveness is completed so that functionality can be optimized.6

The primary purpose of this app was not only to monitor lifestyle logging behaviors, but also to facilitate self-management by obese patients in hospital clinics and assist clinicians toward the effective counseling of patients with regard to cognitive behavioral therapy. A few studies have investigated whether the use of obesity-related behavioral monitoring apps helps patients as well as clinicians in time-limited hospital clinics. Moreover, if apps for weight management do not engage participants enough for them to want to continue using the app, the intended weight loss and maintenance effects might not be achieved.7 Therefore, we first evaluated the usability of the app in this study before conducting a full-scale effectiveness assessment.

Development of the Dr. Youth app

The app’s overall content and usability were formulated and developed via qualitative investigation and supported by a team approach. The study team consisted of two family physicians, one neuropsychiatric specialist, and a nutritionist. The team met to discuss the project plan, and an app developer was employed to write the software for the app.

Obesity represents a heterogeneous condition according to demographic, clinical, and behavioral factors. Based on a study of the demographic, clinical, and behavioral typology of obesity in the United States,8 we paid particular attention to the content that differences in the demographic characteristics, clinical conditions, and behavioral factors create distinct subtypes of obesity. Hence, stratifying obesity by subtypes or by assessing different risks across distinct groups may benefit weight management. Obesity subtyping helps to identify which behaviors need to be targeted and in whom, and who will be helped by the target obesity management program. A study used data from the Yorkshire Health Study to investigate six distinct groups of obese individuals, namely heavy drinking males, young healthy females, affluent and healthy elderly, physically sick but happy elderly, unhappy and anxious middle-aged persons, and a cluster of people with very poor health.9 Considering clinical experiences as well as several studies using obesity subtypes, we defined seven categories of obesity subtypes according to demographic factors and behavioral factors to better understand the lifestyle differences of individuals and to provide personalized lifestyle logging and feedback for each participant. The following obesity subtypes were suggested for a hospital weight management clinic setting: heavy drinking, stress-related eating, sedentary lifestyle, depression or sleep problems, age-related weight gain, shift work-related weight gain, and binge eating patterns (Table 1). The main reason for subtyping obesity was to focus on the kind of behavior monitoring that should be emphasized for each type of obese participant.

Typically, behavior change techniques for weight management include many complex and diverse components. Behavior change taxonomies have been used to identify the basic components that modify behavior within theoretical frameworks.10 The incorporation of evidence-based behavior change techniques in an app reportedly facilitates increased engagement and effectiveness.11 Thus, we incorporated several evidence-based behavioral techniques while developing our app, including commitment and goal setting (e.g., a participant could select the option “I will attempt to reach the target weight by a predefined date” when a participant initiated the app). We also added self-monitoring of corresponding behaviors for seven domains (monitoring “critical food” [a concept defined specifically for this study and explained later in this section], sleep, regular meal times, nighttime eating, sleep, stress level, and alcohol intake) and monitored feedback for each behavior (weekly feedback on the above-mentioned seven domains based on the logged information). Suggestions were provided accordingly for behavior improvements or encouraging positive behavior changes.10 We also encouraged participants to monitor their weight and provided feedback based on the weight changes.

The app was designed to collect patients’ behavioral information through dietary habit questionnaires12 and sleep time and stress evaluation using the Korean-translated Brief Encounter Psychosocial Instrument.13 The aim was to categorize participants into the obesity subtypes shown in Table 1. Dietary habit questionnaires were used to identify the dietary patterns of the patients and assess whether they have preferences for foods such as fried foods, instant foods such as ramen or fast foods such as pizza. We used the Korean-translated Brief Encounter Psychosocial Instrument13 to construct the questions, given that stress and sleep affect eating habits12 and that experiencing psychological problems can reduce the motivation to lose weight, leading to a high dropout rate in the weight management program.

We selected behavioral monitoring indicators based on previous studies and reports.8,14 Categories of behavioral monitoring indicators include regular meal intake, regular sleep time, frequency of alcohol drinking, frequency of critical foods intake, body weight, exercise frequency, and stress levels according to several obesity subtypes. Therefore, the app was designed in a “to-do list” format that included a life-log, critical foods, and other behavioral monitoring indicator information.

“Critical foods” is a concept we defined, consisting of foods that the patient is recommended to avoid. Unlike conventional food diaries, the critical food list to be avoided is set based on participants’ choices to monitor their “risk” foods. The list of critical foods included simple carbohydrates, such as white bread, cakes, cookies, and sugar-sweetened beverages, and red meats with high levels of saturated fat, such as pork belly or ribs. Participants can choose up to three items for their critical food lists and log their frequency of critical food intake so that they can effectively monitor this list instead of keeping a food diary. For example, if one participant eats too many cookies, they can add cookies to their critical food list and would be required to monitor how frequently they ate cookies and how many cookies they ate in a day.

The app had an algorithm and message pool configuration to improve user motivation. According to their previous adherence to weekly behavior monitoring, participants were categorized as nonusers (participants whose logging adherence was less than 30%), moderate users (participants whose logging adherence was 30%–50%), and highly active users (participants who logged at least 50% of their suggested behavior monitoring indicators). Motivation messages were randomly selected from the targeted pool and sent to personalize users’ adherence to the app. At the end of 1 month, participants were evaluated for their achievements in bodyweight reduction, and new goals or behavior monitoring indicators were modified to adapt to their new goal or behavior target.

The app also had a reporting section, which showed how participants had been monitoring their behaviors as well as body weight according to their performance and adherence levels. Management plans consisted of guided exercise programs as well as low-calorie healthy recipes, which were suggested to participants as shown in Figs. 1 and 2.

Participants

Patients at a university hospital outpatient weight management clinic were invited to participate in the study from November 1 to 30, 2018. The eligible population was patients with a body mass index (BMI) of 25 kg/m2 or more who were seeking weight control, had their own smartphones, and were able to download the Dr. Youth app (YSBio Co., Ltd., Seoul, Korea). Patients were excluded if they had a drug addiction or alcohol use problem or uncontrolled psychotic disorders. Patients who could not use the app or who were unable to answer the questionnaire were also excluded. This study was approved by the Ethics Committee of Seoul National University Bundang Hospital (IRB No. B-1810-499-306).

Typically, usability testing conducted with five participants can identify at least 85% of usability problems.15 Since our app was developed to provide personalized care linked with an outpatient weight management clinic, the number of problems found in the app was considered to be an important factor.16 Hence, we performed our usability study with at least 50 participants, a study size at which 98%–100% of the problems would be detectable. Table 2 presents the basic characteristics of the 50 participants (14 men and 36 women, aged between 20 and 59 years) The participants’ mean age was 42.6 years (standard deviation [SD], 10.8 years); their mean BMI was 29.6 kg/m2 (SD, 5.7 kg/m2).

Procedure: assessment of usability

Participants who met the eligibility criteria were recruited from a university weight management clinic. If patients agreed to participate in the study, they provided informed consent, and then were asked to download the app from the iPhone App Store or Google Play store. Participants were asked to perform several tasks and complete the 10-item System Usability Scale (SUS) questionnaire.17

Usability was defined as the extent to which the app could be used by a clinical cohort of obese adults to achieve specified tasks with technical effectiveness, efficiency, and satisfaction.18 This definition is in line with the standard ISO9241-11.19 Usability testing refers to the evaluation of information systems that involve participants who are representative of the target user population.20 Effectiveness refers to whether people can complete their tasks and achieve their goals. User efficiency is the extent to which users expend resources toward achieving their goals. User satisfaction is the level of comfort users experience in achieving those goals.

Participants were given eight tasks to complete in order to test the app’s technical effectiveness, user efficiency, and user satisfaction. Each task required, or requested, that the participant obtain or enter specific data. The tasks required the user to enter personal settings, use the goal-setting feature, and enter details regarding their lifestyle.

Technical effectiveness

The participants carried out eight tasks to test the app’s technical effectiveness. A task was completed when the task goal was obtained successfully without a critical error. A critical error was defined as an error resulting in an incorrect or incomplete outcome. A noncritical error was an error that would not affect the final output of the task, but caused it to be completed less efficiently. We analyzed the success and failure rates of participants’ task completion. A success rate of 80% or higher was considered to lower the importance of technical factors in usability assessment.

User efficiency

The user efficiency of the app was tested, with users being timed as they undertook tasks in order to examine whether the app was easy to navigate. User efficiency was measured by recording the mean time (in seconds) a user took to complete a task.21 The Single Ease Question (SEQ) was asked of participants at the end of testing as a quantitative evaluation of difficulty.22 The SEQ is a seven-point rating scale to assess how difficult users find a task, where a rating of 1 indicates that the task was very difficult and a rating of 7 indicates that it was very easy. The SEQ was administered immediately after a user completed a task.

User satisfaction

After the task was complete, the overall usability score was measured using the SUS questionnaires.17 SUS is a brief quantitative assessment of usability, which can be carried out using the questionnaire, and the results are valid and reliable. SUS is a scale ranging from 0 to 100, with scores over 80 and below 70 suggesting high usability and low usability, respectively.23 The ten SUS items were ranked according to a 5-point Likert scale.

In addition, after using the app for a week, the participants were asked to submit online feedback through a Google questionnaire system. Descriptive analysis was performed where the median values and 25%–75% intervals were displayed in continuous variables and frequencies with the percentages as categorical variables. When analyzing data according to sex, age group, or BMI group, we performed the Wilcoxon rank-sum test between two independent groups and the Kruskal-Wallis test among more than two groups. All analyses were performed using SAS version 9.4 (SAS Institute Inc., Cary, NC, USA).

Technical effectiveness

All tasks were completed successfully, except for one task for one participant with a type 2 error. The total success rate of all tasks was about 99%.

User efficiency

The results of user efficiency are presented in Table 3. The longest time taken to complete a task was 120.5 seconds for task 1, and the shortest time was 9.5 seconds for task 6. Regarding the difficulty evaluation, the SEQ ratings indicated that the most difficult case was task 6, and the easiest one was task 1.

User satisfaction and usability

The SUS scores corresponding to the ten items are presented in Fig. 3. Odd-numbered items indicate a positive response, while even-numbered items indicate a negative response. Higher numbers indicate increasing degrees of participant agreement.

The results of the SUS are presented in Table 4. The overall mean SUS score was 76.65 (SD, 15.43). The SUS scores of 37 participants (74%) were 70 or higher, which indicated that the app was an acceptable system.24 There were no significant differences in SUS scores among different sex, age, or BMI groups.

Thirty-eight of the 50 participants submitted responses to our Google questionnaires after using the app for a week (a 76% response rate). The most frequent comments made by participants throughout testing of the app to each question were as follows. When asked, “What are the good points when using the app?,” the most frequent response was “It was simple and good.” When asked, “What did you dislike when using the app?,” the most frequent response was “I felt sorry that I couldn’t enter the details.” When asked, “Are there any improvements or suggestions to this app?,” the most common response was “I hope the programs and recording section of the app will be more specific.”

Overall, our weight management app was rated as easy to use, and each of the test tasks was successfully completed without critical errors. Evaluation of the usability of smartphone apps is critical for their success, allowing developers to adapt and improve them against the rapidly changing backdrop of mobile technology. App functions that are difficult to use or to understand might result in a significant decrease in usability. These factors are common problems and shortcomings of weight-control mobile apps that are currently available.

Very few studies have reported on the development and use of mobile phone apps for individual behavioral change or weight monitoring.25 Hebden et al.26 developed and tested four mobile phone apps to improve nutrition and physical activity behaviors during young adulthood. Qualitative feedback provided few suggestions for content changes in these apps, with the major concerns being their slow running speed and login requirements. Dennison et al.27 reported that healthy young adults displayed some interest in using health care apps that support behavior change. Legitimacy, effort required, and immediate effects on mood emerged as important influences relating to app usage, as users of behavior-change apps may lose motivation over time and engage in only intermittent use. Whether this view is applicable to obese patients and to long-term users remains to be seen. O’Malley et al.6 tested the usability of a mobile app used in adolescent obesity management. They described the importance of usability testing in refining the development of an app for use in the treatment of obesity.

Considering these earlier studies, usability testing is clearly paramount for the optimal design and development of mobile apps used in clinical settings. The objective of SUS, therefore, was to provide us with a measure of people’s subjective perceptions of system usability in the very short time available during an evaluation session. Tullis and Stetson’s research showed that using SUS provides a measure of the perceived usability of a system with a small sample and allows us to be fairly confident in obtaining a good assessment of how people see the system.28

Overall, the test results were promising, and participants rated the Dr. Youth app as an acceptable, easy-to-use system. Each test task was completed successfully, with only one critical error, indicating the app’s technical effectiveness. Users were satisfied with the app and suggested several ways to improve it further. The mean SUS score of 76.65 and the 74% rating for acceptable usability were promising. Given the current popularity of mobile apps, we anticipated that a mobile app would be a useful tool for obesity treatment, and this study’s results support our view.

Some limitations of this study must be acknowledged. Usability has an initial assessment process that is followed by a feedback process when the app is constantly used and verified. This study comprises the initial assessment process. Thus, further research will be needed to establish clinical evidence with continuous feedback on the app’s usability. The development and the usability testing of the app in the study were two different processes, which makes it difficult to define the distribution of obesity subtypes or Korean-translated Brief Encounter Psychosocial Instrument (BEPSI-K) scores of the participants recruited for the usability testing of the app. Additionally, we did not collect participants’ educational or socioeconomic statuses, which might be a potential limitation of the study. Furthermore, obesity subtyping in this study was not based on domestic data. Considering differences in the demographic characteristics of obese patients, it may be meaningful to define obesity subtypes using domestic data unique to Korea in the future, followed by usability testing of an app for obese patients. In addition, the app should be tested in a cohort of obese adults who are not attending a clinic for weight management, as we do not know whether the users’ level of motivation for lifestyle change affects their perceptions of technical usability. Finally, we assessed the users’ satisfaction with using the app as a whole, rather than their satisfaction with completing each task. Future work to explore individual components of the app may also be warranted. Despite these limitations, our study demonstrated the good usability performance of the Dr. Youth app for weight management.

Overall, the Dr. Youth app performed well in usability testing, with results providing support for its usability by end users. The results of this study guided the final development cycle of the app prior to its use. Usability testing of mobile apps designed to address clinical problems is vital, as it enables developers to account for users’ needs and to better optimize the mobile app’s acceptability and utility.

Seung Ju You and You Soo Kim from YSBio Co., Ltd. declare that they have no conflict of interest. The other authors declare no conflict of interest.

This study was supported by the Ministry of Small- and Medium-sized Enterprises and Startups (grant no. C0564334).

Study concept and design: JYL, JYK, SJY, YSK, HYK, JHK, SK (Sohye Kim), and SK (Siye Kil); acquisition of data: SJY, SK (Siye Kil), and HK; analysis and interpretation of data: JYK, YSK, JHP, and JSH; drafting of the manuscript: JYL, JYK, and SJY; critical revision of the manuscript: JYK, JHP, SK (Sohye Kim), HK, and YSY; statistical analysis: JYL and KML; obtained funding: JYK, SJY, and YSK; administrative, technical, or material support: SK (Siye Kil) and HK; and study supervision: JYK and YSK.

Fig. 1. Screenshots of health behavior monitoring in the Dr. Youth application.
Fig. 2. Screenshots of the life log and body weight report.
Fig. 3. Mean System Usability Scale scores corresponding to the 10 items. app, application.

Obesity subtypes according to the obesity-related behavioral components

CategoryObesity subtypeBehavioral component
1Frequent drinking/company night outI frequently eat and drink with my coworkers or friends after work.
2StressI frequently overeat when I feel like I’ve been stressed out/under stress.
3Work at night/night shiftI work late at night or in night shifts and thus eat late.
4Sedentary lifestyleI eat a lot but don’t exercise much.
5Depression or sleep problemI eat too much when I feel depressed, or I have a sleep problem and usually eat to sleep.
6Age-related weight gainI’ve gained weight as I’ve gotten older. I put on weight when I started menopause.
7Snacks/refreshmentsI am not really hungry, but I need something to nibble on.

General characteristics of the study participants

VariableNo. (%)
Total number50

Sex
 Male14 (28)
 Female36 (72)

Age (yr)
 20–298 (16)
 30–399 (18)
 40–4917 (34)
 50–5916 (32)

BMI (kg/m2)
 ≥25.0 and <3034 (68)
 ≥30.016 (32)

BMI, body mass index.

Types of tasks and user efficiencies

TaskDescriptionTime (sec)Difficulty (mean SEQ)
MedianQ1Q3
1Sign up for membership and sign in.120.5991556.16
2Fill out and submit the consent form.129206.3
3Go to the “Obesity subtype” section and select one. Then check the prescribed monitoring index.14.510256.36
4Record your meal times and whether you eat snacks or eat late at night.2616326.24
5Go to the “Sleep time records” section and enter your daily sleep record.1812316.34
6Record the amount of alcohol you drink.9.55126.46
7Record your stress level.129146.37
8Set your own physical exercise goal and enter performance status.14.510256.4

Q, quartile; SEQ, Single Ease Question.

SUS scores according to sex, age, and BMI

VariableSUSP

MedianQ1Q3
Sex0.442*
 Male81.2572.5087.50
 Female75.0065.0087.50

Age (yr)0.798
 20–2978.7576.2586.25
 30–3972.5065.0087.50
 40–4980.0065.0097.50
 ≥5073.7567.5086.25

BMI (kg/m2)0.700*
 ≥25.0 and <3077.5065.0087.50
 ≥3076.2571.2592.50

SUS, System Usability Scale; BMI, body mass index; Q, quartile.

  1. World Health Organization. WHO factsheet: obesity and overweight [Internet]. Geneva: World Health Organization; 2018. [cited 2019 Aug 30]. Available from: http://www.who.int/mediacentre/factsheets/fs311/en/
  2. NCD-RisC. Body mass index, evolution of BMI over time [Internet]. London: NCD-RisC; 2017. [cited 2019 Aug 30]. Available from: http://ncdrisc.org/obesity-prevalence-line.html
  3. Mokdad AH, Ford ES, Bowman BA, Dietz WH, Vinicor F, Bales VS, et al. Prevalence of obesity, diabetes, and obesity-related health risk factors, 2001. JAMA 2003;289:76-9.
    Pubmed CrossRef
  4. Kang JH, Jeong BG, Cho YG, Song HR, Kim KA. Socioeconomic costs of overweight and obesity in Korean adults. J Korean Med Sci 2011;26:1533-40.
    Pubmed KoreaMed CrossRef
  5. Bacigalupo R, Cudd P, Littlewood C, Bissell P, Hawley MS, Buckley Woods H. Interventions employing mobile technology for overweight and obesity: an early systematic review of randomized controlled trials. Obes Rev 2013;14:279-91.
    Pubmed KoreaMed CrossRef
  6. O’Malley G, Dowdall G, Burls A, Perry IJ, Curran N. Exploring the usability of a mobile app for adolescent obesity management. JMIR Mhealth Uhealth 2014;2:e29.
    Pubmed KoreaMed CrossRef
  7. Dounavi K, Tsoumani O. Mobile health applications in weight management: a systematic literature review. Am J Prev Med 2019;56:894-903.
    Pubmed CrossRef
  8. Jimenez MP, Green MA, Subramanian SV, Razak F. A demographic, clinical, and behavioral typology of obesity in the United States: an analysis of National Health and Nutrition Examination Survey 2011–2012. Ann Epidemiol 2018;28:175-81.
    Pubmed KoreaMed CrossRef
  9. Green MA, Strong M, Razak F, Subramanian SV, Relton C, Bissell P. Who are the obese?: a cluster analysis exploring subgroups of the obese. J Public Health (Oxf) 2016;38:258-64.
    Pubmed CrossRef
  10. Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med 2013;46:81-95.
    Pubmed CrossRef
  11. Michie S, Abraham C, Whittington C, McAteer J, Gupta S. Effective techniques in healthy eating and physical activity interventions: a meta-regression. Health Psychol 2009;28:690-701.
    Pubmed CrossRef
  12. Jo JS, Kim KN. Development of a questionnaire for dietary habit survey of Korean adults. Korean J Community Nutr 2014;19:258-73.
    CrossRef
  13. Huh BY, Yim JH, Bae JM, Choi SS, Kim SW, Hwang SH. The validity of modified Korean-translated BEPSI (Brief Encounter Psychosocial Instrument) as instrument of stress measurement in outpatient clinic. Kor J Fam Med 1996;17:42-53.
  14. Field AE, Camargo CA, Ogino S. The merits of subtyping obesity: one size does not fit all. JAMA 2013;310:2147-8.
    Pubmed CrossRef
  15. Six JM, Macefield R. How to determine the right number of participants for usability studies. San Francisco (CA): UXmatters; 2016.
  16. Macefield R. How to specify the participant group size for usability studies: a practitioner’s guide. J Usability Stud 2009;5:34-45.
  17. Borsci S, Federici S, Lauriola M. On the dimensionality of the System Usability Scale: a test of alternative measurement models. Cogn Process 2009;10:193-7.
    Pubmed CrossRef
  18. Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004;37:56-76.
    Pubmed CrossRef
  19. Brooke J. SUS: a quick and dirty usability scale. Usability evaluation in industry, Jordan PW, Thomas B, McClelland IL, Weerdmeester B. London: Taylor and Francis; 1996 p. 189-94.
  20. Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces. Proc AMIA Annu Fall Symp 1997:218-22.
    Pubmed KoreaMed
  21. Abraham C, Michie S. A taxonomy of behavior change techniques used in interventions. Health Psychol 2008;27:379-87.
    Pubmed CrossRef
  22. Sauro J, Dumas JS. Comparison of three one-question, post-task usability questionnaires, Proceedings of the 27th SIGCHI Conference on Human Factors in Computing Systems, 2009 Apr 4–9, Boston (MA), USA.
  23. Brooke J. SUS: a retrospective. J Usability Stud 2013;8:29-40.
  24. Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009;4:114-23.
  25. Knight-Agarwal C, Davis DL, Williams L, Davey R, Cox R, Clarke A. Development and pilot testing of the Eating4two mobile phone app to monitor gestational weight gain. JMIR Mhealth Uhealth 2015;3:e44.
    Pubmed KoreaMed CrossRef
  26. Hebden L, Cook A, van der Ploeg HP, Allman-Farinelli M. Development of smartphone applications for nutrition and physical activity behavior change. JMIR Res Protoc 2012;1:e9.
    Pubmed KoreaMed CrossRef
  27. Dennison L, Morrison L, Conway G, Yardley L. Opportunities and challenges for smartphone applications in supporting health behavior change: qualitative study. J Med Internet Res 2013;15:e86.
    Pubmed KoreaMed CrossRef
  28. Tullis TS, Stetson JN. A comparison of questionnaires for assessing website usability, Proceedings of the Usability Professionals Association (UPA) 2004 Conference, 2004, Minneapolis (MN), USA.