If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Smartphone applications are valid and reliable in assessing cognition in pwMS.
•
Smartphone versions of the symbol digit modalities test are most used.
•
Short and long-term practice effects are present.
•
Longitudinal data, predictive and ecological validity are lacking.
Abstract
Background
Current cognitive monitoring of people with multiple sclerosis (pwMS) is sporadic, resource intensive and insensitive for detection of real-world cognitive performance and decline. Smartphone applications may provide us with a more sensitive biomarker for cognitive decline that reflects real-world performance. The goal of this study was to perform a systematic review and qualitative synthesis of all current smartphone apps monitoring cognition in pwMS.
Methods
A systematic search of five major online databases (PubMed/Medline, Scopus, Web of Science, Cumulative Index of Nursing and Allied Health Literature and IEEE Xplore) was performed in accordance with the Cochrane Handbook and Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement. We included all studies with at least one measure of phone-based digital biomarkers for monitoring cognition in pwMS above the age of 18.
Two authors independently screened the articles retrieved. Data on test-retest reliability, validity coefficients, feasibility and practice effects were extracted from the studies identified. Critical appraisal of the studies was performed using the National Institute of Health quality assessment tool for observational cohort and cross-sectional studies.
Results
12 articles covering six smartphone apps were included in this review. All articles had a low risk of bias, though sample size calculation was rarely performed. Of the six apps, five used smartphone versions of the symbol digit modalities test. The final app examined keystroke features passively.
Test-retest reliability ranged from good to excellent. Concurrent validity was demonstrated through moderate to strong correlation with neuropsychological tests and weak to moderate correlations with EDSS, radiological biomarkers and patient-reported outcomes. Mobile apps performed comparably, and in some cases outperformed established cognitive tests. Whilst reported acceptability was high, significant attrition rates were present in longitudinal cohorts. There were significant short and long-term practice effects.
Overall, smartphone versions of the SDMT showed strong psychometric properties across multiple apps.
Conclusion
Smartphone applications are reliable and valid biomarkers of real-world cognition in pwMS. Further longitudinal data would allow for a better understanding of their predictive and ecological validity.
). Current neuropsychiatric batteries require significant time and trained staff to administer and interpret the results. Access to neuropsychologists is limited and a formal, comprehensive neuropsychological assessment can take several hours. Whilst brief, simple-to-administer cognitive tests such as the symbol digit modalities test (SDMT) and the Brief International Cognitive Assessment for MS (BICAMS) have been developed to reduce the demand on busy healthcare professionals (HCP) (
), they are challenging to perform routinely in the outpatient clinic setting. Testing usually takes place in a clinical setting instead of the home environment, which can underestimate the cognitive demands required in a dynamic, real-world setting (
In response to this, computerization of neuropsychological testing has gathered pace in the past two decades. There are now several computerized neuropsychological assessment devices that are valid and reliable supplements to a full neuropsychological assessment (
). With further technological advances, several smartphone applications have been developed to monitor cognition in pwMS. Given the ubiquity of smartphones, these apps are ideal for ecological assessment of pwMS’ cognition. However, many of these applications lack validation and reliability data. The advent of COVID-19 and its impact on immunocompromised pwMS has further emphasized the need for valid and reliable remote monitoring.
Our objective was to comprehensively review smartphone applications monitoring cognition in MS and their psychometric properties. In particular, we focused on test–retest reliability, validity (including discriminant, ecological, predictive, concurrent validity), practice effects and feasibility.
2. Methods
2.1 Search protocol and inclusion criteria
The systematic review is reported in accordance with the Cochrane Handbook and Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement (
). A systematic literature search in PubMed/Medline, Scopus, Web of Science, Cumulative Index of Nursing and Allied Health Literature (CINAHL), and IEEE Xplore was performed. The search terms included: multiple sclerosis OR MS AND Cognition OR Cognitive Reserve OR cognit* OR Memory OR Executive Function OR Language OR processing speed AND Smartphone OR Cell Phone OR phone OR Mobile Application. The search algorithm was adapted according to each database and filters were used to refine the searches. Searches were made in May 2022. Appendix 1 shows the specific search strategy for each database.
Two authors (YF) and (FB) independently screened each title and abstract to identify potential articles investigating the use of smartphone to monitor cognition in pwMS. Full text articles were then retrieved and reviewed based on the eligibility criteria to determine articles for inclusion. A manual search in the reference lists (forward and backward searches) of included articles was conducted to identify other relevant studies. Any discrepancies were discussed until consensus was reached with a third author (DM).
To be included in the systematic review, the following criteria had to be met: (1) Inclusion of pwMS above the age of 18 (2) Included at least one measure of a mobile-phone based digital biomarker for monitoring cognition (3) Includes sufficient data to analyse one or more of: test-retest reliability, validity, practice effects and feasibility. Review articles, abstracts, protocol papers, and patient-reported outcomes were not included in this review.
2.2 Data extraction and quality assessment
The following data were extracted: app name, authors, publication year, study design, expanded disability status scale (EDSS), sex, duration of disease, type of MS, age, number of pwMS in the study, smartphone test features, cognitive outcome measures and measures of validity and reliability. The following smartphone test features were used: mean number of correct responses on the smartphone SDMT, passive keystroke features for Neurokeys, median number of correct responses and standard deviation of time to identify a digit for elevateMS. The extraction of smartphone test features was based on the features examined in the selected studies.
To aid readability, the following cut-offs were used to colour the cells of Table 2:
•
Pearson's r or Spearman's rho correlation coefficients (
): Acceptable was defined as p-values <0.05 (coloured green) and poor as values >/=0.05 (coloured red).
Where multiple correlation coefficients were reported for a range of smartphone test features, the best value was taken.
The methodological quality of the included studies was evaluated using the Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies developed by the
. This tool includes 14 criteria that are rated according to three responses: Yes, No, or other. Studies with greater than 50% of responses (eight or more) being “yes” are considered “good” quality, 30–50% are considered ‘fair quality’ (five to seven), and less than 30% (four or less) are considered “poor” quality based on previous systematic reviews (
Effectiveness of physical therapy interventions in reducing fear of falling among individuals with neurologic diseases: a systematic review and meta-analysis.
There were 2189 citations identified from the initial search of pre-specified databases and an additional citation was found through backward and forward searching. 230 duplicates were removed. Following title and abstract screening by two review authors a further 1925 were excluded. The remaining 35 articles were assessed for eligibility after retrieving full text articles. 12 articles were found that met the study inclusion criteria, covering six smartphone applications. The remaining 23 articles were excluded and reasons for exclusion have been specified. The PRISMA flow diagram is shown in Fig. 1.
Table 1 presents the characteristics of the studies included in this review. 12 studies were identified, covering six smartphone apps (FLOODLIGHT, MSSherpa, MSCopilot, Neurokeys, NeuFun, elevateMS). Six were prospective cohort studies, five were cross-sectional and one had both a cross-sectional and prospective cohort component. Both MSCopilot and FLOODLIGHT had two papers arising out of the same cohort, whilst MSSherpa and Neurokeys had initial cross-sectional studies with longitudinal follow-up reports with minor differences in participant demographics between the two timepoints. Six studies utilised the participant's own smartphones. Three studies examined only participants with RRMS, eight included all types of MS and one did not specify the types of MS included. The number of pwMS ranged from 21 to 495. Percentage of females ranged from 53 to 95%.
Adherence and satisfaction of smartphone- and smartwatch-based remote active testing and passive monitoring in people with multiple sclerosis: nonrandomized interventional feasibility study.
Practice effects of mobile tests of cognition, dexterity, and mobility on patients with multiple sclerosis: data analysis of a smartphone-based observational study.
Evaluating the utility of smartphone-based sensor assessments in persons with multiple sclerosis in the real-world using an App (elevateMS): observational, prospective pilot digital health study.
-Mean (Range) EDSS (for subset of 112 pwMS): 4.9 (0–7) -Mean (Range) Age: 54.3 (18.5 - 77.2) -Female%: 58% -Mean time since onset (±SD): NA -Type of MS: all
Google Pixel XL/2XL phones running Android 9 and above
Of the six apps, five used smartphone versions of the SDMT to assess cognition. A figure showcasing the subtle differences between different versions of smartphone SDMTs are attached in Figure Appendix 1. Of these, only elevateMS utilised a voice-controlled version. Neurokeys examined passive keystroke features.
3.3 Psychometric properties of smartphone applications
3.3.1 Smartphone SDMTs
The MSSherpa version of the smartphone SDMT has been shown to correlate moderate to strongly with paper-based SDMT (r = 0.622 to 0.784), although there were concerns regarding incorrectly administered SDMT in one of the validation studies (
). Both pwMS and healthy controls (HC) scored lower on the smartphone versions on the SDMT, likely due to the response time required in between symbols (
The MS Copilot version was shown to correlate strongly with MSFC (r = 0.81) (250) and moderately with EDSS (r = 0.65) as part of a combined test battery (
), with comparable performance to the traditional MSFC. It also displayed comparable discriminative validity as part of MSCopilot compared to MSFC with significant differences between HC, pwMS with mild disability and pwMS with moderate disability (
). There was no significant difference between the discriminative validity of the smartphone SDMT compared to traditional SDMT. Test-retest reliability was good (ICC = 0.9). Acceptability was high amongst both participants and HCP, with 85% of participants willing to use it at least monthly (
The FLOODLIGHT version correlates strongly with traditional SDMT (r = 0.85) and moderately with EDSS (r = −0.43), magnetic resonance imaging (MRI) measures (r = 0.42 to 0.54) and multiple sclerosis impact scale (MSIS-29) (r = −0.52) (
Adherence and satisfaction of smartphone- and smartwatch-based remote active testing and passive monitoring in people with multiple sclerosis: nonrandomized interventional feasibility study.
). However, adherence decreased over time, with only 60.5% of pwMS adhering to at least 3 days of testing per week after 24 weeks. There was evidence of both short and long-term practice effects extending beyond five repetitions (
Practice effects of mobile tests of cognition, dexterity, and mobility on patients with multiple sclerosis: data analysis of a smartphone-based observational study.
Correlation of the NeuFun version with traditional SDMT was good (CCC = 0.84) and weak to moderate with clinical composite measures (Rho = −0.24 to −0.49), MRI measures (Rho = 0.45–0.46) and unique smartphone-based measures of upper limb function (r = 0.45)(
). The magnitude of the correlations was comparable with traditional SDMT and exceeded that of the paced auditory serial addition test (PASAT). Test-retest reliability was good (ICC 0.87 to 0.9) (
). There was a significant difference between test scores of HC and pwMS and a practice effect was noted at up to eight repetitions.
The only voice-controlled digit substitution test is utilized by Elevate MS. This has shown limited discriminative validity only - it was only able to distinguish pwMS with severe impairment, and not those with mild or moderate impairment (
Evaluating the utility of smartphone-based sensor assessments in persons with multiple sclerosis in the real-world using an App (elevateMS): observational, prospective pilot digital health study.
). Other forms of validity and reliability were not examined in the study.
Predictors of smartphone based SDMT performance is an emerging area of research. Pham et al. showed through elastic net regression that age, vision, eye movement, MRI measures (brain volume and T2 lesion volume (T2LV)) and cerebellar function predict smartphone SDMT performance (
A passive monitoring approach towards cognitive testing has been tested with the Neurokeys app. This app continuously collected keystroke data whenever the keyboard was used. Cross-sectional analysis of eight passive keystroke features revealed test-retest reliability in the good to excellent range. Correlation with SDMT, EDSS and 9-hole peg test (9HPT) was moderate and Release-Release Latency had the strongest correlation with SDMT with a Pearson's r value of −0.553 (
). In the follow-up longitudinal analysis, a further seven features were included and a total of 15 features were analysed. In longitudinal analysis, the variables examined such as emoji sentiments had an AUC <0.7 for prediction of clinically relevant change in SDMT (
). It should be noted that this exceeded that of the traditional clinical benchmark (EDSS) used in the study. Other test features predicted clinically relevant change in the EDSS and new or enhancing MRI lesions with AUC exceeding 0.7 and again outperforming the traditional clinical benchmarks of 9HPT and EDSS respectively. This remains an extremely promising field given its ability to convert real-world smartphone use into a surrogate for cognitive function without the need for active participation by pwMS.
A summary of the validity and reliability of the aforementioned apps can be found in Table 2. For each app, if multiple studies examining reliability and validity were found, each study was allocated a separate row. Detailed information including duration in-between test and retest and exact correlation coefficients are included in Appendix Table 1.
Table 2Validity and reliability of smartphone apps monitoring cognition.
Smartphone app
Concurrent validity with established cognitive tests
Concurrent validity with clinical composite measures
Evaluating the utility of smartphone-based sensor assessments in persons with multiple sclerosis in the real-world using an App (elevateMS): observational, prospective pilot digital health study.
Abbreviations: 9HPT, 9-hole peg test; BICAMS, brief international cognitive assessment for MS; BV, brain volume; BVMT-R, brief visuospatial memory test-revised; CI, cognitively impaired; CVLT, California verbal learning test; EDSS, expanded disability status scale; FLAIR, fluid attenuated inversion recovery; HC, healthy controls; ICC, intra-class correlation; MSFC, MS functional composite; MSIS-29, MS impact scale; pwMS, people with MS; pwMSwoCI, people with MS without cognitive impairment; SDMT, symbol digit modalities test; T2LV, T2 lesion volume.
For the purposes of this study, strong correlation was defined as Pearson's r or Spearman's rho >0.7 (coloured green), moderate between 0.4 and 0.7 (coloured yellow), and weak between 0.1 and 0.4 (coloured red).
With regards to ICC, strong was defined as >0.9 (coloured green), good as >0.75–0.9 (coloured yellow) and weak as </= 0.75.
With regards to AUC, any score >/= 0.7 was considered acceptable (coloured green), scores <0.7 were considered poor (coloured red).
With regards to t-tests or Wilcoxon's signed-rank test used to discriminate pwMS VS controls, a p-value <0.05 was considered acceptable (coloured green), scores >/=0.05 was considered poor (coloured red).
Where a range of values were reported, the best possible value was taken for the colour coding.
*With regards to discriminant/known-groups validity, the known groups are separated by the symbol &.
Practice effects were examined for FLOODLIGHT, MSSherpa and Neufun. For MSSherpa, prominent practice effects were noted between the first and second test, with a cohen's d value of 0.477 reported (
). Subsequent repetitions revealed cohen's d values of less than 0.2. However, with the increase in frequency of testing, it has become apparent that significant practice effects continue for longer than previously thought. In the longitudinal one year follow up study of the same MSSherpa cohort, a local linear trend model was applied and this showed that practice effects lasted for up to two months before plateauing (
Practice effects of mobile tests of cognition, dexterity, and mobility on patients with multiple sclerosis: data analysis of a smartphone-based observational study.
). The majority (19.7%) occurred within the first five repetitions but there was still a significant long-term practice effect noted beyond the first five repetitions. In fact, in a multivariate model the number of repetitions had the largest beta coefficient for the difference between the fifth to last score. However, the R-squared value was only 0.3, suggesting that other factors may be at play. A long-term learning curve analysis predicted that with more tests performed, further improvement in scores can occur with an eventual average improvement over baseline of 40.8% (
Practice effects of mobile tests of cognition, dexterity, and mobility on patients with multiple sclerosis: data analysis of a smartphone-based observational study.
In Neufun, practice effects were examined in 15 pwMS who had longitudinal data with over 20 test sittings. Non-linear regression was utilised to show that on average, practice effects plateaued at eight repetitions, although it should be noted that two participants continued to experience ongoing practice effects beyond 20 repetitions.
It should be noted that the possibility of participants practising in between tests was not explicitly excluded. Details including short versus long-term practice effects is included in Appendix Table 1.
3.3.4 Feasibility
Feasibility was reported for FLOODLIGHT, MSSherpa, elevateMS and MSCopilot. In FLOODLIGHT, 84.7% of participants reported at least acceptable impact on their daily activities. However actual adherence to testing (defined as at least 3 days per week of valid testing) was lower, with 70% to active and 79% to passive testing. After six months, 63.9% of participants indicated willingness to continue using the app. Of the test battery, the 2 min walk test (2MWT) was the least acceptable with 45% indicating that it was the one component of the battery they would remove. It should be noted that this cohort was asked to perform the tests daily. The actual frequency required to detect a clinically significant change is likely to be much less frequent, and thus real-world adherence and acceptability may be higher (
Adherence and satisfaction of smartphone- and smartwatch-based remote active testing and passive monitoring in people with multiple sclerosis: nonrandomized interventional feasibility study.
In elevateMS, only 46.1 to 50.0% of pwMS were compliant (defined as at least one sensor-based test per week in this study) over a 12-week follow up period (
Evaluating the utility of smartphone-based sensor assessments in persons with multiple sclerosis in the real-world using an App (elevateMS): observational, prospective pilot digital health study.
). Participants who were referred from MS clinics had a significantly higher retention rate compared to self-referred pwMS.
In MSCopilot, 67.6% preferred the smartphone app to the traditional MSFC and 87.1% indicated that the test was easy to use. Acceptable frequency was at least once a month in 85.5% (
Details with regards to feasibility is included in Appendix Table 1.
3.3.5 Quality of studies
The results of the methodological quality assessment of the included studies are presented in Appendix Table 2. Of the 12 included studies, all were classified as having good methodological quality with a low risk of bias. Blinding, levels of exposure and assessment of exposure prior to outcome assessment were not relevant to any of the studies. A lack of sample size calculation was common, affecting 10 out of the 12 included studies.
4. Discussion
Remote smartphone-based testing of cognition represents an attractive option for ecological assessment of cognition in pwMS. This is particularly relevant in the COVID-19 pandemic, which has emphasized the need for valid, reliable remote assessment of immunocompromised pwMS. This review shows that these smartphone apps are generally valid and reliable. Concurrent validity was comparable, if not better than contemporary cognitive tests. They were highly acceptable to pwMS, although actual compliance was lower across all apps with significant attrition rates. The ability to predict longitudinal change has not yet been demonstrated, as many of the studies have been cross-sectional. Some studies have also shown that MCID was smaller than SDC, suggesting that further work can be done to increase the sensitivity of these monitoring tools, particularly at an individual level.
A smartphone version of the SDMT is utilised by the FLOODLIGHT, MSSherpa, MSCopilot, ElevateMS and NeuFun apps. Smartphone SDMTs lack standardization across different apps. There are also subtle yet key differences between smartphone and traditional SDMTs. Depending on the app, some smartphone SDMTs are not restricted to six keys for first 26 symbols, include no practice keys and have different symbols. Pixel sizes for symbols differ, and sizes of mobile buttons for answering are not standardized. The impact of these minor variations in smartphone SDMTs are unclear, and future studies may examine correlations between different versions of smartphone SDMTs. The raw overall score for smartphone SDMTs was significantly lower than the traditional written SDMT. The primary reason for this is likely due to the delay in display and reaction time as smartphone SDMTs only display one symbol at a time. Other possible reasons suggested include memorization from prior SDMT attempts and the use of only six symbols for the first 26 symbols in the traditional SDMT (
Concurrent validity with traditional SDMT and other established neuropsychological tests were examined in four out of the six apps and were generally strong. Concurrent validity with other clinical measures of MS such as EDSS, radiological features and other clinical measures was examined in five out of the six apps. This revealed moderate correlation coefficients, and crucially when compared to other traditional neuropsychiatric tests such as the PASAT and traditional SDMT, smartphone SDMT outperformed the PASAT and performed comparably, if not slightly better than the traditional SDMT.
Discriminant validity was examined in five out of six apps and has shown the ability to discriminate not just between HC and pwMS, but also different levels of disability in pwMS. In particular, the sensitivity and specificity to distinguish pwMS with mild disability from those with more severe disability was greater than traditional measures such as PASAT and oral SDMT in MSCopilot (
). Test-retest reliability was reported for five out of the six apps and were generally in the good to excellent range, though specific test features on passive keystroke testing were less reliable. Given the exponential increase in the number of test features that can be derived from smartphone apps, further validity and reliability testing will allow us to select the best test features from these apps.
The quality of the studies were generally good as assessed by the critical appraisal tool. Major limitations of the studies include the lack of a sample size calculation in 10 of the 12 studies. This would be particularly relevant when assessing longitudinal outcomes as sample sizes must be sufficient to have enough participants with a clinically significant change.
There are still significant gaps in the literature with regards to the validity of smartphone apps monitoring cognition in MS. Ecological validity (correlation with employment and activities of daily living) has not been demonstrated. Much of the literature is cross-sectional and longitudinal data is lacking at this stage due to the short follow-up periods and small proportion of participants experiencing clinically significant improvement or deterioration. For example, in the FLOODLIGHT study, only seven participants experienced a significant change in their EDSS score over six months. Of the apps where longitudinal data was available, predictive validity was not shown, with AUC below 0.7 for prediction of a clinically significant change in the benchmark cognitive test (BICAMS and SDMT) in MSSherpa and Neurokeys. Furthermore, MCID exceeded SDC at the group level only in the Neurokeys app. Nevertheless, the results were still promising given that the AUC for Neurokeys still exceeded that of the clinical benchmark used in the study, the EDSS. Future studies with larger cohorts and longer follow-up periods will be able to investigate the ability of smartphone apps to predict clinically relevant outcomes longitudinally, and whether they have sufficient sensitivity for MCID to exceed the SDC. By proving ecological and predictive validity, HCPs would be more likely to base their clinical decision-making on smartphone-based digital biomarkers.
The volume of data generated by smartphone applications have also allowed for novel methods to adjust for potential confounders of cognitive function in MS. This has been an area of ongoing controversy with conflicting findings on the effect of mood and fatigue on cognition (
), which ignores the day-to-day fluctuations of mood and fatigue. The use of an in-app generated fatigue score and upper limb function surrogate as demonstrated by NeuFun allowed for adjustment of this at the time of testing without the need for collection of additional data. The passive monitoring of emoji sentiments demonstrated by
) is another promising method for real-world mood monitoring and subsequent adjustment. Other apps such as FLOODLIGHT have a daily mood question as part of its battery. Further studies may examine the effect of adjusting for a smartphone app-based mood surrogate and its effect on cognitive performance.
Long-term practice effects can occur with the smartphone SDMT as shown by FLOODLIGHT, NeuFun and MSSherpa. With new modelling techniques and increased test repetitions, it is now apparent that practice effects last longer than previously thought. Whilst the largest practice effect is still within the first two to three repetitions (consistent with studies in traditional SDMT), it has been demonstrated that eight or more repetitions may be required prior to stabilization of practice effects. A future study may examine the use of a non-linear correction to counteract the practice effect. There may also be a proportion of patients where this improvement in test score is a real effect, such as a response to their treatment regimen or recovery from a relapse. This is a known phenomenon with the EDSS, where a significant proportion of patients experience regression of their EDSS (
Greater sensitivity to multiple sclerosis disability worsening and progression events using a roving versus a fixed reference value in a prospective cohort study.
), thus reducing the sensitivity of the test to detect a clinically significant change. A potential solution may be the application of a roving smartphone SDMT score, such as that suggested with the EDSS. A future study may use this to determine if this increases the sensitivity of smartphone apps to detect clinically significant cognitive change.
Whilst acceptability was largely excellent amongst pwMS, compliance with testing decreased significantly with time and was lower than self-reported acceptability. In the FLOODLIGHT study, only 60.5% of pwMS adhered to at least three days of testing per week after 24 weeks. Real-world compliance is likely to be even lower than trial settings. However, it should be noted that test frequency in these studies were high – most required a minimum of two to three tests every week. It is likely that minimum test frequency required to detect a clinically relevant change in cognition will be less frequent, and this may reduce the risk of non-compliance. For example, analysis of longitudinal MSSherpa data suggests that testing frequency of once every 12 days is sufficient to detect a 4-point change in SDMT scores (
). Other strategies such as automated text reminders and passive monitoring may also improve compliance. Further studies may also explore the acceptability to HCP and healthcare organisations, particularly in light of the potential cost savings highlighted by a recent study (
). This test involves a novel cognitive test where a number or letter is displayed on the screen from a random selection of 10 letters and five digits. The subject has to tap the screen when a letter appears, but do nothing if a number appeared instead. Whilst this was only subject to a small pilot study of 21 participants with radiologically isolated syndrome (RIS), it was able to discriminate HC from patients with RIS. This ability to detect subclinical cognitive dysfunction is promising and a follow-up study on early MS patients is planned. Other smartphone-based cognitive tests with a similar mechanism include the choice reaction test (part of the MSReactor battery) and further studies on this platform will be of interest (
Limitations of this study include the lack of a specific critical appraisal tool for smartphone apps. Whilst critical appraisal tools exist for validity and reliability studies, they are based largely on traditional in-clinic assessments and are not optimized for remote assessment tools such as smartphone apps with frequent repetitions. Another limitation specific to the Neurokeys app was that the most promising keystroke features for prediction of cognitive change were emoji sentiments, and the potential for depression and mood as a confounder for cognitive function was not considered in the study. Finally, none of the included studies specifically examined the effect of device type on results. Future studies could consider sensitivity analyses to confirm validity across different smartphone models.
5. Conclusion
This review shows that remote monitoring via smartphone apps is a feasible, valid and reliable method to collect quantifiable, objective data on cognition. Compared to contemporary cognitive tests, it may represent a more sensitive method to detect subclinical cognitive dysfunction. This may pave the way for more efficient trial recruitment in progressive MS. Whilst this does not replace formal neuropsychological testing by a trained neuropsychologist, it offers a low-cost, widely accessible way to assess cognition in pwMS. Further research should investigate the predictive and ecological validity of these smartphone apps and integrate them with other biomarkers to develop more accurate predictive tools.
Funding
This work was supported by the National Health and Medical Research Council (NHMRC), Multiple Sclerosis Research Australia (MSRA), AVANT Foundation, Australia and New Zealand Association of Neurologists (ANZAN).
Declaration of Competing Interest
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:
Yi Chao Foong reports a relationship with Biogen that includes: travel reimbursement. Francesca Bridge reports a relationship with Biogen that includes: travel reimbursement. Anneke van der Walt reports a relationship with Novartis Pharmaceuticals Corporation that includes: consulting or advisory, funding grants, speaking and lecture fees, and travel reimbursement. Anneke van der Walt reports a relationship with Biogen that includes: consulting or advisory, funding grants, speaking and lecture fees, and travel reimbursement. Anneke van der Walt reports a relationship with Merck & Co Inc that includes: consulting or advisory, funding grants, speaking and lecture fees, and travel reimbursement. Anneke van der Walt reports a relationship with F Hoffmann-La Roche Ltd that includes: consulting or advisory, funding grants, speaking and lecture fees, and travel reimbursement. Anneke van der Walt reports a relationship with National Health and Medical Research Council that includes: funding grants. Anneke van der Walt reports a relationship with Multiple Sclerosis Research Australia that includes: funding grants. Melissa Gresle reports a relationship with Biogen that includes: funding grants. Melissa Gresle reports a relationship with F Hoffmann-La Roche Ltd that includes: funding grants. Daniel Merlo reports a relationship with Novartis that includes: speaking and lecture fees. Helmut Butzkueven reports a relationship with Novartis that includes: consulting or advisory, funding grants, and speaking and lecture fees. Helmut Butzkueven reports a relationship with Biogen Inc that includes: consulting or advisory, funding grants, and speaking and lecture fees. Helmut Butzkueven reports a relationship with Merck & Co Inc that includes: consulting or advisory, funding grants, and speaking and lecture fees. Helmut Butzkueven reports a relationship with F Hoffmann-La Roche Ltd that includes: consulting or advisory, funding grants, and speaking and lecture fees. Helmut Butzkueven reports a relationship with UCB Pharma SA that includes: consulting or advisory and speaking and lecture fees. Helmut Butzkueven reports a relationship with National Health and Medical Research Council that includes: funding grants. Helmut Butzkueven reports a relationship with Medical Research Future Fund that includes: funding grants. Helmut Butzkueven reports a relationship with Monash partners that includes: funding grants. Helmut Butzkueven reports a relationship with Trish MS research foundation that includes: funding grants. Helmut Butzkueven reports a relationship with Pennycook Foundation that includes: funding grants. Helmut Butzkueven reports a relationship with Multiple Sclerosis Research Australia that includes: funding grants. Helmut Butzkueven reports a relationship with MSBase Foundation that includes: consulting or advisory. Helmut Butzkueven reports a relationship with Oxford Health Policy Forum Brain Health Initiative that includes: consulting or advisory. Yi Chao Foong reports a relationship with National Health and Medical Research Council that includes: funding grants. Yi Chao Foong reports a relationship with Avant Foundation that includes: funding grants. Yi Chao Foong reports a relationship with Multiple Sclerosis Research Australia that includes: funding grants. Yi Chao Foong reports a relationship with Australian and New Zealand Association of Neurologists that includes: funding grants.
Effectiveness of physical therapy interventions in reducing fear of falling among individuals with neurologic diseases: a systematic review and meta-analysis.
Greater sensitivity to multiple sclerosis disability worsening and progression events using a roving versus a fixed reference value in a prospective cohort study.
Adherence and satisfaction of smartphone- and smartwatch-based remote active testing and passive monitoring in people with multiple sclerosis: nonrandomized interventional feasibility study.
Evaluating the utility of smartphone-based sensor assessments in persons with multiple sclerosis in the real-world using an App (elevateMS): observational, prospective pilot digital health study.
Practice effects of mobile tests of cognition, dexterity, and mobility on patients with multiple sclerosis: data analysis of a smartphone-based observational study.
00178-5&id=gr1.jpg){kind=link}