Teleneuropsychology in the time of COVID-19: The experience of The Australian Epilepsy Project

  • Chris Tailby
    Correspondence
    Corresponding author at: The Florey Institute of Neuroscience and Mental Health, Austin Campus, 245 Burgundy Street, Heidelberg 3084, VIC, Australia.
    Affiliations
    The Florey Institute of Neuroscience and Mental Health, Heidelberg, Australia

    Department of Clinical Neuropsychology, Austin Health, Heidelberg, Australia
    Search for articles by this author
  • Alana J. Collins
    Affiliations
    The Florey Institute of Neuroscience and Mental Health, Heidelberg, Australia
    Search for articles by this author
  • David N. Vaughan
    Affiliations
    The Florey Institute of Neuroscience and Mental Health, Heidelberg, Australia

    Department of Neurology, Austin Health, Heidelberg, Australia
    Search for articles by this author
  • David F. Abbott
    Affiliations
    The Florey Institute of Neuroscience and Mental Health, Heidelberg, Australia

    The Florey Department of Neuroscience and Mental Health, Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Parkville, Australia
    Search for articles by this author
  • Marie O’Shea
    Affiliations
    Department of Clinical Neuropsychology, Austin Health, Heidelberg, Australia

    School of Psychological Sciences, University of Melbourne, Parkville, Australia
    Search for articles by this author
  • Christoph Helmstaedter
    Affiliations
    Department of Epileptology, University Hospital Bonn (UKB), Bonn, Germany
    Search for articles by this author
  • Graeme D. Jackson
    Affiliations
    The Florey Institute of Neuroscience and Mental Health, Heidelberg, Australia

    Department of Neurology, Austin Health, Heidelberg, Australia

    The Florey Department of Neuroscience and Mental Health, Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Parkville, Australia
    Search for articles by this author
Open AccessPublished:October 15, 2020DOI:https://doi.org/10.1016/j.seizure.2020.10.005

      Highlights

      • TeleNP represents a medium through which clinical research and important clinical care can continue.
      • We discuss benefits, challenges, and practical considerations in the use of teleNP.
      • We describe how the Australian Epilepsy Project has adapted to the COVID-19 environment.
      • TeleNP data acquired to date reveals deficits typical of those observed in epilepsy.
      • TeleNP presents an opportunity to expand the reach and breadth of neuropsychological services.

      Abstract

      Purpose

      Traditional neuropsychological testing carries elevated COVID-19 risk for both examinee and examiner. Here we describe how the pilot study of the Australian Epilepsy Project (AEP) has transitioned to tele-neuropsychology (teleNP), enabling continued safe operations during the pandemic.

      Methods

      The AEP includes adults (age 18–60) with a first unprovoked seizure, new diagnosis of epilepsy or drug resistant focal epilepsy. Shortly after launching the study, COVID-related restrictions necessitated adaptation to teleNP, including delivery of verbal tasks via videoconference; visual stimulus delivery via document camera; use of web-hosted, computerised assessment; substitution of oral versions for written tests; online delivery of questionnaires; and discontinuation of telehealth incompatible tasks.

      Results

      To date, we have completed 24 teleNP assessments: 18 remotely (participant in own home) and six on-site (participant using equipment at research facility). Five face-to-face assessments were conducted prior to the transition to teleNP. Eight of 408 tests administered via teleNP (1.9 %) have been invalidated, for a variety of reasons (technical, procedural, environmental). Data confirm typical patterns of epilepsy-related deficits (p < .05) affecting processing speed, executive function, language and memory. Questionnaire responses indicate elevated rates of patients at high risk of mood (34 %) and anxiety disorder (38 %).

      Conclusion

      Research teleNP assessments reveal a typical pattern of impairments in epilepsy. A range of issues must be considered when introducing teleNP, such as technical and administrative set up, test selection and delivery, and cohort suitability. TeleNP enables large-scale neuropsychological research during periods of social distancing (and beyond), and offers an opportunity to expand the reach and breadth of neuropsychological services.

      Abbreviations:

      AEP (Australian Epilepsy Project), ASM (antiseizure medication), TeleNP (teleneuropsychology)

      Keywords

      1. Introduction

      The emergence of COVID-19 has resulted in significant changes in the day-to-day functioning of society the world over. Social distancing policies have had profound effects on people’s day-to-day movements and interpersonal interactions [
      • Fowler G.
      Smartphone data reveal which Americans are social distancing (and not).
      ] with enormous implications for workplaces, organisations, institutions and social activities. In the health sector this has meant that, wherever possible, clinical consultations have moved from face-to-face to a virtual environment (e.g. telephone, videoconferencing). It also appears that fear of infection is leading to avoidance of the hospital system by individuals who would otherwise seek medical care for conditions such as stroke and cardiac arrest [
      • Krumholz H.M.
      Where have all the heart attacks gone?.
      ,
      • Spinney L.
      Concern as heart attack and stroke patients delay seeking help.
      ].
      Thus, one might ask how can the health sector, including clinical research, best continue to operate in this environment? In this communication we describe how a large-scale clinical research project in epilepsy has adapted to the emergence of COVID-19, in recognition of the fact that face-to-face interactions in clinical research will have to be reduced for the foreseeable future. We focus on the use of teleneuropsychology (TeleNP) – the application of audiovisual technologies to enable remote clinical encounters with patients to conduct neuropsychological (NP) assessments [
      • Bilder R.M.
      • Postal K.S.
      • Barisa M.
      • Aase D.M.
      • Cullum C.M.
      • Gillaspy S.R.
      • et al.
      Inter organizational practice committee Recommendations/Guidance for teleneuropsychology in response to the COVID-19 pandemic.
      ] – to acquire research-based neuropsychological datasets. While our emphasis is on research-based data collection via teleNP, we also touch upon issues relevant to the clinical application of TeleNP in epilepsy.

      2. The Australian Epilepsy Project

      The Australian Epilepsy Project (AEP) is a large-scale clinical research project shortlisted for funding by the Medical Research Future Fund of the Australian Government. The vision of the AEP is to develop predictive epilepsy-specific decision support tools for use by clinicians. Machine learning / artificial intelligence (AI) methods will be applied to prospectively acquired neuropsychological, genetics and advanced imaging data obtained from 8000+ adults living with epilepsy, to predict their epilepsy-related two-year outcomes (Fig. 1). Sharing of de-identified datasets will further maximise breakthrough opportunities in research. We have commented elsewhere [
      • Pedersen M.
      • Verspoor K.
      • Jenkinson M.
      • Law M.
      • Abbott D.F.
      • Jackson G.D.
      Artificial intelligence for clinical decision support in neurology.
      ] on the role of machine learning/AI in the analysis of such datasets, and in the health sector more broadly, and do not consider this issue further here.
      Fig. 1
      Fig. 1Conceptual overview of multimodal data collection, integration, analysis, and clinical reporting in the Australian Epilepsy Project.
      The AEP commenced a pilot study in February of 2020 to evaluate recruitment feasibility and participant tolerability of the protocols for collection of neuropsychological and imaging data. The first case of COVID-19 was reported in Australia in late January 2020, and a State of Emergency was declared in Victoria in mid-March, around six weeks into AEP recruitment. Despite the introduction of Government-mandated COVID-related restrictions, the AEP pilot study has been able to continue by switching to the use of TeleNP for all participants.

      3. TeleNP can enable research in the era of COVID-19

      Prior to the arrival of COVID-19 in Australia, the AEP Pilot Study relied almost exclusively on face-to-face interactions for its data collection and analysis activities. The institutional and governmental response to COVID-19 in Australia demanded a re-evaluation and adjustment to each of these activities to ensure the safety of all persons involved, while preserving the scientific integrity and health care objectives. All our activities accord with guidance from Federal and State regulatory authorities, Institutional clinical governance, and local human research ethics committee approval. Box 1 provides a condensed description of our current operating protocol, outlining the key conceptual features.
      Data collection activities of the AEP Pilot Study during COVID-19.
      Referral
      • Recruitment occurs through neurologists during routine clinical practice. For the majority of patients this is now via telehealth consultation.
      Recruitment
      • Information provision and the establishing of informed consent are conducted via email and telephone. Participant consent is confirmed verbally and documented electronically and in writing by the researcher.
      Data collection
      • Cognition: Neuropsychological assessment is performed entirely via TeleNP. This includes a combination of neuropsychologist-supervised oral and computer-assisted testing, in conjunction with purely computer-administered web-based testing. Further detail is provided in the main text.
      • MRI: MRI is performed in-person, at research-dedicated scanners. All participants are screened for COVID-19 symptoms or risk factors before they attend the premises. On-site, physical distancing strategies, appropriate personal protective equipment use, and cleaning procedures are all applied according to up-dated research facility protocols. Images are transmitted electronically to the hospital radiology departments for standard reporting and clinical use. This approach eliminates the need for participants to physically attend hospital premises for clinical scans and reduces the burden on hospital radiology at a time of increased strain on the hospital system.
      • While genetics and epilepsy follow-up (e.g. seizure diaries, medications, psychological and quality of life questionnaires, adverse events, health economic data) are not collected in the AEP Pilot Study, in the full project genetic samples will be obtained by blood-draw at a local community pathology provider (typically at the same time as routine clinical blood tests), and epilepsy follow-up will occur via smart device app/web survey and telephone call.
      Reporting
      • Data analysis and transfer is performed by research staff accessing secure server platforms remotely via encrypted network connections, enabling this work to be safely performed from home. Key team decision making activities are supported via teleconferencing.

      4. Selecting epilepsy-relevant tools for TeleNP

      The most substantive protocol changes have involved the transition to collecting all neuropsychological data via TeleNP. Traditional, face-to-face neuropsychological testing carries elevated COVID-19 risk, both for participants and the neuropsychologist, and is clearly unacceptable from both a community safety and occupational health viewpoint. The examiner and examinee may spend several hours in close proximity, passing materials back and forth (e.g. stimulus materials, response forms), usually in a small enclosed room for privacy. Further, the examinee must also travel to the physical premises for the assessment, which can necessitate additional interpersonal interactions (e.g. public transport, waiting areas).
      The neuropsychological measures used in our TeleNP protocol are listed in Table 1. These measures were selected for their evidence base in epilepsy (as acquired through traditional, face-to-face assessments), and their compatibility with TeleNP administration. The experience gained from the AEP Pilot Study will be used to further empirically refine instrument selection.
      Table 1Neuropsychological measures used in the AEP Pilot TeleNP protocol.
      Test of Premorbid Functioning [
      • Wechsler D.
      Test of premorbid functioning. UK version (TOPF UK).
      ]: an estimate of premorbid intellect based on irregular word reading, used here both as a measure of intellect, and due to the elevated incidence of reading disorders in epilepsy [
      • Schachter S.C.
      • Galaburda A.M.
      • Ransil B.J.
      Associations of dyslexia with epilepsy, handedness, and gender.
      ,
      • Tailby C.
      • Weintrob D.L.
      • Saling M.M.
      • Fitzgerald C.
      • Jackson G.D.
      Reading difficulty is associated with failure to lateralize temporooccipital function.
      ].
      WASI-II FSIQ 2 (Matrix Reasoning, Vocabulary) [
      • Wechsler D.
      Manual for the Wechsler abbreviated intelligence scale (WASI).
      ]*: a short form intelligence measure with excellent psychometric properties.
      Oral Symbol Digit Modalities Test [
      • Hinton-Bayre A.
      • Geffen G.
      Comparability, reliability, and practice effects on alternate forms of the Digit Symbol Substitution and Symbol Digit Modalities tests.
      ]: the SDMT is a sensitive measure of processing speed and, in its written form, has been used widely in epilepsy trials [
      • Dodrill C.B.
      A neuropsychological battery for epilepsy.
      ].
      Oral Trail Making Test [
      • Army US
      Army individual test battery. Manual of Directions and Scoring.
      ]: a measure of divided attention and speed, sensitive (in its original written form) to dysfunction in epilepsy [
      • Dodrill C.B.
      A neuropsychological battery for epilepsy.
      ,
      • Lutz M.T.
      • Helmstaedter C.
      EpiTrack: tracking cognitive side effects of medication on attention and executive functions in patients with epilepsy.
      ].
      Reverse Digit Span [
      • Wechsler D.
      WAIS-IV technical and interpretive manual.
      ]*: a measure of working memory, sensitive to dysfunction in epilepsy [
      • Lutz M.T.
      • Helmstaedter C.
      EpiTrack: tracking cognitive side effects of medication on attention and executive functions in patients with epilepsy.
      ].
      Verbal Fluency [
      • Benton L.A.
      • Hamsher K.D.
      • Sivan A.B.
      Controlled oral word association test.
      ]*: letter and category based verbal fluency are sensitive markers of dysfunction in epilepsy, both in focal epilepsy [
      • Metternich B.
      • Buschmann F.
      • Wagner K.
      • Schulze-Bonhage A.
      • Kriston L.
      Verbal fluency in focal epilepsy: a systematic review and meta-analysis.
      ] and as an effect of antiseizure medications [
      • Lutz M.T.
      • Helmstaedter C.
      EpiTrack: tracking cognitive side effects of medication on attention and executive functions in patients with epilepsy.
      ], and also useful as a predictor of cognitive risk from surgery [
      • Sarkis R.A.
      • Busch R.M.
      • Floden D.
      • Chapin J.S.
      • Kalman Kenney C.
      • Jehi L.
      • et al.
      Predictors of decline in verbal fluency after frontal lobe epilepsy surgery.
      ].
      Boston Naming Test [
      • Kaplan E.
      • Goodglass H.
      • Weintraub S.
      The Boston naming test.
      ]*: a measure of confrontation naming, sensitive to lateralised dysfunction in focal epilepsy [
      • Loring D.W.
      • Strauss E.
      • Hermann B.P.
      • Barr W.B.
      • Perrine K.
      • Trenerry M.R.
      • et al.
      Differential neuropsychological test sensitivity to left temporal lobe epilepsy.
      ,
      • Umfleet L.G.
      • Janecek J.K.
      • Quasney E.
      • Sabsevitz D.S.
      • Ryan J.J.
      • Binder J.R.
      • et al.
      Sensitivity and specificity of memory and naming tests for identifying left temporal-lobe epilepsy.
      ], and also useful as a predictor of cognitive risk from surgery [
      • Hermann B.
      • Davies K.
      • Foley K.
      • Bell B.
      Visual confrontation naming outcome after standard left anterior temporal lobectomy with sparing versus resection of the superior temporal gyrus: a randomized prospective clinical trial.
      ].
      Rey Auditory Verbal Learning Test (learning and delayed recall trials)*: a word list learning task sensitive to memory dysfunction across a range of epilepsy syndromes [
      • Witt J.-A.
      • Helmstaedter C.
      Should cognition be screened in new-onset epilepsies? A study in 247 untreated patients.
      ], to lateralisation in temporal lobe epilepsy [
      • Loring D.W.
      • Strauss E.
      • Hermann B.P.
      • Barr W.B.
      • Perrine K.
      • Trenerry M.R.
      • et al.
      Differential neuropsychological test sensitivity to left temporal lobe epilepsy.
      ,
      • Saling M.M.
      Verbal memory in mesial temporal lobe epilepsy: beyond material specificity.
      ], and predictive of post-operative cognitive outcome [
      • Baxendale S.
      • Thompson P.
      • Harkness W.
      • Duncan J.
      Predicting memory decline following epilepsy surgery: a multivariate approach.
      ].
      CANTAB Connect (web-based testing) [
      • CANTAB®
      Cognitive assessment software.
      ]: a computerised assessment battery with an emphasis on executive functions. While the CANTAB does not have an extensive research base in epilepsy [
      • Witt J.-A.
      • Alpherts W.
      • Helmstaedter C.
      Computerized neuropsychological testing in epilepsy: overview of available tools.
      ], it has been used widely in other neurological conditions and is sensitive to frontal and temporal lobe dysfunction [
      • Strauss E.
      • Sherman E.M.S.
      • Spreen O.
      A compendium of neuropsychological tests: administration, norms and commentary.
      ]. We include the following measures from the CANTAB:
      • Spatial Working Memory: a measure of spatial working memory; shown to predict postoperative psychological outcomes in epilepsy [
        • Pope R.A.
        • Thompson P.J.
        • Rantell K.
        • Stretton J.
        • Wright M.-A.
        • Foong J.
        Frontal lobe dysfunction as a predictor of depression and anxiety following temporal lobe epilepsy surgery.
        ].
      • Rapid Visual Information Processing: a measure of information processing speed and sustained attention; computerised assessment of sustained attention has been recommended for large scale epilepsy research [
        • Baker G.A.
        • Marson A.G.
        Cognitive and behavioural assessments in clinical trials: what type of measure?.
        ].
      • Intra/Extra Dimensional Set Shift: a measure of cognitive flexibility and set shifting
      • One Touch Stockings of Cambridge: a measure of planning and problem solving
      • Paired Associate Learning: a non-verbal measure of arbitrary associate learning. While numerous different measures of non-verbal memory have been used in epilepsy there is no broadly agreed upon, recommended measure [
        • Saling M.M.
        Verbal memory in mesial temporal lobe epilepsy: beyond material specificity.
        ,
        • Loring D.W.
        • Lowenstein D.H.
        • Barbaro N.M.
        • Fureman B.E.
        • Odenkirchen J.
        • Jacobs M.P.
        • et al.
        Common data elements in epilepsy research: development and implementation of the NINDS epilepsy CDE project.
        ,
        • Barr W.B.
        • Chelune G.J.
        • Hermann B.P.
        • Loring D.W.
        • Perrine K.
        • Strauss E.
        • et al.
        The use of figural reproduction tests as measures of nonverbal memory in epilepsy surgery candidates.
        ,
        • Barr W.
        • Morrison C.
        • Zaroff C.
        • Devinsky O.
        Use of the Brief Visuospatial Memory Test—revised (BVMT-R) in neuropsychological evaluation of epilepsy surgery candidates.
        ]. Arbitrary associate learning has been shown to be a sensitive marker of mesial temporal lobe function in temporal lobe epilepsy [
        • Saling M.M.
        Verbal memory in mesial temporal lobe epilepsy: beyond material specificity.
        ,
        • Saling M.M.
        • Berkovic S.F.
        • O’Shea M.F.
        • Kalnins R.
        • Darby D.
        • Bladin P.
        Lateralization of verbal memory and unilateral hippocampal sclerosis - evidence of task-specific effects.
        ,
        • Weintrob D.L.
        • Saling M.M.
        • Berkovic S.F.
        • Reutens D.C.
        Impaired verbal associative learning after resection of left perirhinal cortex.
        ,
        • Weintrob D.L.
        • Saling M.M.
        • Berkovic S.F.
        • Berlangieri S.U.
        • Reutens D.C.
        Verbal memory in left temporal lobe epilepsy: evidence for task-related localization.
        ]. The paired associate learning subtest of the CANTAB has been shown to be a sensitive marker of pathology in the mesial temporal region in other forms of neurological disease targeting the mesial temporal lobe [
        • Fowler K.S.
        • Saling M.M.
        • Conway E.L.
        • Semple J.M.
        • Louis W.J.
        Paired associate performance in the early detection of DAT.
        ].
      The following psychological measures, administered via REDCap [
      • Harris P.A.
      • Taylor R.
      • Thielke R.
      • Payne J.
      • Gonzalez N.
      • Conde J.G.
      Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support.
      ,
      • Harris P.A.
      • Taylor R.
      • Minor B.L.
      • Elliott V.
      • Fernandez M.
      • O’Neal L.
      • et al.
      The REDCap consortium: building an international community of software platform partners.
      ] online surveys, are also reviewed by the neuropsychologist during the TeleNP testing session:
      Neurological Disorders Depressions Inventory for Epilepsy (NDDI-E) [
      • Gilliam F.G.
      • Barry J.J.
      • Hermann B.P.
      • Meador K.J.
      • Vahle V.
      • Kanner A.M.
      Rapid detection of major depression in epilepsy: a multicentre study.
      ]*: a widely used mood screening tool developed specifically for epilepsy, validated against psychiatric interview determined clinical diagnosis [
      • Gilliam F.G.
      • Barry J.J.
      • Hermann B.P.
      • Meador K.J.
      • Vahle V.
      • Kanner A.M.
      Rapid detection of major depression in epilepsy: a multicentre study.
      ].
      Generalised Anxiety Disorder 7-item scale (GAD7) [
      • Spitzer R.L.
      • Kroenke K.
      • Williams J.B.
      • Löwe B.
      A brief measure for assessing generalized anxiety disorder: the GAD-7.
      ]*: a brief generalised anxiety screening tool that has been validated in epilepsy against psychiatric interview determined clinical diagnosis [
      • Seo J.-G.
      • Cho Y.W.
      • Lee S.-J.
      • Lee S.-J.
      • Kim J.-E.
      • Moon H.-J.
      • et al.
      Validation of the generalized anxiety disorder-7 in people with epilepsy: a MEPSY study.
      ,
      • Micoulaud-Franchi J.-A.
      • Lagarde S.
      • Barkate G.
      • Dufournet B.
      • Besancon C.
      • Trébuchon-Da Fonseca A.
      • et al.
      Rapid detection of generalized anxiety disorder and major depression in epilepsy: validation of the GAD-7 as a complementary tool to the NDDI-E in a French sample.
      ,
      • Scott A.J.
      • Sharpe L.
      • Thayer Z.
      • Miller L.A.
      • Hunt C.
      • MacCann C.
      • et al.
      Design and validation of two measures to detect anxiety disorders in epilepsy: the Epilepsy Anxiety Survey Instrument and its brief counterpart.
      ].
      Epilepsy Anxiety Survey Instrument (EASI) [
      • Scott A.J.
      • Sharpe L.
      • Thayer Z.
      • Miller L.A.
      • Hunt C.
      • MacCann C.
      • et al.
      Design and validation of two measures to detect anxiety disorders in epilepsy: the Epilepsy Anxiety Survey Instrument and its brief counterpart.
      ]: an anxiety screening tool designed specifically for use in epilepsy, validated against psychiatric interview determined clinical diagnosis.
      * Recommended measure in the NINDS Epilepsy Common Data Elements [
      • Loring D.W.
      • Lowenstein D.H.
      • Barbaro N.M.
      • Fureman B.E.
      • Odenkirchen J.
      • Jacobs M.P.
      • et al.
      Common data elements in epilepsy research: development and implementation of the NINDS epilepsy CDE project.
      ].
      In our pre−COVID face-to-face protocol we had been administering EpiTrack [
      • Lutz M.T.
      • Helmstaedter C.
      EpiTrack: tracking cognitive side effects of medication on attention and executive functions in patients with epilepsy.
      ] and the Reaction Time task from the CANTAB. The trail making test (TMT), inhibition task and maze task within EpiTrack cannot be administered via telehealth, and the Reaction Time task is not available via CANTAB Connect (indeed the variability in the hardware possessed by participants would almost certainly preclude accurate measurement of reaction times in any home delivered, web based platform). We include the oral version of the TMT in our telehealth protocol, as a measure comparable to the written TMT [
      • Mrazik M.
      • Millis S.
      • Drane D.L.
      The oral trail making test: effects of age and concurrent validity.
      ]. We have also trialled various versions of the Stroop task (Victoria Stroop [
      • Troyer A.K.
      • Leach L.
      • Strauss E.
      Aging and response inhibition: normative data for the Victoria stroop Test.
      ], Dodrill Stroop [
      • Dodrill C.B.
      A neuropsychological battery for epilepsy.
      ]), to use as a measure of inhibition, but have ultimately abandoned it due to insensitivity (Victoria version) and difficulty presenting the stimuli appropriately via videoconference (Dodrill version).

      5. Practical considerations for TeleNP

      The potential benefits of TeleNP have long been recognised, including convenience, user satisfaction, potential cost-reductions and improved access (geographic; availability of interpreter services [
      • Haralambous B.
      • Subramaniam S.
      • Hwang K.
      • Dow B.
      • LoGiudice D.
      A narrative review of the evidence regarding the use of telemedicine to deliver video-interpreting during dementia assessments for older people.
      ]). Nonetheless, the neuropsychological community has not uniformly embraced the adoption of TeleNP necessitated by the emergence of COVID-19. One of the most obvious concerns relates to whether TeleNP departs sufficiently from standardised face-to-face administration to invalidate test results and interpretation. There is accumulating evidence that telehealth delivered neuropsychological assessments can yield reliable and valid evaluations [
      • Brearly T.W.
      • Shura R.D.
      • Martindale S.L.
      • Lazowski R.A.
      • Luxton D.D.
      • Shenal B.V.
      • et al.
      Neuropsychological test administration by videoconference: a systematic review and meta-analysis.
      ,
      • Grosch M.C.
      • Gottlieb M.C.
      • Cullum C.M.
      Initial practice recommendations for teleneuropsychology.
      ,
      • Wadsworth H.E.
      • Dhima K.
      • Womack K.B.
      • Hart Jr J.
      • Weiner M.F.
      • Hynan L.S.
      • et al.
      Validity of teleneuropsychological assessment in older patients with cognitive disorders.
      ]. Since the emergence of COVID-19, a number of journal articles [
      • Bilder R.M.
      • Postal K.S.
      • Barisa M.
      • Aase D.M.
      • Cullum C.M.
      • Gillaspy S.R.
      • et al.
      Inter organizational practice committee Recommendations/Guidance for teleneuropsychology in response to the COVID-19 pandemic.
      ,
      • The Inter Organizational Practice Committee
      Recommendations/Guidance for teleneuropsychology (TeleNP) in response to the COVID-19 pandemic.
      ,
      • Hewitt K.C.
      • Loring D.W.
      Emory university telehealth neuropsychology development and implementation in response to the COVID-19 pandemic.
      ,
      • Marra D.E.
      • Hamlet K.M.
      • Bauer R.M.
      • Bowers D.
      Validity of teleneuropsychology for older adults in response to COVID-19: a systematic and critical review.
      ] and statements from professional bodies have provided guidelines and experience-based recommendations regarding the use of TeleNP (via position papers [

      British Psychological Society D of N. Division of Neuropsychology Professional Standards Unit Guidelines to colleagues on the use of Tele-neuropsychology 2020.

      ], webinars and online resources; see, for instance, the Australian Psychological Society [https://www.psychology.org.au/Event/21454], the International Neuropsychological Society [https://www.the-ins.org/webinars/], the Inter Organizational Practice Committee [https://iopc.online]). While the purpose of the present paper is not to provide a comprehensive review of TeleNP, we do provide a brief discussion on some issues relevant to our implementation.
      The best TeleNP evidence concerns the use of tasks that are predominantly verbal in nature [
      • Brearly T.W.
      • Shura R.D.
      • Martindale S.L.
      • Lazowski R.A.
      • Luxton D.D.
      • Shenal B.V.
      • et al.
      Neuropsychological test administration by videoconference: a systematic review and meta-analysis.
      ]. This encompasses the majority of measures we have selected for use (Table 1), and includes measures such as paragraph and word list learning tasks [
      • Cullum C.M.
      • Hynan L.S.
      • Grosch M.
      • Parikh M.
      • Weiner M.F.
      Teleneuropsychology: evidence for video teleconference-based neuropsychological assessment.
      ,
      • Stead A.
      • Vinson M.
      Cognitive assessment using face-to-face and videoconferencing methods.
      ,
      • Stain H.J.
      • Payne K.
      • Thienel R.
      • Michie P.
      • Carr V.
      • Kelly B.
      The feasibility of videoconferencing for neuropsychological assessments of rural youth experiencing early psychosis.
      ,
      • JACOBSEN S.E.
      • SPRENGER T.
      • ANDERSSON S.
      • KROGSTAD J.-M.
      Neuropsychological assessment and telemedicine: a preliminary study examining the reliability of neuropsychology services performed via telecommunication.
      ], verbal span/working memory tasks (such as digit span) [
      • Cullum C.M.
      • Hynan L.S.
      • Grosch M.
      • Parikh M.
      • Weiner M.F.
      Teleneuropsychology: evidence for video teleconference-based neuropsychological assessment.
      ,
      • Stead A.
      • Vinson M.
      Cognitive assessment using face-to-face and videoconferencing methods.
      ,
      • Stain H.J.
      • Payne K.
      • Thienel R.
      • Michie P.
      • Carr V.
      • Kelly B.
      The feasibility of videoconferencing for neuropsychological assessments of rural youth experiencing early psychosis.
      ,
      • JACOBSEN S.E.
      • SPRENGER T.
      • ANDERSSON S.
      • KROGSTAD J.-M.
      Neuropsychological assessment and telemedicine: a preliminary study examining the reliability of neuropsychology services performed via telecommunication.
      ], verbal fluency tasks [
      • Cullum C.M.
      • Hynan L.S.
      • Grosch M.
      • Parikh M.
      • Weiner M.F.
      Teleneuropsychology: evidence for video teleconference-based neuropsychological assessment.
      ,
      • Stead A.
      • Vinson M.
      Cognitive assessment using face-to-face and videoconferencing methods.
      ,
      • Stain H.J.
      • Payne K.
      • Thienel R.
      • Michie P.
      • Carr V.
      • Kelly B.
      The feasibility of videoconferencing for neuropsychological assessments of rural youth experiencing early psychosis.
      ,
      • JACOBSEN S.E.
      • SPRENGER T.
      • ANDERSSON S.
      • KROGSTAD J.-M.
      Neuropsychological assessment and telemedicine: a preliminary study examining the reliability of neuropsychology services performed via telecommunication.
      ,
      • Hildebrand R.
      • Chow H.
      • Williams C.
      • Nelson M.
      • Wass P.
      Feasibility of neuropsychological testing of older adults via videoconference: implications for assessing the capacity for independent living.
      ], and measures of crystallized intelligence (e.g. measures of word reading and vocabulary [
      • JACOBSEN S.E.
      • SPRENGER T.
      • ANDERSSON S.
      • KROGSTAD J.-M.
      Neuropsychological assessment and telemedicine: a preliminary study examining the reliability of neuropsychology services performed via telecommunication.
      ,
      • Hildebrand R.
      • Chow H.
      • Williams C.
      • Nelson M.
      • Wass P.
      Feasibility of neuropsychological testing of older adults via videoconference: implications for assessing the capacity for independent living.
      ,
      • Kirkwood K.T.
      • Peck D.F.
      • Bennie L.
      The consistency of neuropsychological assessments performed via telecommunication and face to face.
      ,
      • Rebchuk A.D.
      • Deptuck H.M.
      • O’Neill Z.R.
      • Fawcett D.S.
      • Silverberg N.D.
      • Field T.S.
      Validation of a novel telehealth administration protocol for the NIH toolbox-cognition battery.
      ]). There is also evidence for tasks that rely upon verbal responses to visually presented stimuli, such as visual confrontation naming [
      • Cullum C.M.
      • Hynan L.S.
      • Grosch M.
      • Parikh M.
      • Weiner M.F.
      Teleneuropsychology: evidence for video teleconference-based neuropsychological assessment.
      ,
      • Stead A.
      • Vinson M.
      Cognitive assessment using face-to-face and videoconferencing methods.
      ] and visuoperceptual reasoning tasks (e.g. WAIS Matrix Reasoning [
      • JACOBSEN S.E.
      • SPRENGER T.
      • ANDERSSON S.
      • KROGSTAD J.-M.
      Neuropsychological assessment and telemedicine: a preliminary study examining the reliability of neuropsychology services performed via telecommunication.
      ,
      • Hildebrand R.
      • Chow H.
      • Williams C.
      • Nelson M.
      • Wass P.
      Feasibility of neuropsychological testing of older adults via videoconference: implications for assessing the capacity for independent living.
      ]). While supported by good evidence, it is worth noting that purely verbal tasks do not guarantee immunity to issues when administered via telehealth. Transient interruptions of the connection can interfere, especially with ‘one shot’ (e.g. digit span) or timed tasks (e.g. oral versions of SDMT and TMT). Our experience to date has been that poor connections are often apparent from the outset, and in many instances can be remedied simply by re-establishing the call or asking other users on the network to minimise their own network usage (e.g. streaming). We have had one participant whose computer microphone proved to be faulty but were able to proceed by using a concurrent telephone call accompanying the computer’s video feed without appreciable lag (a solution, incidentally, that we have employed in clinical practice also). Another participant was unable to establish a videoconference link from home, despite repeated attempts, and ultimately completed their teleNP assessment onsite.
      More difficult to administer are tasks that require physical interaction with stimuli provided by the examiner (e.g. paper forms, three dimensional blocks). While there is some evidence for administration of such tasks via TeleNP (e.g. Grooved Pegboard [
      • JACOBSEN S.E.
      • SPRENGER T.
      • ANDERSSON S.
      • KROGSTAD J.-M.
      Neuropsychological assessment and telemedicine: a preliminary study examining the reliability of neuropsychology services performed via telecommunication.
      ]; written version of the Symbol Digit Modalities Test [
      • JACOBSEN S.E.
      • SPRENGER T.
      • ANDERSSON S.
      • KROGSTAD J.-M.
      Neuropsychological assessment and telemedicine: a preliminary study examining the reliability of neuropsychology services performed via telecommunication.
      ,
      • Settle J.R.
      • Robinson S.A.
      • Kane R.
      • Maloni H.W.
      • Wallin M.T.
      Remote cognitive assessments for patients with multiple sclerosis: a feasibility study.
      ], Complex Figure Copy and Recall Tests [
      • Galusha-Glasscock J.M.
      • Horton D.K.
      • Weiner M.F.
      • Cullum C.M.
      Video teleconference administration of the repeatable battery for the assessment of neuropsychological status.
      ], Clock Drawing Test [
      • Brearly T.W.
      • Shura R.D.
      • Martindale S.L.
      • Lazowski R.A.
      • Luxton D.D.
      • Shenal B.V.
      • et al.
      Neuropsychological test administration by videoconference: a systematic review and meta-analysis.
      ,
      • Grosch M.C.
      • Gottlieb M.C.
      • Cullum C.M.
      Initial practice recommendations for teleneuropsychology.
      ,
      • Galusha-Glasscock J.M.
      • Horton D.K.
      • Weiner M.F.
      • Cullum C.M.
      Video teleconference administration of the repeatable battery for the assessment of neuropsychological status.
      ]) we have ultimately elected not to include them for a variety of reasons (impracticalities of providing materials to participants: Grooved Pegboard; poor sensitivity and reliability: Rey Complex Figure [
      • Barr W.B.
      • Chelune G.J.
      • Hermann B.P.
      • Loring D.W.
      • Perrine K.
      • Strauss E.
      • et al.
      The use of figural reproduction tests as measures of nonverbal memory in epilepsy surgery candidates.
      ]; suitable oral version available: SDMT). In other instances, we opted to retain the conceptual element of a traditional pen-and-paper task, but change the mode of delivery and response (e.g. using the oral versions of the Trail Making Test and Symbol Digit Modalities Test [
      • Mrazik M.
      • Millis S.
      • Drane D.L.
      The oral trail making test: effects of age and concurrent validity.
      ,
      • JACOBSEN S.E.
      • SPRENGER T.
      • ANDERSSON S.
      • KROGSTAD J.-M.
      Neuropsychological assessment and telemedicine: a preliminary study examining the reliability of neuropsychology services performed via telecommunication.
      ,
      • Settle J.R.
      • Robinson S.A.
      • Kane R.
      • Maloni H.W.
      • Wallin M.T.
      Remote cognitive assessments for patients with multiple sclerosis: a feasibility study.
      ,
      • Wadsworth H.E.
      • Galusha-Glasscock J.M.
      • Womack K.B.
      • Quiceno M.
      • Weiner M.F.
      • Hynan L.S.
      • et al.
      Remote neuropsychological assessment in rural American Indians with and without cognitive impairment.
      ,
      • Strober L.
      • DeLuca J.
      • Benedict R.H.
      • Jacobs A.
      • Cohen J.A.
      • Chiaravalloti N.
      • et al.
      Symbol digit modalities test: a valid clinical trial endpoint for measuring cognition in multiple sclerosis.
      ]), though we note that this likely alters what the task is actually measuring (e.g. pen-and-paper versus oral Trail Making Test [
      • Mrazik M.
      • Millis S.
      • Drane D.L.
      The oral trail making test: effects of age and concurrent validity.
      ]).
      TeleNP is not without its challenges. Familiarity with the required technology – on the part of both the examinee and clinician – can influence the degree of engagement with, and the flow and ease of, the interaction. Indeed, we have found it essential to factor in ∼15 min of initial set up time at the beginning of appointments to ensure participants are able to log onto the videoconference call and that their technology is functioning appropriately (assisting them via phone as necessary). A single neuropsychologist administered the TeleNP assessments for our protocol, with this individual completing multiple supervised practice administrations prior to commencing participant data collection (and reviewing the aforementioned TeleNP webinars provided by the Australian Psychological Association and the International Neuropsychological Society once these were available). These practice sessions were essential to ensuring familiarity with the testing technology and practicalities of administration via telehealth. We have also developed a set of Standard Operating Procedures for telehealth to facilitate the training of new staff as the project expands.
      The suitability of TeleNP for specific patient groups is an important issue, such as paediatric populations, people with intellectual disability or severe cognitive compromise, and linguistically and culturally diverse groups. Indeed, many of these concerns are also relevant to traditional face-to-face consultations. This complexity has not yet been fully addressed by the field and remains a critical challenge to the broad application of clinical TeleNP. However, the acquisition of uniform test data for machine-learning analysis is a narrower problem, where these issues are partly avoided through assessment of a necessarily more targeted cohort in which TeleNP administration is appropriate.
      Access to technology is another issue of concern, since not all individuals possess the hardware required to support videoconference-based TeleNP. The use of technology at a local facility (e.g. GP clinic or research site) can increase availability and address issues of social equity, while simultaneously ensuring the quality of technology and connectivity [
      • Brearly T.W.
      • Shura R.D.
      • Martindale S.L.
      • Lazowski R.A.
      • Luxton D.D.
      • Shenal B.V.
      • et al.
      Neuropsychological test administration by videoconference: a systematic review and meta-analysis.
      ,
      • Grosch M.C.
      • Gottlieb M.C.
      • Cullum C.M.
      Initial practice recommendations for teleneuropsychology.
      ]. Indeed, the majority of evidence for telehealth administration comes from studies where examinees are tested via technology at a local research facility [
      • Brearly T.W.
      • Shura R.D.
      • Martindale S.L.
      • Lazowski R.A.
      • Luxton D.D.
      • Shenal B.V.
      • et al.
      Neuropsychological test administration by videoconference: a systematic review and meta-analysis.
      ]. We have made this approach available to participants, in order to improve participant access to the study (we offer free parking for participants who are able to travel to the facility by car and offer taxi vouchers for those who require transport). To date, roughly 20 % of participants have opted to complete their TeleNP testing on-site at the research facility (at the time of their MRI scan); the remainder have completed the TeleNP assessment using their own technology at home.
      Operational changes we implemented to enable TeleNP are outlined in Box 2. TeleNP appointments are conducted via Zoom (using a HIPAA compliant Education account; zoom.us), with the ‘password’ and ‘waiting room’ features enabled. TeleNP sessions are delivered by a qualified clinical neuropsychologist. For tasks using oral or screenshare-based stimulus delivery, responses are recorded by the examiner on original test record forms. The web-delivered, computer-administered CANTAB tests are recorded and scored by the software itself. Throughout the TeleNP session (including CANTAB testing), the examinee remains in audiovisual contact with the examiner, enabling monitoring of behaviours and the occurrence of potential distractions. To date, our TeleNP participants have responded positively, reporting the experience to be smooth and efficient, and appreciating the opportunity to carry out the assessment without leaving their home. Those who have previously undergone face-to-face assessment have noted the telehealth experience to be similar.
      TeleNP within the AEP.
      Prior to their TeleNP appointment participants are:
      • Screened to check that they have adequate technology to support TeleNP, defined at minimum as:
        • o
          Computer or iPad with web cam, microphone and internet connection (smartphones and non-iPad tablets are not suitable, given software requirements of the web-delivered computer-administered testing used in the AEP, see below).
        • o
          Quiet, distraction free room in which to complete the TeleNP assessment
      • Emailed a link to a set of electronically-hosted surveys tapping elements of psychosocial functioning germane to epilepsy (e.g. Neurological Disorders Depression in Epilepsy [
        • Gilliam F.G.
        • Barry J.J.
        • Hermann B.P.
        • Meador K.J.
        • Vahle V.
        • Kanner A.M.
        Rapid detection of major depression in epilepsy: a multicentre study.
        ]; Patient Health Questionnaire GAD-7 [
        • Spitzer R.L.
        • Kroenke K.
        • Williams J.B.
        • Löwe B.
        A brief measure for assessing generalized anxiety disorder: the GAD-7.
        ]; Epilepsy Anxiety Survey Instrument [
        • Scott A.J.
        • Sharpe L.
        • Thayer Z.
        • Miller L.A.
        • Hunt C.
        • MacCann C.
        • et al.
        Design and validation of two measures to detect anxiety disorders in epilepsy: the Epilepsy Anxiety Survey Instrument and its brief counterpart.
        ]; QoLiE-31 [
        • Cramer J.A.
        • Perrine K.
        • Devinsky O.
        • Bryant-Comstock L.
        • Meador K.
        • Hermann B.
        Development and cross-cultural translations of a 31-item quality of life in epilepsy inventory.
        ]; Liverpool Adverse Events Profile [
        • Baker G.
        Development of a patient-based symptom check list to quantify adverse effects in persons receiving antiepileptic drugs.
        ,
        • Baker G.A.
        Initial development, reliability and validity of a patient-based adverse drug event scale.
        ,
        • Baker G.A.
        The Liverpool adverse drug events profile.
        ]; ABNAS [
        • Brooks J.
        • Baker G.A.
        • Aldenkamp A.P.
        The A–B neuropsychological assessment schedule (ABNAS): the further refinement of a patient-based scale of patient-perceived cognitive functioning.
        ]). The surveys are hosted on a REDCap [
        • Harris P.A.
        • Taylor R.
        • Thielke R.
        • Payne J.
        • Gonzalez N.
        • Conde J.G.
        Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support.
        ,
        • Harris P.A.
        • Taylor R.
        • Minor B.L.
        • Elliott V.
        • Fernandez M.
        • O’Neal L.
        • et al.
        The REDCap consortium: building an international community of software platform partners.
        ] database server at our institution.
      • Emailed a telehealth information sheet, along with text describing an ‘agreement to telehealth’ whereby participant and researcher agree that they will not “record, reproduce, publish or make copies of the materials used during the neuropsychology telehealth session” [
        Pearson telepractice no objection letter.
        ]. Participants are advised that their TeleNP session cannot proceed until they confirm acceptance of this agreement.
      At the beginning of the TeleNP session the neuropsychologist:
      • Verifies the participant’s identity
      • Checks the participant’s telehealth technology setup (e.g. microphone and webcam setup and testing; quality of connection; disabling of other apps and notifications; suitability of environment)
      • Explains what will happen in the event of lost connection (attempt to reconnect; if unable to, will call mobile phone; if no contact within 10 min, session considered aborted and will be rescheduled)
      • Confirms the participant’s current location and obtains additional contact information in event of emergency (e.g. seizure). We explain that in the event of a seizure, if we cannot reach one of the contacts provided, or if we feel a more urgent response is appropriate, that we will call an ambulance. This information is summarised in a teleNP information sheet provided to all participants in advance of their session. We are yet to have a participant experience a seizure during testing.
      • Re-iterates terms of agreement to participate in telehealth (e.g. participant will not record or reproduce any materials)
      Neuropsychological testing is administered via the following methods [
      • Brearly T.W.
      • Shura R.D.
      • Martindale S.L.
      • Lazowski R.A.
      • Luxton D.D.
      • Shenal B.V.
      • et al.
      Neuropsychological test administration by videoconference: a systematic review and meta-analysis.
      ,
      • The Inter Organizational Practice Committee
      Recommendations/Guidance for teleneuropsychology (TeleNP) in response to the COVID-19 pandemic.
      ,

      British Psychological Society D of N. Division of Neuropsychology Professional Standards Unit Guidelines to colleagues on the use of Tele-neuropsychology 2020.

      ,
      • Cullum C.M.
      • Hynan L.S.
      • Grosch M.
      • Parikh M.
      • Weiner M.F.
      Teleneuropsychology: evidence for video teleconference-based neuropsychological assessment.
      ,
      • Hildebrand R.
      • Chow H.
      • Williams C.
      • Nelson M.
      • Wass P.
      Feasibility of neuropsychological testing of older adults via videoconference: implications for assessing the capacity for independent living.
      ,
      Pearson telepractice no objection letter.
      ,
      • Temple V.
      • Drummond C.
      • Valiquette S.
      • Jozsvai E.
      A comparison of intellectual assessments over video conferencing and in-person for individuals with ID: preliminary data.
      ,
      • Luxton D.D.
      • Pruitt L.D.
      • Osenbach J.E.
      Best practices for remote psychological assessment via telehealth technologies.
      ]:
      • 1
        via ScreenShare linked to a high resolution document camera: Test of Premorbid Function [
        • Wechsler D.
        Test of premorbid functioning. UK version (TOPF UK).
        ]; Matrix Reasoning from WASI-II [
        • Wechsler D.
        Manual for the Wechsler abbreviated intelligence scale (WASI).
        ]; Boston Naming Test [
        • Kaplan E.
        • Goodglass H.
        • Weintraub S.
        The Boston naming test.
        ]; oral Symbol Digit Modalities Test [
        • Strober L.
        • DeLuca J.
        • Benedict R.H.
        • Jacobs A.
        • Cohen J.A.
        • Chiaravalloti N.
        • et al.
        Symbol digit modalities test: a valid clinical trial endpoint for measuring cognition in multiple sclerosis.
        ].
      • 2
        via oral stimulus delivery: letter and category verbal fluency, reverse digit span, oral Trail Making Test [
        • Mrazik M.
        • Millis S.
        • Drane D.L.
        The oral trail making test: effects of age and concurrent validity.
        ], Rey Auditory Verbal Learning Test [
        • Strauss E.
        • Sherman E.M.S.
        • Spreen O.
        A compendium of neuropsychological tests: administration, norms and commentary.
        ], Vocabulary from WASI-II [
        • Wechsler D.
        Manual for the Wechsler abbreviated intelligence scale (WASI).
        ].
      • 3
        via web-delivered computer-administered testing: using the CANTAB [
        • CANTAB®
        Cognitive assessment software.
        ] web platform. While testing via this platform can be completed by the participant in a standalone manner, we have the examiner remain on the videocall throughout, to handle any unanticipated problems that might arise and also to monitor behaviour during the testing.
      For each task, the neuropsychologist records observations of anything that might invalidate a test (e.g. temporary connection loss; distraction). All data is recorded on response forms coding using a random six digit participant identification code, and then transcribed onto a secure central database (REDCap).

      6. TeleNP reveals a typical pattern of impairments in epilepsy

      Fig. 2 summarises the cognitive data we have acquired at the time of writing. Given the relatively small sample to date (n = 29), data have not been subgrouped according to AEP referral type (first unprovoked seizure: n = 17, new diagnosis epilepsy: n = 6, refractory epilepsy: n = 6), or into method of neuropsychological test administration (teleNP-home: n = 18, teleNP-onsite: n = 6; face-to-face: n = 5). Data are expressed as z-scores, calculated relative to normative data. The distributions shown in Fig. 2 are as expected for an epilepsy cohort, with reductions in the domains of processing speed, working memory, executive function, language, and anterograde memory [
      • Lutz M.T.
      • Helmstaedter C.
      EpiTrack: tracking cognitive side effects of medication on attention and executive functions in patients with epilepsy.
      ,
      • Witt J.-A.
      • Helmstaedter C.
      Should cognition be screened in new-onset epilepsies? A study in 247 untreated patients.
      ].
      Fig. 2
      Fig. 2Boxplots summarising performance across the sample of participants collected to date, colour coded according to cognitive domain. ToPF = Test of Premorbid Functioning; FSIQ = WASI-II FSIQ 2; SDMT = Symbol Digit Modalities Test; RVP = CANTAB Rapid Visual Information Processing; DSB = Digit Span Backwards; SWM = CANTAB Spatial Working Memory; SWM: be468 = SWM between errors; SWM: Strategy = SWM strategy score; TMT = Trail Making Test; OTS = CANTAB One Touch Stockings of Cambridge; OTS: 1st Try = OTS problems solved on first choice; IED = CANTAB Intra/Extradimensional Set Shift; IED: yerta = IED total errors adjusted; COWAT = letter fluency; Animals = animal fluency; BNT = Boston Naming Test; RAVLT = Rey Auditory Verbal Learning Test; RAVLT: loss = words lost between trial 3 and delay; PAL = CANTAB Paired Associate Learning; PAL: tea28 = PAL total errors adjusted; PAL: fams28 = PAL first attempt memory score. RAVLT z-scores are derived from a local sample of 72 participants: mean age = 30.8 ± 10.9 years, total words recalled across trials A1-A3 = 26.8 ± 5.3; delay score = 9.0 ± 2.8; words lost across delay = 2.1 ± 1.6. Among the notably reduced (z < -4) BNT scores in four participants, one is from an individual in whom English is a second language, spoken for approximately 4 years; one is from an individual with a suspected left parietal focus; one is from an individual with a suspected reading difficulty; and one is from an individual with suspected left TLE.
      Table 2 summarises the data for each cognitive task. Performance is considered for the sample as a whole (overall mean z-score, SD, n and result of one sample t-test/Wilcoxon signed rank test [relative to a mean/median z-score = 0, one-sided test], and percentage of cases performing >1.5 SDs below expectation), and also separately for each method of administration: teleNP-at home, teleNP-onsite, face-to-face (n for each method of administration, result of a one-way ANOVA/Kruskall-Wallis test).
      Table 2Neuropsychological test scores for the overall sample, and separately for each method of administration (at home TeleNP; onsite TeleNP, and face-to-face).
      TESTμSDp (t/T)nn: z<-1.5 (%)μ(home)μ(onsite)μ(f2f)p (F/H)
      ToPF0.340.750.99290 (0.0)0.59 (18)−0.19 (6)0.11 (5)0.07
      FSIQ-2−0.110.820.23281 (3.6)0.06 (17)−0.57 (6)−0.16 (5)0.36
      RVP: a’−0.250.890.07282 (7.1)−0.08 (18)−0.97 (5)−0.16 (5)0.47
      Oral SDMT−1.131.180.0193 (33.3)−0.93 (8)−2.73 (1)NaN (0)0.16
      ^DSB−0.100.970.07240 (0.0)−0.15 (18)0.06 (6)NaN (0)0.71
      ^SWM: be468−0.211.450.15283 (10.7)−0.35 (18)0.04 (5)0.02 (5)0.68
      ^SWM: Strategy−0.161.120.08282 (7.1)−0.15 (18)0.02 (5)−0.41 (5)0.78
      ^Oral TMT B−0.930.910.00247 (29.2)−0.75 (18)−1.45 (6)NaN (0)0.11
      OTS: 1st Try0.240.990.89280 (0.0)0.31 (18)0.27 (5)−0.04 (5)0.53
      ^IED: yerta−0.140.940.29282 (7.1)−0.04 (18)−0.27 (5)−0.40 (5)0.67
      COWAT−1.300.870.002411 (45.8)−1.11 (18)−1.86 (6)NaN (0)0.07
      Animals−0.371.400.09295 (17.2)−0.26 (18)−0.56 (6)−0.52 (5)0.65
      ^BNT−1.771.650.002813 (46.4)−1.90 (18)−1.76 (6)−1.18 (4)0.89
      RAVLT: sum(A1-A3)−0.921.050.00298 (27.6)−0.77 (18)−1.22 (6)−1.10 (5)0.42
      RAVLT: delay−1.271.280.002914 (48.3)−1.02 (18)−1.69 (6)−1.66 (5)0.23
      RAVLT: loss−1.071.270.002912 (41.4)−0.97 (18)−1.29 (6)−1.18 (5)0.66
      PAL: tea28−0.121.170.29283 (10.7)−0.06 (18)−0.54 (2)0.05 (5)0.95
      PAL: fams28−0.241.240.15284 (14.3)−0.18 (18)−0.72 (2)0.01 (5)0.97
      ^ = nonparametric test (Wilcoxon, Kruskall-Wallis), used when Shapiro-Wilk test p < .05; “p (t/T)” column reports p-value from one sample t-test (T-statistic if nonparametric test used) on z-scores collapsed across administration method; “n z < -1.5 (%)” = number (and percentage) of participants with z < -1.5 (expected in 6.7 % of a normally distributed sample); “home” = teleNP with participant at home; “onsite” = teleNP with participant at research site; “f2f” = traditional face-to-face neuropsychological testing; parenthetic numbers in “μ(home)”,” μ(onsite)”, and “μ(f2f)” columns report n for those conditions; “p (F/H)” column reports the p-value from a one-way ANOVA (based on F; H-value if nonparametric test used) assessing effect of administration method; ToPF = Test of Premorbid Functioning; FSIQ-2 = WASI-II FSIQ 2; SDMT = Symbol Digit Modalities Test; RVP = CANTAB Rapid Visual Information Processing; DSB = Digit Span Backwards; SWM = CANTAB Spatial Working Memory; SWM: be468 = SWM between errors; SWM: Strategy = SWM strategy score; TMT = Trail Making Test; OTS = CANTAB One Touch Stockings of Cambridge; OTS: 1st Try = OTS problems solved on first choice; IED = CANTAB Intra/Extradimensional Set Shift; IED: yerta = IED total errors adjusted; COWAT = letter fluency; Animals = animal fluency; BNT = Boston Naming Test; RAVLT = Rey Auditory Verbal Learning Test; RAVLT: loss = words lost between trial 3 and delay; PAL = CANTAB Paired Associate Learning; PAL: tea28 = PAL total errors adjusted; PAL: fams28 = PAL first attempt memory score. RAVLT z-scores are derived from a local sample of 72 participants: mean age = 30.8 ± 10.9 years, total words recalled across trials A1-A3 = 26.8 ± 5.3; delay score = 9.0 ± 2.8; words lost across delay = 2.1 ± 1.6.
      The ANOVA/Kruskall-Wallis tests indicate that no cognitive test shows a significant effect of administration method (p > .05, albeit with small group sizes). Collapsed across administration method, one sample t-tests/Wilcoxon signed rank tests confirm a pattern of impairments typical of those seen in epilepsy, with significant (p < 0.05) reductions in processing speed (oral SDMT), executive function (oral TMT B, COWAT), language (BNT, Animal fluency) and anterograde memory (RAVLT measures). Trends (p < 0.08) were also apparent for sustained rapid information processing (CANTAB RVP) and working memory (DSB, CANTAB SWM).
      Screening of mood and anxiety also confirmed a relatively high proportion of individuals at risk for these disorders: 10 of 29 participants (34 %) were at high risk of mood disorder based on the NDDIE [total score > 15, see reference
      • Gilliam F.G.
      • Barry J.J.
      • Hermann B.P.
      • Meador K.J.
      • Vahle V.
      • Kanner A.M.
      Rapid detection of major depression in epilepsy: a multicentre study.
      ]; and between 8 and 11 (28 % and 38 %) were at high risk of anxiety disorder based on the brEASI [total score > 7, see reference
      • Scott A.J.
      • Sharpe L.
      • Thayer Z.
      • Miller L.A.
      • Hunt C.
      • MacCann C.
      • et al.
      Design and validation of two measures to detect anxiety disorders in epilepsy: the Epilepsy Anxiety Survey Instrument and its brief counterpart.
      ] and GAD7 [total score > 7, see reference
      • Micoulaud-Franchi J.-A.
      • Lagarde S.
      • Barkate G.
      • Dufournet B.
      • Besancon C.
      • Trébuchon-Da Fonseca A.
      • et al.
      Rapid detection of generalized anxiety disorder and major depression in epilepsy: validation of the GAD-7 as a complementary tool to the NDDI-E in a French sample.
      ], respectively.

      7. The future of TeleNP in epilepsy research and clinical trials

      The social distancing requirements stemming from COVID-19 are likely to be with us for a long time. This highlights the importance of expanding the testing options available to neuropsychology, by developing assessment tools explicitly designed for use via telehealth. Such developments would be of great benefit even once the need for social distancing has passed, improving access to neuropsychological services. Existing, evidence-based screening tools already used in epilepsy (e.g. EpiTrack [
      • Lutz M.T.
      • Helmstaedter C.
      EpiTrack: tracking cognitive side effects of medication on attention and executive functions in patients with epilepsy.
      ]) could be adapted and validated for delivery via online platforms. In the process, such tools could be extended, ensuring coverage of epilepsy relevant neuropsychological domains and exploiting the response sampling available via a computerised medium. The guiding principle should be to target those domains most affected or important to people with epilepsy [
      • Vogt V.L.
      • Äikiä M.
      • Del Barrio A.
      • Boon P.
      • Borbély C.
      • Bran E.
      • et al.
      Current standards of neuropsychological assessment in epilepsy surgery centers across Europe.
      ], using measures sensitive to the lifetime variability of the condition, from disease onset through introduction of anti-seizure medications (ASMs) to chronic refractoriness and surgery. These domains include:
      • Anterograde memory: the most common cognitive complaint in epilepsy [
        • Hall K.E.
        • Isaac C.L.
        • Harris P.
        Memory complaints in epilepsy: an accurate reflection of memory impairment or an indicator of poor adjustment? A review of the literature.
        ]; the majority of focal epilepsies affect the temporal lobe [
        • Semah F.
        • Picot M.-C.
        • Adam C.
        • Broglin D.
        • Arzimanoglou A.
        • Bazin B.
        • et al.
        Is the underlying cause of epilepsy a major prognostic factor for recurrence?.
        ].
      • Executive functions/fluid intelligence: sensitive to ASM effects [
        • Lutz M.T.
        • Helmstaedter C.
        EpiTrack: tracking cognitive side effects of medication on attention and executive functions in patients with epilepsy.
        ]; the frontal lobes are frequently involved in focal [
        • Semah F.
        • Picot M.-C.
        • Adam C.
        • Broglin D.
        • Arzimanoglou A.
        • Bazin B.
        • et al.
        Is the underlying cause of epilepsy a major prognostic factor for recurrence?.
        ] and genetic [
        • Baulac S.
        Genetics advances in autosomal dominant focal epilepsies: focus on DEPDC5. Progress in brain research, vol. 213.
        ] epilepsies.
      • Crystallized intelligence: considered less susceptible to ASM effects [
        • Helmstaedter C.
        • Witt J.-A.
        • Hoppe C.
        Evaluating the mediating role of executive functions for antiepileptic drugs’ effects on IQ in children and adolescents with epilepsy.
        ]; can be affected by age of epilepsy onset [
        • Strauss E.
        • Loring D.
        • Chelune G.
        • Hunter M.
        • Hermann B.
        • Perrine K.
        • et al.
        Predicting cognitive impairment in epilepsy: findings from the bozeman epilepsy consortium.
        ]; provides a measure of cognitive reserve [
        • Helmstaedter C.
        • Kurthen M.
        • Lux S.
        • Reuber M.
        • Elger C.E.
        Chronic epilepsy and cognition: a longitudinal study in temporal lobe epilepsy.
        ].
      • Mood: frequently disturbed in people with epilepsy [
        • Kanner A.M.
        Depression in epilepsy: prevalence, clinical semiology, pathogenic mechanisms, and treatment.
        ].
      • Adverse treatment side effects
      Such screening assessments cannot replace comprehensive assessments and may miss subtle problems for some individuals. However, unlike other existing tools that have been developed for dementia screening, the tools would be validated for epilepsy, be age and education adjusted, address functions most often affected during the course of epilepsy and its treatment, and be useable for the remote assessment of patients unable to attend face-to-face assessment or who would otherwise be lost to follow-up. Ultimately this kind of approach will facilitate large-scale collection of data that would not otherwise be practical using traditional methods.

      8. TeleNP in the clinical management of epilepsy

      To this point we have emphasised the role of TeleNP in research. What of the role of TeleNP in clinical care? While COVID-19 strains the health systems of many countries around the world, individuals nonetheless continue to experience seizures and associated cognitive and psychological comorbidities. In response to COVID-19 and the rush towards telehealth, the ILAE Neuropsychology Task Force underscored that comprehensive telehealth neuropsychological assessments for epilepsy surgery candidates have not yet been carried out or validated [

      International League Against Epilepsy Neuropsycholoigcal assessments for epilepsy surgery during COVID-19 Restrictions n.d.

      ]. The Task Force recommends that any surgical candidates proceed to surgery only after a comprehensive, face-to-face, neuropsychological work up, concluding that, “whilst compromise and new ways of working are necessary for urgent neurosurgical procedures, epilepsy surgery should not be conducted as an emergency procedure.” While this position aspires to an ideal, the effects of COVID-19 will in all likelihood be with us for a long time yet, and epilepsy surgery cannot simply cease to occur for the foreseeable future, especially not if the only barrier is failure to carry out a neuropsychological evaluation. In our view, a reasonable position would accommodate TeleNP using epilepsy-relevant, evidence-based instruments, augmented by shorter face-to-face assessments where required to address specific clinical issues.
      The impetus to move towards telehealth stimulated by COVID-19 should be viewed as an opportunity to expand the reach and breadth of neuropsychology [
      • Brearly T.W.
      • Shura R.D.
      • Martindale S.L.
      • Lazowski R.A.
      • Luxton D.D.
      • Shenal B.V.
      • et al.
      Neuropsychological test administration by videoconference: a systematic review and meta-analysis.
      ,
      • Cullum C.M.
      • Hynan L.S.
      • Grosch M.
      • Parikh M.
      • Weiner M.F.
      Teleneuropsychology: evidence for video teleconference-based neuropsychological assessment.
      ]. The clinical neuropsychologist’s role in an epilepsy surgery program extends beyond psychometric documentation. The delivery of counselling, psychoeducation, advocacy, and psychotherapy via telehealth has a solid evidence base [
      • Backhaus A.
      • Agha Z.
      • Maglione M.L.
      • Repp A.
      • Ross B.
      • Zuest D.
      • et al.
      Videoconferencing psychotherapy: a systematic review.
      ], including in epilepsy specific contexts [
      • Meyer B.
      • Weiss M.
      • Holtkamp M.
      • Arnold S.
      • Brückner K.
      • Schröder J.
      • et al.
      Effects of an epilepsy-specific Internet intervention (Emyna) on depression: results of the ENCODE randomized controlled trial.
      ,
      • Bahrani K.
      • Singh M.B.
      • Bhatia R.
      • Prasad K.
      • Vibha D.
      • Shukla G.
      • et al.
      Telephonic review for outpatients with epilepsy—a prospective randomized, parallel group study.
      ,
      • Rasmusson K.A.
      • Hartshorn J.C.
      A comparison of epilepsy patients in a traditional ambulatory clinic and a telemedicine clinic.
      ,
      • Ahmed S.N.
      • Mann C.
      • Sinclair D.B.
      • Heino A.
      • Iskiw B.
      • Quigley D.
      • et al.
      Feasibility of epilepsy follow-up care through telemedicine: a pilot study on the patient’s perspective.
      ], and is encouraged in recent set of epilepsy-specific consensus recommendations [
      • French J.A.
      • Brodie M.J.
      • Caraballo R.
      • Devinsky O.
      • Ding D.
      • Jehi L.
      • et al.
      Keeping people with epilepsy safe during the Covid-19 pandemic.
      ]. Speaking from the local perspective, within the Department of Clinical Neuropsychology at Austin Health we already employ telehealth (telephone, videoconferencing using the Coviu platform) routinely in the pre- and post-surgical counselling of patients [
      • Wilson Sj
      • Kincade P.
      • Saling Mm
      • Bladin Pf.
      Patient readmission and support utilization following anterior temporal lobectomy.
      ]. Anecdotally, a number of our patients have commented that they feel more comfortable in their home environment, and find it easier to be open about their experiences when communicating through the intermediary of technology. The average age of our surgical cohort is in the mid-30′s representing a generation for whom technology is ‘second-nature’. Approximately 20 % of our patients live outside of metropolitan centres. Further expanding the role of TeleNP would be of great benefit to such a patient demographic, improving timely access to care. For example, it would enable important post-operative evaluations and follow-up without the burden of travel and the attendant costs and psychosocial disruption.
      These anecdotal benefits are substantiated by our recent experience in Germany using phone or videoconference telemedicine in the counselling of people with epilepsy during the COVID pandemic [
      • von Wrede R.
      • Moskau-Hartmann S.
      • Baumgartner T.
      • Helmstaedter C.
      • Surges R.
      Counseling of people with epilepsy via telemedicine: experiences at a German tertiary epilepsy center during the COVID-19 pandemic.
      ]. Overall 82 % of the 239 adult epilepsy patients participating in the audit were satisfied with their telemedicine experience, with high rates of satisfaction especially for time, comprehensibility, and opportunity to get answers to current questions. The participants considered immediate convenience and shortfall of travel expenses as advantages of telemedicine. Approximately three quarters of participants reported that they would appreciate the opportunity for future telemedical counselling, but at the same consider telemedicine as an add-on service rather than a permanent substitute to visits onsite.

      9. Conclusion

      COVID-19 has abruptly and dramatically changed the way that society functions, including the operation of the health and medical research sector. Our experience shows that it is possible to continue to perform evidence-based, epilepsy-related neuropsychological research while at the same time fully supporting public health strategies aimed at containing and mitigating the effects of COVID-19. In the event that sustaining such policies into the medium or longer-term is necessary, the strategies adopted by the AEP have positioned it to continue to grow and expand, with no impact on the feasibility, integrity or safety of the project. Indeed, this model of telehealth-based operations provides a template for the healthcare of tomorrow, while decreasing the burden on traditional hospital systems. The challenges posed by COVID-19 are immense, and we must respond swiftly and creatively, where possible converting the adversity of the present into opportunities for the future.

      Declaration of Competing Interest

      Christoph Helmstaedter has received grants from the EU, travel support by Desitin, honoraries for talks, counselling, and advisory boards by GW pharmaceuticals, Eisai, UCB pharma, and Precisis, as well as license fees by EISAI, UCB pharma and Precisis. The remaining authors have no conflicts of interest.

      Acknowledgements

      This project was supported by an Australian Government Medical Research Future Fund Frontier Health and Medical Research Program Stage One grant ( MRFF75908 ); the National Health and Medical Research Council of Australia (Program Grant 1091593 and Project Grant APP1157145 ); a grant from Brain Australia ; and the Victorian Government including Victorian-led Frontier Health and Medical Research project funding and funding from the Operational Infrastructure Support Grant . The authors acknowledge the facilities and scientific and technical assistance of the National Imaging Facility, a National Collaborative Research Infrastructure Strategy (NCRIS) capability, at The Florey Institute of Neuroscience and Mental Health. We would also like to thank Saul Mullen, Paul Lightfoot, Marta Taranto, Olivia Lavalette and Gloria Nkhoma for helpful discussions, and two anonymous reviewers of an earlier version of this manuscript.

      References

        • Fowler G.
        Smartphone data reveal which Americans are social distancing (and not).
        Washington Post, 2020 (accessed April 21, 2020)
        • Krumholz H.M.
        Where have all the heart attacks gone?.
        The New York Times, 2020
        • Spinney L.
        Concern as heart attack and stroke patients delay seeking help.
        The Guardian, 2020
        • Bilder R.M.
        • Postal K.S.
        • Barisa M.
        • Aase D.M.
        • Cullum C.M.
        • Gillaspy S.R.
        • et al.
        Inter organizational practice committee Recommendations/Guidance for teleneuropsychology in response to the COVID-19 pandemic.
        Arch Clin Neuropsychol. 2020;
        • Pedersen M.
        • Verspoor K.
        • Jenkinson M.
        • Law M.
        • Abbott D.F.
        • Jackson G.D.
        Artificial intelligence for clinical decision support in neurology.
        Brain Communications. 2020;
        • Wechsler D.
        Test of premorbid functioning. UK version (TOPF UK).
        Pearson Corporation, UK2011
        • Schachter S.C.
        • Galaburda A.M.
        • Ransil B.J.
        Associations of dyslexia with epilepsy, handedness, and gender.
        Annals-New York Academy of Sciences. 1993; 682 (402–402)
        • Tailby C.
        • Weintrob D.L.
        • Saling M.M.
        • Fitzgerald C.
        • Jackson G.D.
        Reading difficulty is associated with failure to lateralize temporooccipital function.
        Epilepsia. 2014; 55: 746-753https://doi.org/10.1111/epi.12607
        • Wechsler D.
        Manual for the Wechsler abbreviated intelligence scale (WASI).
        The Psychological Corporation, San Antonio, TX1999
        • Hinton-Bayre A.
        • Geffen G.
        Comparability, reliability, and practice effects on alternate forms of the Digit Symbol Substitution and Symbol Digit Modalities tests.
        Psychol Assess. 2005; 17: 237
        • Dodrill C.B.
        A neuropsychological battery for epilepsy.
        Epilepsia. 1978; 19: 611-623
        • Army US
        Army individual test battery. Manual of Directions and Scoring.
        1944
        • Lutz M.T.
        • Helmstaedter C.
        EpiTrack: tracking cognitive side effects of medication on attention and executive functions in patients with epilepsy.
        Epilepsy Behav. 2005; 7: 708-714
        • Wechsler D.
        WAIS-IV technical and interpretive manual.
        Pearson, San Antonio, TX2008
        • Benton L.A.
        • Hamsher K.D.
        • Sivan A.B.
        Controlled oral word association test.
        Multilingual aphasia examination. 1994
        • Metternich B.
        • Buschmann F.
        • Wagner K.
        • Schulze-Bonhage A.
        • Kriston L.
        Verbal fluency in focal epilepsy: a systematic review and meta-analysis.
        Neuropsychol Rev. 2014; 24: 200-218
        • Sarkis R.A.
        • Busch R.M.
        • Floden D.
        • Chapin J.S.
        • Kalman Kenney C.
        • Jehi L.
        • et al.
        Predictors of decline in verbal fluency after frontal lobe epilepsy surgery.
        Epilepsy Behav. 2013; 27: 326-329https://doi.org/10.1016/j.yebeh.2013.02.015
        • Kaplan E.
        • Goodglass H.
        • Weintraub S.
        The Boston naming test.
        Lea & Febiger, Philadelphia, PA1983
        • Loring D.W.
        • Strauss E.
        • Hermann B.P.
        • Barr W.B.
        • Perrine K.
        • Trenerry M.R.
        • et al.
        Differential neuropsychological test sensitivity to left temporal lobe epilepsy.
        J Int Neuropsychol Soc. 2008; 14: 394-400
        • Umfleet L.G.
        • Janecek J.K.
        • Quasney E.
        • Sabsevitz D.S.
        • Ryan J.J.
        • Binder J.R.
        • et al.
        Sensitivity and specificity of memory and naming tests for identifying left temporal-lobe epilepsy.
        Appl Neuropsychol Adult. 2015; 22: 189-196
        • Hermann B.
        • Davies K.
        • Foley K.
        • Bell B.
        Visual confrontation naming outcome after standard left anterior temporal lobectomy with sparing versus resection of the superior temporal gyrus: a randomized prospective clinical trial.
        Epilepsia. 1999; 40: 1070-1076
        • Witt J.-A.
        • Helmstaedter C.
        Should cognition be screened in new-onset epilepsies? A study in 247 untreated patients.
        J Neurol. 2012; 259: 1727-1731
        • Saling M.M.
        Verbal memory in mesial temporal lobe epilepsy: beyond material specificity.
        Brain. 2009; (awp012)
        • Baxendale S.
        • Thompson P.
        • Harkness W.
        • Duncan J.
        Predicting memory decline following epilepsy surgery: a multivariate approach.
        Epilepsia. 2006; 47: 1887-1894
        • CANTAB®
        Cognitive assessment software.
        Cambridge Cognition, 2019
        • Witt J.-A.
        • Alpherts W.
        • Helmstaedter C.
        Computerized neuropsychological testing in epilepsy: overview of available tools.
        Seizure. 2013; 22: 416-423
        • Strauss E.
        • Sherman E.M.S.
        • Spreen O.
        A compendium of neuropsychological tests: administration, norms and commentary.
        3rd ed. Oxford University Press, New York2006
        • Pope R.A.
        • Thompson P.J.
        • Rantell K.
        • Stretton J.
        • Wright M.-A.
        • Foong J.
        Frontal lobe dysfunction as a predictor of depression and anxiety following temporal lobe epilepsy surgery.
        Epilepsy Res. 2019; 152: 59-66
        • Baker G.A.
        • Marson A.G.
        Cognitive and behavioural assessments in clinical trials: what type of measure?.
        Epilepsy Res. 2001; 45: 163-167
        • Loring D.W.
        • Lowenstein D.H.
        • Barbaro N.M.
        • Fureman B.E.
        • Odenkirchen J.
        • Jacobs M.P.
        • et al.
        Common data elements in epilepsy research: development and implementation of the NINDS epilepsy CDE project.
        Epilepsia. 2011; 52: 1186-1191
        • Barr W.B.
        • Chelune G.J.
        • Hermann B.P.
        • Loring D.W.
        • Perrine K.
        • Strauss E.
        • et al.
        The use of figural reproduction tests as measures of nonverbal memory in epilepsy surgery candidates.
        J Int Neuropsychol Soc. 1997; 3: 435-443
        • Barr W.
        • Morrison C.
        • Zaroff C.
        • Devinsky O.
        Use of the Brief Visuospatial Memory Test—revised (BVMT-R) in neuropsychological evaluation of epilepsy surgery candidates.
        Epilepsy Behav. 2004; 5: 175-179
        • Saling M.M.
        • Berkovic S.F.
        • O’Shea M.F.
        • Kalnins R.
        • Darby D.
        • Bladin P.
        Lateralization of verbal memory and unilateral hippocampal sclerosis - evidence of task-specific effects.
        J Clin Exp Neuropsychol. 1993; 15: 608-618https://doi.org/10.1080/01688639308402582
        • Weintrob D.L.
        • Saling M.M.
        • Berkovic S.F.
        • Reutens D.C.
        Impaired verbal associative learning after resection of left perirhinal cortex.
        Brain. 2007; 130: 1423-1431
        • Weintrob D.L.
        • Saling M.M.
        • Berkovic S.F.
        • Berlangieri S.U.
        • Reutens D.C.
        Verbal memory in left temporal lobe epilepsy: evidence for task-related localization.
        Ann Neurol. 2002; 51: 442-447
        • Fowler K.S.
        • Saling M.M.
        • Conway E.L.
        • Semple J.M.
        • Louis W.J.
        Paired associate performance in the early detection of DAT.
        J Int Neuropsychol Soc. 2002; 8: 58
        • Harris P.A.
        • Taylor R.
        • Thielke R.
        • Payne J.
        • Gonzalez N.
        • Conde J.G.
        Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support.
        J Biomed Inform. 2009; 42: 377-381
        • Harris P.A.
        • Taylor R.
        • Minor B.L.
        • Elliott V.
        • Fernandez M.
        • O’Neal L.
        • et al.
        The REDCap consortium: building an international community of software platform partners.
        J Biomed Inform. 2019; 95103208
        • Gilliam F.G.
        • Barry J.J.
        • Hermann B.P.
        • Meador K.J.
        • Vahle V.
        • Kanner A.M.
        Rapid detection of major depression in epilepsy: a multicentre study.
        Lancet Neurol. 2006; 5: 399-405
        • Spitzer R.L.
        • Kroenke K.
        • Williams J.B.
        • Löwe B.
        A brief measure for assessing generalized anxiety disorder: the GAD-7.
        Arch Intern Med. 2006; 166: 1092-1097
        • Seo J.-G.
        • Cho Y.W.
        • Lee S.-J.
        • Lee S.-J.
        • Kim J.-E.
        • Moon H.-J.
        • et al.
        Validation of the generalized anxiety disorder-7 in people with epilepsy: a MEPSY study.
        Epilepsy Behav. 2014; 35: 59-63
        • Micoulaud-Franchi J.-A.
        • Lagarde S.
        • Barkate G.
        • Dufournet B.
        • Besancon C.
        • Trébuchon-Da Fonseca A.
        • et al.
        Rapid detection of generalized anxiety disorder and major depression in epilepsy: validation of the GAD-7 as a complementary tool to the NDDI-E in a French sample.
        Epilepsy Behav. 2016; 57: 211-216
        • Scott A.J.
        • Sharpe L.
        • Thayer Z.
        • Miller L.A.
        • Hunt C.
        • MacCann C.
        • et al.
        Design and validation of two measures to detect anxiety disorders in epilepsy: the Epilepsy Anxiety Survey Instrument and its brief counterpart.
        Epilepsia. 2019; 60: 2068-2077
        • Mrazik M.
        • Millis S.
        • Drane D.L.
        The oral trail making test: effects of age and concurrent validity.
        Arch Clin Neuropsychol. 2010; 25: 236-243
        • Troyer A.K.
        • Leach L.
        • Strauss E.
        Aging and response inhibition: normative data for the Victoria stroop Test.
        Aging Neuropsychol Cogn. 2006; 13: 20-35
        • Haralambous B.
        • Subramaniam S.
        • Hwang K.
        • Dow B.
        • LoGiudice D.
        A narrative review of the evidence regarding the use of telemedicine to deliver video-interpreting during dementia assessments for older people.
        Asia-pacific Psychiatry. 2019; 11e12355https://doi.org/10.1111/appy.12355
        • Brearly T.W.
        • Shura R.D.
        • Martindale S.L.
        • Lazowski R.A.
        • Luxton D.D.
        • Shenal B.V.
        • et al.
        Neuropsychological test administration by videoconference: a systematic review and meta-analysis.
        Neuropsychol Rev. 2017; 27: 174-186
        • Grosch M.C.
        • Gottlieb M.C.
        • Cullum C.M.
        Initial practice recommendations for teleneuropsychology.
        Clin Neuropsychol. 2011; 25: 1119-1133
        • Wadsworth H.E.
        • Dhima K.
        • Womack K.B.
        • Hart Jr J.
        • Weiner M.F.
        • Hynan L.S.
        • et al.
        Validity of teleneuropsychological assessment in older patients with cognitive disorders.
        Arch Clin Neuropsychol. 2018; 33: 1040-1045
        • The Inter Organizational Practice Committee
        Recommendations/Guidance for teleneuropsychology (TeleNP) in response to the COVID-19 pandemic.
        2020
        • Hewitt K.C.
        • Loring D.W.
        Emory university telehealth neuropsychology development and implementation in response to the COVID-19 pandemic.
        Clin Neuropsychol. 2020; : 1-15
        • Marra D.E.
        • Hamlet K.M.
        • Bauer R.M.
        • Bowers D.
        Validity of teleneuropsychology for older adults in response to COVID-19: a systematic and critical review.
        Clin Neuropsychol. 2020; : 1-42
      1. British Psychological Society D of N. Division of Neuropsychology Professional Standards Unit Guidelines to colleagues on the use of Tele-neuropsychology 2020.

        • Cullum C.M.
        • Hynan L.S.
        • Grosch M.
        • Parikh M.
        • Weiner M.F.
        Teleneuropsychology: evidence for video teleconference-based neuropsychological assessment.
        J Int Neuropsychol Soc. 2014; 20: 1028-1033
        • Stead A.
        • Vinson M.
        Cognitive assessment using face-to-face and videoconferencing methods.
        Nurs Older People. 2019; 31
        • Stain H.J.
        • Payne K.
        • Thienel R.
        • Michie P.
        • Carr V.
        • Kelly B.
        The feasibility of videoconferencing for neuropsychological assessments of rural youth experiencing early psychosis.
        J Telemed Telecare. 2011; 17: 328-331
        • JACOBSEN S.E.
        • SPRENGER T.
        • ANDERSSON S.
        • KROGSTAD J.-M.
        Neuropsychological assessment and telemedicine: a preliminary study examining the reliability of neuropsychology services performed via telecommunication.
        J Int Neuropsychol Soc. 2003; 9: 472-478
        • Hildebrand R.
        • Chow H.
        • Williams C.
        • Nelson M.
        • Wass P.
        Feasibility of neuropsychological testing of older adults via videoconference: implications for assessing the capacity for independent living.
        J Telemed Telecare. 2004; 10: 130-134
        • Kirkwood K.T.
        • Peck D.F.
        • Bennie L.
        The consistency of neuropsychological assessments performed via telecommunication and face to face.
        J Telemed Telecare. 2000; 6: 147-151
        • Rebchuk A.D.
        • Deptuck H.M.
        • O’Neill Z.R.
        • Fawcett D.S.
        • Silverberg N.D.
        • Field T.S.
        Validation of a novel telehealth administration protocol for the NIH toolbox-cognition battery.
        Telemed E-health. 2019; 25: 237-242
        • Settle J.R.
        • Robinson S.A.
        • Kane R.
        • Maloni H.W.
        • Wallin M.T.
        Remote cognitive assessments for patients with multiple sclerosis: a feasibility study.
        Mult Scler. 2015; 21: 1072-1079
        • Galusha-Glasscock J.M.
        • Horton D.K.
        • Weiner M.F.
        • Cullum C.M.
        Video teleconference administration of the repeatable battery for the assessment of neuropsychological status.
        Arch Clin Neuropsychol. 2016; 31: 8-11
        • Wadsworth H.E.
        • Galusha-Glasscock J.M.
        • Womack K.B.
        • Quiceno M.
        • Weiner M.F.
        • Hynan L.S.
        • et al.
        Remote neuropsychological assessment in rural American Indians with and without cognitive impairment.
        Arch Clin Neuropsychol. 2016; 31: 420-425
        • Strober L.
        • DeLuca J.
        • Benedict R.H.
        • Jacobs A.
        • Cohen J.A.
        • Chiaravalloti N.
        • et al.
        Symbol digit modalities test: a valid clinical trial endpoint for measuring cognition in multiple sclerosis.
        Mult Scler. 2019; 25: 1781-1790
        • Cramer J.A.
        • Perrine K.
        • Devinsky O.
        • Bryant-Comstock L.
        • Meador K.
        • Hermann B.
        Development and cross-cultural translations of a 31-item quality of life in epilepsy inventory.
        Epilepsia. 1998; 39: 81-88
        • Baker G.
        Development of a patient-based symptom check list to quantify adverse effects in persons receiving antiepileptic drugs.
        Epilepsia. 1993; 34: 18
        • Baker G.A.
        Initial development, reliability and validity of a patient-based adverse drug event scale.
        Epilepsia. 1994; 35: S20
        • Baker G.A.
        The Liverpool adverse drug events profile.
        Epilepsia. 1995; 36: S59
        • Brooks J.
        • Baker G.A.
        • Aldenkamp A.P.
        The A–B neuropsychological assessment schedule (ABNAS): the further refinement of a patient-based scale of patient-perceived cognitive functioning.
        Epilepsy Res. 2001; 43: 227-237
      2. Pearson telepractice no objection letter.
        Pearson, 2020
        • Temple V.
        • Drummond C.
        • Valiquette S.
        • Jozsvai E.
        A comparison of intellectual assessments over video conferencing and in-person for individuals with ID: preliminary data.
        J Intellect Disabil Res. 2010; 54: 573-577
        • Luxton D.D.
        • Pruitt L.D.
        • Osenbach J.E.
        Best practices for remote psychological assessment via telehealth technologies.
        Prof Psychol Res Pr. 2014; 45: 27
        • Vogt V.L.
        • Äikiä M.
        • Del Barrio A.
        • Boon P.
        • Borbély C.
        • Bran E.
        • et al.
        Current standards of neuropsychological assessment in epilepsy surgery centers across Europe.
        Epilepsia. 2017; 58: 343-355
        • Hall K.E.
        • Isaac C.L.
        • Harris P.
        Memory complaints in epilepsy: an accurate reflection of memory impairment or an indicator of poor adjustment? A review of the literature.
        Clin Psychol Rev. 2009; 29: 354-367
        • Semah F.
        • Picot M.-C.
        • Adam C.
        • Broglin D.
        • Arzimanoglou A.
        • Bazin B.
        • et al.
        Is the underlying cause of epilepsy a major prognostic factor for recurrence?.
        Neurology. 1998; 51: 1256-1262
        • Baulac S.
        Genetics advances in autosomal dominant focal epilepsies: focus on DEPDC5. Progress in brain research, vol. 213.
        Elsevier, 2014: 123-139
        • Helmstaedter C.
        • Witt J.-A.
        • Hoppe C.
        Evaluating the mediating role of executive functions for antiepileptic drugs’ effects on IQ in children and adolescents with epilepsy.
        Epilepsy Behav. 2019; 96: 98-103
        • Strauss E.
        • Loring D.
        • Chelune G.
        • Hunter M.
        • Hermann B.
        • Perrine K.
        • et al.
        Predicting cognitive impairment in epilepsy: findings from the bozeman epilepsy consortium.
        J Clin Exp Neuropsychol. 1995; 17: 909-917https://doi.org/10.1080/01688639508402439
        • Helmstaedter C.
        • Kurthen M.
        • Lux S.
        • Reuber M.
        • Elger C.E.
        Chronic epilepsy and cognition: a longitudinal study in temporal lobe epilepsy.
        Ann Neurol. 2003; 54: 425-432
        • Kanner A.M.
        Depression in epilepsy: prevalence, clinical semiology, pathogenic mechanisms, and treatment.
        Biol Psychiatry. 2003; 54: 388-398https://doi.org/10.1016/S0006-3223(03)00469-4
      3. International League Against Epilepsy Neuropsycholoigcal assessments for epilepsy surgery during COVID-19 Restrictions n.d.

        • Backhaus A.
        • Agha Z.
        • Maglione M.L.
        • Repp A.
        • Ross B.
        • Zuest D.
        • et al.
        Videoconferencing psychotherapy: a systematic review.
        Psychol Serv. 2012; 9: 111
        • Meyer B.
        • Weiss M.
        • Holtkamp M.
        • Arnold S.
        • Brückner K.
        • Schröder J.
        • et al.
        Effects of an epilepsy-specific Internet intervention (Emyna) on depression: results of the ENCODE randomized controlled trial.
        Epilepsia. 2019; 60: 656-668
        • Bahrani K.
        • Singh M.B.
        • Bhatia R.
        • Prasad K.
        • Vibha D.
        • Shukla G.
        • et al.
        Telephonic review for outpatients with epilepsy—a prospective randomized, parallel group study.
        Seizure. 2017; 53: 55-61
        • Rasmusson K.A.
        • Hartshorn J.C.
        A comparison of epilepsy patients in a traditional ambulatory clinic and a telemedicine clinic.
        Epilepsia. 2005; 46: 767-770
        • Ahmed S.N.
        • Mann C.
        • Sinclair D.B.
        • Heino A.
        • Iskiw B.
        • Quigley D.
        • et al.
        Feasibility of epilepsy follow-up care through telemedicine: a pilot study on the patient’s perspective.
        Epilepsia. 2008; 49: 573-585
        • French J.A.
        • Brodie M.J.
        • Caraballo R.
        • Devinsky O.
        • Ding D.
        • Jehi L.
        • et al.
        Keeping people with epilepsy safe during the Covid-19 pandemic.
        Neurology. 2020; 94: 1032-1037
        • Wilson Sj
        • Kincade P.
        • Saling Mm
        • Bladin Pf.
        Patient readmission and support utilization following anterior temporal lobectomy.
        Seizure. 1999; 8: 20-25
        • von Wrede R.
        • Moskau-Hartmann S.
        • Baumgartner T.
        • Helmstaedter C.
        • Surges R.
        Counseling of people with epilepsy via telemedicine: experiences at a German tertiary epilepsy center during the COVID-19 pandemic.
        Epilepsy Behav. 2020; 112107298https://doi.org/10.1016/j.yebeh.2020.107298