Automatic assessment of the myoclonus severity from videos recorded according to standardized Unified Myoclonus Rating Scale protocol and using human pose and body movement analysis

Open ArchivePublished:January 24, 2020DOI:https://doi.org/10.1016/j.seizure.2020.01.014

      Highlights

      • Automatic detection of myoclonic jerks from video footage is feasible.
      • Algorithms use human pose and body movement for myoclonus detection.
      • Myoclonic jerks and movement smoothness can be automatically quantified.
      • Automatic analysis is in high agreement with clinical myoclonus rating.

      Abstract

      Purpose

      Myoclonus in progressive myoclonus epilepsy type 1 (EPM1) patients shows marked variability, which presents a substantial challenge in devising treatment and conducting clinical trials. Consequently, fast and objective myoclonus quantification methods are needed.

      Methods

      Ten video-recorded unified myoclonus rating scale (UMRS) myoclonus with action tests were performed on EPM1 patients who were selected for the development and testing of the automatic myoclonus quantification method. Human pose and body movement analyses of the videos were used to identify body keypoints and further analyze movement smoothness and speed. The automatic myoclonus rating scale (ARMS) was developed. It included the jerk count during movement score and the log dimensionless jerk (LDLJ) score to evaluate changes in the smoothness of movement.

      Results

      The scores obtained with the automatic analyses showed moderate to strong significant correlation with the UMRS myoclonus with action scores. The jerk count of the primary keypoints and the LDLJ scores were effective in the evaluation of the myoclonic jerks during hand movements. They also correlated moderately to strongly with the total UMRS test panel scores (r2 = 0,77, P = 0,009 for the jerk count score and r2 = 0,88, P = 0,001 for the LDLJ score). The automatic analyses was weaker in quantification of the neck, trunk, and leg myoclonus.

      Conclusion

      Automatic quantification of myoclonic jerks using human pose and body movement analysis of patients’ videos is feasible and was found to be quite consistent with the accepted clinical gold standard quantification method. Based on the results of this study, the automatic analytical method should be further developed and validated to improve myoclonus severity follow-up for EPM1 patients.

      Keywords

      1. Introduction

      Many neurological diseases such as epilepsy cause abnormal body posture or abnormal movement of body parts. These abnormal involuntary movements, such as muscle twitches, myoclonic jerks or other motor manifestations during epileptic seizures, provide important information that can be used to diagnose and assess disease severity and progression.
      A variety of clinical rating scales has been developed for physicians to evaluate body pose and/or body movements [
      • Frucht S.J.
      • Leurgans S.E.
      • Hallett M.
      • Fahn S.
      The unified myoclonus rating scale.
      ,
      • Pietracupa S.
      • Bruno E.
      • Cavanna A.E.
      • Falla M.
      • Zappia M.
      • Colosimo C.
      Scales for hyperkinetic disorders: a systematic review.
      ]. However, this type of evaluation and analysis is not a trivial task and requires expert knowledge. Some of the designed metrics are imprecise and often subjective. Thus, it is difficult to compare the measurements made by different observers (e.g., physicians and specialists) and even measurements made by the same observer at different times. Very often, patients require continuous monitoring, and body pose and movement must be repeatedly evaluated and analyzed to track disease progression or the effectiveness of a particular treatment. This makes current rating scales and follow-up approaches subjective, time-consuming, and difficult to apply in a consistent manner. Therefore, physicians, researchers, and the medical industry have been seeking automatic tools that can efficiently and reliably analyze human body pose and body part movements with some degree of consistency.
      Computer vision is a scientific field that develops methods for acquiring, processing, analyzing and understanding digital images. Over the past few years, it has achieved remarkable success due to the rapid development of certain deep learning techniques [
      • Goodfellow I.
      • Bengio Y.
      • Courville A.
      Deep learning.
      ]. Deep learning, which relies on deep neural network (DNN) architecture, is based on a cascade of multiple layers of convolutional neural networks that can effectively analyze audio signals, images, videos and other types of sequential data. The technology has become tremendously effective in natural language processing, speech recognition, image and video recognition, and object detection.
      Deep learning techniques have been used to improve the accuracy of human pose estimation and the detection of keypoints (i.e., detection of regions of high importance, such as joints) [
      • Cao Z.
      • Simon T.
      • Wei S.
      • Sheikh Y.
      Realtime multi-person 2D pose estimation using part affinity fields.
      ,
      • Insafutdinov E.
      • Pishchulin L.
      • Andres B.
      • Andriluka M.
      • Schiele B.
      DeeperCut: a deeper, stronger, and faster multi-person pose estimation model.
      ,
      • Kendall A.
      • Grimes M.
      • Cipolla R.
      Convolutional networks for real-time 6-DOF camera relocalization.
      ]. For example, the method was adapted, fine-tuned, and tested for pose and movement recognition of motor seizures in an epilepsy monitoring unit [
      • Chen K.
      • Gabriel P.
      • Alasfour A.
      • Gong C.
      • Doyle W.K.
      • Devinsky O.
      • et al.
      Patient-specific pose estimation in clinical environments.
      ]. Machine learning was also applied to develop automated evaluation methods for quantifying patients’ movement disorders associated with Parkinson’s disease [
      • Kubota K.J.
      • Chen J.A.
      • Little M.A.
      Machine learning for large-scale wearable sensor data in Parkinson’s disease: concepts, promises, pitfalls, and futures.
      ]. It was used for the automatic sub-task segmentation of timed up and go test videos of patients with Parkinson’s disease [
      • Li T.
      • Chen J.
      • Hu C.
      • Ma Y.
      • Wu Z.
      • Wan W.
      • et al.
      Automatic timed up-and-go sub-task segmentation for Parkinson’s disease patients using video-based activity classification.
      ], and automatic pose estimation from videos has also been successfully applied in the evaluation of the general movements of infants [
      • Marchi V.
      • Hakala A.
      • Knight A.
      • D’Acunto F.
      • Scattoni M.L.
      • Guzzetta A.
      • et al.
      Automated pose estimation captures key aspects of General Movements at eight to 17 weeks from conventional videos.
      ].
      Progressive myoclonus epilepsy type 1 (EPM1) belongs to a group of progressive myoclonus epilepsies (PMEs) characterized by myoclonus, epilepsy, and progressive neurological deterioration [
      • Kälviäinen R.
      • Khyuppenen J.
      • Koskenkorva P.
      • Eriksson K.
      • Vanninen R.
      • Mervaala E.
      Clinical picture of EPM1-Unverricht-Lundborg disease.
      ]. The disease manifests with generalized tonic-clonic seizures and myoclonus. The tonic-clonic seizures can usually be effectively controlled using antiepileptic drugs (AEDs), though myoclonic jerks tend to be particularly progressive and disabling. It was previously reported that the disease symptoms progress within first 5–10 years from their onset, then later reach a plateau [
      • Magaudda A.
      • Ferlazzo E.
      • Nguyen V.H.
      • Genton P.
      Unverricht-Lundborg disease, a condition with self-limited progression: long-term follow-up of 20 patients.
      ]. Therefore, it is crucial to find the best suitable medical treatment for this disease, especially myoclonus.
      Several clinical scales are employed to quantitate myoclonus, including the myoclonus rating scale, the unified myoclonus rating scale (UMRS), and modified versions of the UMRS [
      • Frucht S.J.
      • Leurgans S.E.
      • Hallett M.
      • Fahn S.
      The unified myoclonus rating scale.
      ,
      • Koskiniemi M.
      • Van Vleymen B.
      • Hakamies L.
      • Lamusuo S.
      • Taalas J.
      Piracetam relieves symptoms in progressive myoclonus epilepsy: a multicentre, randomised, double blind, crossover study comparing the efficacy and safety of three dosages of oral piracetam with placebo.
      ,
      • Goldsmith D.
      • Minassian B.A.
      Efficacy and tolerability of perampanel in ten patients with Lafora disease.
      ]. The UMRS is widely considered to be the gold standard for evaluating myoclonus. The UMRS test panel includes an evaluation of both the extent of the myoclonic jerk and myoclonic jerk frequency over a 10 s observation period. The inter-rater agreement associated with the UMRS test panel has been shown to be relatively high. For the myoclonus with action portion, the Cronbach’s α-score was reported with a frequency score of 0.85 and an amplitude score of 0.86 [
      • Frucht S.J.
      • Leurgans S.E.
      • Hallett M.
      • Fahn S.
      The unified myoclonus rating scale.
      ,
      • Pietracupa S.
      • Bruno E.
      • Cavanna A.E.
      • Falla M.
      • Zappia M.
      • Colosimo C.
      Scales for hyperkinetic disorders: a systematic review.
      ]. However, this evaluation largely depends on the experience of the scoring physician. Moreover, previous drug trials have shown that it fails to account for the global complexity of myoclonus in patients with myoclonic epilepsy [
      • Kalviainen R.
      • Genton P.
      • Andermann E.
      • Andermann F.
      • Magaudda A.
      • Frucht S.J.
      • et al.
      Brivaracetam in Unverricht-Lundborg disease (EPM1): results from two randomized, double-blind, placebo-controlled studies.
      ]. Myoclonus in EPM1 patients is known to be stimulus sensitive, and stress, sleep deprivation, anxiety, time of the day, and many other factors can influence the severity of myoclonic jerks. Therefore, objective serial quantification of myoclonus in EPM1 patients using the UMRS remains suboptimal due to subjectivity of the evaluation and the individual fluctuations of myoclonus.
      In this study, an automatic tool was developed to obtain a myoclonic jerk score from video-recordings using human body keypoint detection and human pose estimation. EPM1 patients are exceptionally well suited for this application because the determination of their clinical myoclonus rating is so challenging. The UMRS myoclonus with action scores were compared with scores obtained from the new automatic myoclonus rating scale (AMRS) in adult EPM1 patients. Novel terms are introduced in this paper to describe the proposed analytical method, including jerk count and log dimensionless jerk (LDLJ) score. Jerk count was developed for this study, and it was defined as the number of velocity spikes during the movement. There are several measurements for evaluation of movement smoothness [
      • Balasubramanian S.
      • Melendez-Calderon A.
      • Roby-Brami A.
      • Burdet E.
      On the analysis of movement smoothness.
      ]. The LDLJ was adapted for myoclonus evaluation as part of the new AMRS.

      2. Materials and methods

      2.1 Subjects

      EPM1 patients with different degrees of myoclonus severity were selected from our previously described EPM1 phenotype-genotype study [
      • Hypponen J.
      • Aikia M.
      • Joensuu T.
      • Julkunen P.
      • Danner N.
      • Koskenkorva P.
      • et al.
      Refining the phenotype of Unverricht-Lundborg disease (EPM1): a population-wide Finnish study.
      ]. Ten EPM1 patients (eight homozygous for the expansion mutation in the CSTB gene, three men and seven women, aged 14–47 years, mean age 30.5 ± 11 years) were included in this pilot study. The principal investigator of the original EPM1 phenotyping study (RK) selected the cases, and there were no previous attempts to develop an automatic quantification method to assess myoclonic jerks. Therefore, this study employed a proof of concept approach to demonstrate the feasibility of myoclonus quantification from patient video recordings. The myoclonus patients with different degrees of severity were selected to evaluate whether detecting different myoclonic jerks from video footage was possible by using the automatic method described below.
      Based on the previous UMRS myoclonus with action scores, three patients were found to have mild disability (1–30), three patients exhibited moderate disability (31–59), and four patients had severe disability (score > 60). All patients were being treated with individually adjusted AEDs. Three patients were wheelchair bound, and two patients occasionally needed a wheelchair.
      The original study was conducted at the Kuopio Epilepsy Center at Kuopio University Hospital jointly with the Folkhälsan Institute of Genetics and Neuroscience Center at the University of Helsinki. The Kuopio University Hospital’s ethics committee approved the study protocol, and informed consent was properly obtained from all participants. For adolescent patients (age 12–14 years), the informed consent was provided by their guardians.
      Ten video-recorded UMRS myoclonus with action test panels were available for automatic scoring and analysis. Developers of the AMRS were unfamiliar with the clinical history of the selected EPM1 patients and were unaware of the patients’ previous UMRS scoring results. All UMRS scoring results were obtained from the previously conducted EPM1 phenotyping study [
      • Hypponen J.
      • Aikia M.
      • Joensuu T.
      • Julkunen P.
      • Danner N.
      • Koskenkorva P.
      • et al.
      Refining the phenotype of Unverricht-Lundborg disease (EPM1): a population-wide Finnish study.
      ]. Clinical neurophysiologist (JH) completed all UMRS scoring for the original study. She received training and certification to administer and score the UMRS test panel for the brivaracetam drug-trial [
      • Kalviainen R.
      • Genton P.
      • Andermann E.
      • Andermann F.
      • Magaudda A.
      • Frucht S.J.
      • et al.
      Brivaracetam in Unverricht-Lundborg disease (EPM1): results from two randomized, double-blind, placebo-controlled studies.
      ]. A re-evaluation of the original UMRS scores was not conducted for this pilot study because previously obtained scores were deemed acceptable for the analyses.
      Original videos included all tasks associated with the UMRS’s stimulus sensitivity, myoclonus with action, and functional test panels. However, only six of the UMRS myoclonus with action tasks were selected for the automatic analyses, including the neck, trunk, right arm, left arm, right leg, and left leg tasks [
      • Frucht S.J.
      • Leurgans S.E.
      • Hallett M.
      • Fahn S.
      The unified myoclonus rating scale.
      ], while close eyelids, arising, standing, and walking tasks were excluded. In the original video, patients closing their eyes were recorded using the Zoom option, which affects the reliability of keypoint recognition. Arising, standing, and walking tasks were excluded because some patients with severe myoclonus were unable to perform the tasks, and therefore, the video recordings did not contain the data required to conduct an automatic analysis for these tasks.

      2.2 Automatic human pose and body movement analysis

      For an image that contains one or more people, a human keypoint detector can locate the keypoints of the human body. These systems use deep convolutional neural network (CNN) architecture, such as VGG or ResNet [
      • Simonyan K.
      • Zisserman A.
      Very deep convolutional networks for large-scale image recognition.
      ,
      • He K.
      • Zhang X.
      • Ren S.
      • Sun J.
      Deep residual learning for image recognition.
      ], as the backbone, and extra layers are then added to predict the location of the keypoints. After the initial detection of keypoints, the system further associates the detected keypoints to separate patients from one another and improve the accuracy of the estimated location of the keypoints. To train the system, it was first initiated using a pretrained model, which is often a model trained for image classification that uses an ImageNet [
      • Deng J.
      • Dong W.
      • Socher R.
      • Li L.
      • Li K.
      • Fei-fei L.
      Imagenet: a large-scale hierarchical image database.
      ] dataset. Trainable parameters were fine-tuned for human keypoint detection using a human pose dataset such as MPII [
      • Andriluka M.
      • Pishchulin L.
      • Gehler P.
      • Schiele B.
      2D human pose estimation: New benchmark and state of the art analysis.
      ] or COCO [
      • Lin T.
      • Maire M.
      • Belongie S.J.
      • Bourdev L.D.
      • Girshick R.B.
      • Hays J.
      • et al.
      Microsoft COCO: common objects in context.
      ]. In our study, OpenPose [
      • Cao Z.
      • Simon T.
      • Wei S.
      • Sheikh Y.
      Realtime multi-person 2D pose estimation using part affinity fields.
      ] was used as the human keypoint detector to detect and locate 18 keypoints on the human body, as shown in Fig. 1.
      Fig. 1
      Fig. 1Eighteen keypoints detected by a human keypoints detector.
      The primary keypoint is the body part that is directly involved in the movement. Auxiliary keypoints are the body parts that may indicate body stability during the movement. For each UMRS task, the corresponding primary keypoint and auxiliary keypoints were selected for recognition and tracking by the algorithm during movement. Details related to the investigated keypoints are presented in Table 1.
      Table 1Description of primary and secondary keypoints used for the movement assessment during the UMRS test panel tasks. *Due to the primary video setup required by the UMRS video recording instructions, the auxiliary keypoints could not be used for the automated analysis of the leg action test panel.
      UMRS taskMovement performed during the taskPrimary keypoint (“moving”)Auxiliary keypoints (“not moving”)
      B. NeckHead movement in flexion-extension and side to side rotationNose tipRight shoulder, Left shoulder
      C. TrunkTrunk flexion when sittingNeckRight shoulder, Left shoulder
      D. Right arm – movingFinger to nose movementRight wristNose tip, Left shoulder
      D. Right arm - staticFinger on the noseRight wristNose tip, Left shoulder
      E. Left arm - movingFinger to nose movementLeft wristNose tip, Right shoulder
      E. Left arm - staticFinger on the noseLeft wristNose tip, Right shoulder
      D.E. arms forwardBoth arms forward with palms down and then wrists extendedRight wrist, Left wristNose tip
      F. Right legHeel to toe shinRight ankleNA*
      G. Left legHeel to toe shinLeft ankleNA*
      UMRS hand tasks include both static (keeping finger on the nose) and moving (finger-to-nose) actions [
      • Frucht S.J.
      • Leurgans S.E.
      • Hallett M.
      • Fahn S.
      The unified myoclonus rating scale.
      ]. It also includes the performance of a static task, which is the extension of both arms, first with palms down and then wrists extended. The keypoints were detected and analyzed separately for each part of the task, as shown in the Table 1.
      After keypoint detection, the pose of the human body, the trajectory of the keypoints during an action, and the smoothness of the movement can be analyzed. However, the exact locations of body keypoints in an image are sometimes ambiguous (e.g., the shoulder is more like a large homogeneous area than a precisely defined point). Since each frame is processed separately, the estimated keypoint locations obtained from a video are often unstable. In addition, occlusion of the keypoints, for example clothing, affects the precision of keypoint detection, which can be imprecise or even impossible if the selected keypoint is not clearly visible in the video. For example, the patient was sitting while performing the UMRS “heel to toe shin” tasks, and it was impossible to reliably detect the auxiliary keypoints such as the hip joint. All these factors can induce noise during keypoint detection and tracking because the detected keypoint locations might incidentally vary in subsequent images.
      The induced noise affects the tracking of keypoint movement, especially when the speed of the movement is estimated. Fig. 2(a) shows the right wrist movement when the patient is asked to complete an action. The estimated speed based on the location of the right wrist, as shown in Fig. 2(b), contains a high level of frequency noise due to keypoint detection instability. To address this issue, the optical flow method was used to estimate the speed of keypoints.
      Fig. 2
      Fig. 2(a) Right wrist movement when the patient was asked to perform an action; (b) the speed of the movement estimated from raw keypoint locations; (c) the speed of the movement estimated from optical flow.
      Optical flow can be used to estimate motion in a sequence of images by finding corresponding pixels in the images and calculating their trajectories [
      • Fortun D.
      • Bouthemy P.
      • Kervrann C.
      Optical flow modeling and computation: a survey.
      ]. The method assumes that the flow is constant in a local area and estimates the flow by minimizing the least square error. To achieve a stable estimation when an object moves along a trajectory between two frames, the system builds a pyramid of images and applies a coarse-to-fine approach, whereby the estimation from higher layers can be used to correct the estimation in lower layers. Using the detected keypoints, the speed of the keypoint movement was estimated by analyzing the optical flow around the keypoint area. The speed estimation using optical flow, as shown in Fig. 2(c), was found to be more stable than the estimation based on raw keypoint locations, as shown in Fig. 2(b).
      The keypoint detector and the optical flow algorithm introduced noise to the acquired signals. Thus, the raw signals were further processed with a low pass moving average filter (window size = 7).

      2.3 Automatic myoclonus rating scale (AMRS)

      The UMRS score for each task is a multiplied measure of the myoclonus amplitude and frequency [
      • Frucht S.J.
      • Leurgans S.E.
      • Hallett M.
      • Fahn S.
      The unified myoclonus rating scale.
      ]. The score for each UMRS task was first calculated separately, and the total sum of all the tasks was then used for general evaluation. In accordance with the original UMRS setup, the automatic scores were generated separately for each task. For the arm tasks, the moving and static scores were obtained separately. These separate scores for the arm myoclonic jerks and their sum were used for the analyses. We also calculated the sum of the individual task’s jerk count for the primary keypoint and the sum of the LDLJ scores to match and compare the automatic scores with the total score of the UMRS myoclonus with action panel.
      For this pilot study, jerk count was used as the main evaluation metric. The concept of jerk count was developed for this study, and it was defined as the number of velocity spikes above a predefined threshold during the movement. Since keypoint detection can introduce low-magnitude changes in speed, an empirically determined threshold was used to adequately remove these changes. The analyses first included a count of all the velocity spikes detected by the algorithm during the task. The results were than linearly scaled from 0 to 5 independently for each task. The score obtained from the analyses was used to evaluate the myoclonic jerks throughout the entire task or movement. A higher value is indicative of more prominent myoclonus.
      The other important characterization of sensorimotor control is the smoothness of the movement. A number of measures have been described that can be used to evaluate movement smoothness, especially during point-to-point or goal-directed tasks. In general, smoothness measurements evaluate the speed profile of the movement, which is usually bell-shaped in healthy subjects. In subjects with neurological disorders the movement speed can be highly intermittent including interchanging episodes of acceleration and deceleration. Smoothness measurements quantify the amount of intermittency during the movement. Therefore, in addition to jerk count, we estimated the log dimensionless jerk (LDLJ) scores of the primary keypoint [
      • Balasubramanian S.
      • Melendez-Calderon A.
      • Roby-Brami A.
      • Burdet E.
      On the analysis of movement smoothness.
      ]. Since this metric was used to evaluate the smoothness of the movement in general, it was more affected by small and high-frequency jerks, and it was also more sensitive to noise.

      2.4 Statistical analysis

      Statistical analyses were performed with IBM SPSS Statistics, version 22. Spearman’s correlation tests were used to determine the correlation between continuous variables, and the correlation was considered strong, moderate, fair, or poor if r2 was 0.8–0.9, 0.6–0.7, 0.3–0.5, or 0.1–0.2, respectively [
      • Akoglu H.
      User’s guide to correlation coefficients.
      ].

      3. Results

      The main characteristics for each patient are listed in Table 2.
      Table 2Basic characteristics of the EPM1 patients included in the study.
      PatientAgeWheelchair useAge at EPM1 onsetDisease durationNr of AEDsUMRS Myoclonus with Action: total scoreUMRS Functional test: total scoreAMRS Primary keypoint jerk count: total scoreAMRS LDLJ score of the primary keypoint: total score
      1Male
      Expansion mutation homozygote.
      47Wheelchair bound10376 AEDs1202834,227,3
      2Male
      Expansion mutation homozygote.
      25Walks11142 AEDs1224,38
      3Female
      Compound heterozygote.
      34Wheelchair bound10244 AEDs6998,836,8
      4Male
      Expansion mutation homozygote.
      25Walks10152 AEDs29313,513,8
      5Female
      Expansion mutation homozygote.
      18Occasionally1173 AEDs431418,521,5
      6Female
      Expansion mutation homozygote.
      41Walks10314 AEDs816,510,8
      7Female
      Expansion mutation homozygote.
      23Wheelchair bound8153 AEDs901824,927
      8Female
      Expansion mutation homozygote.
      38Occasionally10284 AEDs631716,630
      9Female
      Compound heterozygote.
      14Walks682 AEDs40810,815,1
      10Female
      Expansion mutation homozygote.
      40Walks13275 AEDs3474,917,9
      AED, anti-epileptic drug, UMRS, Unified Myoclonus rating scale, AMRS, Automatic Myoclonus Rating Scale.
      a Expansion mutation homozygote.
      b Compound heterozygote.
      The automatically calculated jerk count scores for the listed keypoints were correlated with the UMRS myoclonus with action test panel scores. The jerk count scores for the primary keypoints showed significant moderate to strong correlation with the UMRS scores of the arms and total scores (Table 3, Fig. 3). The automatic leg movement jerk count scores and UMRS leg tests scores were moderately correlated for the right leg, but did not reach the significance level for the left leg. The LDLJ scores of the primary keypoints exhibited significant moderate to strong correlation with all the UMRS subscores and the total scores (Table 3, Fig. 3). Detailed correlation results for the jerk count of the primary keypoints and LDLJ scores of the primary keypoints are presented in Table 3.
      Table 3Main results of the UMRS Myoclonus with Action scores correlation with the AMRS jerk count scores of the primary keypoints and LDLJ scores of the primary keypoint.
      Jerk count of the primary keypointLDLJ score for the primary keypoint
      B. NeckC. TrunkD. Right armE. Left armF. Right legG. Left legTotal sumB. NeckC. TrunkD. Right armE. Left armF. Right legG. Left legTotal sum
      Myoclonus with Action: neck−0,1100,664
      0,7630,036
      Myoclonus with Action: trunk−0,0400,767
      0,9120,010
      Myoclonus with Action: right arm0,7300,763
      0,0160,010
      Myoclonus with Action:

      left arm
      0,8810,786
      0,0010,007
      Myoclonus with Action: right leg0,6860,743
      0,0280,014
      Myoclonus with Action:

      left leg
      0,5540,689
      0,0970,027
      Myoclonus with Action: total score0,7700,879
      0,0090,001
      Data presented as Spearman’s correlation coefficient, P-value.
      Fig. 3
      Fig. 3(a) Correlation between the UMRS myoclonus with action total score and the total jerk count score of the primary keypoints. (b) Correlation between the UMRS myoclonus with action total score and the total LDLJ score for the primary keypoint.
      By contrast, the jerk count scores for the first and the second auxiliary keypoints had much weaker correlation with the UMRS scores (Supplementary Table 1). Only the auxiliary scores for the arm jerk count were moderately correlated with the UMRS myoclonus with action scores of the arm movement (Supplementary Table 1). The auxiliary keypoints’ jerk counts did not correlate with the total UMRS myoclonus with action score (r2 = 0.333, P = 0.347 and r2 = 0.224, P = 0.533 for the first and second auxiliary keypoints, respectively). In addition, due to the primary video setup required by the UMRS video recording instructions, the auxiliary keypoints could not be used for the automated analysis of the leg action test panel.
      In accordance with the above, the automatic total jerk count of primary keypoints and total LDLJ score were significantly moderately to strongly correlated with the results of the UMRS panel’s functional test results (Supplementary Table 2).

      4. Discussion

      To the best of our knowledge, this is the first study that has attempted to use pose estimation and body movement analysis to evaluate myoclonus from individual patient videos. In this pilot study, the analysis algorithms were adapted for the scoring of myoclonus during active movement to obtain the AMRS scores.
      The results indicated that the system was capable of automatically quantifying myoclonic jerks in six tasks of the UMRS myoclonus with action panel. The correlation results of the automatically obtained AMRS scores for the arm tasks and the total scores with the original UMRS scores were moderate to strong. There was a high level of agreement between the original clinical UMRS scores and the automatic analyses. Our results suggest that the pose estimation and keypoint detection method used for myoclonic jerk count developed in this study could be a valid method to quantify myoclonus severity in EPM1 patients. However, since it was a small-scale proof of concept study, the reproducibility of the automatic analyses of serial measurements were not evaluated. One scorer originally evaluated the videos used for the development of the automated method. Therefore, the comparison of the method with the multiple scorings performed by the certified physicians was outside the scope of this pilot study. The automatic analyses could have a lower intra-observer variability, but more comparisons of the automatic analyses with the original UMRS scores performed by different observers would be needed to validate this statement.
      It is important to note that the nature of myoclonic jerks could be characterized in a more subtle way by the automatic analyses. Myoclonic jerks can vary widely in severity and distribution. In EPM1 patients, myoclonic jerks range from minimal and modest, clinically restricted (e.g., one limb myoclonias) to a very large generalized myoclonic jerks that can prevent the patient from being able to walk. The automatic primary keypoint jerk count and LDLJ score analyses performed differently for some of our patients (e.g. Table 2, patients 3, 8 and 10). The LDLJ score was more likely to detect small amplitude and high-frequency myoclonic jerks, which means that the myoclonus in these patients had a significant effect on the smoothness of movement without causing significant speed spikes. Clinical quantification of the smoothness of movement is difficult. However, it is an important characteristic of motor learning and sensorimotor control. Therefore, considering the heterogeneous nature of myoclonic jerks, using both the jerk count score and the LDLJ score could be a beneficial and more sensitive method of detecting subtle changes in patients’ myoclonus count or their smoothness of movement.
      In this study, strong to moderate results were obtained for the automatic analyses of the arms and total myoclonus scores, while the developed algorithm workflow performed more poorly in the quantifications of the myoclonic jerks during neck, trunk and leg tasks. As previously state in the Methods section, a portion of the tasks were excluded from analysis due to the limitations of the original videos. The automatic scoring also exhibited better performance in the evaluation of moving primary keypoints for the jerk count compared with the selected auxiliary “static” keypoints. The detection and tracking of the keypoints depends on the video recording quality including the setup of the video recorded tasks. The position of the subject during the movement or clothing might also substantially affect the reliability of the keypoint detection and increase the signal-to-noise ratio. Therefore, obtaining a standardized video recording with a clear and unobscured view of the selected keypoints is an important component of any future development of the method. Nevertheless, the automatic analysis was reliably consistent and showed that the limited set of tasks may offer an accurate reflection of the overall myoclonus severity assessment found in UMRS videos. Future studies that employ and evaluate this method should use high-definition video footage, and the patient’s entire body and their legs should be in view. The keypoint detection system could be reinforced with an embedded keypoint tracking system. Currently, the keypoints are detected independently in each frame, resulting in a normally distributed point cloud around the true keypoint, whereas a combined system would provide more accurate results by adapting the system to frame sequences instead of independent frames. This could significantly improve system’s speed estimation performance, and thus, the jerk count and LDLJ assessment.
      Numerous methods are used to evaluate various movement disorders, which require patients to wear different types of sensors [
      • Kubota K.J.
      • Chen J.A.
      • Little M.A.
      Machine learning for large-scale wearable sensor data in Parkinson’s disease: concepts, promises, pitfalls, and futures.
      ,
      • Rissanen S.M.
      • Ruonala V.
      • Pekkonen E.
      • Kankaanpaa M.
      • Airaksinen O.
      • Karjalainen P.A.
      Signal features of surface electromyography in advanced Parkinson’s disease during different settings of deep brain stimulation.
      ]. Only a limited number of attached sensors is usually used in the evaluation to minimize the patients’ discomfort. Therefore, the information collected by the wearable devices has limitations. Our automatic evaluation only requires video of the patient performing certain simple motor tasks. The patient is not required to wear any additional sensors or devices. Therefore, it is less burdensome for the patient, and results can be obtained faster during an outpatient visit. Video recordings can be used to holistically evaluate myoclonic jerks, while simultaneously isolating and evaluating different parts of the body.
      On the other hand, myoclonus in EPM1 patients is known to fluctuate during the day, and many factors, such as unexpected external stimuli, stress, and sleep deprivation, may significantly aggravate myoclonus. Therefore, the limitation of the current protocol is the length of the video recording of the test panel, which provides an evaluation of myoclonus in a relatively short period of time. The method is also designed to evaluate certain simple motor tasks, and therefore, it is not directly applicable for long-term follow up. On the other hand, the wearable sensors are well suited for long-term home-based monitoring. Therefore, using separately the video recording and wearable sensors can provide valuable but limited information for the clinical evaluation, and it is unlikely that one selected method will provide a comprehensive evaluation of such a complex symptom phenotype as myoclonus in EPM1 patients. One aim of future development is to combine different types of automatic follow-up and evaluation methods. The automatic analyses of the videos described in this study may be used to plan the sensor recording, and these methods can be combined to validate the data obtained from the sensors.
      It is always challenging to conduct clinical treatment trials using the best standardized and preferably objective quantitative method to evaluate changes in disease severity or clinical phenotype, and myoclonus is no exception to this rule. Our automatic myoclonus scoring system is the first step to offer a standardized quantitative method. As previously discussed, promising results were found for quantifying the smoothness of movement by detecting smaller, high-frequency myoclonic jerks. When testing new drugs, the patients’ subjective evaluation of disability and possible improvement in their symptoms are both taken into account. Some of the previous automated methods for analyzing symptom have exhibited weaker correlation with patients’ subjective feelings compared to clinical evaluation scales [
      • Rodriguez-Blazquez C.
      • Forjaz M.J.
      • Kurtis M.M.
      • Balestrino R.
      • Martinez-Martin P.
      Rating scales for movement disorders with sleep disturbances: a narrative review.
      ]. Therefore, our method must also be tested by taking EPM1 patients’ subjective evaluation of their symptom severity and disability into account.
      The results of this pilot study strongly suggest that this method is promising for the quantification of myoclonus. However, it must still be validated by a study that includes a larger patient population, and its capacity for prospective follow-up must also be tested. To address the problem of evaluating fluctuations’ of myoclonus symptoms throughout the day, the automatic scoring must be modified for continuous assessment throughout the day, possibly in combination with other recording techniques. It is hoped that the automatic methods described above can be further improved and standardized for better and faster performance in the clinical setting. For example, adjusting the original UMRS test panel to better meet the demands of the automatic pose estimation and keypoint recognition analyses would greatly enhance the method’s effectiveness.

      5. Conclusions

      The results of this pilot study indicated that the automated keypoint detection method and pose estimation from video method reliably quantified myoclonic jerks in EPM1 patients. The automatic quantification of myoclonus demonstrated a high level of agreement with the previous clinical evaluation. It also effectively quantified the smoothness of movement, and it was sensitive enough to detect small-amplitude and high-frequency myoclonic jerks. In the future, the method should be validated with a study of a larger patient population, and its feasibility should be further tested. Finally, updating the video recording requirements and possibly the test panel tasks may yield the best possible results from the automated analyses.

      Disclosure of Conflict of Interest

      Jelena Hyppönen: none.
      Anna Hakala, Kaapo Annala and Honglei Zhang: AH and KA are employees of Neuro Event Labs, the company that provided the equipment and technology used in the study. HZ works as data science consultant for Neuro Event Labs. Neuro Event Labs has received research grant for this study from Eisai.
      Jukka Peltola: JP has participated in clinical trials for Eisai, UCB, and Bial; received research grants from Eisai, Medtronic, UCB, and Cyberonics/Livanova; received speaker honoraria from Cyberonics/Livanova, Eisai, Medtronic, Orion Pharma, and UCB; received support for travel to congresses from Cyberonics/Livanova, Eisai, Medtronic, and UCB; and participated in advisory boards for Cyberonics/Livanova, Eisai, Medtronic, UCB, and Pfizer. In addition, JP is a shareholder of Neuro Event Labs.
      Esa Mervaala: none.
      Reetta Kälviäinen: RK has received grants from the Academy of Finland and the Saastamoinen Foundation, a speaker’s honoraria from Eisai, UCB, and Orion and honoraria for the membership of the advisory boards from Eisai, GW Pharmaceuticals, Marinus Pharmaceuticals, Sage Therapeutics, Takeda and UCB.

      Acknowledgments

      This study was partially funded by a grant from Eisai .

      Appendix A. Supplementary data

      The following is Supplementary data to this article:

      References

        • Frucht S.J.
        • Leurgans S.E.
        • Hallett M.
        • Fahn S.
        The unified myoclonus rating scale.
        Adv Neurol. 2002; 89: 361
        • Pietracupa S.
        • Bruno E.
        • Cavanna A.E.
        • Falla M.
        • Zappia M.
        • Colosimo C.
        Scales for hyperkinetic disorders: a systematic review.
        J Neurol Sci. 2015; 358 (November 15): 9-21
        • Goodfellow I.
        • Bengio Y.
        • Courville A.
        Deep learning.
        MIT Press, 2016
        • Cao Z.
        • Simon T.
        • Wei S.
        • Sheikh Y.
        Realtime multi-person 2D pose estimation using part affinity fields.
        CoRR. 2016; (abs/1611.08050)
        • Insafutdinov E.
        • Pishchulin L.
        • Andres B.
        • Andriluka M.
        • Schiele B.
        DeeperCut: a deeper, stronger, and faster multi-person pose estimation model.
        CoRR. 2016; (abs/1605.03170)
        • Kendall A.
        • Grimes M.
        • Cipolla R.
        Convolutional networks for real-time 6-DOF camera relocalization.
        CoRR. 2015; (abs/1505.07427)
        • Chen K.
        • Gabriel P.
        • Alasfour A.
        • Gong C.
        • Doyle W.K.
        • Devinsky O.
        • et al.
        Patient-specific pose estimation in clinical environments.
        IEEE J Transl Eng Health Med. 2018; 6 (October 10:2101111)
        • Kubota K.J.
        • Chen J.A.
        • Little M.A.
        Machine learning for large-scale wearable sensor data in Parkinson’s disease: concepts, promises, pitfalls, and futures.
        Mov Disord. 2016; 31 (September 01): 1314-1326
        • Li T.
        • Chen J.
        • Hu C.
        • Ma Y.
        • Wu Z.
        • Wan W.
        • et al.
        Automatic timed up-and-go sub-task segmentation for Parkinson’s disease patients using video-based activity classification.
        IEEE Trans Neural Syst Rehabil Eng. 2018; 26 (November 01): 2189-2199
        • Marchi V.
        • Hakala A.
        • Knight A.
        • D’Acunto F.
        • Scattoni M.L.
        • Guzzetta A.
        • et al.
        Automated pose estimation captures key aspects of General Movements at eight to 17 weeks from conventional videos.
        Acta Paediatr. 2019; (March 18)
        • Kälviäinen R.
        • Khyuppenen J.
        • Koskenkorva P.
        • Eriksson K.
        • Vanninen R.
        • Mervaala E.
        Clinical picture of EPM1-Unverricht-Lundborg disease.
        Epilepsia. 2008; 49 (April 01): 549
        • Magaudda A.
        • Ferlazzo E.
        • Nguyen V.H.
        • Genton P.
        Unverricht-Lundborg disease, a condition with self-limited progression: long-term follow-up of 20 patients.
        Epilepsia. 2006; 47 (May 01): 860
        • Koskiniemi M.
        • Van Vleymen B.
        • Hakamies L.
        • Lamusuo S.
        • Taalas J.
        Piracetam relieves symptoms in progressive myoclonus epilepsy: a multicentre, randomised, double blind, crossover study comparing the efficacy and safety of three dosages of oral piracetam with placebo.
        J Neurol Neurosurg Psychiatr. 1998; 64 (March 01): 344
        • Goldsmith D.
        • Minassian B.A.
        Efficacy and tolerability of perampanel in ten patients with Lafora disease.
        Epilepsy Behav. 2016; 62 (September 01): 132-135
        • Kalviainen R.
        • Genton P.
        • Andermann E.
        • Andermann F.
        • Magaudda A.
        • Frucht S.J.
        • et al.
        Brivaracetam in Unverricht-Lundborg disease (EPM1): results from two randomized, double-blind, placebo-controlled studies.
        Epilepsia. 2016; 57 (February 01): 210-221
        • Balasubramanian S.
        • Melendez-Calderon A.
        • Roby-Brami A.
        • Burdet E.
        On the analysis of movement smoothness.
        J Neuroeng Rehabil. 2015; 12 (December 09:112-015)
        • Hypponen J.
        • Aikia M.
        • Joensuu T.
        • Julkunen P.
        • Danner N.
        • Koskenkorva P.
        • et al.
        Refining the phenotype of Unverricht-Lundborg disease (EPM1): a population-wide Finnish study.
        Neurology. 2015; 84 (April 14): 1529-1536
        • Simonyan K.
        • Zisserman A.
        Very deep convolutional networks for large-scale image recognition.
        arXiv. 2020; 1409 (Invalid date Invalid date): 1556
        • He K.
        • Zhang X.
        • Ren S.
        • Sun J.
        Deep residual learning for image recognition.
        CoRR. 2015; (abs/1512.03385)
        • Deng J.
        • Dong W.
        • Socher R.
        • Li L.
        • Li K.
        • Fei-fei L.
        Imagenet: a large-scale hierarchical image database.
        CVPR. 2009;
        • Andriluka M.
        • Pishchulin L.
        • Gehler P.
        • Schiele B.
        2D human pose estimation: New benchmark and state of the art analysis.
        in: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, USA: IEEE Computer Society2014
        • Lin T.
        • Maire M.
        • Belongie S.J.
        • Bourdev L.D.
        • Girshick R.B.
        • Hays J.
        • et al.
        Microsoft COCO: common objects in context.
        CoRR. 2014; (abs/1405.0312)
        • Fortun D.
        • Bouthemy P.
        • Kervrann C.
        Optical flow modeling and computation: a survey.
        Comput Vis Image Underst. 2015; 134 (May 01): 21
        • Akoglu H.
        User’s guide to correlation coefficients.
        Turk J Emerg Med. 2018; 18 (August 07): 91-93
        • Rissanen S.M.
        • Ruonala V.
        • Pekkonen E.
        • Kankaanpaa M.
        • Airaksinen O.
        • Karjalainen P.A.
        Signal features of surface electromyography in advanced Parkinson’s disease during different settings of deep brain stimulation.
        Clin Neurophysiol. 2015; 126 (December 01): 2290-2298
        • Rodriguez-Blazquez C.
        • Forjaz M.J.
        • Kurtis M.M.
        • Balestrino R.
        • Martinez-Martin P.
        Rating scales for movement disorders with sleep disturbances: a narrative review.
        Front Neurol. 2018; 9 (June 07): 435