If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
A novel method presented determines if a clinical trial is reversible in time.
•
Reversible trials suggests natural variability in seizure frequency.
•
In 3 datasets, reversibility was present during a placebo condition.
•
Trial placebo response rates may reflect natural variability in seizure frequency.
Abstract
Purpose
Clinical epilepsy drug trials have been measuring increasingly high placebo response rates, up to 40%. This study was designed to examine the relationship between the natural variability in epilepsy, and the placebo response seen in trials. We tested the hypothesis that ‘reversing' trial direction, with the baseline period as the treatment observation phase, would reveal effects of natural variability.
Method
Clinical trial simulations were run with time running forward and in reverse. Data sources were: SeizureTracker.com (patient reported diaries), a randomized sham-controlled TMS trial, and chronically implanted intracranial EEG electrodes. Outcomes were 50%-responder rates (RR50) and median percentage change (MPC).
Results
The RR50 results showed evidence that temporal reversal does not prevent large responder rates across datasets. The MPC results negative in the TMS dataset, and positive in the other two.
Conclusions
Typical RR50s of clinical trials can be reproduced using the natural variability of epilepsy as a substrate across multiple datasets. Therefore, the placebo response in epilepsy clinical trials may be attributable almost entirely to this variability, rather than the “placebo effect”.
Institute of Medicine (U.S.), Committee on the Public Health Dimensions of the Epilepsies, England MJ. Epilepsy across the spectrum promoting health and understanding. Washington, D.C.: National Academies Press, 2012 http://www.nap.edu/catalog.php?record_id=13379 (Accessed 28 August 2014).
]. In spite of this, there has been great difficulty in providing patients new treatments that can improve seizure control. One significant challenge has been financial; clinical trials are increasingly expensive [
], and the risk of conducting a trial that fails to show superiority over placebo is a significant concern. This is compounded by the increasing placebo response rates over recent decades [
Factors determining response to antiepileptic drugs in randomized controlled trials. A systematic review and meta-analysis: Response to AEDs in Randomized Trials.
]. Thus, finding ways to reduce the placebo response can help contain costs for clinical trials, and in turn accelerate development of new therapies for epilepsy.
The term “placebo response” is used here to capture the effect size measured in the placebo arm of a clinical trial. The “placebo effect” for our purposes is defined as a measureable effect relevant to the disease that is directly attributable to the placebo given during the trial [
]. Psychological causes include the “placebo effect”, classical conditioning, the Hawthorne effect, symbols and expectations, and social learning. Regression-to-the-mean for our purposes refers to the impact of patients who are sicker than their usual registering for a trial. Such patients are expected to subsequently return to their “average disease” state without any intervention. Both regression-to-the-mean and psychological causes produce an improvement in disease. Natural variability in the context of epilepsy trials relates to the expected variation in seizure frequency over time, even in the absence of a change in treatment. Although these causes have been well described for many years, no study has attempted to dissect the relative contribution of each in epilepsy trials. Yet the time-course of these causes is expected to differ significantly. Psychological effects would be expected to occur after placebo administration and gradually wear off [
], regression to the mean would be expected to occur prior to placebo administration and continue afterwards, and natural variability would show no clear pattern before or after placebo administration. Moreover, if one reversed the direction of time, the first two of these effects would exhibit minimal responder rates, while the third may result in similar rates as the forward direction. Of note, the reverse temporal responder rate is not predictable given the traditional responder rate, because different baselines are used, and they represent different fractions of the overall trial population.
Recently, our group reported preliminary evidence that placebo response may simply reflect natural variability in epilepsy [
]. Based on that observation, we sought to validate this claim using several data sources that range from one of the world’s largest patient-managed seizure diary database (SeizureTracker.com) [
]. We tested the hypothesis that ‘reversing' trial direction, with the baseline period as the treatment observation phase, would show large and roughly similar responder rates to ‘forward’ trial analysis due to natural variability (Fig. 1).
Fig. 1Time reversal. A model for calculating outcomes from a clinical trial in forward and reverse direction. Small vertical arrows represent seizures times. Baseline and treatment periods can be redefined for “reverse” calculation of effect. In this hypothetical patient example, there are 5 seizures shown in the baseline period, 2 seizures during titration and 2 during treatment. For this illustration, we assume an 8-week baseline, 4-week titration, and 8-week treatment period. The percentage change for this patient is therefore 60% (using (5–2)/5). In the reverse direction, the patient has a negative 50% change (using (2–5)/2). Thus, in the forward calculation, this patient would be a 50%-responder, but not so in the reverse calculation.
To illustrate the key concepts being tested, 3 artificial datasets were constructed: (A) “psychological effects”, (B) “regression-to-the-mean”, and (C) “natural variability” (Fig. 2). In each, 150 simulated patient diaries (5 entries per patient) were generated with random number generators to represent 5 months. In all cases, the patient’s “usual” monthly seizure rate was 15. For all three, zero-mean normally distributed noise was added (A: std. dev. = 8, B: std. dev. = 8, C: std. dev = 10). There was more variability in the random noise added to C intentionally. For A, each patient had the following added to their 5 diary entries: 0, 0, −8, −7, −3. For B, each patient had the following added to their diary entries: 8, 6, 2, 0, 0. For C, nothing was added to the diary values. These artificial sets of patients were analyzed in the forward and reverse time analysis of 50% responder rate, in order to demonstrate the pattern expected depending on the cause of placebo response.
Fig. 2Components of placebo response. The data presented here is artificially constructed to demonstrate the differences between causes of the placebo response. Left set of graphs: examples of monthly seizure frequencies measured in a hypothetical clinical trial with 2 months of baseline and 3 months of therapy (treatment) over 150 simulated patients. Thick vertical bar: onset of the treatment condition. Dashed horizontal bar: the patient’s “usual” seizure frequency = 15/month. Middle graphs: boxplots of the range of baseline and test values collected in the simulations. Right graphs: for forward and reverse calculations, % change between the baseline and treatment conditions. In the case of psychological effect (A), the onset of the treatment phase decreases the usual seizure frequency, and gradually over time the monthly frequency increases to the usual level. In regression-to-the-mean (B), the baseline begins much higher than the patient’s usual, and gradually decreases. In the case of natural fluctuations (C), the frequency goes up and down haphazardly. All three situations would result some patients showing a decrease from baseline to treatment, but for very different reasons. Of note, only (C) will have a large expected reverse effect size (relative to the forward effect size). The other two (A and B) would be expected to have a minimal effect in the reverse calculation, as seen in the right graphs. Simulations A and B did not achieve a statistically significant positive τ value while C did. This confirms that C is the only example that is temporally reversible.
2.2 General simulation technique with realistic data
For each of the datasets, a common simulation framework was used. A trial-sized segment of with 8 weeks of baseline and 8 weeks of treatment was used from the start of each patient’s diary. A “titration period” was included merely to be more clinically realistic. In the case of SeizureTracker and NeuroVista, a 4-week “titration” period was inserted in between the two. In the TMS study, the structure of the study included a 2-week intervention period, roughly analogous to the 4-week titration period. For all datasets, patients were only included if they had ≥8 seizures during the baseline period (i.e. at least 1 seizure per week on average). Data were then analyzed for 2 outcomes, 50%-responder rate (RR50) and median% seizure frequency change (MPC). RR50 represented the proportion of patients that had a 50% or larger reduction in 28-day seizure rate during the intervention period compared with their baseline. MPC represented the median (across patients) % change in 28-day seizure rate between baseline and intervention. When time flowed in the forward direction, this was referred to as “forward.” All patients' seizure diaries were then temporally reversed, such that the final day became the first day, and the first day became the last, and so forth (Fig. 1). When analyzing the reverse temporal flow, the eligibility criteria were re-applied. RR50 and MPC were then calculated on these reversed diaries, referred to as “reverse.”
We introduce a metric, τ, for determining if temporal reversal appears to matter. For the RR50 rate in the forward direction denoted F, and the RR50 rate in the reverse direction denoted R:
(1)
If τ ≥ 0, the relative contribution of natural variability to placebo response could be said to be large, whereas if τ < 0, then it could be small (compared to other influences). The methodology for calculating p-values is described in the Appendix.
2.3 SeizureTracker based realistic simulation
SeizureTracker is a free online and mobile patient reported database of seizure diaries [
]. Data was exported from December 2007 through May 2016. Of note, this represents an expanded set of data compared with that originally studied previously [
], adding an additional 2 years. Patients with no seizures, no age reported, or absurd ages (i.e. >200 years old) were excluded. Seizures with invalid dates or identically repeated records were excluded.
All SeizureTracker simulations began with time zero representing the first recorded seizure in an individual’s diary [
]. For each simulation, eligibility criteria were applied in the forward direction and patients were selected. To be eligible, patients needed on average at least 1 seizure per week during the baseline period [
]. Selected patients were then enrolled in a simulated clinical trial lasting 5 months (2-month baseline, 1-month titration, and a 2-month treatment period), to match typical modern clinical trials [
]. The outcome measures of RR50 and MPC in 28-day seizure frequency were calculated in the typical fashion using the baseline and treatment periods. The same patients’ diaries were then reversed temporally, such that the last moment of the trial became the first, and the first moment became the last. A special requirement was imposed only in the SeizureTracker dataset that any patient who became “seizure-free” during the treatment period was not included in final calculations. This was done to avoid the possibility of including patients with “diary fatigue,” i.e. patients who simply stopped recording events partway through the diary.
2.4 NeuroVista based realistic simulation
In the NeuroVista study of a long-term seizure prediction device, 15 patients were implanted with chronic subdural electrodes for 7–23 (median 12) months, and continuous intracranial EEG data was recorded along with external microphone audio. These two inputs were analyzed according to previously described methods [
], to obtain monthly seizure counts for each patient. Simulations were run from each patient starting at the beginning of subdural recording, similar to SeizureTracker, utilizing 2-month baseline, 1-month titration, and 2-month treatment periods. Thus, a total of 5 months of seizure data was used from each patient regardless of total implantation time. For this study, a “seizure” included any electrographically confirmed seizure, with or without clinical correlate.
The same simulations were run a second time using only those seizures classified as “clinical” and excluding those that were electrographic only. This was done to determine if the results differ from the simulations that include all seizures from the NeuroVista data.
2.5 TMS study based realistic simulation
A randomized clinical trial performed at NIH, testing the question of clinical effectiveness of one hertz TMS on reducing seizures has been previously reported [
]. This trial comprised 12 patients who were given “sham” TMS, and 12 who received TMS treatment. The question the original study asked was: could a statistically significant difference be appreciated between the two groups. The answer from their original analysis was no. There was an 8-week baseline seizure count, 1 week of TMS or sham treatment, followed by an 8-week follow-up. The original study reported percentage change, rather than MPC or 50%-responder rate values. Our simulation used the original raw data to calculate RR50 and MPC values for both the forward and reverse time methods from the sham group only. We assumed that the conclusion of the 2002 study was correct, that the two arms were essentially “placebo”, however we treated the arms of the trial separately in case there was a subtle unmeasured difference between them.
3. Results
The artificial simulation is shown in Fig. 2. The RR50 in the forward direction for both group A (psychological causes) and B (regression-to-the-mean) was very large, while the reverse RR50 was very small. In group C (natural variability), the RR50 was approximately equal in both directions. The value of τ was −0.50 for A (p > 0.10), −0.92 for B (p > 0.10), and 1.17 for C (p = 0.008). Therefore, groups A and B do not show strong evidence of reversibility (because τ < 0 and p > 0.05), while C does (τ > 0, and p < 0.05).This result is consistent with expectation, given that natural variability should be temporally reversible.
Of the 22,360 patients initially exported for SeizureTracker, 13,488 patients were retained after initial exclusion criteria were applied. Of the 1,210,036 seizures in the database, 1,112,343 seizures were retained after exclusions were applied. For SeizureTracker, 1552 patients were eligible in the forward simulation while 1007 were eligible for the reverse. For NeuroVista, 11 of the original 15 patients were used in the forward and reverse simulation. For the TMS study, all 12 “sham” patients from the TMS study were included in the forward analysis while 10 were included in the reverse analysis.
Fig. 3A summarizes the RR50 responses. τ values were: Neurovista 4.0 (p = 0.002), TMS 2.6 (p = 0.044), SeizureTracker 0.6 (p < 0.001). Because τ > 0 and p < 0.05, all 3 datasets demonstrated statistically significant evidence of reversibility. All three were compatible with natural variability being the largest influence (similar to group C in Fig. 2).
Fig. 3The forward and reverse calculations. A: the 50% responder rate (RR50) is shown for each dataset. In order for τ to be positive, the reverse RR50 must be >50% of the forward value. This is true in all 3 cases, and each reached statistical signficance (p < 0.05). B: The median% change (MPC) is shown for each dataset. In NeuroVista and SeizureTracker, MPC values are similar regardless of direction of temporal flow. In the TMS case, the MPC values show opposite signs.
Fig. 3B summarizes the MPC responses. NeuroVista and SeizureTracker both demonstrated large positive MPC values in both forward and reverse computations, while the TMS study showed a positive “forward” MPC and a negative “reverse” MPC.
When using the “clinical” seizures alone from the NeuroVista data, the RR50 values were 14.3% forward, and 33.3% reverse. The MPC values were 24.2% forward, and 38.1% reverse. The forward computation used 7 patients, while the reverse used 9. τ was 3.7 with p = 0.022. As with the complete NeuroVista dataset, τ > 0, confirming statistically significant temporal reversibility.
4. Discussion
This study used ‘Big Data’ and formal clinical trial data to present evidence that natural variability may account for the majority of the placebo response in epilepsy trials. Indeed, the RR50 in both forward and reverse (Fig. 3A) were similar to placebo arm values seen in typical clinical trials [
]. This suggests that RR50 may be less optimal as a trial outcome metric, because the natural variability (which is always present) is not adjusted for (Fig. 2). Given that in two of the three cases, positive MPC values were obtained using the “reverse” calculation, it may be worth considering alternatives to MPC as well.
Our analysis highlights the connection between inclusion criteria and placebo response. In the reverse analysis, MPC appears positive and similar to the forward time analysis only if eligibility criteria are re-applied. The decreased number of patients in the reverse direction suggests that some patients had a significant decrease in average seizure counts in the forward direction. The reverse eligibility criteria enforced a form of regression-to-the-mean influence by selecting “sicker” moments in some patients. Selection of less strict eligibility criteria may allow more patients in their “usual state” and thus avoid excessive regression-to-the-mean influence. By design, the regression-to-the-mean effects seen here were caused by natural variability. This is in contradistinction to the effect of acutely ill patients enrolling in trials due to higher than average seizure rates – such a circumstance would not be easily reversible in this analysis.
The NeuroVista dataset was unique because it included information about seizures that were clinically detectable and clinically silent. The latter were not included in patient-reported outcomes (such as SeizureTracker, or typical randomized controlled trials for drugs or devices), so it is unknown what influence including them would have on outcome measures such as RR50 and MPC. When we re-analyzed the data using only the subset of seizures with clinical manifestations that were also reported by the patient, we found that the overall results were unchanged. This shows that despite concerns about under-reporting and over-reporting of seizures in patient reported outcomes [
], natural variability in epilepsy influencing placebo responses is likely to have prominent influence on trial results.
A meta-analysis of topiramate, tiagabine and levetiracetam studies supports our results. Between 14.5% and 19% of patients experienced 50% or greater increase in their baseline seizure-frequency [
]. Those patients would be considered to have achieved a 50%-response if the studies were analyzed in reverse time. Compared with our 50%-responder rates (Fig. 3), the same range of values was found regardless of the direction that time was flowing. The RR50 for epilepsy trials has been 4–40% [
Very little is known about the elemental constituents of the placebo response in epilepsy. Although a number of studies suggest possibilities such as regression-to-the-mean [
Nocebo effect in refractory partial epilepsy during pre-surgical monitoring: systematic review and meta-analysis of placebo-controlled clinical trials.
Nocebo effect in refractory partial epilepsy during pre-surgical monitoring: systematic review and meta-analysis of placebo-controlled clinical trials.
]. No long-term study has attempted to quantify the expected variability within and across patients.
The techniques described in this manuscript represent a novel method of approaching clinical trial data to derive new insights into disease variability. Indeed, many other diseases lead to episodic symptoms with poorly characterized longitudinal variability. Examples abound in neurology (e.g. headache, TIA, stroke, multiple sclerosis, cataplexy, stiff-person syndrome, autoimmune encephalitis, etc.), psychiatric conditions (e.g. psychotic episodes, depression, manic episodes, PTSD, panic attacks) and general medical conditions (e.g. hypoglycemic episodes in diabetes, flares in asthma, congestive heart failure, inflammatory bowel disease). It remains unknown if the components of the “placebo effect” in epilepsy are representative or unique.
Interestingly, all three datasets were able to produce reasonable results that mirror typical “placebo effects”, though only the TMS study included a form of placebo, and this was not true in the reverse calculation of the TMS study. Thus, it appears from these data that the presence or absence of a true placebo may be irrelevant to the typical placebo response seen in clinical trials.
4.1 Limitations
There are several limitations to our study. Since it is retrospective, incomplete control over the data biases and quality was inherent. By evaluating data obtained from very different sources in a common framework as well as applying preprocessing to increase signal-to-noise ratios, we sought to counteract these problems.
Since the multiple modalities for seizure recording had similar general results, the combined data seem to represent consistent representation of natural variability with differing levels of superimposed noise. The data derived from SeizureTracker can be thought of as having high noise levels, due to incomplete patient reporting, as well as reporting biases (under-reporting and over-reporting) [
]. The TMS data also suffers from the biases of patient reported data, though due to close follow-up, the quality of that data is expected to be higher. Unlike the other datasets, the TMS study had all the elements of a typical clinical trial: strict eligibility criteria that could promote regression-to-the-mean, close observation that could result in the Hawthorne effect [
]. Therefore, the TMS study included an intermediate noise level. The NeuroVista data comprised the lowest level of noise, because regardless of recall, seizures were objectively recorded and analyzed longitudinally with implanted electrodes. Reviewing convergent results from data of various noise levels allows increased confidence in the results.
The re-application of eligibility criteria in both temporal directions was necessary to make a “fair” comparison between the two. Simultaneously, it imposes a form of regression-to-the-mean influence that is driven by natural variability. For instance, suppose a patient typically has 2 seizures per month, but by chance alone happens to have a couple of months with 4 seizures per month during the baseline period. Such a patient would be eligible for the hypothetical study we did here, but would appear to improve despite no intervention after the baseline period. This patient would be counted among the 50%-responders as well. An analogous situation can occur in the reverse temporal analysis. Without imposing eligibility criteria in both directions, the populations comparison would be inappropriate because the regression-to-the-mean influence described above would only impact the forward but not the reverse population.
This study also was limited by the sample size of NeuroVista and TMS. Datasets with larger samples of patients may identify different patterns that those found in the present study. However, the p values suggest that the results obtained were unlikely to be due to chance alone.
Fundamentally, a retrospective study of this kind cannot provide strong evidence that placebo response is unrelated to placebo. At best, it can show suggestive evidence, which this study does. A prospective randomized clinical trial with seizure reduction as an outcome measure, with placebo and no intervention, would offer the best possible evidence, as has been done in several other diseases [
]. That study found no difference between any group for seizure reduction.
A final consideration is the potential for placebo to produce harm, i.e. the “nocebo” effect. In a clinical trial, participants provided with placebo may improve or worsen for all the reasons discussed above, but some may experience worsening of seizure frequency due to nocebo influence. Although not yet demonstrated directly in epilepsy, evidence is mounting that nocebo influences may be present in controlled trials of anti-seizure drugs. For example, one meta-analysis found that 3.2% of placebo treated patients withdrew from randomized trials due to adverse events [
]. The quantitative impact of nocebo on clinical trial outcomes remains an open area for future studies.
5. Conclusions
The natural variability of epilepsy plays a major role in generating the response seen in placebo arms of clinical trials. This concept has been demonstrated in 3 very different datasets with similar results. The implication is that a similar pattern could be achieved without a placebo at all, suggesting that, at least in epilepsy, the “placebo effect” may not exist. Moreover, use of strict eligibility requirements may inadvertently increase regression-to-the-mean effects. Further confirmatory studies will be needed to test these ideas in larger sets of clinical trial data. A randomized clinical trial with both a placebo arm and a “no intervention” arm may represent the ideal confirmatory trial
Author contributions
Dr. Goldenholz: study design, acquisition of secondary data sets, analysis, manuscript writing.
Dr. Strashny: development of the p-value calculation; calculation of the p-values for actual and simulated datasets.
Dr. Cook: collection and analysis of primary dataset, editing manuscript.
Mr. Moss: collection of primary dataset, editing manuscript.
Dr. Theodore: collection and analysis of primary dataset, editing manuscript.
Ethics statement
We confirm that we have read the Journal’s position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.
Conflicts of interest
None of the authors has any conflict of interest to disclose.
Acknowledgements:
This study in full was supported by the National Institute of Neurological Disorders and Stroke Division of Intramural Research. This study was approved by the Office of Human Subject Research Protection under protocol #12301. Use of the NeuroVista and SeizureTracker data was facilitated by the International Seizure Diary Consortium (https://sites.google.com/site/isdchome/). The findings and conclusions in this paper are those of the authors and do not necessarily represent the official position of the National Center for Health Statistics or the Centers for Disease Control.
Appendix
This appendix explains the calculation of p-values for τ. Let nD , D∈{F, R}, be the number of patients in the forward and reverse directions respectively. Let PD be the probability that, moving in a particular direction, a patient achieves the RR50 criterion (a 50% or larger reduction). Then, the number of patients, moving in a particular direction, who achieve the RR50 criterion has the binomial distribution: KD∼ B (nD, PD).
Conditional on kF, the observed number of patients, moving in the forward direction, who achieve the RR50 criterion, the expected value of PF is E [PF]=kF⁄nF.
The null hypothesis is that the expected value of PR is less than or equal to half of the expected value of PF: H0:E[PR] ≤ 0.5E[PF]; H1:E [PR] > 0.5E[PF]. Thus, under the null hypothesis, the expected value of PR is E[PR |H0] = 0.5 kF⁄nF . The distribution of KR under the null hypothesis is KR |H0∼ B(nR, 0.5 kF⁄nF). The p-value is the probability that KR |H0 is greater or equal to the observed number of patients, moving in the reverse direction, who achieve the RR50 criterion: p= Pr (KR |H0 ≥ kR).
Institute of Medicine (U.S.), Committee on the Public Health Dimensions of the Epilepsies, England MJ. Epilepsy across the spectrum promoting health and understanding. Washington, D.C.: National Academies Press, 2012 http://www.nap.edu/catalog.php?record_id=13379 (Accessed 28 August 2014).
Factors determining response to antiepileptic drugs in randomized controlled trials. A systematic review and meta-analysis: Response to AEDs in Randomized Trials.
Nocebo effect in refractory partial epilepsy during pre-surgical monitoring: systematic review and meta-analysis of placebo-controlled clinical trials.
30597-6&id=gr1.jpg){kind=link}
30597-6&id=gr2.jpg){kind=link}
30597-6&id=gr3.jpg){kind=link}