Research Article| Volume 53, P31-36, December 2017
• PDF [992 KB]PDF [992 KB]
• Top

# A multi-dataset time-reversal approach to clinical trial placebo response and the relationship to natural variability in epilepsy

Open ArchivePublished:October 23, 2017

## Highlights

• A novel method presented determines if a clinical trial is reversible in time.
• Reversible trials suggests natural variability in seizure frequency.
• In 3 datasets, reversibility was present during a placebo condition.
• Trial placebo response rates may reflect natural variability in seizure frequency.

## Abstract

### Purpose

Clinical epilepsy drug trials have been measuring increasingly high placebo response rates, up to 40%. This study was designed to examine the relationship between the natural variability in epilepsy, and the placebo response seen in trials. We tested the hypothesis that ‘reversing' trial direction, with the baseline period as the treatment observation phase, would reveal effects of natural variability.

### Method

Clinical trial simulations were run with time running forward and in reverse. Data sources were: SeizureTracker.com (patient reported diaries), a randomized sham-controlled TMS trial, and chronically implanted intracranial EEG electrodes. Outcomes were 50%-responder rates (RR50) and median percentage change (MPC).

### Results

The RR50 results showed evidence that temporal reversal does not prevent large responder rates across datasets. The MPC results negative in the TMS dataset, and positive in the other two.

### Conclusions

Typical RR50s of clinical trials can be reproduced using the natural variability of epilepsy as a substrate across multiple datasets. Therefore, the placebo response in epilepsy clinical trials may be attributable almost entirely to this variability, rather than the “placebo effect”.

## 1. Background

Nearly 1 out of 3 of patients with epilepsy do not yet have control over their seizures [
• Kwan P.
• Brodie M.J.
Early identification of refractory epilepsy.
], and face 1.6-3 fold increase in mortality compared to the general population [

Institute of Medicine (U.S.), Committee on the Public Health Dimensions of the Epilepsies, England MJ. Epilepsy across the spectrum promoting health and understanding. Washington, D.C.: National Academies Press, 2012 http://www.nap.edu/catalog.php?record_id=13379 (Accessed 28 August 2014).

]. In spite of this, there has been great difficulty in providing patients new treatments that can improve seizure control. One significant challenge has been financial; clinical trials are increasingly expensive [

PhRMA. Profile Biopharmaceutical Research Industry. 2015 http://www.phrma.org/sites/default/files/pdf/2015_phrma_profile.pdf.

], and the risk of conducting a trial that fails to show superiority over placebo is a significant concern. This is compounded by the increasing placebo response rates over recent decades [
• Rheims S.
• Perucca E.
• Cucherat M.
• et al.
Factors determining response to antiepileptic drugs in randomized controlled trials. A systematic review and meta-analysis: Response to AEDs in Randomized Trials.
]. Thus, finding ways to reduce the placebo response can help contain costs for clinical trials, and in turn accelerate development of new therapies for epilepsy.
The term “placebo response” is used here to capture the effect size measured in the placebo arm of a clinical trial. The “placebo effect” for our purposes is defined as a measureable effect relevant to the disease that is directly attributable to the placebo given during the trial [
• Goldenholz D.M.
• Goldenholz S.R.
Response to placebo in clinical epilepsy trials-old ideas and new insights.
,
• Goldenholz D.M.
• Moss R.
• Scott J.
• et al.
Confusing placebo effect with natural history in epilepsy: a big data approach.
]. The placebo response has been thought to comprise a number of unrelated causes, principally: (A) psychological factors [
• Fisher R.S.
• Blum D.E.
• DiVentura B.
• et al.
Seizure diaries for clinical research and practice: limitations and future prospects.
,
• Theodore W.H.
• Hunter K.
• Chen R.
• et al.
Transcranial magnetic stimulation for the treatment of seizures. A controlled study.
,
• Cook M.J.
• O’Brien T.J.
• Berkovic S.F.
• et al.
Prediction of seizure likelihood with a long-term, implanted seizure advisory system in patients with drug-resistant epilepsy: a first-in-man study.
,
• Cook M.J.
• Karoly P.J.
• Freestone D.R.
• et al.
Human focal seizures are characterized by populations of fixed duration and interval.
,
• Somerville E.R.
Aggravation of partial seizures by antiepileptic drugs − is there evidence from clinical trials.
,
• French J.A.
• Krauss G.L.
• Wechsler R.T.
• et al.
Perampanel for tonic-clonic seizures in idiopathic generalized epilepsy.
,
• Rheims S.
• Cucherat M.
• Arzimanoglou A.
• et al.
Greater response to placebo in children than in adults: a systematic review and meta-analysis in drug-resistant partial epilepsy.
,
• Morton V.
• Torgerson D.J.
Regression to the mean: treatment effect without the intervention.
,
• McCambridge J.
• Witton J.
• Elbourne D.R.
Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects.
,
• Jarius S.
• Wildemann B.
And Pavlov still rings a bell: summarising the evidence for the use of a bell in Pavlov’s iconic experiments on classical conditioning.
], (B) regression-to-the-mean [
• Espay A.J.
• Norris M.M.
• Eliassen J.C.
• et al.
Placebo effect of medication cost in Parkinson disease: a randomized double-blind study.
] and (C) natural variability of disease [
• Theodore W.H.
• Hunter K.
• Chen R.
• et al.
Transcranial magnetic stimulation for the treatment of seizures. A controlled study.
]. Psychological causes include the “placebo effect”, classical conditioning, the Hawthorne effect, symbols and expectations, and social learning. Regression-to-the-mean for our purposes refers to the impact of patients who are sicker than their usual registering for a trial. Such patients are expected to subsequently return to their “average disease” state without any intervention. Both regression-to-the-mean and psychological causes produce an improvement in disease. Natural variability in the context of epilepsy trials relates to the expected variation in seizure frequency over time, even in the absence of a change in treatment. Although these causes have been well described for many years, no study has attempted to dissect the relative contribution of each in epilepsy trials. Yet the time-course of these causes is expected to differ significantly. Psychological effects would be expected to occur after placebo administration and gradually wear off [
• Benedetti F.
• Maggi G.
• Lopiano L.
• et al.
Open versus hidden medical treatments: the patient’s knowledge about a therapy affects the therapy outcome.
], regression to the mean would be expected to occur prior to placebo administration and continue afterwards, and natural variability would show no clear pattern before or after placebo administration. Moreover, if one reversed the direction of time, the first two of these effects would exhibit minimal responder rates, while the third may result in similar rates as the forward direction. Of note, the reverse temporal responder rate is not predictable given the traditional responder rate, because different baselines are used, and they represent different fractions of the overall trial population.
Recently, our group reported preliminary evidence that placebo response may simply reflect natural variability in epilepsy [
• Benedetti F.
Placebo effects: from the neurobiological paradigm to translational implications.
]. Based on that observation, we sought to validate this claim using several data sources that range from one of the world’s largest patient-managed seizure diary database (SeizureTracker.com) [
• Benedetti F.
Placebo effects: from the neurobiological paradigm to translational implications.
,
• a French J.
• Krauss G.L.
• Biton V.
• et al.
Adjunctive perampanel for refractory partial-onset seizures: randomized phase III study 304.
], and a transcranial magnetic stimulation (TMS) clinical trial [
• Callaghan B.C.
• Anand K.
• Hesdorffer D.
• et al.
Likelihood of seizure remission in an adult population with refractory epilepsy.
], to one of the world’s most accurate seizure diary dataset based on longitudinal intracranial monitoring (NeuroVista data) [
• Callaghan B.
• Schlesinger M.
• Rodemer W.
• et al.
Remission and relapse in a drug-resistant epilepsy population followed prospectively.
]. We tested the hypothesis that ‘reversing' trial direction, with the baseline period as the treatment observation phase, would show large and roughly similar responder rates to ‘forward’ trial analysis due to natural variability (Fig. 1).

## 2. Methods

### 2.1 Artificial simulation

To illustrate the key concepts being tested, 3 artificial datasets were constructed: (A) “psychological effects”, (B) “regression-to-the-mean”, and (C) “natural variability” (Fig. 2). In each, 150 simulated patient diaries (5 entries per patient) were generated with random number generators to represent 5 months. In all cases, the patient’s “usual” monthly seizure rate was 15. For all three, zero-mean normally distributed noise was added (A: std. dev. = 8, B: std. dev. = 8, C: std. dev = 10). There was more variability in the random noise added to C intentionally. For A, each patient had the following added to their 5 diary entries: 0, 0, −8, −7, −3. For B, each patient had the following added to their diary entries: 8, 6, 2, 0, 0. For C, nothing was added to the diary values. These artificial sets of patients were analyzed in the forward and reverse time analysis of 50% responder rate, in order to demonstrate the pattern expected depending on the cause of placebo response.

### 2.2 General simulation technique with realistic data

For each of the datasets, a common simulation framework was used. A trial-sized segment of with 8 weeks of baseline and 8 weeks of treatment was used from the start of each patient’s diary. A “titration period” was included merely to be more clinically realistic. In the case of SeizureTracker and NeuroVista, a 4-week “titration” period was inserted in between the two. In the TMS study, the structure of the study included a 2-week intervention period, roughly analogous to the 4-week titration period. For all datasets, patients were only included if they had ≥8 seizures during the baseline period (i.e. at least 1 seizure per week on average). Data were then analyzed for 2 outcomes, 50%-responder rate (RR50) and median% seizure frequency change (MPC). RR50 represented the proportion of patients that had a 50% or larger reduction in 28-day seizure rate during the intervention period compared with their baseline. MPC represented the median (across patients) % change in 28-day seizure rate between baseline and intervention. When time flowed in the forward direction, this was referred to as “forward.” All patients' seizure diaries were then temporally reversed, such that the final day became the first day, and the first day became the last, and so forth (Fig. 1). When analyzing the reverse temporal flow, the eligibility criteria were re-applied. RR50 and MPC were then calculated on these reversed diaries, referred to as “reverse.”
We introduce a metric, τ, for determining if temporal reversal appears to matter. For the RR50 rate in the forward direction denoted F, and the RR50 rate in the reverse direction denoted R:
$τ=2R−FF$
(1)

If τ ≥ 0, the relative contribution of natural variability to placebo response could be said to be large, whereas if τ < 0, then it could be small (compared to other influences). The methodology for calculating p-values is described in the Appendix.

### 2.3 SeizureTracker based realistic simulation

SeizureTracker is a free online and mobile patient reported database of seizure diaries [
• a French J.
• Krauss G.L.
• Biton V.
• et al.
Adjunctive perampanel for refractory partial-onset seizures: randomized phase III study 304.
]. Data was exported from December 2007 through May 2016. Of note, this represents an expanded set of data compared with that originally studied previously [
• Benedetti F.
Placebo effects: from the neurobiological paradigm to translational implications.
], adding an additional 2 years. Patients with no seizures, no age reported, or absurd ages (i.e. >200 years old) were excluded. Seizures with invalid dates or identically repeated records were excluded.
All SeizureTracker simulations began with time zero representing the first recorded seizure in an individual’s diary [
• Benedetti F.
Placebo effects: from the neurobiological paradigm to translational implications.
]. For each simulation, eligibility criteria were applied in the forward direction and patients were selected. To be eligible, patients needed on average at least 1 seizure per week during the baseline period [
• Callaghan B.C.
• Anand K.
• Hesdorffer D.
• et al.
Likelihood of seizure remission in an adult population with refractory epilepsy.
]. Selected patients were then enrolled in a simulated clinical trial lasting 5 months (2-month baseline, 1-month titration, and a 2-month treatment period), to match typical modern clinical trials [
• Choi H.
• a Heiman G.
• Munger Clary H.
• et al.
Seizure remission in adults with long-standing intractable epilepsy: an extended follow-up.
]. The outcome measures of RR50 and MPC in 28-day seizure frequency were calculated in the typical fashion using the baseline and treatment periods. The same patients’ diaries were then reversed temporally, such that the last moment of the trial became the first, and the first moment became the last. A special requirement was imposed only in the SeizureTracker dataset that any patient who became “seizure-free” during the treatment period was not included in final calculations. This was done to avoid the possibility of including patients with “diary fatigue,” i.e. patients who simply stopped recording events partway through the diary.

### 2.4 NeuroVista based realistic simulation

In the NeuroVista study of a long-term seizure prediction device, 15 patients were implanted with chronic subdural electrodes for 7–23 (median 12) months, and continuous intracranial EEG data was recorded along with external microphone audio. These two inputs were analyzed according to previously described methods [
• Callaghan B.
• Schlesinger M.
• Rodemer W.
• et al.
Remission and relapse in a drug-resistant epilepsy population followed prospectively.
,
• Luciano A.L.
• Shorvon S.D.
Results of treatment changes in patients with apparently drug-resistant chronic epilepsy.
], to obtain monthly seizure counts for each patient. Simulations were run from each patient starting at the beginning of subdural recording, similar to SeizureTracker, utilizing 2-month baseline, 1-month titration, and 2-month treatment periods. Thus, a total of 5 months of seizure data was used from each patient regardless of total implantation time. For this study, a “seizure” included any electrographically confirmed seizure, with or without clinical correlate.
The same simulations were run a second time using only those seizures classified as “clinical” and excluding those that were electrographic only. This was done to determine if the results differ from the simulations that include all seizures from the NeuroVista data.

### 2.5 TMS study based realistic simulation

A randomized clinical trial performed at NIH, testing the question of clinical effectiveness of one hertz TMS on reducing seizures has been previously reported [
• Callaghan B.C.
• Anand K.
• Hesdorffer D.
• et al.
Likelihood of seizure remission in an adult population with refractory epilepsy.
]. This trial comprised 12 patients who were given “sham” TMS, and 12 who received TMS treatment. The question the original study asked was: could a statistically significant difference be appreciated between the two groups. The answer from their original analysis was no. There was an 8-week baseline seizure count, 1 week of TMS or sham treatment, followed by an 8-week follow-up. The original study reported percentage change, rather than MPC or 50%-responder rate values. Our simulation used the original raw data to calculate RR50 and MPC values for both the forward and reverse time methods from the sham group only. We assumed that the conclusion of the 2002 study was correct, that the two arms were essentially “placebo”, however we treated the arms of the trial separately in case there was a subtle unmeasured difference between them.

## 3. Results

The artificial simulation is shown in Fig. 2. The RR50 in the forward direction for both group A (psychological causes) and B (regression-to-the-mean) was very large, while the reverse RR50 was very small. In group C (natural variability), the RR50 was approximately equal in both directions. The value of τ was −0.50 for A (p > 0.10), −0.92 for B (p > 0.10), and 1.17 for C (p = 0.008). Therefore, groups A and B do not show strong evidence of reversibility (because τ < 0 and p > 0.05), while C does (τ > 0, and p < 0.05).This result is consistent with expectation, given that natural variability should be temporally reversible.
Of the 22,360 patients initially exported for SeizureTracker, 13,488 patients were retained after initial exclusion criteria were applied. Of the 1,210,036 seizures in the database, 1,112,343 seizures were retained after exclusions were applied. For SeizureTracker, 1552 patients were eligible in the forward simulation while 1007 were eligible for the reverse. For NeuroVista, 11 of the original 15 patients were used in the forward and reverse simulation. For the TMS study, all 12 “sham” patients from the TMS study were included in the forward analysis while 10 were included in the reverse analysis.
Fig. 3A summarizes the RR50 responses. τ values were: Neurovista 4.0 (p = 0.002), TMS 2.6 (p = 0.044), SeizureTracker 0.6 (p < 0.001). Because τ > 0 and p < 0.05, all 3 datasets demonstrated statistically significant evidence of reversibility. All three were compatible with natural variability being the largest influence (similar to group C in Fig. 2).
Fig. 3B summarizes the MPC responses. NeuroVista and SeizureTracker both demonstrated large positive MPC values in both forward and reverse computations, while the TMS study showed a positive “forward” MPC and a negative “reverse” MPC.
When using the “clinical” seizures alone from the NeuroVista data, the RR50 values were 14.3% forward, and 33.3% reverse. The MPC values were 24.2% forward, and 38.1% reverse. The forward computation used 7 patients, while the reverse used 9. τ was 3.7 with p = 0.022. As with the complete NeuroVista dataset, τ > 0, confirming statistically significant temporal reversibility.

## 4. Discussion

This study used ‘Big Data’ and formal clinical trial data to present evidence that natural variability may account for the majority of the placebo response in epilepsy trials. Indeed, the RR50 in both forward and reverse (Fig. 3A) were similar to placebo arm values seen in typical clinical trials [
• Theodore W.H.
• Hunter K.
• Chen R.
• et al.
Transcranial magnetic stimulation for the treatment of seizures. A controlled study.
]. This suggests that RR50 may be less optimal as a trial outcome metric, because the natural variability (which is always present) is not adjusted for (Fig. 2). Given that in two of the three cases, positive MPC values were obtained using the “reverse” calculation, it may be worth considering alternatives to MPC as well.
Our analysis highlights the connection between inclusion criteria and placebo response. In the reverse analysis, MPC appears positive and similar to the forward time analysis only if eligibility criteria are re-applied. The decreased number of patients in the reverse direction suggests that some patients had a significant decrease in average seizure counts in the forward direction. The reverse eligibility criteria enforced a form of regression-to-the-mean influence by selecting “sicker” moments in some patients. Selection of less strict eligibility criteria may allow more patients in their “usual state” and thus avoid excessive regression-to-the-mean influence. By design, the regression-to-the-mean effects seen here were caused by natural variability. This is in contradistinction to the effect of acutely ill patients enrolling in trials due to higher than average seizure rates – such a circumstance would not be easily reversible in this analysis.
The NeuroVista dataset was unique because it included information about seizures that were clinically detectable and clinically silent. The latter were not included in patient-reported outcomes (such as SeizureTracker, or typical randomized controlled trials for drugs or devices), so it is unknown what influence including them would have on outcome measures such as RR50 and MPC. When we re-analyzed the data using only the subset of seizures with clinical manifestations that were also reported by the patient, we found that the overall results were unchanged. This shows that despite concerns about under-reporting and over-reporting of seizures in patient reported outcomes [
• a French J.
• Krauss G.L.
• Biton V.
• et al.
Adjunctive perampanel for refractory partial-onset seizures: randomized phase III study 304.
], natural variability in epilepsy influencing placebo responses is likely to have prominent influence on trial results.
A meta-analysis of topiramate, tiagabine and levetiracetam studies supports our results. Between 14.5% and 19% of patients experienced 50% or greater increase in their baseline seizure-frequency [
• Schmidt D.
• Sillanpää M.
Evidence-based review on the natural history of the epilepsies.
]. Those patients would be considered to have achieved a 50%-response if the studies were analyzed in reverse time. Compared with our 50%-responder rates (Fig. 3), the same range of values was found regardless of the direction that time was flowing. The RR50 for epilepsy trials has been 4–40% [
• Theodore W.H.
• Hunter K.
• Chen R.
• et al.
Transcranial magnetic stimulation for the treatment of seizures. A controlled study.
,
• Jehi L.
• Sarkis R.
• Bingaman W.
• et al.
When is a postoperative seizure equivalent to ‘epilepsy recurrence’ after epilepsy surgery.
• Geerts A.
• Arts W.F.
• Stroink H.
• et al.
Course and outcome of childhood epilepsy: a 15-year follow-up of the Dutch Study of Epilepsy in Childhood.
].
Very little is known about the elemental constituents of the placebo response in epilepsy. Although a number of studies suggest possibilities such as regression-to-the-mean [
• Espay A.J.
• Norris M.M.
• Eliassen J.C.
• et al.
Placebo effect of medication cost in Parkinson disease: a randomized double-blind study.
], psychological influences [
• Fisher R.S.
• Blum D.E.
• DiVentura B.
• et al.
Seizure diaries for clinical research and practice: limitations and future prospects.
,
• Cook M.J.
• O’Brien T.J.
• Berkovic S.F.
• et al.
Prediction of seizure likelihood with a long-term, implanted seizure advisory system in patients with drug-resistant epilepsy: a first-in-man study.
,
• Cook M.J.
• Karoly P.J.
• Freestone D.R.
• et al.
Human focal seizures are characterized by populations of fixed duration and interval.
,
• Somerville E.R.
Aggravation of partial seizures by antiepileptic drugs − is there evidence from clinical trials.
,
• Rheims S.
• Cucherat M.
• Arzimanoglou A.
• et al.
Greater response to placebo in children than in adults: a systematic review and meta-analysis in drug-resistant partial epilepsy.
], geographic variability [
• Sillanpää M.
• Schmidt D.
Natural history of treated childhood-onset epilepsy: prospective, long-term population-based study.
] and natural variability [
• Benedetti F.
Placebo effects: from the neurobiological paradigm to translational implications.
,
• Schmidt D.
• Sillanpää M.
Evidence-based review on the natural history of the epilepsies.
,
• Brodie M.J.
• Barry S.J.E.
• Bamagous G a.
• et al.
Patterns of treatment response in newly diagnosed epilepsy.
,
• Hróbjartsson A.
• Gøtzsche P.C.
Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment.
,
• Kaptchuk T.J.
• Friedlander E.
• Kelley J.M.
• Sanchez M.N.
• Kokkotou E.
• Singer J.P.
• et al.
Placebos without deception: a randomized controlled trial in irritable bowel syndrome.
,
• Hróbjartsson A.
• Gøtzsche P.C.
Placebo interventions for all clinical conditions.
,
• Tan S.Y.
• Bruni J.
Cognitive-behavior therapy with adult patients with epilepsy: a controlled outcome study.
,
• Zis P.
• Shafiq F.
• Mitsikostas D.D.
Nocebo effect in refractory partial epilepsy during pre-surgical monitoring: systematic review and meta-analysis of placebo-controlled clinical trials.
], this is the first study to address the question directly using clinical data.
The natural variability in epilepsy has been studied previously, focusing on spontaneous prolonged remissions, their durations and frequencies [
• Brodie M.J.
• Barry S.J.E.
• Bamagous G a.
• et al.
Patterns of treatment response in newly diagnosed epilepsy.
,
• Hróbjartsson A.
• Gøtzsche P.C.
Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment.
,
• Kaptchuk T.J.
• Friedlander E.
• Kelley J.M.
• Sanchez M.N.
• Kokkotou E.
• Singer J.P.
• et al.
Placebos without deception: a randomized controlled trial in irritable bowel syndrome.
,
• Hróbjartsson A.
• Gøtzsche P.C.
Placebo interventions for all clinical conditions.
,
• Tan S.Y.
• Bruni J.
Cognitive-behavior therapy with adult patients with epilepsy: a controlled outcome study.
,
• Zis P.
• Shafiq F.
• Mitsikostas D.D.
Nocebo effect in refractory partial epilepsy during pre-surgical monitoring: systematic review and meta-analysis of placebo-controlled clinical trials.
]. No long-term study has attempted to quantify the expected variability within and across patients.
The techniques described in this manuscript represent a novel method of approaching clinical trial data to derive new insights into disease variability. Indeed, many other diseases lead to episodic symptoms with poorly characterized longitudinal variability. Examples abound in neurology (e.g. headache, TIA, stroke, multiple sclerosis, cataplexy, stiff-person syndrome, autoimmune encephalitis, etc.), psychiatric conditions (e.g. psychotic episodes, depression, manic episodes, PTSD, panic attacks) and general medical conditions (e.g. hypoglycemic episodes in diabetes, flares in asthma, congestive heart failure, inflammatory bowel disease). It remains unknown if the components of the “placebo effect” in epilepsy are representative or unique.
Interestingly, all three datasets were able to produce reasonable results that mirror typical “placebo effects”, though only the TMS study included a form of placebo, and this was not true in the reverse calculation of the TMS study. Thus, it appears from these data that the presence or absence of a true placebo may be irrelevant to the typical placebo response seen in clinical trials.

### 4.1 Limitations

There are several limitations to our study. Since it is retrospective, incomplete control over the data biases and quality was inherent. By evaluating data obtained from very different sources in a common framework as well as applying preprocessing to increase signal-to-noise ratios, we sought to counteract these problems.
Since the multiple modalities for seizure recording had similar general results, the combined data seem to represent consistent representation of natural variability with differing levels of superimposed noise. The data derived from SeizureTracker can be thought of as having high noise levels, due to incomplete patient reporting, as well as reporting biases (under-reporting and over-reporting) [
• a French J.
• Krauss G.L.
• Biton V.
• et al.
Adjunctive perampanel for refractory partial-onset seizures: randomized phase III study 304.
,
• Callaghan B.
• Schlesinger M.
• Rodemer W.
• et al.
Remission and relapse in a drug-resistant epilepsy population followed prospectively.
]. One way to decrease noise is to exclude data that seems to suggest seizure freedom, because this may reflect diary fatigue [
• Benedetti F.
Placebo effects: from the neurobiological paradigm to translational implications.
]. The TMS data also suffers from the biases of patient reported data, though due to close follow-up, the quality of that data is expected to be higher. Unlike the other datasets, the TMS study had all the elements of a typical clinical trial: strict eligibility criteria that could promote regression-to-the-mean, close observation that could result in the Hawthorne effect [
• Cook M.J.
• Karoly P.J.
• Freestone D.R.
• et al.
Human focal seizures are characterized by populations of fixed duration and interval.
], and psychological cues [
• Cook M.J.
• O’Brien T.J.
• Berkovic S.F.
• et al.
Prediction of seizure likelihood with a long-term, implanted seizure advisory system in patients with drug-resistant epilepsy: a first-in-man study.
]. Therefore, the TMS study included an intermediate noise level. The NeuroVista data comprised the lowest level of noise, because regardless of recall, seizures were objectively recorded and analyzed longitudinally with implanted electrodes. Reviewing convergent results from data of various noise levels allows increased confidence in the results.
The re-application of eligibility criteria in both temporal directions was necessary to make a “fair” comparison between the two. Simultaneously, it imposes a form of regression-to-the-mean influence that is driven by natural variability. For instance, suppose a patient typically has 2 seizures per month, but by chance alone happens to have a couple of months with 4 seizures per month during the baseline period. Such a patient would be eligible for the hypothetical study we did here, but would appear to improve despite no intervention after the baseline period. This patient would be counted among the 50%-responders as well. An analogous situation can occur in the reverse temporal analysis. Without imposing eligibility criteria in both directions, the populations comparison would be inappropriate because the regression-to-the-mean influence described above would only impact the forward but not the reverse population.
This study also was limited by the sample size of NeuroVista and TMS. Datasets with larger samples of patients may identify different patterns that those found in the present study. However, the p values suggest that the results obtained were unlikely to be due to chance alone.
Fundamentally, a retrospective study of this kind cannot provide strong evidence that placebo response is unrelated to placebo. At best, it can show suggestive evidence, which this study does. A prospective randomized clinical trial with seizure reduction as an outcome measure, with placebo and no intervention, would offer the best possible evidence, as has been done in several other diseases [
• Hróbjartsson A.
• Gøtzsche P.C.
Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment.
,
• Kaptchuk T.J.
• Friedlander E.
• Kelley J.M.
• Sanchez M.N.
• Kokkotou E.
• Singer J.P.
• et al.
Placebos without deception: a randomized controlled trial in irritable bowel syndrome.
]. According to a 2010 Cochrane meta-analysis [
• Hróbjartsson A.
• Gøtzsche P.C.
Placebo interventions for all clinical conditions.
], only one such study has been published, comparing cognitive behavioral therapy to “placebo” therapy and no intervention in a total of 27 patients [
• Tan S.Y.
• Bruni J.
Cognitive-behavior therapy with adult patients with epilepsy: a controlled outcome study.
]. That study found no difference between any group for seizure reduction.
A final consideration is the potential for placebo to produce harm, i.e. the “nocebo” effect. In a clinical trial, participants provided with placebo may improve or worsen for all the reasons discussed above, but some may experience worsening of seizure frequency due to nocebo influence. Although not yet demonstrated directly in epilepsy, evidence is mounting that nocebo influences may be present in controlled trials of anti-seizure drugs. For example, one meta-analysis found that 3.2% of placebo treated patients withdrew from randomized trials due to adverse events [
• Hróbjartsson A.
• Gøtzsche P.C.
Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment.
]. The quantitative impact of nocebo on clinical trial outcomes remains an open area for future studies.

## 5. Conclusions

The natural variability of epilepsy plays a major role in generating the response seen in placebo arms of clinical trials. This concept has been demonstrated in 3 very different datasets with similar results. The implication is that a similar pattern could be achieved without a placebo at all, suggesting that, at least in epilepsy, the “placebo effect” may not exist. Moreover, use of strict eligibility requirements may inadvertently increase regression-to-the-mean effects. Further confirmatory studies will be needed to test these ideas in larger sets of clinical trial data. A randomized clinical trial with both a placebo arm and a “no intervention” arm may represent the ideal confirmatory trial

## Author contributions

Dr. Goldenholz: study design, acquisition of secondary data sets, analysis, manuscript writing.
Dr. Strashny: development of the p-value calculation; calculation of the p-values for actual and simulated datasets.
Dr. Cook: collection and analysis of primary dataset, editing manuscript.
Mr. Moss: collection of primary dataset, editing manuscript.
Dr. Theodore: collection and analysis of primary dataset, editing manuscript.

## Ethics statement

We confirm that we have read the Journal’s position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.

## Conflicts of interest

None of the authors has any conflict of interest to disclose.

## Acknowledgements:

This study in full was supported by the National Institute of Neurological Disorders and Stroke Division of Intramural Research. This study was approved by the Office of Human Subject Research Protection under protocol #12301. Use of the NeuroVista and SeizureTracker data was facilitated by the International Seizure Diary Consortium (https://sites.google.com/site/isdchome/). The findings and conclusions in this paper are those of the authors and do not necessarily represent the official position of the National Center for Health Statistics or the Centers for Disease Control.

## Appendix

This appendix explains the calculation of p-values for τ. Let nD , D∈{F, R}, be the number of patients in the forward and reverse directions respectively. Let PD be the probability that, moving in a particular direction, a patient achieves the RR50 criterion (a 50% or larger reduction). Then, the number of patients, moving in a particular direction, who achieve the RR50 criterion has the binomial distribution: KD∼ B (nD, PD).
Conditional on kF, the observed number of patients, moving in the forward direction, who achieve the RR50 criterion, the expected value of PF is E [PF]=kFnF.
The null hypothesis is that the expected value of PR is less than or equal to half of the expected value of PF: H0:E[PR] ≤ 0.5E[PF]; H1:E [PR] > 0.5E[PF]. Thus, under the null hypothesis, the expected value of PR is E[PR |H0] = 0.5 kF⁄nF . The distribution of KR under the null hypothesis is KR |H0∼ B(nR, 0.5 kF⁄nF). The p-value is the probability that KR |H0 is greater or equal to the observed number of patients, moving in the reverse direction, who achieve the RR50 criterion: p= Pr (KR |H0 ≥ kR).

## References

• Kwan P.
• Brodie M.J.
Early identification of refractory epilepsy.
N Engl J Med. 2000; 342: 314-319
1. Institute of Medicine (U.S.), Committee on the Public Health Dimensions of the Epilepsies, England MJ. Epilepsy across the spectrum promoting health and understanding. Washington, D.C.: National Academies Press, 2012 http://www.nap.edu/catalog.php?record_id=13379 (Accessed 28 August 2014).

2. PhRMA. Profile Biopharmaceutical Research Industry. 2015 http://www.phrma.org/sites/default/files/pdf/2015_phrma_profile.pdf.

• Rheims S.
• Perucca E.
• Cucherat M.
• et al.
Factors determining response to antiepileptic drugs in randomized controlled trials. A systematic review and meta-analysis: Response to AEDs in Randomized Trials.
Epilepsia. 2011; (no–no)
• Goldenholz D.M.
• Goldenholz S.R.
Response to placebo in clinical epilepsy trials-old ideas and new insights.
Epilepsy Res. 2016; 122: 15-25
• Goldenholz D.M.
• Moss R.
• Scott J.
• et al.
Confusing placebo effect with natural history in epilepsy: a big data approach.
Ann Neurol. 2015; 78: 329-336
• Fisher R.S.
• Blum D.E.
• DiVentura B.
• et al.
Seizure diaries for clinical research and practice: limitations and future prospects.
Epilepsy Behav. 2012; 24: 304-310
• Theodore W.H.
• Hunter K.
• Chen R.
• et al.
Transcranial magnetic stimulation for the treatment of seizures. A controlled study.
2002: 2001-2003
• Cook M.J.
• O’Brien T.J.
• Berkovic S.F.
• et al.
Prediction of seizure likelihood with a long-term, implanted seizure advisory system in patients with drug-resistant epilepsy: a first-in-man study.
Lancet Neurol. 2013; 12: 563-571
• Cook M.J.
• Karoly P.J.
• Freestone D.R.
• et al.
Human focal seizures are characterized by populations of fixed duration and interval.
Epilepsia. 2015; (n/a–n/a)
• Somerville E.R.
Aggravation of partial seizures by antiepileptic drugs − is there evidence from clinical trials.
Neurology. 2002; 59: 79-83
• French J.A.
• Krauss G.L.
• Wechsler R.T.
• et al.
Perampanel for tonic-clonic seizures in idiopathic generalized epilepsy.
2015 (0)
• Rheims S.
• Cucherat M.
• Arzimanoglou A.
• et al.
Greater response to placebo in children than in adults: a systematic review and meta-analysis in drug-resistant partial epilepsy.
PLoS Med. 2008; 5: 1223-1237
• Morton V.
• Torgerson D.J.
Regression to the mean: treatment effect without the intervention.
J Eval Clin Pract. 2005; 11: 59-65
• McCambridge J.
• Witton J.
• Elbourne D.R.
Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects.
J Clin Epidemiol. 2014; 67: 267-277
• Jarius S.
• Wildemann B.
And Pavlov still rings a bell: summarising the evidence for the use of a bell in Pavlov’s iconic experiments on classical conditioning.
J Neurol [Internet]. 2015; 262: 2177-2178
• Espay A.J.
• Norris M.M.
• Eliassen J.C.
• et al.
Placebo effect of medication cost in Parkinson disease: a randomized double-blind study.
Neurology. 2015; 84: 794-802
• Benedetti F.
• Maggi G.
• Lopiano L.
• et al.
Open versus hidden medical treatments: the patient’s knowledge about a therapy affects the therapy outcome.
Prevent Treat. 2003; : 6https://doi.org/10.1037/1522-3736.6.1.61a
• Benedetti F.
Placebo effects: from the neurobiological paradigm to translational implications.
Neuron. 2014; 84: 623-637
• a French J.
• Krauss G.L.
• Biton V.
• et al.
Adjunctive perampanel for refractory partial-onset seizures: randomized phase III study 304.
Neurology. 2012; 79: 589-596
• Callaghan B.C.
• Anand K.
• Hesdorffer D.
• et al.
Likelihood of seizure remission in an adult population with refractory epilepsy.
Ann Neurol. 2007; 62: 382-389
• Callaghan B.
• Schlesinger M.
• Rodemer W.
• et al.
Remission and relapse in a drug-resistant epilepsy population followed prospectively.
Epilepsia. 2011; 52: 619-626
• Choi H.
• a Heiman G.
• Munger Clary H.
• et al.
Seizure remission in adults with long-standing intractable epilepsy: an extended follow-up.
Epilepsy Res. 2011; 93: 115-119
• Luciano A.L.
• Shorvon S.D.
Results of treatment changes in patients with apparently drug-resistant chronic epilepsy.
Ann Neurol. 2007; 62: 375-381
• Schmidt D.
• Sillanpää M.
Evidence-based review on the natural history of the epilepsies.
Curr Opin Neurol. 2012; 25: 159-163
• Jehi L.
• Sarkis R.
• Bingaman W.
• et al.
When is a postoperative seizure equivalent to ‘epilepsy recurrence’ after epilepsy surgery.
Epilepsia. 2010; 51: 994-1003
• Geerts A.
• Arts W.F.
• Stroink H.
• et al.
Course and outcome of childhood epilepsy: a 15-year follow-up of the Dutch Study of Epilepsy in Childhood.
Epilepsia. 2010; 51: 1189-1197
• Sillanpää M.
• Schmidt D.
Natural history of treated childhood-onset epilepsy: prospective, long-term population-based study.
Brain. 2006; 129: 617-624
• Brodie M.J.
• Barry S.J.E.
• Bamagous G a.
• et al.
Patterns of treatment response in newly diagnosed epilepsy.
Neurology. 2012; 78: 1548-1554
• Hróbjartsson A.
• Gøtzsche P.C.
Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment.
N Engl J Med. 2001; 344: 1594-1602
• Kaptchuk T.J.
• Friedlander E.
• Kelley J.M.
• Sanchez M.N.
• Kokkotou E.
• Singer J.P.
• et al.
Placebos without deception: a randomized controlled trial in irritable bowel syndrome.
PLoS One. 2010; 5: e15591
• Hróbjartsson A.
• Gøtzsche P.C.
Placebo interventions for all clinical conditions.
Cochrane database Syst Rev [Internet]. 2010; 20 (CD003974)
• Tan S.Y.
• Bruni J.
Cognitive-behavior therapy with adult patients with epilepsy: a controlled outcome study.
Epilepsia. 1986; 27: 225-233
• Zis P.
• Shafiq F.
• Mitsikostas D.D.
Nocebo effect in refractory partial epilepsy during pre-surgical monitoring: systematic review and meta-analysis of placebo-controlled clinical trials.
Seizure. 2017; 45: 95-99