Overview of the Role of Statistic Analysis in the Design of Spine-related Studies

Statistics is a part of the study design

Perhaps one of the most important concepts is that statistics is not just a painful step toward the end of a study when a data file is provided to someone given the instructions to analyze it. It is very important for someone who is familiar with study design and data analysis to be included as a part of the study design from the inception. This goes well beyond just performing a power analysis for sample size. Ideally, someone familiar with studies in the particular discipline being investigated should be involved. They can provide valuable input into what types of data to collect, which versions of questionnaires to use, data file structure, discourage trying to collect too much data for which there are no plans on how to use, and may be able to provide insight as to problems encountered with previous studies in the same area.

Creating data entry files

There is likely no one better acquainted with the importance of data file structure that those who have to analyze the data. Many small items such as recording height in inches, or having a feet field and an inches field that can be computed into one field later, rather that in mixed units of measure such as 5'10” can be addressed early if someone who is familiar with the format needed for analysis is involved with creating the files from the start. Each such entry has to be re-entered as a single numerical value. It is doubtful that unless someone has been through the exercise of having to re-enter multiple variables for a large number of patients can the frustration of this be appreciated. Sometimes it is helpful to design the database file with drop down menus for entry, particularly for string (text) data. This tends to create consistent entry and avoids time wasted going through the file to correct typographical errors, variation in terminology (such as using herniated disc, HNP, disc herniation all for the same diagnosis but will be recognized by software as three distinct diagnostic groups) and being consistent with capitalization. Many software programs will identify “male” and “Male” as 2 distinct values for a gender variable. For large multicenter trials, especially FDA IDE trials, the file structure is often determined by a database manager. It is important for this person as well as the statistician to have a role in the protocol development and design of case report forms and other data collection instruments, be they on paper or electronic format.

Electronic data capture has many potential advantages over traditional paper collection. First, there are no data entry errors when entering from other sources, patients or clinicians completing the forms cannot select 2 responses for the same item or write in qualifiers to explain their answer, and data is collected and available for review in real-time. However, despite the enthusiasm and benefits of electronic data capture, it should be made clear that it cannot overcome problems with using non-validated questionnaires, poor study design, and in some instances it actually introduces non-validated collection methods.

What test should be used?

Seeing all of the test options available in a statistical software package can be overwhelming. Not only are there many tests, but many have options to modify each by changing the default settings. The type of test(s) used to analyze data for a particular study should be derived based primarily on 2 key factors: 1) the characteristics of the data (continuous, categorical, parametric, etc.) and the question to be addressed (comparing groups, determining change over time or after intervention, looking for association between variables, etc.).

The characteristics of the data are essential in determining which test is most appropriate to use. Several terms are important in this arena. Continuous data are those that are measured on a continuous scale such as age or visual analog scale (VAS) value. Arithmetic properties apply; that is, one can add, subtract, calculate percentage change for variables with continuous values.

One frequently-overlooked item with continuous data is the distribution of the data. Generally, everyone is familiar with normally-distributed data or the “bell curve.” However, not all continuous data fits into this category. The data may be distributed to be skewed to the left or right, meaning that a large number of patients tend to cluster at one end of the scale. The reason why this is important is that many commonly used analyses, such as a t test, are based on the assumption of normally distributed data. If this assumption is violated, the results from a t test may not be valid.

The other commonly encountered data type is categorical, which has several subgroups. Categorical data may be ordinal, which means that the values assigned imply a ranking. For example, a 4-point scale of poor, fair, good, or excellent for assessing outcome implies a ranking that provides information related to which patients did better, or worse, than others after treatment intervention. While numerical values may be assigned to the four categories, unlike continuous data, arithmetic properties are not applicable. One cannot assume that an “excellent” outcome is achieved by doing exactly twice as well as a patient with a “fair” outcome. Non-ordinal categorical data has no ranking and includes variables such as gender, insurance type, and diagnosis.

The other important factor in determining which test is most appropriate to use is what does one want to learn from the data. Such questions include: did patients improve after treatment, is there a difference in outcome between different treatment groups, and are 2 variables related.

Types of test to use for data analysis

One of the most commonly used tests is the very familiar t test. There are 2 basic versions available. The independent sample t test is used when comparing the mean values of a variable with normally distributed data from 2 groups, such as comparing VAS scores from investigational and control groups to determine if the groups are significantly different. If more than 2 groups are to be compared, ANOVA is preferred to multiple t tests between sets of 2 groups. A paired t test is used when comparing the values of the same variable at 2 different times such as VAS scores before and after treatment.

As described earlier, one of the often-overlooked underlying assumptions for applying the t test is that the data is normally distributed. If it is not, then testing involving median values, rather than mean values, is more appropriate. Although variables, such as VAS scores, may be normally distributed prior to an intervention, this does not mean that they will be normally distributed after intervention. This was well described by Geisler when reviewing and re-analyzing data from the CHARITÉ FDA IDE trial.⁵ The postoperative VAS and Oswestry scores were heavily skewed toward the lower end of the scale, indicating that the majority of patients had very low pain and disability scores after surgery. When reanalyzing the data adding the non-randomized training cases and using the non-parametric Wilcoxon rank sum test, which does not require the data to be normally distributed, it was found that the artificial disc group had VAS and Oswestry scores significantly less than those in the fusion group at all of the follow-up periods.

The Wilcoxon rank sum test, also called the Mann–Whitney U, is designed primarily to compare groups based on a variable with ordinal categorical or non-parametric continuous values. It is more appropriate to use than a t test to compare between groups outcome that is classified on an ordinal scale such as the four-point poor to excellent outcome scale described earlier.

One of the other commonly used tests is χ² (the chi-square test). This is most suitably applied to categorical data. It is designed to compare the proportion of subjects associated with a particular response in one group to that in another group. A Fisher's exact test may be used with the same type of data as a χ² but when one or more of the cells in the data table have an expected value of 5 or less.

Frequently-overlooked tests

One test that seems to be underutilized is the McNemar. McNemar may be loosely thought of as a non-parametric categorical version of the paired t test in that it is designed to test for change when subjects respond to the same question on more than one occasion. One application of this test would be in determining if there has been a significant change in the proportions of patients working and not working before and after an intervention.

One very powerful, yet seemingly rarely-used tool, is regression analysis. When working with multiple variables, this test can be used to identify which are significantly related to the variable of interest and which are not. For example, to identify factors related to the postoperative work status for patients enrolled in a particular study, the postoperative work status can be assigned as the dependent variable and all the variables of age, gender, educational level, smoking status, insurance type, length of time off work preoperatively, job demand classification, level(s) operated, preoperative work status, number of levels operated, surgery type, etc. as independent variables. Regression analysis is particularly helpful when some of the independent variables are possibly related such as insurance type, job demands, preoperative work status, and the length of time off work before surgery. The analysis will select the one of these variables that is most significantly related to postoperative work status and, then once the effects of that variable have been accounted for, determine if any of the remaining items are significantly related to the dependent variable of work status. This is preferable to multiple univariate analyses in which each variable is assessed separately to determine its potential relationship to the independent variable. As such several of the variables may be found to be related to the independent variable of interest, such as return to work. However the results may be somewhat misleading and too many dependent variables identified as significantly related to the independent variable if they are interrelated.

Logistic regression analysis (used to determine which variables if any are significantly related to a dichotomous variable – can only take on one of 2 values) was used by Guyer et al. in determining factors differentiating patients with the best and worst outcomes following lumbar total disc replacement.⁶ In that study it was found that the length of time off work prior to surgery was the variable most strongly related to the best versus worst outcome classification. After the effect of that variable was accounted for, none of the other variables investigated had an impact on best versus worst outcome classification.

Lack of consistency in study design

There is little consistency in the design and reporting of results of spine studies. One good example of part of the problem are the recent total disc replacement (TDR) trials.^{7, 8} When reviewed individually, the overall design of each of these studies was based on the traditional format for FDA device trials and a blending of real-life factors of available control groups, costs, patient recruitment, and likely input from key investigators and the clinical research organizations contracted to conduct the studies. However, due to the study designs used in the trials, a very great opportunity was likely missed. The studies used different versions of questionnaires, different methods of assessing satisfaction and work status, and different definitions of success. While there was certainly nothing wrong with the way the data were reported in either paper, it is frustrating for those who later want to do meta-analyses that the data cannot be combined from 2 studies that intuitively should be very similar.

The problem with inconsistency should improve through the use of electronic data capture systems. However based on the author's experience during the past several years, it is not safe to presume that all such systems are using validate instruments in a validated method. One should carefully review the questionnaires and how they are administered before committing to using any data collection system. One of the easiest items to check is which version of the Oswestry is used.⁹ If this is not a current or validated version, use of the program should probably be avoided.

In spine, there seems to be broad acceptance for t tests and χ², with familiarity to a lesser extent for correlation coefficients and ANOVA analyses. Unfortunately, it often seems that anything beyond this rather limited tool set is thought of as statistical voodoo or data manipulation to gets the desired results. While no doubt the “supermarket” approach to data analysis (that is, conducting various tests on the data until the one that produces the highly sought after P < 0.05 is identified) has been used abusively, there is nothing wrong with using techniques beyond most readers’ comfort zones. Readers and reviewers need to become more comfortable with tests beyond those that are currently the most commonly used. There are occasions, although likely to be frowned upon by statistical purist, to present and analyze the data in more than one form. For example, if in the total disc replacement trials, to use the Wilcoxon analysis which is more appropriate than the t test based on the data distribution, but also report the mean values for the VAS scores since these have been used in so many other spine studies and may be very helpful to readers to interpret the data in the context of previous work. However, in such scenarios, the authors must make it very clear that the mean values were not used for analyses and are provided purely as reference data.