Any rheumatology research is initiated by a problem in mind: this could be anything that the researcher seeks an answer for, a situation that s/he finds unsettled, with inconclusive evidence, or a condition/therapy/method that needs to be improved. Usually, research question is initially posed as a question, which serves as the focus of further investigation. Success in medical research is an end result of teamwork, with gathering of the research team early in the process. Any research project will significantly benefit from a well-balanced, multidisciplinary, complementary research team and avoid “reinventing the wheel.”

Research question is developed after a comprehensive review of the literature and careful feasibility evaluation. Literature review helps with shaping the study objectives and preventing unnecessary repetitions of work on already conclusive evidence. An effective literature review includes a review of all potential literature sources using targeted search terms, accessing and reviewing the articles and, if needed, accessing and reviewing the primary sources of referenced articles, and noting key points in these articles. Apart from Medline, OVID, and Embase databases, evidence-based research reviews can be identified through the Cochrane Collaboration topic reviews and Evidence-Based Medicine Reviews (EBMR). In reviewing the literature, it is important to consider the potential for publication bias: the published literature is typically limited to significant findings, and nonsignificant studies are not published. For example, Pocock et al. revealed that p values obtained from a systematic review of epidemiologic studies had a peak around p values greater than 0.01 and less than 0.05, rather than a normal distribution pattern, and concluded that statistically significant findings are more publishable [1]. Also, foreign language journals and publications can be difficult to access and review.

Once a study question is deemed worth pursing, then the next question is “feasibility” assessment in terms of study design, time frame, available resources (financial resources and manpower), and ethical implications. The types of study design typically fit under three headings (see Panel 1). Since each of these designs has different feasibility and resource implications, preliminary investigations are important. For example, for a clinical study, the preliminary counts of patients with the disease or surgical procedure of interest and the likelihood of participation (for prospective studies) will define the potential study population. Even for a registry or an existing database study, preliminary counts are informative in terms of the number of years to be included in the study. Since most rheumatologic conditions are rare, the minimum sample size requirements may necessitate multicenter studies. The study questions may also dictate whether population-based samples are needed (e.g., incidence, prevalence studies). Feasibility checks are repeated in other stages of study development.

Detailed definition of the main exposure, outcome variables, potential confounders, and covariates determines what data need to be collected. Most studies attempt to estimate two types of parameters: the frequency of disease occurrence in a particular population and/or the effect size (if any) of a given exposure on disease occurrence. Some research questions do not focus on a relationship at all, and are simply descriptive, such as the prevalence of a particular rheumatologic condition in the population. In all situations, exposure and outcome definitions should be objective, standard, and comprehensive, including exclusion criteria (if any). For example, in a study examining the risk of rheumatoid arthritis associated with obesity, the main exposure variable is obesity and the outcome variable is rheumatoid arthritis. Yet, it is important to explain in detail what is meant by obesity, both in terms of body mass index cutoff values and exposure period in relation to the onset of rheumatoid arthritis (obesity during adolescence, during adulthood, past history of obesity, current obesity, etc.). The definition of rheumatoid arthritis is also specified as new onset, based on self-reported physician diagnosis or based on classification criteria.

In case of a hypothesis testing research design, all exposure and outcome variables are clearly defined, including whether they will be continuous measures or categorical measures and what to do in case of missing data. Some common mistakes are (a) improper grouping of quantitative exposure/outcome variables into several ordered groups, where the number of categories and justification of cut points are often unrealistic, such as, grouping age as below 15, 15–25, 25–35, 35, or above, where categories are overlapped and the group sizes are not similar or using a cut point of 130 mg/dl for identifying individuals with high fasting blood glucose, rather than using the widely accepted cut point of 126 mg/dl; (b) identification of confounders is often perceived as a task in statistical analysis stage and the requirements to be fulfilled for a variable to be a potential confounder is often ignored [2]. However, it is not possible to account for confounders in statistical analyses unless the relevant data are collected.

The selection of potential confounders is based on input from the clinicians and needs clarity, consistency, and explanation. At this stage, the investigator may also consider if there is any possibility for effect modification, i.e., effect of an exposure varies in subgroups of patients, such as men and women or young and old. If effect modification is likely, the subgroups of the effect modifier and the type of interaction (synergistic versus antagonistic, multiplicative versus additive) need to be defined, to the extent possible and power considerations, and sample size should be planned accordingly. As an example, Park et al. [3] found in their recent work that the risk of radiographic progression in rheumatoid arthritis was statistically significantly associated with LDL cholesterol and triglyceride levels (leading up to 5.6-fold risk in the third tertile of both groups). Moreover, LDL cholesterol synergistically increased the adjusted probability of radiographic progression in patients with high serum leptin levels but not in those without. In this situation, leptin is considered to modify the effect of LDL cholesterol on radiographic progression. The assessment of effect modification is important for properly specifying the predictors in statistical models, for making inferences about possible biological (causal) interactions between exposures (e.g., synergy), for generalizing the study findings to other populations, and also affect the minimum sample size requirement [4].

All variables should be measurable, objective, and validated, as appropriate. All indices/questionnaires/inventories, etc., must (known to) be validated in the source population. Diagnostic/therapeutic/preventive thresholds, cutoff points, and “risk zones” should be comparable to those in the literature, unless needed/intended to be used otherwise. Issues related to data collection tools are further detailed below.

It is important to emphasize that the quality of data collection depends upon the measurement tools. Collection of diagnostic-classification criteria can be difficult, in particular, in chart review studies. In prospective studies involving multiple individuals, intra- and/or interobserver reliability can be significantly low, hampering the quality of data collection [5, 6]. Therefore, interobserver reliability studies are common in rheumatology, in particular, in studies involving imaging [7, 8].

Study population is typically a population-based sample (epidemiological studies), convenience sample (analytical studies), or volunteers (interventional studies, studies collecting biospecimens). If the goal is to estimate disease frequency in a particular population, such as the occurrence of new cases and/or deaths (incidence, mortality), or to study the presence of existing cases (prevalence), the base population of the study is the group of all individuals who, if they developed the disease, would become cases. In such situations, it is important that the study is conducted on an “adequate” size population, representatively chosen from the source population. Unfortunately, true population-based studies are feasible in only selected countries where routine healthcare data are available [9].

The investigator should attempt to clarify the rationale for selection of a particular population, the characteristics of the population, with emphasis on representativeness (if any), so that the generalizability (external validity) of the results could be estimated in advance. Representativeness depends upon the source of participants and the proportion participating, i.e., exclusions, refusals to participate, dropouts, or a discontinuity in preplanned follow-ups will hamper the generalizability of the final study findings. Due to some common sources of bias, representativeness is not always guaranteed. Studies in hospital settings may be prone to Berkson bias [10]. This type of selection bias arises in case–control studies in hospital settings. When both the exposure and disease/outcome under study increase the likelihood of admission to the hospital, then the exposure prevalence among hospital cases will be systematically higher than hospital controls and will in turn distort the odds ratio. Similarly, Neyman bias (synonyms: incidence–prevalence bias, selective survival bias) may distort the results of a rheumatologic study when a series of survivors is selected, if the exposure is related to prognostic factors or the exposure itself is a prognostic determinant, the sample of cases offers a distorted frequency of the exposure [11]. This bias can occur in both cross-sectional and case–control studies, if the risk factor influences mortality from the disease being studied. Detection bias may arise in cohort studies when exposed and unexposed individuals have different surveillance intensity to identify outcomes. For example, in a cohort study examining hypertension risk in rheumatic diseases, patients with rheumatic diseases may visit their doctor more frequently and may have a higher likelihood of hypertension being diagnosed than the comparison cohort of subjects without a rheumatic disease. Selection bias may distort findings also if study subjects or participants of a study are different than the pool of all potential diseased individuals. For example, volunteers who agree to participate in research studies are typically different than those who do not. Thus, it is important to carefully consider representativeness and appropriate procedures, where needed.

The study design is chosen based on the research questions. It can also be a combination of several designs:

In case–control studies, the focus is on selection of new onset cases and controls and matching, if needed. In cohort studies, the focus is defining exposure groups as accurately as possible, whereas in randomized interventional studies, the focus is on randomization methods and blinding. Irrespective of study design, selection bias can be a major threat to validity. The various study designs are outlined in other chapters, with emphasis on advantages and limitations of each. A few points are worth mentioning in this section. Randomized controlled trials (RCTs) provide the best scientific evidence but it is important to note that RCTs-based results are hampered in generalizability due to relatively small sample sizes and short follow-up periods of RCTs. Well-conducted observational studies can provide a wealth of epidemiologic information when randomized controlled trials are either not feasible or too expensive. Such observational studies are very useful in studying the real life effectiveness and/or safety of drugs. Post-marketing safety of many of the drugs commonly used in rheumatology is now being conducted using data from large administrative claims databases [12].

Sample size calculation should precede the study start. Sample size and power calculations are also required for grant applications to justify the proposed study size, unless a convenience sample is used, where the power of the study can be estimated backwards.

The minimum sample size will be the least number of individuals to be recruited for the study in order to reach robust estimates (prevalence, incidence, a particular effect size) for the population. Any sample size below the required minimum number will be prone to type II (beta) error, i.e., declaring a difference does not exist between the groups compared when in fact it does. Even in situations where an association is detected in a study with less than required number of study participants, its occurrence due to a type I error often cannot be ruled out. In other words, the authors may conclude by chance that a difference between the groups exist when in fact it does not.

A sample size calculation demands information on study design, the expected probability of the outcome in the source population, the minimum effect size in checking for an association, the desired precision in detecting an estimate, preset confidence interval/alpha error, the requested power (1-beta), ratio of controls to cases, number of hypotheses, and ad hoc analyses to be performed. There might still be other considerations. In calculation of sample sizes for rheumatologic studies, a common difficulty is the lack of knowledge of the standard deviation of the index variables. There are well-recognized tables established for calculating the sample size [13], yet the best approach will be a collaboration of a sampling expert and the clinician with a grasp of the related literature.

Sample size calculation is often conducted for the main hypothesis alone. Any ad hoc estimations (data dredging) or further control in multivariate analyses will lead to larger confidence intervals than initially aimed for, i.e., require adjustments in the minimum sample size. The minimum sample size calculated should also be inflated based on estimated completion rates (due to access problems, refusals), dropouts, etc.

Sample size calculation is a “must” to control for type I and type II errors. However, in rheumatologic studies, researchers should be aware of the clear distinction between the requirements for a “minimal sample size” and “representativeness” for a selected group of individuals (such as, cases/controls, cohort, the general population) for whom the study findings will be interpreted for. Studies conducted on sample sizes equal or above the required numbers do not always guarantee error-free estimations. Large simple trial designs, characterized by large sample sizes, are preferred study designs for drug safety research because it is considered as to control for biases inherent to observational research while still providing results that are generalizable to “real-world use” [14, 15]. Such studies often provide the investigator to control for many potential confounders that may be detected in a “large and heterogeneous” group of participants and maximize the benefit of using statistical models for robust estimations of adjusted risk estimates in studying potential associations. However, even in such situations, evaluations of the similarities between the study participants and all the eligible in the population will be needed to judge the representativeness of the study population. Thus, in gathering the “study population,” the representativeness and sample size issues should be handled separately, but yet hand in hand. The expert support of the biostatistician is critical at this phase of study development.

A study is externally valid, or generalizable, if it can produce unbiased inferences regarding the target population (beyond the study participants). It is noteworthy that activities related to selection of study population (the sample) and completion of data collection from all selected cases are prone to a variety of biases that should be carefully evaluated and be controlled for, as much as possible [16–18]. These potential biases that hinder external validity of the study findings are listed in Table 1. These different forms of bias can create systematic errors that, even though the required sample size is achieved, results do not represent the general population because each eligible individual will not have a well-defined probability of selection chance to participate in a given study.