Every day, people make claims based on data: a new drink improves focus, a poll predicts an election, athletes who sleep more perform better, students who use flashcards score higher. The big question is not just what the data say, but how the data were collected. Two studies can report similar-looking numbers and still support completely different conclusions. Understanding study design is what keeps statistics from becoming guesswork.
Statistics is about using data to learn about a larger situation. But the method used to collect that data changes what you are allowed to conclude. If researchers ask people questions, that is different from watching what people naturally do. Both are also different from assigning people to different treatments and comparing the results. A good statistical thinker always asks: Was this a survey, an observational study, or an experiment?
These three methods all belong to the study of inference, which means drawing conclusions from data. However, they answer different types of questions. A survey is usually best for learning what people think, believe, own, or experience. An observational study is best for discovering patterns and associations in the real world. An experiment is best for testing whether one factor actually causes a change in another.
A sample survey collects information by asking questions to a sample chosen from a population.
An observational study collects data by observing individuals and measuring variables without assigning treatments.
An experiment imposes a treatment on individuals and compares outcomes across groups.
Although these methods all use data, they do not have equal power. That does not mean one is always better than another. Instead, each has a purpose. The key is matching the method to the question and then respecting the limits of the conclusion.
A sample survey is designed to learn about a larger group by questioning a smaller group. A school might survey 400 students to estimate the percentage who prefer later start times. The goal is not to change anything, only to measure opinions or characteristics.
An observational study looks at what is already happening. A researcher might record how many hours students sleep and how they perform on tests. No one is told to sleep more or less. The researcher simply observes patterns in naturally occurring data.
An experiment goes one step further. In an experiment, researchers impose a treatment. For example, a scientist might randomly assign one group of plants to receive a new fertilizer and another group to receive regular fertilizer, then compare growth. Because the treatment is assigned, experiments are the strongest tool for studying cause and effect.
A survey begins with a population, the entire group the researcher wants to understand, and a sample, a smaller group selected from that population. The sample should represent the population well. If the sample is representative, conclusions from the sample can be used to estimate characteristics of the full population.
Suppose a city wants to estimate the proportion of residents who use public transportation at least three times a week. Asking every resident may be too expensive or too slow, so the city surveys a sample. If the sample is selected well, the city can use the sample proportion, often written as \(\hat{p}\), to estimate the true population proportion, often written as \(p\). In other words, the city can use the percentage found in the sample to estimate the percentage in the full population.
Sampling method matters enormously. A random sample gives every individual in the population a known chance of being selected. Random sampling helps reduce bias because it avoids choosing only the easiest, loudest, or most convenient people to reach. If a poll about school lunch quality only asks students standing near the cafeteria at one moment, the results may not represent the whole school.
Surveys are especially useful for questions about opinions, habits, preferences, and self-reported experiences. They can estimate values such as the percentage of voters supporting a candidate, the average number of hours students spend on homework, or the proportion of customers satisfied with a product.
But surveys also have limits. A badly worded question can influence answers. A voluntary response survey, such as an online poll where people choose whether to participate, often attracts people with strong opinions. This creates bias, a systematic tendency for results to miss the truth in one direction. Even with a large sample, bias can make conclusions unreliable.
Some of the most famous polling errors in history happened not because the calculations were difficult, but because the sample was unrepresentative. A huge sample does not fix poor sampling.
Another important point is that surveys do not establish causation. If a survey finds that students who play an instrument also report higher grades, the survey does not prove that music lessons cause better grades. It only describes a relationship or a difference in reported outcomes.
In an observational study, researchers measure variables without telling participants what to do. A scientist may record exercise habits and blood pressure, or a school counselor may compare attendance and course performance. The researcher is not assigning treatments; instead, the researcher observes the world as it already exists.
Observational studies are powerful because many important questions cannot be studied experimentally. It would be unethical to assign people to smoke cigarettes for years just to test long-term health effects. Instead, researchers observe people who already smoke and compare them with people who do not. Observational studies are common in medicine, economics, education, environmental science, and sociology.
These studies often look for an association, meaning that two variables tend to change together. For example, students who sleep more may tend to earn higher test scores. That pattern can be informative. It may suggest a real relationship worth studying more carefully.
However, an observational study cannot usually prove cause and effect because of confounding. A third factor may influence both variables. Students who sleep more may also have steadier schedules, lower stress, or stronger study habits. These hidden influences are often called lurking variables. They make it difficult to know whether one variable truly causes the other.
This is why statisticians say, association is not the same as causation. If an observational study finds that teens who spend more time on screens also sleep less, it does not prove screen time causes less sleep. Screen use might matter, but so might stress, activities, homework load, or other factors.
Still, observational studies are extremely valuable. They reveal patterns, generate hypotheses, and provide evidence when experiments are impractical or unethical. In fact, much of modern public health depends on carefully designed observational research. The key is stating conclusions honestly: observational studies support claims about association, not strong claims about direct causation.
An experiment is different because the researcher actively applies a treatment and compares outcomes. If a new study app is being tested, some students might use the app and others might not. By controlling who receives the treatment, the researcher can make fairer comparisons.
Experiments are built around variables. The explanatory variable is the factor the researcher changes, such as using a new medicine or a new teaching strategy. The response variable is the measured outcome, such as recovery time or test score. The goal is to see whether changes in the explanatory variable produce changes in the response variable.
A strong experiment uses random assignment. This means subjects are assigned to groups by chance. If 100 patients are in a medical trial, random assignment might place about half in a treatment group and half in a control group. Random assignment helps make the groups similar at the start, so differences at the end are more likely to be due to the treatment rather than preexisting differences.
The control group is the group that does not receive the treatment or receives a standard treatment. It provides a baseline for comparison. In medicine, the control group may receive a placebo, which looks like the real treatment but has no active ingredient. This helps researchers separate the effect of the treatment from the effect of expectations.
Good experiments also try to reduce bias through blinding. In a single-blind study, subjects do not know which group they are in. In a double-blind study, neither the subjects nor the researchers interacting with them know. Blinding helps prevent expectations from affecting the outcomes or the measurements.
When an experiment is well designed, it gives the strongest evidence for causation. If the treatment group and control group differ in a meaningful way after random assignment and careful control, the treatment is a plausible cause of the difference. That is something a survey or observational study usually cannot establish.
Why experiments are strongest for cause and effect
Random assignment spreads many hidden differences across groups by chance. Instead of one group having all the highly motivated students or healthiest patients, those traits are more likely to be mixed between groups. This does not guarantee perfection, but it greatly improves the fairness of the comparison.
Even experiments have limitations. If the sample itself is not representative, the results may show a causal effect for the participants in the study, but not necessarily for the whole population. This leads directly to one of the most important distinctions in statistics: random sampling and random assignment are not the same thing.
The word randomization appears in several statistical settings, but its meaning depends on the design. In surveys, randomness usually refers to how the sample is chosen from the population. In experiments, randomness usually refers to how subjects are assigned to treatments. In observational studies, researchers usually do not assign treatments at all.
Random sampling helps researchers generalize from a sample to a population. If the sample is randomly selected, it is more likely to represent the population fairly. This strengthens external validity, meaning the results may be more safely extended to the larger group.
Random assignment helps researchers compare groups fairly within an experiment. It balances lurking variables as much as possible between treatment groups. This strengthens internal validity, meaning differences in outcomes are more likely to be caused by the treatment.
An observational study may still use random sampling to select participants from a population, but it does not use random assignment to impose treatments. A survey may also use random sampling, but since there is no treatment, random assignment is not part of the design. An experiment can use random assignment without random sampling, random sampling without random assignment, both, or neither. The two ideas solve different problems.
Here is the key distinction: random sampling supports generalization, while random assignment supports causal conclusions. If you remember only one sentence from this topic, that is the one to keep.
The three designs can be compared by purpose, how data are collected, and what kind of conclusion is justified.
| Method | Main purpose | Researcher imposes treatment? | Typical use of randomness | Strongest conclusion |
|---|---|---|---|---|
| Sample survey | Estimate opinions or characteristics of a population | No | Random sampling | Description of population values |
| Observational study | Detect associations in naturally occurring data | No | Sometimes random sampling | Association, not strong causation |
| Experiment | Test effects of treatments | Yes | Random assignment, sometimes random sampling | Cause-and-effect evidence |
Table 1. Comparison of sample surveys, observational studies, and experiments by purpose, treatment, randomization, and conclusions.
The same topic can often be studied in more than one way. For example, to study student stress, a survey could ask students how stressed they feel, an observational study could compare stress levels with sleep or workload, and an experiment could test whether a specific stress-reduction program changes outcomes. The question determines the best method.
Classifying studies is easier when you focus on two questions: Are people being asked questions? and Is a treatment being imposed? If neither is true, the study is likely observational.
Worked example 1: Classifying a school poll
A principal selects 250 students at random from the school roster and asks whether they support a later lunch period.
Step 1: Identify what the researchers do.
The principal asks selected students a question. No treatment is imposed.
Step 2: Classify the design.
This is a sample survey because information is collected by asking questions to a sample.
Step 3: Decide what conclusion is justified.
If the sample was chosen randomly and responses are honest, the results can be used to estimate the proportion of all students who support the change.
The correct classification is sample survey.
Notice that this design is useful for estimating opinion, but it cannot test whether a later lunch period would improve student focus. That would require a different design.
Worked example 2: Sleep and grades
A researcher records the nightly sleep and final exam scores of 180 students, then checks whether students who sleep longer tend to score higher.
Step 1: Identify whether a treatment is imposed.
No one is assigned a sleep schedule. The researcher only measures existing behavior.
Step 2: Classify the design.
This is an observational study.
Step 3: Decide what conclusion is justified.
The researcher may conclude there is an association between sleep and exam scores if the data support it. The researcher may not conclude that extra sleep directly causes higher scores based on this study alone.
The correct classification is observational study.
This example connects back to the earlier illustration of an observational study, where the researcher records naturally occurring values without telling participants what to do. That is the signature of an observational study.
Worked example 3: Testing an energy drink
A company recruits 80 volunteers. The volunteers are randomly assigned so that 40 receive a new energy drink and 40 receive a similar-looking drink without the active ingredient. After 30 minutes, all volunteers complete the same reaction-time test.
Step 1: Identify the treatment.
The treatment is receiving the new energy drink.
Step 2: Look for random assignment.
The volunteers are assigned to groups by chance, which is random assignment.
Step 3: Classify the design.
This is an experiment because a treatment is imposed and outcomes are compared.
Step 4: Decide what conclusion is justified.
If the treatment group performs better, the study provides evidence that the drink caused improved reaction time for the volunteers in the study.
The correct classification is experiment.
The group flow in the earlier experiment diagram matches this structure: subjects enter the study, are randomly assigned, and then their outcomes are compared.
Worked example 4: Random sample or random assignment?
A health researcher randomly selects 500 adults from a city list and then asks each person about exercise habits and cholesterol level. No program is assigned.
Step 1: Identify the kind of randomness.
The adults were randomly selected from the city list, so the study uses random sampling.
Step 2: Check for imposed treatment.
No treatment or program is assigned.
Step 3: Classify the design and its conclusions.
This is an observational study with random sampling. It may support generalization to the city's adults, but it does not support a strong causal claim.
This study uses random sampling, not random assignment.
Students often confuse those two ideas, which is why the side-by-side comparison is so useful. One kind of randomness selects people; the other kind assigns treatments.
Election polling is a classic use of sample surveys. Pollsters want to estimate support in a population, so they focus on sampling methods, response rates, and question wording. If the sample is not representative, the poll may miss the true opinion of voters.
Public health often relies on observational studies. Researchers may track diet, exercise, pollution exposure, or stress and compare these factors with health outcomes over time. Such studies have uncovered major health risks, but careful researchers avoid claiming direct causation unless stronger evidence exists.
Clinical trials for medicines and vaccines are experiments. Researchers randomly assign participants to treatment and control groups, often using placebos and blinding. These designs help determine whether a treatment truly works. Modern medicine depends heavily on this experimental logic.
Education researchers also use all three methods. A survey might measure student attitudes toward homework. An observational study might examine the relationship between attendance and performance. An experiment might test whether a new tutoring program improves algebra scores compared with standard instruction.
When judging any claim, first identify the variables, then ask whether the researcher merely observed them, asked about them, or actively changed one of them.
Sports analytics gives another good example. Teams may survey fans about ticket prices, conduct observational studies on sleep and recovery patterns in athletes, and run experiments on training plans or nutrition strategies. The same field uses all three designs for different purposes.
One common mistake is thinking that any study with the word random automatically proves causation. That is false. A random sample does not create causation by itself. Only random assignment in a well-designed experiment supports strong causal conclusions.
Another mistake is believing that a large sample guarantees truth. Large samples reduce random variation, but they do not remove bias. A large voluntary response poll can still be badly misleading.
A third mistake is using causal language for observational results. If a headline says, "Students who join clubs earn better grades," a careful reader should ask whether joining clubs causes better grades or whether students who are already organized and engaged are simply more likely to join clubs. Confounding may be at work.
The best habit is to ask three questions whenever you see a statistical claim: Who was studied? How were they selected? Was any treatment imposed? Those questions quickly reveal whether the study is a survey, an observational study, or an experiment, and what kind of conclusion is justified.
