How we change what others think, feel, believe and do
Experimental design principles
> Social Research >
Design > Experinental design principles
In designing a social research experiment there are a number of considerations and principles that are worth taking into account that help to create valid results. The conclusions that you can draw from your analysis will depend strongly on the principles you use.
Researchers can easily introduce bias into results by accidental, subconscious or otherwise reduce the validity of the methods used. Many of the methods used seek to reduce this bias.
The subjects of research may also introduce their own bias, for example through Social Desirability Bias. What is said to them and what they understand may also need careful consideration, although this may raise ethical concerns.
It is seldom possible to research an entire population, so a sample must be selected to represent the population. After the experiment, statistical analysis can be used to enable generalization back to the population.
There are many sampling methods that can be used and the appropriate method needs to be carefully selected.
Ideally, sampling is done randomly. In practice, samples are often selected based more on convenience and what is feasible. This need not invalidate experiments, but the analysis must be done carefully and conclusions drawn with care also.
If a random sample can be selected, then statistics can be used to generalize, drawing conclusions about the entire population. Randomization is thus at the heart of a true experiment. Randomization also helps remove bias, where choices by the researcher, subject or others may lead to invalid results.
Selection and assignment
Randomization starts with selection of target (random selection). It may also apply in deciding what treatments are applied to which subjects, and when (random assignment). Where there are multiple groups, pre-selected subjects may be randomly assigned to groups.
Random assignment is sometimes difficult, for example in a study of two school classes, in which natural groups have to be used.
Randomization creates probabilistic equivalence, where multiple test groups are declared statistically equivalent (two groups of people are never equal in all respects).
A comparison of pre-tests (for example with a t-test) will tell you how different groups in parallel experiments are. Typically, a 95% equivalence (alpha <= 0.05) is sought. This is particularly important when groups are not randomly assigned and probabilistic equivalence cannot be claimed.
In electronics theory, a signal is made up of the original signal plus added unwanted noise, such as hissing on a radio. FM and DAB are methods to remove electronic noise and improve the 'signal to noise ratio'. The greater the signal-to-noise, the greater the effect.
There are two ways of improving the signal-to-noise ratio: increasing the signal or decreasing the noise. In social research, factorial design experiments increase the signal, whilst covariance or blocking designs aim to decrease noise.
Comparison is a common way of eliminating noise. If two experiments are the same in all respects except for changes in target variables, then a comparison (effectively subtracting one from another) allows the 'signal' variables to be highlighted.
Experiments often use multiple groups, with each group forming a mini-experiment on its own, and with the findings from each group then being compared for further comparison, analysis and conclusion.
There are two main ways groups are used:
The same group may also be given the same treatment, but at different times or in different contexts, to see if they have changed or if they are affected by contextual factors (the context thus becomes part of the treatment).
A control group is one which includes the same type of people (preferably randomly assigned) as those in the treatment group.
The control group is a useful way of eliminating extraneous variables and reducing noise.
Experiments that apply some kind of treatment can apply a test or measurement in two places: before the treatment, which is a pre-test and after the treatment, which is a post-test.
This allows for identifying change caused by the treatment by comparing before and after results from a similar test (for example a change in skill, attitude, etc.).
A post-test only may be used sometimes, for example in a 'taste-test', where the subject is asked about their experience in the treatment.
The term 'pre-test' is also used to describe a trial of a test before it is applied 'for real'.
Showing something to be true does not make it always true, even though you may repeat the experiment many times. If, however, you can show that it cannot be false, then you need go no further.
Thus, if you can show that the null hypothesis is false, then you have also proven that the primary hypothesis is true.