Skip to main content

Statistical Sources: Evaluating Statistics

More on Primary Sources

Evaluating Statistics Checklist

  1. Always go back to the original source of the statistic to check for accuracy.
  2. Find out the definition of what was counted. For example, if the statistic refers to students doing well in school, how is "well" defined?
  3. Check to see if the sample population fairly represents the greater population. 
    • Were volunteers taken or was the sample random?
    • Is there a demographic missing from the sample population?
  4. Check to see if enough people were surveyed compared to the total population. For quick estimates, use this calculator.
  5. Try to find out who funded the study.
  6. If you can, read the actual questions for surveys.
  7. Think critically about the conclusions drawn from the study. Do the conclusions represent a correlation or causation?
    • Correlation: a trend between two potentially independent variables
    • Causation: a variable that directly results from another variable
  8. Beware of visual cheats in charts and graphs as evidenced by "Working remotely increases performance."

Unfair practices in visual information

Visual information can be very convincing. It is important that your visual information is fair. Here are some ways that data visualization may go wrong:
 
Vertical axis starts above zero - This creates a chart that magnifies the visual difference between two numbers, making slight increases or decreases look significant.

Comparison of 2 allergy medicines - Happajoy and Pollaway. 30.6% of people reported Happajoy as effective and 30.4% of people reported Pollaway as effective. The y-axis starts as 30.2 so the difference in effectiveness looks larger than it is.

 

Missing figures - If there is no context for the visual information with numbers, there is no credibility.

Graph with no labels on the axes but has a increasing arrow that says 13%. Title says working remotely increases performance by at least 13%.

 

Height versus width - Beware of a greater change in width versus height that can make the difference seem greater than it really is.

Graph showing cones A at 1.5, B at 3.5, C at 5.5, and D at 7.5. Cone A is wider than the other ones shown.Cones from Microsoft Excel mislead the reader

Changing the unit of comparison - This is comparing apples to oranges instead of apples to apples. Of course, if the chart doesn't label the unit of comparison, you wouldn't know if this is the case, which should be a red flag.

 

Changing the height of vertical values - This is when the height of a unit (like a bar on a chart) doesn't match up with the vertical axis.

 

Axis units are not consistent - Units should remain a consistent size or there is no visual frame of reference for increases or decreases.

 

Changing the conventions - People make assumptions about the information based on the type of chart used. Changing the conventional use can create a situation in which people are easily tricked. 

Y-axis on graph has 0 at the top and 1,000 at the bottom which is flipped from how most graphs appear.