Chapter 2: Generalizations: How Broadly do the Results Apply?

Section 2.3: Errors & Significance

PROJECT: PSYCHIC ABILITY

For this project, you may work alone or in small groups. Each student will submit their own project as a word-processed file. Type your answers into this document and submit your completed project in Canvas on the Assignment page “Project: Psychic Ability.”

Parapsychology Studies

Statistician Jessica Utts has conducted an extensive analysis of Ganzfeld studies that have investigated psychic functioning. Ganzfeld studies involve a “sender” and a “receiver.” Two people are placed in separate, acoustically-shielded rooms. The sender looks at a “target” image on a television screen (which may be a static photograph or a short movie segment playing repeatedly) and attempts to transmit information about the target to the receiver. The receiver is then shown four possible choices of targets, one of which is the correct target and the other three are “decoys.” The receiver must choose the one he or she thinks best matches the description transmitted by the sender. If the correct target is chosen by the receiver, the session is a “hit.” Otherwise, it is a miss. Utts reported that her analysis considered a total of 2,124 sessions and found a total of 709 “hits” (Utts, 2010).

1. If the subjects in these studies have no psychic ability, what would be the long-run probability that the receiver would identify the correct target?

709/2124

2.
State the appropriate null and alternative hypotheses in both words and symbols for testing whether the data provide strong evidence of psychic ability.
[Hint: The wording in #1 should help with this.]

Null: The receiver chooses the correct target, the session is a hit

Alternative: The receiver does not choose the correct target; the session is a miss.

3. Calculate the proportion of “hits” (successful transmissions of the target image) among the 2,124 sessions that the researcher analyzed. Use the appropriate symbol. Is this proportion larger than the null-hypothesized value for the probability of a successful transmission?

709:2124

4. Use a simulation-based method to determine an approximate p-value for testing the hypotheses stated in #2.

approximate p-value = _________________

5. Are the validity conditions for a theory-based method satisfied? Justify your claim.

6. Use the theory-based method to calculate a standardized statistic and p-value for testing the hypotheses stated in #2. [Hint: You can check the “Normal Approximation” box or use the “Theory-Based Inference” applet.]

7. Based on the p-values and theory-based standardized statistic, would you characterize the evidence against the null hypothesis as weak, moderate, strong, or very strong? Explain.

8. What if you had to make a yes-or-no decision about psychic ability based on these data—would you decide to “reject” the null hypothesis of no psychic ability in favor of the alternative hypothesis that the subjects have some psychic ability? Why would you make this decision?

9. If your friend has a
stricter
criterion than you, requiring stronger evidence before deciding to reject the null hypothesis, would your friend use a
smaller
or a
larger
significance level
than you? Explain.

Key Idea:
The significance level controls the probability that you mistakenly reject a null hypothesis that is true. To achieve a stricter standard for rejecting the null hypothesis, use a smaller significance level.

Definition: Type I and Type II Errors

·
Type I error
(false alarm):

o If the null hypothesis is actuallytruebut the data lead a researcher torejectthe null hypothesis, then the researcher has made an error. This error is sometimes called afalse alarm, because the researcher believes that he or she has discovered an effect/difference when there is actually no effect/difference.

·
Type II error

(missed opportunity)

o On the other hand, if the null hypothesis is actuallyfalsebut the data donotlead the researcher to reject the null hypothesis, then the researcher has made a different kind of error. This error is sometimes called amissed opportunity, because the researcher fails to detect an effect or difference that really is present.

10.
Using a 0.05 significance level, which type of error (false alarm or missed opportunity) could Utts and her colleagues conceivably have made in this ESP study? Explain your answer.
[Hint:First ask whether the data led to rejecting the null hypothesis or not. Then ask what kind of error is possible with that decision.]

A Smaller Study

The article by Utts (2010) also refers to 56 individual studies that led to the combined total of 2,124 sessions that you have analyzed here. Most of these studies involved a sample size of about 50 sessions. One particular study involved 50 sessions and resulted in 15 “hits.”

11. Use a simulation- or theory-based analysis to approximate the p-value for a study that produces 15 hits in 50 sessions. Is this p-value small enough to reject the null hypothesis of no psychic ability at the 0.05 significance level? What about the 0.10 level?

12. What type of error could you be making with your decision in #11? Explain what this error means in the context of this study.

13. What if the researchers used a very small significance level? Say, α = 0.0001. How does this decision affect the probability of making a Type I error? Explain this in context.

Key Idea:

One trade-off for selecting a lower significance level (probability of Type I error if null hypothesis is true) is that the probability of a Type II error will increase.