Skip to content
PaperFren

Research method

Effect Size

7 studies

Ask PaperFrenWatch this

An effect size is a number that says how large a contrast is, not merely whether p dropped below 0.05 — for example Cohen’s d, partial η², or a near-zero programme-level estimate. Size is still not importance: a large pre/post η² without a control can be practice, a tiny exam d can be the policy-relevant finding, and a 'significant' poorer UDL rating is a perception shift, not a ranked educational value. Effect size also does not settle which engagement dimension moved.

Engagement and learning papers report effect sizes when they want to say how much a format, module or group differed. They answer 'how big was this contrast in these units?' Their main limitation is comparability: Partial η² = 0.35 on essays, a FLEX estimate indistinguishable from zero across 133 courses, and high self-reported simulation engagement are not the same kind of 'size'.

Studies

7

Findings

6

6 supporting · 0 challenging · 0 qualifying citations

Open tensions

2

Currently

What we know

  1. A Swiss FLEX Bachelor’s format cut face-to-face time by about 51% versus a part-time programme. Researchers compared 2,822 FLEX exams with 11,638 part-time exams in 133 courses (2015–2019; FLEX n students = 278, part-time = 1,068). The overall effect size was close to zero and not significantly different from zero; exam differences were significant in only 24 of 133 courses, sometimes favouring FLEX and sometimes part-time. Equivalence on exams is not proof of identical soft-skill or career outcomes.
  2. Wageningen’s online peer-feedback module (330 enrolled, 284 completers across five domain courses) improved argumentative-essay scores from pre-test to post-test with Partial η² = 0.35 (Wilks’ λ = 0.65, F(7, 269) = 20.56). That is a large within-sample change without a no-feedback control, so it is not an isolated treatment effect of peer feedback versus practice.
  3. Among 1,731 of 19,752 invited U.S. research-university students, retrospective UDL items found Engagement and Action & Expression significantly poorer after the spring 2020 remote shift (153 respondents eligible for disability services; Representation improved for some of them). Significance here flags a perception contrast, not a Cohen’s d on learning and not a controlled redesign.
  4. Forty intermediate Iranian EFL learners (20 text-based, 20 multimodal; fourteen sessions) split engagement: text-based stronger on autonomous, behavioural and cognitive dimensions; multimodal on emotional and social; satisfaction similar. A single 'engagement effect size' would have hidden that split. The window is fourteen sessions in one EFL context.
  5. Polish primary students (survey n = 333, ages about 8–14) reported widespread home Internet and more screen time in the first COVID wave. Experts rated a randomly selected 60-student subsample (18%) on five national informatics criteria (instrument α = 0.76). Ratings were generally low; the group with school plus out-of-school distance informatics scored better on several criteria, especially programming/troubleshooting. This is not a randomised pedagogy trial.

Largest unresolved question

Near-zero and large are both in this set because the contrasts differ. FLEX’s programme-level exam effect is ~0 despite 51% less classroom time (only 24/133 courses significant). Peer feedback’s Partial η² = 0.35 is a single-group pre/post on essays. Science-simulation undergraduates (n = 1,034; 536 male, 498 female) reported high engagement and satisfaction with no control-group d at all. You cannot rank those as competing estimates of 'whether blended learning works'.

Procedure

How it works

The main stages of the method. Specific protocols and analysis choices vary between studies.

  1. Choose a contrast

    Two formats, a pre/post window, a programme versus a part-time track, or a perception shift after remote teaching. The contrast decides what 'size' can later mean.

  2. Standardise the difference

    The number is Cohen's d, partial η², or a pooled programme-level estimate, not merely whether p dropped below 0.05. Units still belong to that instrument and that design.

  3. Report how large the contrast was

    Papers want to say how much a module, format, or group differed. A large uncontrolled η², a near-zero exam effect after cutting classroom time, and a high self-reported engagement mean are not the same kind of size.

  4. Read the number inside its design

    A pre/post η² without a control can be practice; a significant poorer UDL rating is a perception shift, not a ranked educational value. Effect size also does not settle which engagement dimension moved.

  5. Do not treat a larger number as more important for policy

    Effect size answers how big this contrast was in these units. A tiny exam d after a large time cut can be the policy-relevant finding; a larger uncontrolled rise is a smaller warrant for redesign.

Common misconceptions

  • A larger effect size means the finding matters more for policy.

    FLEX’s near-zero exam effect after cutting classroom time by half is the policy-relevant magnitude: lots of time moved, little exam shift. A large uncontrolled η² on essays is a smaller warrant for requiring peer-feedback modules everywhere.

  • If p is significant, the effect was large.

    UDL Engagement ratings were significantly poorer after remote shift among 1,731 volunteers; that is a detectable perception change, not a demonstrated large learning loss. FLEX’s few significant course contrasts sit inside an overall null.

  • High mean engagement on a survey is an effect size for simulations versus other teaching.

    The 1,034-student simulation paper is a cross-sectional self-report after one semester in one teacher context. Learning style predicted engagement and satisfaction; there is no control-group contrast to convert into d.

Related