Skip to content

bug: --shuffle flag silently ignored after #5438 (sort() in run() overwrites shuffle from loadTests()) #5605

Description

@kapil971390

Bug Report

Introduced in: commit 4f1eb3f (PR #5438) — fix: run-workers --by suite parallelization broken by test file sorting


What was fixed in #5438

testFiles.sort() was moved from loadTests() to run() to fix worker suite distribution. Good fix — but it introduced a regression for --shuffle users.

The regression

In lib/codecept.js, the execution order is now:

// loadTests() — line ~217
if (this.opts.shuffle) {
  this.testFiles = shuffle(this.testFiles)  // ✅ shuffled here
}

// run() — line 293 (added by #5438)
this.testFiles.sort()  // ❌ always sorts — overwrites the shuffle!

// then:
mocha.files = this.testFiles  // receives sorted, not shuffled, files

sort() in run() is unconditional — it runs regardless of whether --shuffle was specified. This means --shuffle is silently ignored on every run after this commit.

Steps to reproduce

# Run twice with --shuffle — expect different order each time
npx codeceptjs run --shuffle
npx codeceptjs run --shuffle

# Both runs will produce identical alphabetical order — shuffle has no effect

Expected behavior

--shuffle randomizes test execution order. Each run should produce a different sequence.

Actual behavior

Both runs produce identical alphabetical order. The shuffle applied in loadTests() is overwritten by sort() in run().

Suggested fix

Guard the sort with a shuffle check:

// run() — only sort if shuffle is not active
if (!this.opts.shuffle) {
  this.testFiles.sort()
}

This preserves alphabetical order for normal runs while respecting shuffle when requested.


Happy to submit a PR if the fix looks right to maintainers.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions