HOW-TO · DEV

How to use AI to write regression test cases for a bug you just fixed

intermediate15 minBy Fredoline Eruo
Target environment
Ubuntu 24.04 · Ollama 0.4.x
PREREQUISITES

Bug fix committed to the local branch, AI coding assistant available, test directory present in the project

What this does

A regression test verifies that a specific bug does not recur after a fix is applied. Rather than writing these tests manually, an AI assistant can generate the test code from the bug description, the fix commit diff, and any existing test patterns in the repository. The resulting tests follow the project's conventions and are immediately runnable, providing a reproducible guard against the original bug returning.

Steps

  1. Identify the exact behavior that changed as a result of the fix. Look at the diff of the fix commit to understand what logic was added, removed, or modified.

  2. Note the bug's original manifestation: the error message, input values that triggered the failure, and any edge cases that were affected.

  3. Open the AI assistant and paste the fix diff along with a brief description of the bug, for example: "The function parseConfig was crashing with a TypeError when passed an empty object. Here is the fix. Write a test that covers this case."

  4. Ask the AI to generate a test case that reproduces the original bug behavior. The test should call the fixed function with the inputs that previously caused the failure and assert the expected correct behavior.

  5. Review the generated test and check that it follows the project's test file naming convention and directory structure. Typical patterns include placing tests in __tests__/, test/, or alongside source files with a .test. suffix.

  6. Copy the generated test into the appropriate file, preserving the existing import statements and test structure.

  7. Run the new test in isolation to confirm it passes with the fix applied:

npx jest path/to/new-test.test.js

  1. Temporarily revert the fix commit to confirm the test fails without it. This proves the test genuinely reproduces the original bug.

  2. Restore the fix commit and run the full test suite to confirm the new test does not break any existing tests.

  3. Commit the new test alongside the fix in the same commit or as a separate commit in the same PR.

Verification

Running npx jest with the test file path should produce output showing one passing test with the test name matching the regression scenario. The test suite exit code should be 0.

Common failures

  • AI generates a test for the fix rather than the bug: The test must assert the correct (post-fix) behavior, not the buggy behavior. If the AI writes an assertion that matches the broken state, the test will pass both before and after the fix, providing no regression protection.
  • Test imports not matching the project structure: Generated tests often use generic import paths. Verify that all imports resolve correctly relative to the test file location before running the test suite.
  • Test framework mismatch: The generated test may use syntax or assertions from a different test framework than the project uses. Confirm the framework (Jest vs. mocha vs. pytest) and adjust the test code to match the project's existing test files.
  • Edge cases not covered: The AI-generated test may only cover the exact bug trigger but miss related edge cases. Review the function's parameter space and ask the AI to add boundary value tests (empty input, null, very large values, etc.).
  • Test not committed with the fix: A regression test that exists only locally is not protected against future accidental removal. Ensure the test file is staged and committed in the same PR as the fix.

Related guides

← All how-to guidesCourses →