HOW-TO · DEV

How to configure a CI/CD pipeline in GitHub Actions to run AI-generated tests automatically

advanced25 minBy Eruo Fredoline
Target environment
Ubuntu 24.04 · Ollama 0.4.x
PREREQUISITES

GitHub repository with GitHub Actions enabled, at least one test file in the repository, and an AI assistant accessible via a CLI tool or API call.

What this does

This guide explains how to configure a GitHub Actions workflow that uses an AI assistant to generate or update test cases and then executes those tests as part of the CI pipeline. The workflow runs on every push and pull request, ensuring that AI-generated tests are validated automatically and never drift behind the current codebase.

Steps

  1. Create a .github/workflows/ai-test.yml workflow file in the repository root.
  2. Define trigger conditions: push and pull_request events targeting the main branch.
  3. Add a matrix strategy to run tests across multiple Python versions (e.g., 3.10, 3.12).
  4. Add a job that calls the AI assistant to generate or update test cases for files modified in the current commit diff.
  5. Configure the workflow to save generated test files to a designated directory and commit the output if a pull request token is available.
  6. Add a step to run pytest on the generated test directory after the AI step completes.
  7. Configure required status checks so that a failing AI-generated test suite blocks merge.
  8. Commit and push the workflow file to verify execution.

Verification

gh run list --workflow=ai-test.yml --limit=1

Expected output:

STATUS  NAME              BRANCH   EVENT  ID
✅  ai-test.yml    main     push    S1234567890

View the run details with gh run view <ID> and confirm the "Run AI tests" step completed with exit code 0.

Common failures

  1. Workflow does not trigger — The workflow file is not in .github/workflows/ or contains YAML syntax errors. Validate the YAML with yamllint and verify the file path.
  2. AI test generation step fails with exit code 1 — The AI service endpoint is unreachable or the API key is missing from repository secrets. Add the key via GitHub repository Settings → Secrets and Variables → Actions.
  3. Test jobs skip because no tests were generated — The diff contained no files matching the target file patterns. Adjust the step that determines which files trigger test generation to match a broader set of source files.
  4. Generated test files have import errors in CI — The CI environment does not have all required packages installed. Add a pip install -r requirements-test.txt step before running pytest.

Related guides

  • Integrate AI test generation into Jenkins pipelines (guide 28)
  • Use AI to optimize Docker build times (guide 29)
← All how-to guidesCourses →