Diagnosing CI test failures

Use 코파일럿 CLI to pull CI logs, correlate failures to local code, and fix issues without leaving the terminal.

이 기사에서

코파일럿 CLI ships with the GitHub MCP server, which gives it direct access to your GitHub Actions workflow runs, job logs, and check statuses. Combined with access to your local files, it can fetch CI failure details, correlate them to your code, and propose fixes from your terminal.

Example scenario 1: Tests pass locally but fail in CI

You have a test that passes on your local machine but fails in CI. You can ask 코파일럿 CLI to investigate this test failure directly.

In this example, the code being tested defines a simple order service (order.py), and there is a corresponding test that checks if an order was created today (test_order_service.py).

Example prompt

Copilot prompt
My CI is failing on this branch. Can you pull the latest workflow run 
logs, figure out what is failing, and help me fix it? The relevant files 
are @order.py and @test_order_service.py

Example response

참고

다음 응답이 예제입니다. 공동 파일럿 채팅 응답은 비결정적이므로 여기에 표시된 응답과 다른 응답을 얻을 수 있습니다.

코파일럿 CLI uses the GitHub MCP server to fetch your latest workflow runs on the current branch, identifies the failed job, and retrieves its logs. It finds the following failure:

___ TestOrderService.test_order_created_today ___
>       assert order["created_date"] == date.today()
E       AssertionError: assert datetime.date(2024, 1, 15) == datetime.date(2024, 1, 16)

test_order_service.py:45: AssertionError

After reading both local files, 코파일럿 CLI notices that the dates are exactly one day apart and identifies this as a timezone boundary issue. The CI runner uses UTC while your local machine uses a different timezone. When the test runs near midnight, date.today() returns different dates in each environment.

코파일럿 CLI suggests standardizing how today is computed by injecting a clock dependency or pinning the timezone:

from datetime import timezone, datetime

def get_today():
    return datetime.now(timezone.utc).date()

It applies the fix to both order.py and test_order_service.py, updating all references to use the new get_today() helper.

After the fix, you can verify the changes and run the tests locally:

  1. Run /diff to review exactly what 코파일럿 CLI changed.
  2. Run !pytest test_order_service.py to confirm tests pass locally.
  3. Push the fix and let CI re-run.

Example scenario 2: Intermittent test failures with race conditions

A test sometimes passes and sometimes fails on the same machine. You want 코파일럿 CLI to compare logs from passing and failing runs to identify the cause.

In this example, the code under test uses a background job in order_service.py to update an order's status asynchronously, and a test in test_order_service.py asserts that the final status is "processed".

Example prompt for race conditions

Copilot prompt
This test passes sometimes and fails sometimes. Can you pull the logs 
from the last passing and failing CI runs for test_order_processed 
on this branch, compare them, and figure out what is going wrong? 
Relevant files: @order_service.py @test_order_service.py

Example response for race conditions

참고

다음 응답이 예제입니다. 공동 파일럿 채팅 응답은 비결정적이므로 여기에 표시된 응답과 다른 응답을 얻을 수 있습니다.

코파일럿 CLI uses the GitHub MCP server to list recent workflow runs on the current branch, identifies one passing and one failing run, and retrieves the logs for both. It compares them:

Passing run:

[DEBUG] Created order #1234
[DEBUG] Background job started for order #1234
[DEBUG] Background job completed (52ms)
[DEBUG] Checking order status
[DEBUG] Order #1234 status: processed

Failing run:

[DEBUG] Created order #1234
[DEBUG] Background job started for order #1234
[DEBUG] Checking order status
[DEBUG] Order #1234 status: pending

코파일럿 CLI spots that in the passing run, the background job completed before the status check, while in the failing run, the status was checked while the job was still running. It identifies this as a race condition because the test does not wait for the background job to finish.

코파일럿 CLI suggests adding an explicit wait mechanism before asserting and proposes a fix using a polling helper:

import time

def wait_for_status(order_id, expected, timeout=5):
    start = time.time()
    while time.time() - start < timeout:
        order = get_order(order_id)
        if order.status == expected:
            return order
        time.sleep(0.1)
    raise TimeoutError(
        f"Order {order_id} did not reach '{expected}' within {timeout}s"
    )

Further reading