Platform

Buildkite Test Engine

Deliver confidently with fast and reliable tests

Buildkite Test Engine speeds up builds with real-time flaky test management and intelligent test splitting. Drive accountability and get more out of your existing CI compute with performance insights and analytics.

Get started Contact us

Test suite reliability: 89.73%

Disabled

Flaky

Flaky assigned to

User Data team

Wishlist items persist across user sessions and devices

Owners: User Data team

main

./spec/wishlist_items_persist_across_user_sessions_and_devices_spec.rb

github↗

💻

Protagonist

Rails monolith

Runs

1,254/week

📱

Mobile app

Runs

473/week

Flaky tests? React in real time

Isolate low quality tests automatically to keep your business unblocked.

Keep the critical path clear

Eliminate wasteful retries and keep merge queues moving by automatically categorizing and quarantining problematic tests, including flaky test detection.

Assign team ownership

Map tests to teams and automatically assign ownership when troublesome test cases are detected.

“ Being able to deliver software to production quickly and consistently is a critical capability for Persona Identities. Flaky tests were significantly impacting our engineering productivity; Buildkite's Test Engine cut flaky test disruptions by 50% in only four weeks, which translated into substantial productivity gains.”

Ian Chesal

Head of Infrastructure, Persona

Find your slowest tests, fast

See which tests are slowing your builds sooner, so you can solve your biggest problems faster.

Understand test performance

Identify and optimize slow or poor performing tests using timing data analysis from all test runs.

Use timing data to split work

Group tests into parallel jobs based on timing data—using the built-in intelligent sorting or your own rules for distributing tests.

Before

10m

Waiting on slowest node

After

Optimal partitioning reduces wait time

Optimize tests on any platform

Get test insights from any CI/CD tool: Buildkite Pipelines, GitHub Actions, Jenkins, CircleCI, and more.

Real-time insights and analytics

Gain critical test insights with a unified view.

Get a comprehensive view of all tests

Understand what’s running and what needs attention across your test suites, all from one place. Unit tests, integration tests, end-to-end tests—see analysis of your regression testing health in a single view.

💻

Protagonist

Rails monolith

Runs

0/week

📱

Mobile app

Runs

0/week

Marketing site

Runs

0/week

Report on progress

Automate reports on the status of unreliable or troublesome tests across teams on a configurable cadence.

Deep tracing and instrumentation

Automatic tracing helps you see why tests are slow or behaving differently between executions.

it 'allows login via OTP backup code' do
  visit '/login/otp-backup-code'
  expect(page).to have_content('Enter an OTP Code')
 
  # Add annotations to test span timelines
  Buildkite::TestCollector.annotate('Rendered OTP Screen')
 
  # Use up one of the valid backup codes
  fill_in 'code', with: validCode
  click_button 'Confirm'
 
  # If all is well, they should see the dashboard
  expect(page).to have_content('Dashboard')

Slowest SQL queries

Slowest HTTP requests

Span timeline

HTTP

GET http://selenium:4444/wd/hub/session

349ms

SQL

SELECT session_id FROM sessions WHERE

8ms

Rendered OTP Screen

HTTP

GET http://selenium:4444/wd/hub/session

25ms

HTTP

POST http://selenium:4444/wd/hub/session

8ms

SQL

SELECT session_id FROM sessions WHERE

8ms

SQL

INSERT INTO otp_attempts (user_id, created_at

8ms

Frequently asked questions

Got a question that’s not on our list? Want a demo? Just want to chat? Get in touch.

Flaky tests are automated tests that produce inconsistent or unreliable results, despite being run on the same code and environment. They cause frustration, decrease confidence in testing, and waste time while you investigate whether the failure is due to a genuine bug.

By default, Test Engine detects flaky tests by surfacing when the same test is run multiple times on the same commit SHA with different results. The tests might run multiple times within a single build or across different builds. Either way, they are detected as flaky if they report both passed and failed results.

Test Engine also offers several other flaky test detection configurations based on those used by top software teams.

Yes! Use our supported collectors for RSpec, Jest, Cypress, pytest, Swift, and more—or easily implement your own collector. Or, use Test Engine with JUnit XML-formatted test results (the industry standard for test result metadata).

Buildkite Test Engine uniquely identifies each test by its combination of test suite, scope, and name. These “managed tests” are used to power Test Engine's analytics, as well as to provide a stable basis for billing purposes.

We use a 90th percentile (P90) billing method for managed tests. This approach encourages continuous improvement of your testing practices while maintaining predictable costs.

To accomplish this, Buildkite measures usage by calculating the number of managed tests that have executed (run) at least once each day, and at month's end, disregard the top 10%. By disregarding the highest usage days, you can freely add new tests, experiment with different testing strategies, or temporarily increase test coverage without worrying about billing spikes.

Yes! All new organizations get a 30-day free trial of the Pro plan to try the key features. See Pricing for all the details on the plans.

Buildkite provides an SLA of 99.95% uptime and a status page to track any incidents.

Resources

Fixing flaky tests

Read blog →