Buildkite Test Engine
Deliver confidently with fast and reliable tests
Buildkite Test Engine speeds up builds with real-time flaky test management and intelligent test splitting. Drive accountability and get more out of your existing CI compute with performance insights and analytics.
- Test suite reliability
- 89.73%
Flaky tests? React in real time
Isolate low quality tests automatically to keep your business unblocked.
Keep the critical path clear
Eliminate wasteful retries and keep merge queues moving by automatically categorizing and quarantining problematic tests, including flaky test detection.
Assign team ownership
Map tests to teams and automatically assign ownership when troublesome test cases are detected.
“ Being able to deliver software to production quickly and consistently is a critical capability for Persona Identities. Flaky tests were significantly impacting our engineering productivity; Buildkite's Test Engine cut flaky test disruptions by 50% in only four weeks, which translated into substantial productivity gains.”
- Ian Chesal
- Head of Infrastructure, Persona
Find your slowest tests, fast
See which tests are slowing your builds sooner, so you can solve your biggest problems faster.
Understand test performance
Identify and optimize slow or poor performing tests using timing data analysis from all test runs.
Use timing data to split work
Group tests into parallel jobs based on timing data—using the built-in intelligent sorting or your own rules for distributing tests.
Optimize tests on any platform
Get test insights from any CI/CD tool: Buildkite Pipelines, GitHub Actions, Jenkins, CircleCI, and more.
Real-time insights and analytics
Gain critical test insights with a unified view.
Get a comprehensive view of all tests
Understand what’s running and what needs attention across your test suites, all from one place. Unit tests, integration tests, end-to-end tests—see analysis of your regression testing health in a single view.
Report on progress
Automate reports on the status of unreliable or troublesome tests across teams on a configurable cadence.
Deep tracing and instrumentation
Automatic tracing helps you see why tests are slow or behaving differently between executions.
Frequently asked questions
Got a question that’s not on our list? Want a demo? Just want to chat? Get in touch.
Flaky tests are automated tests that produce inconsistent or unreliable results, despite being run on the same code and environment. They cause frustration, decrease confidence in testing, and waste time while you investigate whether the failure is due to a genuine bug.
By default, Test Engine detects flaky tests by surfacing when the same test is run multiple times on the same commit SHA with different results. The tests might run multiple times within a single build or across different builds. Either way, they are detected as flaky if they report both passed and failed results.
Test Engine also offers several other flaky test detection configurations based on those used by top software teams.
Yes! Use our supported collectors for RSpec, Jest, Cypress, pytest, Swift, and more—or easily implement your own collector. Or, use Test Engine with JUnit XML-formatted test results (the industry standard for test result metadata).
Buildkite Test Engine uniquely identifies each test by its combination of test suite, scope, and name. These “managed tests” are used to power Test Engine's analytics, as well as to provide a stable basis for billing purposes.
We use a 90th percentile (P90) billing method for managed tests. This approach encourages continuous improvement of your testing practices while maintaining predictable costs.
To accomplish this, Buildkite measures usage by calculating the number of managed tests that have executed (run) at least once each day, and at month's end, disregard the top 10%. By disregarding the highest usage days, you can freely add new tests, experiment with different testing strategies, or temporarily increase test coverage without worrying about billing spikes.
Yes! All new organizations get a 30-day free trial of the Pro plan to try the key features. See Pricing for all the details on the plans.
Buildkite provides an SLA of 99.95% uptime and a status page to track any incidents.
Resources
Get started with real time test management
Create an account to get started with a 30-day free trial. No credit card required.