We're thrilled to introduce Cluster Insights - your new command center for CI performance monitoring. Available now for Enterprise customers.

Cluster Insights delivers real time visibility into your CI infrastructure with powerful metrics that matter. At a glance, see if your teams are facing bottlenecks, track system performance, and make data-driven decisions about your CI resources.

For platform engineering teams balancing cost optimization with developer productivity, Cluster Insights provides the metrics you need. Monitor key performance indicators, identify trends, and ensure your CI infrastructure scales with your team's needs - all from a single, intuitive dashboard.

Key metrics include:

• Queue Wait Time: Track job start delays in real time
• Queued Jobs Waiting: Monitor queue depth and volume trends
• Agent Utilization: Optimize your CI resource allocation
• Job Pass Rate: Identify stability issues before they impact teams

Drill down from organization-wide views to individual queue performance, with detailed p50, p75, p90, and p95 wait time breakdowns to help you spot and resolve bottlenecks. Read our guide to Cluster Insights in the docs for more guidance on how to use these insights to improve developer productivity.

We're excited for the future and anticipate adding expanded insights across both build performance and agent-level metrics, creating an even more comprehensive CI observability solution.

Note: Metrics are only available for agents in clusters. Running unclustered agents? Contact us at support@buildkite.com to discuss cluster migration strategies.

We'd love to hear your feedback, so please let us know what you think at support@buildkite.com.

The build page has been completely reimagined to support modern software delivery at any scale. From light weight test suites to complex infrastructure deployments, this new experience adapts to your workflow while maintaining Buildkite's signature flexibility. The redesigned interface brings powerful navigation through a new sidebar and powerful table view, making it easier than ever to understand and troubleshoot your builds.

Opt in now on the build page! The new build page will become the standard experience on April 24, 2025.

TLDR:

• The new build page is here - opt in and check it out!
• Find everything faster with the new sidebar and a Table view that shows all your jobs
• A new job details drawer available on every page. You can affix to the bottom, right or docked in the center.
• Your annotations now live in the Summary tab, just one click away in the sidebar
• The existing views still work just like before

Enhanced Navigation and Visibility

The new build sidebar transforms how you navigate complex pipelines, providing a state-based view of your build steps that makes it easy to understand progress at a glance. Quick filtering and grouping by state helps you focus on what matters, while intelligent nesting of groups, parallel, and matrix steps keeps everything organized.

The new sidebar listing all steps, grouped by state with additional filters.

We've also introduced an adaptive layout that puts you in control, showing the highest priority items at the top. Immediately see blocked or failed steps at the top of your sidebar. Position the step log drawer on the side, bottom or center of your screen and resize it to your needs. We’ll remember your settings for convenience.

A new table view is perfect for power users looking for all queue and agent information across all jobs in one place. Clicking on column headings sorts by that column for easy browsing.

The new table view showing all jobs in a build.

Easier Build Management

Understanding triggered pipelines is now seamless with trigger steps endowed with their own step drawer details. Monitor downstream builds directly from the parent pipeline without leaving the page - perfect for microservices architectures and fan-out CI/CD workflows where one pipeline kicks off many others.

See the status of a triggered build in the trigger step drawer.

Retries are now easier to track and manage with enhanced visibility throughout the interface. Retried steps are clearly indicated in the sidebar, with hover tooltips for quick navigation between attempts. The step drawer provides a consolidated view of all retries, letting you jump between specific executions effortlessly.

Retried steps are more easily identified in the sidebar, with all jobs available in the logs drawer.

Our customers are often pushing the boundaries and scale of software delivery, so we’ve put in deliberate effort to improve the build pages’ ability to handle large scale builds. This is an ongoing journey, but we’re happy to report faster page loads for large scale builds of 500 to 10,000 steps.

Annotations and more on the Summary tab

Annotations have been moved to the Summary tab, with links to them from the sidebar. This supports annotations in the style of a 'banner' with context of the overall build.

Annotations linked in the sidebar and available on the summary page.

A crowd favorite, we know many customers also use annotations to give context on individual steps, such as neater build outputs or Terraform plans. Which is why we intend to provide additional annotation types in the near future, such as a step annotation with the annotation displayed alongside step logs.

You're now also able to view all the step definitions of your build in YAML. Both initial steps and uploaded steps are available on the Summary tab, under 'Step uploads'.

Getting Started

1. Look for the "Try New Build Experience" button on any build page
2. Toggle between new and classic views anytime during the opt-in period
3. Share your feedback to help us refine the experience at support@buildkite.com
4. Read our docs page on the new build page to read about key features

Note: The classic build UI will be deprecated in two months on April 24, 2025, at which point all users will be automatically upgraded to the new experience.

Built for the Future

This redesign lays the foundation for exciting capabilities coming to the build page in the future. Here are some improvements we're planning:

• Enhanced personalization options to remember your preferred default view settings
• Improved table view with better customization
• Collapsible sidebar for maximizing screen space
• Step-level annotations for more granular build insights
• Multi-log viewing to compare outputs side by side
• Collapse groups of steps on the canvas to optimize rendering
• Unblock builds with one click
• Critical path analysis in the canvas

This new build experience is just the beginning. We're committed to making Buildkite the most powerful and flexible platform for software delivery, and this update sets the stage for even more improvements to come.

Please reach out to support to let us know your feedback at support@buildkite.com once you've tried it. We're keen to ensure this new update suits the wide range of customers and use cases for which Buildkite is known.

We have rolled out a new fix for pipelines that have hanging jobs.

If you are using test splitting queues such as Shopify's open source tool ci-queue (https://github.com/Shopify/ci-queue) at scale, you may have noticed that some jobs are hanging around and causing your builds to take longer than expected.

There is now a new agent cli command to cancel these pesky jobs to improve your build times.

To force cancel all jobs in a step you can run the following from any job within the target build:

1
buildkite-agent step cancel --step <step_key> --force

In this kind of situation you may have a pipeline that looks like this:

1
2
3
4
5
6
7
steps:
  - command: test_reporter.sh
    key: reporter
  - command: test_worker.sh
    key: workers
    parallelism: 500
    soft_fail: true

In this pipeline the workers are responsible for running the tests but the outcome of the tests is reported by the reporter step. The outcome of the workers in Buildkite is not relevant to the build outcome. When a step is marked as soft fail Buildkite still waits for all jobs to finish before marking the build as passed. This can cause the build to take longer than necessary. To fix this you can add a step to cancel the workers after the reporter step has finished.

1
2
3
4
5
6
7
8
9
steps:
  - command:
      - test_reporter.sh
      - buildkite-agent step cancel --step workers --force
    key: reporter
  - command: test_worker.sh
    key: workers
    parallelism: 500
    soft_fail: true

If you find this cancels too quickly, leaving agents unable to upload logs and artefacts, you can set a custom a grace period with the --force-grace-period-seconds flag. This will allow the agents to finish their work before being cancelled.

1
2
3
4
5
6
7
8
9
steps:
  - command:
      - test_reporter.sh
      - buildkite-agent step cancel --step workers --force --force-grace-period-seconds 10
    key: reporter
  - command: test_worker.sh
    key: workers
    parallelism: 500
    soft_fail: true

We've created a new Slack notification service for easier management of your Slack notifications. This new notification service requires only a single authorization for the entire workspace and replaces the need for multiple Slack notification services. See the documentation for enabling a Slack Workspace for your organization. Going forward, we recommend using Slack Workspace. Connecting this new notification service will not impact current implementations.

With this change comes:

• Build notifications now include mentions when triggering builds using an account with an associated email in slack workspace.
• Build notifications can be filtered to only trigger on the first failure with started_failing, rather than notifying on recurring failures. Using YAML, this is possible in the existing slack integration and the new workspace.
• Slack channels are now configured in pipeline YAML when using slack workspace.

We're excited to announce Playwright support for our Test Engine Client, to complement RSpec and Jest as supported test frameworks for test splitting. The client uses timing data obtained from your test suite to intelligently partition your tests across parallel agents, reducing the time taken to run them.

To begin using the client with Playwright, you can follow the instructions in the Playwright documentation in the Test Engine Client repository.

We've introduced team management to the suite settings page.

Users with permission to edit the suite can:

• Add & Remove Teams: Easily control which teams have access to the test suite.
• Edit Permissions: Adjust team permissions directly from the Teams page.
• Team Insights: View the total number of users in each team and how many test suites each team can access.
• Quick Navigation: Click on a team to navigate directly to its details page for managing team members.

This feature streamlines team access management, allowing users to configure permissions and team memberships efficiently within their test suite environment.

We've added a date filter to the flaky tests page, allowing users to refine the list of flaky tests based on set date ranges. Users can now filter flaky test results from the last hour up to the last 28 days.

This enhancement provides greater insight into flaky tests, enabling users to identify recent test instability or track longer-term trends in test flakiness. Enjoy!

We’re excited to release dark mode for all customers!

In your user menu you can set Buildkite to display in light or dark mode, or adapt to your system theme:

Thanks to all the customers whose input shaped this feature. Note that dark mode is experimental and you may encounter issues. Please reach out with any feedback or issues you notice, thanks!

We will be enabling IPv6 support for the agent.buildkite.com domain used by the Buildkite Agent on 2024-11-20. In dual-stack¹ environments, this may mean that agents start connecting from IPv6 addresses where they previously used only IPv4.

If you:

• Have agents running on a dual-stack network, AND
• Use Cluster tokens that restrict agent connection by IP address,

you should ensure that your tokens' IP allowlists include your agents' IPv6 address range(s).

¹ An environment where both IPv4 and IPv6 may be used on the public Internet.

You can now see your Test Analytics failures directly within the build job.

The new Tests tab lists all test executions that have failed in the current job, giving you a quick overview of failures and their summaries.

Navigating to an execution now opens a drawer within the Test's page, enabling you quick access to execution details without leaving the context of the test. This improves workflow by making it easier to explore test details and debug faster.

Pipelines with at least one Test Suite associated have a new Test Digest tab on build pages. This tab displays detailed information about tests that failed during the build runtime.

Legitimate failures are prioritized and listed at the top of the digest, highlighting tests likely affected by the recent code changes on the branch being built.

Tests that flaked during the build are clearly labeled for easy identification.

Organization admins can now disable public pipeline creation for all organization members.

Public pipeline creation is enabled by default.

Disable public pipeline creation

Navigate to your organization settings by selecting 'Settings' in the top nav. In the side nav, under 'Pipelines', select 'Settings'. In the 'Public Pipelines' section, select 'Disable Public Pipeline Creation', and confirm in the dialog box.

Once public pipeline creation is disabled, options to make pipelines public are hidden from the Buildkite UI. If a user attempts to create a public pipeline via our API, an error will be returned.

Disabling public pipeline creation will not affect existing public pipelines - these will remain public, unless you explicitly change their visibility to private. Once this is done, their visibility cannot be made public again unless public pipeline creation is re-enabled.

We've improved filtering on the builds list to help you find builds quicker and easier.

We've made filters more discoverable by showing them in place at the top of the builds list, rather than hiding them away under a menu.

Filter by branch

The improved branch filter allows you to explore builds on any of your projects branches, and even search across multiple branches.

Start typing the name of your branch narrow down the list, and select the branch you want to see builds for.

Use a wildcard, like feature-*, to find builds across all matching branches.

The current branch is displayed in the breadcrumb navigation, so you can quickly jump back to all builds, or the pipeline overview.

Search by commit SHA or message

Using the new search filter, you can paste in a commit SHA to find associated builds. Short SHAs work too!

If you don’t have the SHA handy, just type part of the commit message, and we’ll show you matching builds.

View your builds

Easily switch between seeing all builds and just your own using the My Builds toggle.

Filter by state

Narrow down your search to include only builds that are running, passed, failed, or in any other state.

We will be enabling IPv6 support for Buildkite's APIs on 2024-09-18. This includes api.buildkite.com, graphql.buildkite.com, and analytics-api.buildkite.com. If you have any clients consuming the Buildkite API while operating on a dual-stack network, it is important to check that any API Token IP allowlists, and any Organization API IP allowlists permit your IPv6 network ranges in preparation for the change.

On the build page, jobs waiting on a concurrency group now show a list of the jobs ahead of them in the group.

This list gives you visibility into how soon the current job might run and which jobs are backing up a concurrency group.

For more information on concurrency groups, see the documentation.

You can now view unique flaky test metrics on the suite summary page. This metric reflects the number of unique tests that have been flaky over a given period.

You can now filter flaky tests by branch using the Test Analytics UI or API.

The filter will limit the list of flaky tests to those for which a flake has been detected one or more times on the selected branch.

To learn more, check out the API documentation.

You can now filter your tests on the Test and Flaky test pages by owner.

• owner:team-slug
• owner:team-slug-1,team-slug-2
• owner:none

You can now view your builds on an interactive canvas. The build canvas makes it easier to visualize your pipelines, understand running order and dependencies, and troubleshoot issues. It's available from the Canvas tab on build pages.

You can follow the progress of a running build by selecting Follow mode or pressing j:

Troubleshoot and investigate failed jobs by selecting Go to failure or pressing f:

Hover over a step to see further details and navigate to the logs:

For all the information about the build canvas and a demo of the key features, see Visualize your CI/CD pipeline on a canvas.

If you have feedback or suggestions, please contact support.