Changelog

Cancelling hanging jobs

We have rolled out a new fix for pipelines that have hanging jobs.

If you are using test splitting queues such as Shopify's open source tool ci-queue (https://github.com/Shopify/ci-queue) at scale, you may have noticed that some jobs are hanging around and causing your builds to take longer than expected.

There is now a new agent cli command to cancel these pesky jobs to improve your build times.

To force cancel all jobs in a step you can run the following from any job within the target build:

1
buildkite-agent step cancel --step <step_key> --force

In this kind of situation you may have a pipeline that looks like this:

1
2
3
4
5
6
7
steps:
  - command: test_reporter.sh
    key: reporter
  - command: test_worker.sh
    key: workers
    parallelism: 500
    soft_fail: true

In this pipeline the workers are responsible for running the tests but the outcome of the tests is reported by the reporter step. The outcome of the workers in Buildkite is not relevant to the build outcome. When a step is marked as soft fail Buildkite still waits for all jobs to finish before marking the build as passed. This can cause the build to take longer than necessary. To fix this you can add a step to cancel the workers after the reporter step has finished.

1
2
3
4
5
6
7
8
9
steps:
  - command:
      - test_reporter.sh
      - buildkite-agent step cancel --step workers --force
    key: reporter
  - command: test_worker.sh
    key: workers
    parallelism: 500
    soft_fail: true

If you find this cancels too quickly, leaving agents unable to upload logs and artefacts, you can set a custom a grace period with the --force-grace-period-seconds flag. This will allow the agents to finish their work before being cancelled.

1
2
3
4
5
6
7
8
9
steps:
  - command:
      - test_reporter.sh
      - buildkite-agent step cancel --step workers --force --force-grace-period-seconds 10
    key: reporter
  - command: test_worker.sh
    key: workers
    parallelism: 500
    soft_fail: true

Quinn

Step durations for GitHub commit statuses

We've added step running durations to GitHub commit status step notifications.

A GitHub commit status with running duration

When a step includes a :github_commit_status step notification the step running duration will be included in the commit status description on GitHub. The duration will show when the step is finished.

GitHub commit status step notifications can be configured in your pipeline.yml:

1
2
3
4
steps:
- command: build-the-thing.sh
  notify:
    github_commit_status: true

Grant

New Slack Workspace Integration

We've created a new Slack notification service for easier management of your Slack notifications. This new notification service requires only a single authorization for the entire workspace and replaces the need for multiple Slack notification services. See the documentation for enabling a Slack Workspace for your organization. Going forward, we recommend using Slack Workspace. Connecting this new notification service will not impact current implementations.

With this change comes:

  • Build notifications now include mentions when triggering builds using an account with an associated email in slack workspace.
  • Build notifications can be filtered to only trigger on the first failure with started_failing, rather than notifying on recurring failures. Using YAML, this is possible in the existing slack integration and the new workspace.
  • Slack channels are now configured in pipeline YAML when using slack workspace.

Eleanor

Playwright Test Splitting Support

We're excited to announce Playwright support for our Test Engine Client, to complement RSpec and Jest as supported test frameworks for test splitting. The client uses timing data obtained from your test suite to intelligently partition your tests across parallel agents, reducing the time taken to run them.

To begin using the client with Playwright, you can follow the instructions in the Playwright documentation in the Test Engine Client repository.

Stephen

AWS KMS support for Signed Pipelines

We have added support for AWS KMS signing of pipelines to the buildkite agent and integrated this feature into the Elastic Stack for AWS.

Customers can now easily enable signed pipelines in their elastic stack and take advantage of KMS to securely manage the keys. The settings used to configure this feature is as follows:

signed pipeline settings

For more information on Signed Pipelines using AWS KMS see our documentation.

Mark

Team management in suite settings

We've introduced team management to the suite settings page.

Team management page in suite settings

Users with permission to edit the suite can:

  • Add & Remove Teams: Easily control which teams have access to the test suite.
  • Edit Permissions: Adjust team permissions directly from the Teams page.
  • Team Insights: View the total number of users in each team and how many test suites each team can access.
  • Quick Navigation: Click on a team to navigate directly to its details page for managing team members.

This feature streamlines team access management, allowing users to configure permissions and team memberships efficiently within their test suite environment.

Katie

Date filtering on flaky test page

We've added a date filter to the flaky tests page, allowing users to refine the list of flaky tests based on set date ranges. Users can now filter flaky test results from the last hour up to the last 28 days.

Filter your flakes by date

This enhancement provides greater insight into flaky tests, enabling users to identify recent test instability or track longer-term trends in test flakiness. Enjoy!

Katie

Dark mode

We’re excited to release dark mode for all customers!

In your user menu you can set Buildkite to display in light or dark mode, or adapt to your system theme:

Theme control highlighted in user menu

Thanks to all the customers whose input shaped this feature. Note that dark mode is experimental and you may encounter issues. Please reach out with any feedback or issues you notice, thanks!

Brett

IPv6 Support for the Buildkite Agent

We will be enabling IPv6 support for the agent.buildkite.com domain used by the Buildkite Agent on 2024-11-20. In dual-stack¹ environments, this may mean that agents start connecting from IPv6 addresses where they previously used only IPv4.

If you:

you should ensure that your tokens' IP allowlists include your agents' IPv6 address range(s).

¹ An environment where both IPv4 and IPv6 may be used on the public Internet.

Ellis

Suite repository URL validation

We have updated the validation for the repository URL field on the Suite Settings page. Previously, users could only set GitHub or GitLab URLs. With this update, you can now enter any valid URL as the repository URL.

This provides more flexibility when setting your repository URL, allowing integration with other platforms beyond GitHub and GitLab.

No action is required on your part to benefit from this update. Enjoy the enhanced flexibility!

Katie

Tests tab in job UI

You can now see your Test Analytics failures directly within the build job.

Tests tab

The new Tests tab lists all test executions that have failed in the current job, giving you a quick overview of failures and their summaries.

Meghan

New execution drawer

Navigating to an execution now opens a drawer within the Test's page, enabling you quick access to execution details without leaving the context of the test. This improves workflow by making it easier to explore test details and debug faster.

Execution drawer

Meghan

Test digest on build page

Pipelines with at least one Test Suite associated have a new Test Digest tab on build pages. This tab displays detailed information about tests that failed during the build runtime.

Legitimate failures are prioritized and listed at the top of the digest, highlighting tests likely affected by the recent code changes on the branch being built.

Tests that flaked during the build are clearly labeled for easy identification.

Test digest on the build page

Katie

Disable public pipeline creation

Organization admins can now disable public pipeline creation for all organization members.

Screenshot of public pipeline settings

Public pipeline creation is enabled by default.

Disable public pipeline creation

Navigate to your organization settings by selecting 'Settings' in the top nav. In the side nav, under 'Pipelines', select 'Settings'. In the 'Public Pipelines' section, select 'Disable Public Pipeline Creation', and confirm in the dialog box.

Once public pipeline creation is disabled, options to make pipelines public are hidden from the Buildkite UI. If a user attempts to create a public pipeline via our API, an error will be returned.

Disabling public pipeline creation will not affect existing public pipelines - these will remain public, unless you explicitly change their visibility to private. Once this is done, their visibility cannot be made public again unless public pipeline creation is re-enabled.

Laura

Improved filtering for builds

We've improved filtering on the builds list to help you find builds quicker and easier.

We've made filters more discoverable by showing them in place at the top of the builds list, rather than hiding them away under a menu.

Screenshot of builds list showing filters for branches, commit SHA or message, user, and state

Filter by branch

The improved branch filter allows you to explore builds on any of your projects branches, and even search across multiple branches.

Start typing the name of your branch narrow down the list, and select the branch you want to see builds for.

Use a wildcard, like feature-*, to find builds across all matching branches.

The current branch is displayed in the breadcrumb navigation, so you can quickly jump back to all builds, or the pipeline overview.

Search by commit SHA or message

Using the new search filter, you can paste in a commit SHA to find associated builds. Short SHAs work too!

If you don’t have the SHA handy, just type part of the commit message, and we’ll show you matching builds.

View your builds

Easily switch between seeing all builds and just your own using the My Builds toggle.

Filter by state

Narrow down your search to include only builds that are running, passed, failed, or in any other state.

Angus

Test page header improvements

We have updated the test page to enhance your workflow with several new additions. You can now resolve and assign a flaky test to a team directly from the header.

Test page header

Additionally, you can now view your test within your Git repository through a link in the header. To enable this feature, add your root Git repository URL in the suite settings page.

Test page header Git settings

Meghan

IPv6 Support for the Buildkite API

We will be enabling IPv6 support for Buildkite's APIs on 2024-09-18. This includes api.buildkite.com, graphql.buildkite.com, and analytics-api.buildkite.com. If you have any clients consuming the Buildkite API while operating on a dual-stack network, it is important to check that any API Token IP allowlists, and any Organization API IP allowlists permit your IPv6 network ranges in preparation for the change.

Ellis

Better visibility for jobs waiting on concurrency groups

On the build page, jobs waiting on a concurrency group now show a list of the jobs ahead of them in the group.

Example of a job waiting on a concurrency group

This list gives you visibility into how soon the current job might run and which jobs are backing up a concurrency group.

For more information on concurrency groups, see the documentation.

David

Unique flaky test metric

You can now view unique flaky test metrics on the suite summary page. This metric reflects the number of unique tests that have been flaky over a given period.

Unique flaky tests metric

Meghan

Filter flaky tests by branch from the UI or API

You can now filter flaky tests by branch using the Test Analytics UI or API.

Test Analytics allows users to filter flaky tests by branch on the Flaky test page and API

The filter will limit the list of flaky tests to those for which a flake has been detected one or more times on the selected branch.

To learn more, check out the API documentation.

Malcolm

Start turning complexity into an advantage

Create an account to get started with a 30-day free trial. No credit card required.

Buildkite Pipelines

Platform

  1. Pipelines
  2. Pipeline templates
  3. Public pipelines
  4. Test Engine
  5. Package Registries
  6. Mobile Delivery Cloud
  7. Pricing

Hosting options

  1. Self-hosted agents
  2. Mac hosted agents
  3. Linux hosted agents

Resources

  1. Docs
  2. Blog
  3. Changelog
  4. Webinars
  5. Plugins
  6. Case studies
  7. Events

Company

  1. About
  2. Careers
  3. Press
  4. Brand assets
  5. Contact

Solutions

  1. Replace Jenkins
  2. Workflows for AI/ML
  3. Testing at Scale

Support

  1. System status
  2. Forum