We've just shipped new features that'll help uplevel your build and test workflows with Buildkite, including some key announcements:

• Kubernetes agent stack
• OpenID Connect (OIDC) support
• flaky test tracker for automatically detecting flaky tests

Learn more about what's new in our March 2023 release ✨

We've released a new way to run your Buildkite jobs in Kubernetes natively. The Agent Stack for Kubernetes will allow your Kubernetes cluster to orchestrate your Buildkite Pipeline steps as Kubernetes jobs.

Learn more about the Agent Kubernetes stack

Prompt your users to re-authorize when their origin changes.

With session IP address pinning enabled, authorized sessions can only come from the IP address that created the session. If another IP address attempts to access the organization, the session will be immediately revoked. By pairing IP pinning with SSO session durations, we're taking a proactive approach to combating stolen session cookies.

We're committed to keeping our customers' data secure and are constantly exploring new ways to enhance our security measures.

Learn more about session IP address pinning

Clusters allow you to organize agents into groups. These groups, or clusters, will enable the management of pipelines and queues within that cluster.

Clusters can be turned on by an admin by accessing pipeline settings in the organization settings tab. Note that once clusters is enabled, you will be unable to disable it.

Learn more about clusters

You can now request an OpenID Connect (OIDC) token from the Buildkite Agent 🔑

OIDC tokens are JWTs signed by Buildkite and decode into JSON which includes many attributes like the pipeline slug and the build branch. buildkite-agent oidc request-token will return a token representing the current job that can be exchanged with federated systems to authorize actions like deployments or allow access to context-sensitive information like secrets based on these attributes.

Learn more about OpenID Connect support from the Buildkite Agent

Explore organization change events in your existing AWS monitoring suite.

Enterprise customers can now route Buildkite Audit Log events via the AWS Event Bridge event bus.

Learn more about configuring Buildkite with AWS EventBridge

Restrict API access to IP addresses and CIDR block ranges you trust.

You can now easily create and manage a list of IP addresses and CIDR blocks that are authorized to access your organization via the Buildkite API, improving security and reducing the risk of unauthorized access.

Learn more about configuring IP/CIDR allowlist via the UI, API, or Terraform

Jobs that belong to group steps will now have access to information about their group with three new environment variables:

• BUILDKITE_GROUP_ID
• BUILDKITE_GROUP_KEY
• BUILDKITE_GROUP_LABEL

You could use these variables to upload steps to the same group, or alter the behaviour of jobs based on their group. These environment variables will be absent for jobs that do not belong to group steps.

Learn more about environment variables

Jobs can now be automatically retried based on the signal received by the command process that caused it to exit, in addition to the job's exit code.

This is particularly useful in catching terminated agent hosts, such as you'd see when using EC2 Spot Instances:

1
2
3
4
5
6
7
8
9
10
11
  - label: "Tests"
    command: "tests.sh"
    retry:
      automatic:
        # Catch cleanly-terminated instances
        - limit: 2
          signal_reason: "agent_stop"
        # Catch timed-out agents
        - limit: 2
          exit_status: -1
          signal_reason: none

Learn more about the new attributes

Build Matrix has been extended to support matrix variable interpolation inside the plugins and agents attributes of command steps.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
steps:
- label: "💥 Matrix Build"
  command: "echo {{matrix.os}} {{matrix.arch}}"
  agents:
    queue: "builder-{{matrix.arch}}"
  matrix:
    setup:
      arch:
        - "amd64"
        - "arm64"
      os:
        - "windows"
        - "linux"
  plugins:
    - artifacts#v1.9.0:
        upload: "out/{{matrix.arch}}.gz"

Learn more about increased Build Matrix support

The waterfall view visualizes the timeline of each step in a build. You can see this summary in your Builds page a toggle–enabling to switch between list and waterfall views.

Quickly view insights about failed tests by going directly from a job to its related information in Test Analytics – providing a faster path from fail to fix.

Flaky tests are automated tests that produce inconsistent or unreliable results, despite being run on the same code and environment. They cause frustration, decrease confidence in testing, and waste time while you investigate whether the failure is due to a genuine bug.

Test Analytics finds your flakes by surfacing when the same test is run multiple times on the same commit SHA with different results. The tests might run multiple times within a single build or across different builds. Either way, they are detected as flaky if they report both passed and failed results.

Results are available in the Test Analytics UI and via a new REST API endpoint.

Learn more about flaky test tracker and its API

We've updated the Build UI to be more intuitive with the following improvements:

• The issues tab is called the failures tab, making it easier for you to view failed jobs
• The organization name and avatar are now seen on each Build page, to make it easier view the organization you're editing settings for
• Block steps within groups are now easier to identify in the UI with a thickened purple border
• Pipelines can be bookmarked instead of starred
• Build summary now has a rebuild option, rather than edit steps and new build options
• You can now navigate to all builds by clicking on the builds icon
• Spacing and font fixes

It's also easier to make your pipelines stand out with:

• Pipeline avatars in place of build stats
• Configuration of pipeline color and emoji through pipeline settings

Pipeline tags make it easier to sort through and filter multiple pipelines. Tag names support both text and emojis. You can now tag your pipelines and use the search bar to find tags.

Tags are unique per pipeline, don't contain line breaks and are capped at 64 characters. You can have 5 tags per pipeline.

Tags can also be set through REST and GraphQL APIs.

To set the tags, go to Pipeline Settings > General > Tags

Analyze your usage by pipeline or test suite, with the option to view monthly or daily breakdowns. Delve into previous billing periods, and conveniently export your data for offline analysis.

Learn more about usage breakdown

The latest agent release includes the job-api experiment, which enables an HTTP API within the agent, allowing jobs to inspect and mutate their environment without using bash. This is a big step towards supporting hooks and plugins in other languages.

Other experimental features include:

• allowing spawning agents with a descending priority, rather than the default ascending priority
• artifacts can now be uploaded to S3 compatible endpoints such as MinIO.

Other updates to this release include:

• the AWS Elastic Stack has been updated with a newer version of buildx
• the managed secrets buckets created by the stack now have public ACLs explicitly blocked

For a full list of additions, changes, fixes, and more details, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

We've added a glossary to highlight and explain the core concepts of pipelines.

See Pipelines glossary to check it out. ✨

The 3.44.0 version of the buildkite-agent and the 5.17.0 version of the AWS elastic stack are now available.

Major updates to the agent include:

• New experiment for running jobs under Kubernetes
• Docker images based on Ubuntu 22.04
• The ability to add claims to OIDC token requests

This agent release has been added to the 5.17.0 release of the elastic stack, as well as support for c7gn, m7g, and r7g instance type classes, and updates to Docker, Docker Compose, buildx, git, and the Linux kernel.

For a full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

Update: This change has now been completed.

Over the past two years, GitHub, GitLab, and other Git services have updated their default branch names from "master" to "main" – you can read more about the motivation for the change in this statement from the Software Freedom Conservancy.

In line with this change, we will be updating the default branch for newly created pipelines from "master" to "main" from April 1st, 2023. This will not impact existing pipelines.

You can configure a given pipeline's default branch through the user interface, as well as the REST API and GraphQL.

If you would like to retain "master" as the default branch of new pipelines, you may set an organization-level default branch in Pipeline Settings, which will then be used for new pipelines: