Luc Shelton

In this article I am going to breakdown some of my previous professional experiences in developing and using personal build systems, that enforced robust sanity checks on code submitted to mainline repositories of large multi-team projects

During my time at Splash Damage, I was involved in developing the "personal build pipeline" that has since been highlighted in a talk presented by Valentin Galea, former Technical Director at Splash Damage. You can check out the slides from that presentation here.

For most of my professional career, I've made use of a continuous integration system called TeamCity. It's a closed-source, flexible, user-friendly build and automation system developed by JetBrains. It comes shipped with a wide range of server-sided plugins, including a vendor-supported plugin for running personal builds.

Links to documentation about Personal Builds with TeamCity, and it's servers-sided Remote Run plugin can be found here:

• JetBrains TeamCity: Personal Builds
• JetBrains TeamCity: Personal Builds - Visual Studio Extensions 
• JetBrains TeamCity: Remote Run
• JetBrains TeamCity: Remote Run Command-Line Tool

We made use of these set of vendor supported tools for creating a personal build pipeline, whereby changes made locally on the user's machine, were tested on a remote build machine for verification purposes.

The "personal build pipeline" that I developed was contingent on there being a few tools available.

• Perforce Changelist Tool
  • A user-facing tool that would construct a valid changelist description that would include all relevant information about a change, including a link or build ID (i.e. a build that is succeeding with the changes found in the changelist) to a valid personal build, a valid link or ID (i.e. approved or greenlit) to a Helix Swarm code review.
• Perforce Trigger Handler
  • A Perforce daemon trigger (p4d trigger) that was responsible for verifying that the changelist could be submitted by checking the contents of the changelist and its description.
• Personal Build Tool
  • A tool that would aggregate local changes, and submit those changes directly to the build server for triggering a personal build.
  • Otherwise known as the Content Validation Tool, as it was previously only ever meant to be used for testing changes made to packages and content assets.

You can read more about the personal build pipeline from the UnrealFest presentation here.

Goal

• Enable developers to conveniently run "personal builds" containing changes that are stored locally on the user's machine.
• Prevent developers from breaking the build by ensuring code compiles in a "live" build system environment.
• Ensure that change list descriptions or commit messages contain relevant information that conform to a enforced standard.

Personal Build Tool

The Personal Build Tool is a desktop application responsible for aggregating local changes in a Perforce workspace, submitting those local changes directly to the TeamCity server, and subsequently triggering a new build on TeamCity with the submitted changes. The local changes would be applied on top of the head revision of the repository checkout from mainline.

API

TeamCity exposes some of its functionality through a HTTP based "RESTful" API. This REST API is used in conjunction with the Remote Run plugin to perform a few tasks including:

• Checking the current status of the build, including whether or not it is queued, if it's running, and also retrieving the most recent build log data.
• Starting or queueing a new build with the changes generated with the patch file generated by the Personal Build Tool.
• Cancelling the build if there are any changes.
• Updating the build description or information with a checksum generated from the local modifications on disk.
  • This is used as part of the verification step with the Perforce Trigger Handler.

Information on the TeamCity REST API can be found here.

Decompiling Tools

Like most JetBrains products, the command-line runner was a server-sided plugin that was written in Java. This server-sided plugin was responsible for handling requests for personal builds from another command-line based tool that was also written in Java.

This command-line tool enables a user to specify one or more local files, a build type configuration ID, along with additional parameters (optional), and then submit those changes directly to the TeamCity server for starting a new personal build.

In order to achieve this, the command-line tool constructs a "patch file" which comprises of the user's local changes, along with additional metadata that enables the build system to understand where the files are to be placed in relation to the Version Control System (VCS). In this instance, the VCS was Perforce

Constructing the Patch File

The patch file is a simple binary file that consists of metadata, and the raw contents of local changes from the user's machine. The local changes included in this patch file is what the build system will use, on top of the existing head revision from the mainline of the code repository in the VCS (in this instance, Perforce). The goal of this entire system is to prevent the user from submitting local changes into the repository without first verifying whether or not those changes would compile successfully in a "live" build system environment (e.g. a machine not belonging to the user). This is the entire premise of the personal build system.

The structure of format specification for this patch file are unfortunately not documented anywhere, and it will require you to decompile the Java-based command-line remote run tool in order to understand how the patch file is constructed. The code itself is relatively simple enough to understand even without prior knowledge of Java.

The patch file consisted of information including the following:

• Version control system information, and which branch or stream this was meant to use as the head revision.
• Information about each locally changed file, including their names, total file size, and where they are meant to be placed on a build agent when building in a live environment.

With this knowledge, we were then able to develop a basic serialization mechanism as part of our Personal Build Tool, that was able to aggregate all local changes on the system, and produce a "patch" file that can be used for starting a personal build with.

This patch file would then be submitted directly to the build system, so that the developer can avoid submitting changes to the repository first. The changes found in the patch file would then be combined with the head revision of changes in the repository.

Perforce Changelist Tool

The Perforce Changelist Tool is a simple desktop application that produces a changelist description that conforms to a format or "specification" that the Perforce Trigger Handler expects. The goal of having a specific format or specification for changelist descriptions is to ensure that they are both informative, and consistent. The information found in a changelist description assists in tracking the completion of work.

The Perforce Changelist Tool, as documented in the UnrealFest presentation by Valentin Galea from Splash Damage.

Summary

The Perforce Changelist Tool is a simple desktop application that is launched from P4V, the GUI application "frontend" for Perforce.

Perforce Integration

When the tool is launched for the first-time from a context menu, the tool will automatically integrate itself with P4V by locating the configuration files that are available under the following path.

%USERPROFILE%\.p4qt\customtools.xml

This is an XML based file that gets loaded by P4V upon when the application is started. It provides definitions for custom tools that the client must use, including additional context menu actions that enable you to launch an external application with injected parameters (including the number of a changelist).

The changelist number then gets passed to the Perforce Changelist Tool through a command-line argument (i.e. --changelist 1234), which it then uses to load additional information about a changelist (e.g. files checked out, author, time created).

Generated Output

The purpose of the tool is to populate the description of a changelist with all required information in a format that is recognisable by the Perforce Trigger Handler, which is described later on in this article. The Perforce Changelist Tool makes use of a predetermined "specification", that was designed for ensuring that all subsequent changelists submitted into the repository had clear and concise information associated with them.

This "specification" ensured that the following fields were populated.

• Helix Swarm Review URL and ID
  • This only was required if the changes contained code-based changes.
• Jira Issue ID
  • If the change being submitted was associated with a Jira user story, then this was required.
• Tags
  • These were optional identifiers that simply made it easier to observe change lists at a glance when scrolling through changelist history in P4V.

Resources

• Read more documentation here about integrating "custom tools" with P4V.

Implementation

The tool made use of C#, WPF, and various other third-party libraries for UI controls and styling. In addition, the tool made use of toast notifications for notifying the user when an operation completed, and presented progress information in the task bar.

Perforce Trigger Handler

The Perforce Trigger Handler is a lightweight console-based application that is responsible for validating submitted changelists, by performing a series of checks. If the changelist is considered invalid, the application will terminate with an exit code of 1 (or anything not 0), otherwise if deemed acceptable the application will terminate gracefully with an exit code of 0. The behavior of this application is largely dictated by the specification for Perforce triggers that is documented here.

As stated, the console application performs a series of checks including:

• Validation Checks
  • Does it contain all required information?
  • Does it conform to the specified format for changelist descriptions?
• Time of Day
  • Is this changelist being submitted outside of work hours?
• Build Health
  • Is the latest build on TeamCity currently failing?
    • If so, is the author of the changelist the one who broke the build?

Validation Checks

Each changelist description needed to conform to an enforced standard of what a changelist description should look like. The specification stipulates how a changelist should appear, and what information it should contain. This information is generally populated by the aforementioned Perforce Changelist Tool, that ingests all necessary data from external sources and then produces a conformant changelist description.

Build Health

Before another user can submit code into the repository, it's important that existing build breakages are resolved otherwise subsequent commits to the code repository could worsen the state of the build. If the user submitting the changelist is not the one considered to be responsible for breaking the build, then the user will be prevented from submitting their changelist until the build goes green.

This ensures that existing build breakages are urgently resolved, otherwise there will be a blockage in the pipeline.

Time of Day

This a simple check that ensures that there is no one outside of working hours that is submitting changes into the repository, and that there's no possibility that the build will fail before people arrive at the office the following day.

The message presented to users through P4V if their changelist was considered invalid. The output seen in the image is produced by the Perforce Trigger Handler, a server-sided console application responsible for performing validation checks against the changelist beign submitted.

Implementation

The decision process for submitting the changelist looked a little like this.

The Perforce Trigger Handler had to perform a series of checks before allow for a changelist to be submitted to the Perforce server. The Perforce Trigger Handler could ultimately prevent a changelist from being accepted.

The above diagram encapsulates what happens when a changelist is submitted to the Perforce server.

Conclusion

Personal build pipelines are an extremely powerful tool, but they likely will not suit everyone's needs.

Problems

Personal builds are a great means for ensuring that all changes submitted into the mainline stream of the code repository are vigorously tested, and are less likely to break the build once changes are submitted into the repository.

Of course, the biggest issue with using such a pipeline is that it significantly reduces the speed of iteration across the team. Now, the team must successfully complete multiple steps in order to be able to submit a change into the repository.

There are some instances where this is a hindrance more than a benefit.

Exceptions

With the usage of a specifically configured "magic" keyword, it's possible to enable for the Perforce Trigger Handler to ignore performing all checks before allow a changelist to be submitted to the repository.

There is of course the possibility that such a "magic" keyword could be abused by members of the team, which is why it was imperative for the Perforce Trigger Handler to send out email notifications to team leads each time this particular keyword was used. This was simply to ensure accountability, and so that changes that were submitted using this keyword that were considered problematic can be accounted for (and reverted if necessary).

Splash Damage makes use of a special "[BuildFix]" keyword found in the changelist description to get around the Perforce Trigger Handler's preventative measures.

Build System Resources

As stated in the presentation by Splash Damage, allowing for personal builds at any time of the day, and launched on request by any development team member, means that there is an significantly increased demand on the "build farm" (i.e. systems dedicated for doing nothing other than build the game). Therefore, it's important that there are sufficient build machines made available if a personal build system is adopted, and that those build machines are placed into separate agent pools from the ones dedicated to "mission critical" build type configurations (i.e. ones that must run on a regular basis, or at a particular time of day).

This then ensures that "mission critical" build type configurations will continue to run uninterrupted (and not placed in queues) at any time of day, or whichever time of day that they are configured to run.

Comments

Having a personal build system available to a large-scale development team is an invaluable asset when it is integrated correctly. The fundamental goal is to ensure that a development team can continue to work at a fast and regular cadence without hindrance, and without compromising on the quality of the "product" they are delivering.

Whether or not you wish to integrate a personal build system with your team depends on a few things.

• How often are users committing code?
• How big is the team?
• Does your team have sufficient build system resources available (i.e. build agents)?
• Do you have sufficient development resources for building the aforementioned components?

User Experience

It's important to note that if you do not provide adequate user-friendly tooling around this system, then you will very quickly frustrate and demoralize members of the team. This is understandable, as you are expecting them to adopt an otherwise contrived workflow for results that may not be immediately clear. If you expect users to perform additional steps before submitting their work, then be sure to do as much as you can for them, and with minimal interference.

Tooling

Here's what each of the tools in the pipeline did for the user to mitigate confusion, frustration, or the amount of time required to read documentation.

Perforce Changelist Tool

P4V Integration

• On first-time launch, the tool made itself launchable through P4V by locating P4V's configuration file on disk, and injecting its own configuration into the file, so the tool was available as a context menu option upon right-clicking any changelist in P4V.
• Automatically pulled information about the user and their superior by making use of LDAP or Active Directory, and made "smart" suggestions about who the reviewers were.
• Automatically pulled information about code reviews associated with the changelist (from Helix Swarm).
• Automatically detected which Perforce client the user was using by iterating over all locally available Perforce clients and determining which one the tool resided in (the tool was distributed through Perforce).

Read more documentation here about "custom tools" in P4V.

Typically custom tools configuration are typically stored on the user's machine at the following path.

%USERPROFILE%\.p4qt\customtools.xml

It's a simple XML file that any external tool can simply inject values into, providing that the changes made compliant with the XSD (XML Schema Definition).

Perforce Build Tool

P4V Integration

• Similar to how Perforce Changelist Tool behaved, the tool integrated itself as part of P4V custom tools configuration file that is located at the path mentioned earlier on in this article. 

Personal Builds Support

• The tool made use of the official JetBrains server-sided plugin for performing personal builds, by constructing "patch files" that the TeamCity server instance would recognise. In order to achieve this, we decompiled the existing CLI tooling, and understood how it was constructing the patch files.
• The tool automatically started new builds on the TeamCity server instance with local changes, and provided the user with regular feedback in the form of taskbar progress information or toast notifications through Windows 10.

Perforce Trigger Handler

Remotely Manageable and Configurable

The Perforce Trigger Handler is a command-line application that sits on the same server as the Perforce daemon. More often than not, developers on the team won't have (nor will they require) direct access to the system that powers the Perforce daemon. Therefore, being able to remotely control or alter the behaviour of the Perforce Trigger Handler on the server is important. In our case, the approach for enabling this was simple. We had the application run from a network share that was accessible by both developers, and the server that the Perforce daemon operated from.