Our open supply construct system

• Buck2, our new open source, large-scale build system, is now obtainable on GitHub.
• Buck2 is an extensible and performant construct system written in Rust and designed to make your construct expertise quicker and extra environment friendly. 
• In our inner checks at Meta, we noticed that Buck2 accomplished builds 2x as quick as Buck1.

Buck2, Meta’s open supply large-scale construct system, is now publicly obtainable by way of the Buck2 website and the Buck2 GitHub repository. Whereas it shares some commonalities with different construct programs (like Buck1 and Bazel), Buck2 is a from-scratch rewrite. Buck2 incorporates a full separation of the core and language-specific guidelines, with elevated parallelism, integration with distant execution and digital file programs, and a redesigned console output. All of those modifications are aimed toward serving to engineers and builders spend much less time ready, and extra time iterating on their code.

Hundreds of builders at Meta are already utilizing Buck2 and performing thousands and thousands of builds per day, with builds finishing twice as quick as with Buck1. Our personal inner evaluation has proven that engineers have been capable of produce meaningfully extra code when their builds have been executed by Buck2, and we hope the broader business can even see advantages.

Why rebuild Buck?

Construct programs stand between a programmer and working their code, so something we will do to make the expertise faster or extra productive straight impacts how efficient a developer could be. The objective of Buck2 was to maintain what we favored about Buck1 (the core ideas and workflows), draw inspiration from improvements after Buck1 (together with Bazel, Adapton, and Shake), and concentrate on pace and enabling new experiences.

Buck2’s design relies on the next rules:

• The core construct system has no data of any language-specific guidelines. Having the principles separated from the core signifies that the principles are simpler to vary and perceive. The core of Buck2 is written in Rust, and its language guidelines (like the way to construct C++) are written in Starlark. This separation is in distinction to Buck1 (the place all guidelines are written within the core) and Bazel (the place C++/Java are written within the core).
• The construct system is powered by a single incremental dependency graph, avoiding any phases (in distinction to Buck1 or Bazel). This determination eliminates many varieties of bugs and will increase parallelism.
• The foundations API is designed to include superior options for efficiency, together with dynamic (or monadic) dependency options for expressibility. On the similar time, these options are rigorously restricted to make sure different properties (for instance, quick queries or hermeticity) should not harmed.
• The open supply launch is nearly equivalent to our inner model. The one items swapped out are the toolchains (which level on the inner copies of our compilers) and distant execution (which factors at our inner servers) — each have open supply alternate options provided. We’re additionally releasing all the principles precisely as they’re used internally. Moreover, we’ve separated a few of the logical elements into separate crates (e.g. Starlark, Superconsole, Allocative, Gazebo) in order that they can be utilized outdoors Buck2.
• Buck2 is written to combine with distant execution, with the flexibility to run actions on distant machines. We use the identical API as Bazel, and have been testing distant execution with Buildbarn and EngFlow. Whereas not required (and probably not anticipated for folks beginning out with the open supply model), we’re capable of effectively compute recursive digests and ship them to distant execution effectively.
• Buck2 is written to combine with digital file programs, the place your entire repository will not be all checked out, however fetched on demand because the information are accessed. Specifically, we assist Sapling-based file systems. To combine properly, we look ahead to file notifications (with Watchman) and request each information and file-digests with out direct file operations. The profit is that we will make digital file programs as quick as a full checkout, however with the advantages of a lot quicker checkout and far decrease disk utilization.

The important thing takeaway from all these enhancements is that we’ve designed Buck2 to be quick. In actual world utilization, relying on the construct, Buck2 is considerably quicker than Buck1. If there aren’t any supply code modifications, Buck2 is nearly prompt on subsequent builds. If there’s a number of work to do, Buck2 begins executing quicker and has higher parallelism. This improve in pace is each a consequence of most of the elements above, but in addition care and a spotlight.

The person view

For finish customers, Buck2 works principally the identical as Buck1 (which, to a primary approximation, is pretty just like Bazel). A person defines targets in a BUCK file:

rust_binary(
    identify = “my_binary”,
    srcs = [“main.rs”],
    deps = [“:my_library”],
)

A person can then construct with buck2 construct //:my_binary. The worth essential.rs is a supply file, and :my_library is a dependency outlined in the identical BUCK file. It’s value noting that Buck2 is generally suitable with the BUCK information of Buck1. 

In addition to the rise in pace, there are two main further user-visible variations in comparison with Buck1.

First, the console output has been redesigned on high of the Superconsole library, which we particularly developed for Buck2. The console exhibits a number of extra particulars and feels quite a bit nicer to make use of:

Second, there’s a persistent daemon that maintains a single dependency graph. Whenever you change a BUCK file, a dependency, or a supply file, we invalidate the suitable issues on the dependency graph, then request the output artifacts per the command line. In Buck1 there are a number of distinct dependency graphs, which lead to phases like goal graph building, motion graph building, after which motion graph execution. There are additionally some operations that aren’t carried out on the graph. If sure issues change in Buck1, then total graphs are thrown away, slightly than the minimal items being invalidated. With a single dependency graph, Buck2 is less complicated, avoids extra redundant work, and avoids specific phases. All the things on the dependency graph has a key (how it’s recognized) and a price, together with a operate to compute the worth from the important thing and different associated keys (following the mannequin within the paper, “Build Systems a la Carte”).

The rule creator view

Whereas the person mannequin follows Buck1 very carefully, the method for guidelines is totally completely different. There are many guidelines in Buck, for instance rust_binary used above. Whereas a rule in Buck1 was a Java class, baked into Buck1, a rule in Buck2 is fully decoupled. Buck2 additionally ships with a “prelude” of guidelines that implement many of the Buck1 guidelines. 

Buck1 guidelines have been tuned over time, had a number of efficiency optimizations and highly effective options like graph traversal, however these guidelines have been additionally anticipated to obey a number of advanced invariants—typically breaking these guidelines. For Buck2, the rule API is fully in Starlark, which pressured us to summary these options as generically reusable APIs, aiming to make them secure, expressive, and performant—a difficult stability. We’ll contact on two such examples.

OCaml dependencies

The dependency construction of the OCaml library is tough to precise in Buck1. An OCaml library consists of plenty of OCaml information. These should be compiled in dependency order—so if A.ml makes use of B.ml, you have to compile B.ml first. Bazel requires the dependency of A.ml on B.ml to be written explicitly within the BUCK file. Buck1 and Buck2 each go away that inner dependency implicit and run the software ocamldep to deduce it, which requires much less upkeep because the construction modifications.  What Buck1 did is run ocamldep simply after parsing the BUCK file, which wasn’t actually allowed, and it didn’t monitor dependencies, so should you modified the imports an excessive amount of Buck1 gave spurious compilation failures. With Buck2, we will use the new primitive dynamic_output, which helps you to run a command, learn the output of the file, then wire up the remainder of the graph—placing within the appropriate dependencies between the .ml information routinely.

C++ hyperlink dependencies

Take into account the C++ linking mannequin: To supply a library, you normally must hyperlink collectively its construct output, together with the transitive closure of the construct output of its dependencies. If you happen to merely duplicate the set of dependencies at every layer as you progress up the graph, you find yourself with O(n²) reminiscence utilization. In Buck1, there was customized code in lots of guidelines to seize this sample, counting on the flexibility to share Java values in reminiscence and for the dependencies to be represented in place throughout the rule construction (as there was no reified dependency graph). In Buck2, there are a lot stronger abstraction boundaries, so such reuse must be made extra specific. Due to this fact, we introduced transitive-sets (tsets) to seize this sample of units representing a transitive closure. By making tsets extra summary, we have been additionally capable of wire the tset straight into the underlying dependency graph, that means this illustration is environment friendly in each reminiscence and computation time.

Attempt Buck2 now

We’re eager for folks to present Buck2 a attempt, and we’d be blissful to listen to any suggestions (GitHub issues are one of the best ways). We count on Buck2 will likely be most fascinating to reasonably sized multi-language initiatives. Go to the Buck2 getting started page for extra data.