Accurate Garbage Collection with LLVM

Garbage collection is a widely used technique that frees the programmer from having to know the lifetimes of heap objects, making software easier to produce and maintain. Many programming languages rely on garbage collection for automatic memory management. There are two primary forms of garbage collection: conservative and accurate.

Conservative garbage collection often does not require any special support from either the language or the compiler: it can handle non-type-safe programming languages (such as C/C++) and does not require any special information from the compiler. The Boehm collector is an example of a state-of-the-art conservative collector.

Accurate garbage collection requires the ability to identify all pointers in the program at run-time (which requires that the source-language be type-safe in most cases). Identifying pointers at run-time requires compiler support to locate all places that hold live pointer variables at run-time, including the processor stack and registers.

Conservative garbage collection is attractive because it does not require any special compiler support, but it does have problems. In particular, because the conservative garbage collector cannot know that a particular word in the machine is a pointer, it cannot move live objects in the heap (preventing the use of compacting and generational GC algorithms) and it can occasionally suffer from memory leaks due to integer values that happen to point to objects in the program. In addition, some aggressive compiler transformations can break conservative garbage collectors (though these seem rare in practice).

Accurate garbage collectors do not suffer from any of these problems, but they can suffer from degraded scalar optimization of the program. In particular, because the runtime must be able to identify and update all pointers active in the program, some optimizations are less effective. In practice, however, the locality and performance benefits of using aggressive garbage allocation techniques dominates any low-level losses.

This document describes the mechanisms and interfaces provided by LLVM to support accurate garbage collection.

LLVM's intermediate representation provides garbage collection intrinsics that offer support for a broad class of collector models. For instance, the intrinsics permit:

semi-space collectors
mark-sweep collectors
generational collectors
reference counting
incremental collectors
concurrent collectors
cooperative collectors

We hope that the primitive support built into the LLVM IR is sufficient to support a broad class of garbage collected languages including Scheme, ML, Java, C#, Perl, Python, Lua, Ruby, other scripting languages, and more.

However, LLVM does not itself implement a garbage collector. This is because collectors are tightly coupled to object models, and LLVM is agnostic to object models. Since LLVM is agnostic to object models, it would be inappropriate for LLVM to dictate any particular collector. Instead, LLVM provides a framework for garbage collector implementations in two manners:

At compile time with collector plugins for the compiler. Collector plugins have ready access to important garbage collector algorithms. Leveraging these tools, it is straightforward to emit type-accurate stack maps for your runtime in as little as ~100 lines of C++ code.
At runtime with suggested runtime interfaces, which allow front-end compilers to support a range of collection runtimes.

In general, using a collector implies:

Emitting compatible code, including initialization in the main program if necessary.
Loading a compiler plugin if the collector is not statically linked with your compiler. For llc, use the -load option.
Selecting the collection algorithm by applying the gc "..." attribute to your garbage collected functions, or equivalently with the setGC method.
Linking your final executable with the garbage collector runtime.

This table summarizes the available runtimes.

Collector	`gc` attribute	Linkage	`gcroot`	`gcread`	`gcwrite`
SemiSpace	`gc "shadow-stack"`	TODO FIXME	required	optional	optional
Ocaml	`gc "ocaml"`	provided by ocamlopt	required	optional	optional

The sections for Collection intrinsics and Recommended runtime interface detail the interfaces that collectors may require user programs to utilize.

The ShadowStack backend is invoked with the gc "shadow-stack" function attribute. Unlike many collectors which rely on a cooperative code generator to generate stack maps, this algorithm carefully maintains a linked list of stack root descriptors [Henderson2002]. This so-called "shadow stack" mirrors the machine stack. Maintaining this data structure is slower than using stack maps, but has a significant portability advantage because it requires no special support from the target code generator.

The ShadowStack collector does not use read or write barriers, so the user program may use load and store instead of llvm.gcread and llvm.gcwrite.

ShadowStack is a code generator plugin only. It must be paired with a compatible runtime.

The SemiSpace runtime implements the suggested runtime interface and is compatible with the ShadowStack backend.

SemiSpace is a very simple copying collector. When it starts up, it allocates two blocks of memory for the heap. It uses a simple bump-pointer allocator to allocate memory from the first block until it runs out of space. When it runs out of space, it traces through all of the roots of the program, copying blocks to the other half of the memory space.

This runtime is highly experimental and has not been used in a real project. Enhancements would be welcomed.

The ocaml backend is invoked with the gc "ocaml" function attribute. It supports the Objective Caml language runtime by emitting a type-accurate stack map in the form of an ocaml 3.10.0-compatible frametable. The linkage requirements are satisfied automatically by the ocamlopt compiler when linking an executable.

The ocaml collector does not use read or write barriers, so the user program may use load and store instead of llvm.gcread and llvm.gcwrite.

This section describes the garbage collection facilities provided by the LLVM intermediate representation.

These facilities are limited to those strictly necessary for compilation. They are not intended to be a complete interface to any garbage collector. Notably, heap allocation is not among the supplied primitives. A user program will also need to interface with the runtime, using either the suggested runtime interface or another interface specified by the runtime.

The gc function attribute is used to specify the desired collector algorithm to the compiler. It is equivalent to specifying the collector name programmatically using the setGC method of Function.

Specifying the collector on a per-function basis allows LLVM to link together programs that use different garbage collection algorithms.

The llvm.gcroot intrinsic is used to inform LLVM of a pointer variable on the stack. The first argument must be a value referring to an alloca instruction or a bitcast of an alloca. The second contains a pointer to metadata that should be associated with the pointer, and must be a constant or global value address. If your target collector uses tags, use a null pointer for metadata.

Consider the following fragment of Java code:

       {
         Object X;   // A null-initialized reference to an object
         ...
       }

This block (which may be located in the middle of a function or in a loop nest), could be compiled to this LLVM code:

Entry:
   ;; In the entry block for the function, allocate the
   ;; stack space for X, which is an LLVM pointer.
   %X = alloca %Object*
   
   ;; Tell LLVM that the stack space is a stack root.
   ;; Java has type-tags on objects, so we pass null as metadata.
   %tmp = bitcast %Object** %X to i8**
   call void @llvm.gcroot(i8** %X, i8* null)
   ...

   ;; "CodeBlock" is the block corresponding to the start
   ;;  of the scope above.
CodeBlock:
   ;; Java null-initializes pointers.
   store %Object* null, %Object** %X

   ...

   ;; As the pointer goes out of scope, store a null value into
   ;; it, to indicate that the value is no longer live.
   store %Object* null, %Object** %X
   ...

Some collectors need to be informed when the mutator (the program that needs garbage collection) either reads a pointer from or writes a pointer to a field of a heap object. The code fragments inserted at these points are called read barriers and write barriers, respectively. The amount of code that needs to be executed is usually quite small and not on the critical path of any computation, so the overall performance impact of the barrier is tolerable.

Barriers often require access to the object pointer rather than the derived pointer (which is a pointer to the field within the object). Accordingly, these intrinsics take both pointers as separate arguments for completeness. In this snippet, %object is the object pointer, and %derived is the derived pointer:

    ;; An array type.
    %class.Array = type { %class.Object, i32, [0 x %class.Object*] }
    ...

    ;; Load the object pointer from a gcroot.
    %object = load %class.Array** %object_addr

    ;; Compute the derived pointer.
    %derived = getelementptr %object, i32 0, i32 2, i32 %n

For write barriers, LLVM provides the llvm.gcwrite intrinsic function. It has exactly the same semantics as a non-volatile store to the derived pointer (the third argument).

Many important algorithms require write barriers, including generational and concurrent collectors. Additionally, write barriers could be used to implement reference counting.

The use of this intrinsic is optional if the target collector does use write barriers. If so, the collector will replace it with the corresponding store.

For read barriers, LLVM provides the llvm.gcread intrinsic function. It has exactly the same semantics as a non-volatile load from the derived pointer (the second argument).

Read barriers are needed by fewer algorithms than write barriers, and may have a greater performance impact since pointer reads are more frequent than writes.

As with llvm.gcwrite, a target collector might not require the use of this intrinsic.

LLVM specifies the following recommended runtime interface to the garbage collection at runtime. A program should use these interfaces to accomplish the tasks not supported by the intrinsics.

Unlike the intrinsics, which are integral to LLVM's code generator, there is nothing unique about these interfaces; a front-end compiler and runtime are free to agree to a different specification.

Note: This interface is a work in progress.

void llvm_gc_initialize(unsigned InitialHeapSize);

The llvm_gc_initialize function should be called once before any other garbage collection functions are called. This gives the garbage collector the chance to initialize itself and allocate the heap. The initial heap size to allocate should be specified as an argument.

void *llvm_gc_allocate(unsigned Size);

The llvm_gc_allocate function is a global function defined by the garbage collector implementation to allocate memory. It returns a zeroed-out block of memory of the specified size, sufficiently aligned to store any object.

void llvm_gc_collect();

The llvm_gc_collect function is exported by the garbage collector implementations to provide a full collection, even when the heap is not exhausted. This can be used by end-user code as a hint, and may be ignored by the garbage collector.

void llvm_cg_walk_gcroots(void (*FP)(void **Root, void *Meta));

The llvm_cg_walk_gcroots function is a function provided by the code generator that iterates through all of the GC roots on the stack, calling the specified function pointer with each record. For each GC root, the address of the pointer and the meta-data (from the llvm.gcroot intrinsic) are provided.

User code specifies which GC code generation to use with the gc function attribute or, equivalently, with the setGC method of Function.

To implement a GC plugin, it is necessary to subclass llvm::GCStrategy, which can be accomplished in a few lines of boilerplate code. LLVM's infrastructure provides access to several important algorithms. For an uncontroversial collector, all that remains may be to emit the assembly code for the collector's unique stack map data structure, which might be accomplished in as few as 100 LOC.

This is not the appropriate place to implement a garbage collected heap or a garbage collector itself. That code should exist in the language's runtime library. The compiler plugin is responsible for generating code which is compatible with that runtime library.

To subclass llvm::GCStrategy and register it with the compiler:

// lib/MyGC/MyGC.cpp - Example LLVM GC plugin

#include "llvm/CodeGen/GCStrategy.h"
#include "llvm/CodeGen/GCMetadata.h"
#include "llvm/Support/Compiler.h"

using namespace llvm;

namespace {
  class VISIBILITY_HIDDEN MyGC : public GCStrategy {
  public:
    MyGC() {}
  };
  
  GCRegistry::Add<MyGC>
  X("mygc", "My bespoke garbage collector.");
}

Using the LLVM makefiles (like the sample project), this can be built into a plugin using a simple makefile:

# lib/MyGC/Makefile

LEVEL := ../..
LIBRARYNAME = MyGC
LOADABLE_MODULE = 1

include $(LEVEL)/Makefile.common

Once the plugin is compiled, code using it may be compiled using llc -load=MyGC.so (though MyGC.so may have some other platform-specific extension):

$ cat sample.ll
define void @f() gc "mygc" {
entry:
        ret void
}
$ llvm-as < sample.ll | llc -load=MyGC.so

It is also possible to statically link the collector plugin into tools, such as a language-specific compiler front-end.

The boilerplate collector above does nothing. More specifically:

llvm.gcread calls are replaced with the corresponding load instruction.
llvm.gcwrite calls are replaced with the corresponding store instruction.
No stack map is emitted, and no safe points are added.

Collector provides a range of features through which a plugin collector may do useful work. This matrix summarizes the supported (and planned) features and correlates them with the collection techniques which typically require them.

Algorithm	Done	shadow stack	refcount	mark-sweep	copying	incremental	threaded	concurrent
stack map	✔			✘	✘	✘	✘	✘
initialize roots	✔	✘	✘	✘	✘	✘	✘	✘
derived pointers	NO						✘*	✘*
custom lowering	✔
gcroot	✔	✘	✘
gcwrite	✔		✘			✘		✘
gcread	✔							✘
safe points
in calls	✔			✘	✘	✘	✘	✘
before calls	✔						✘	✘
for loops	NO						✘	✘
before escape	✔						✘	✘
emit code at safe points	NO						✘	✘
output
assembly	✔			✘	✘	✘	✘	✘
JIT	NO			✘	✘	✘	✘	✘
obj	NO			✘	✘	✘	✘	✘
live analysis	NO			✘	✘	✘	✘	✘
register map	NO			✘	✘	✘	✘	✘
* Derived pointers only pose a hazard to copying collectors. ✘ in gray denotes a feature which could be utilized if available.

To be clear, the collection techniques above are defined as:

Shadow Stack: The mutator carefully maintains a linked list of stack root descriptors.
Reference Counting: The mutator maintains a reference count for each object and frees an object when its count falls to zero.
Mark-Sweep: When the heap is exhausted, the collector marks reachable objects starting from the roots, then deallocates unreachable objects in a sweep phase.
Copying: As reachability analysis proceeds, the collector copies objects from one heap area to another, compacting them in the process. Copying collectors enable highly efficient "bump pointer" allocation and can improve locality of reference.
Incremental: (Including generational collectors.) Incremental collectors generally have all the properties of a copying collector (regardless of whether the mature heap is compacting), but bring the added complexity of requiring write barriers.
Threaded: Denotes a multithreaded mutator; the collector must still stop the mutator ("stop the world") before beginning reachability analysis. Stopping a multithreaded mutator is a complicated problem. It generally requires highly platform specific code in the runtime, and the production of carefully designed machine code at safe points.
Concurrent: In this technique, the mutator and the collector run concurrently, with the goal of eliminating pause times. In a cooperative collector, the mutator further aids with collection should a pause occur, allowing collection to take advantage of multiprocessor hosts. The "stop the world" problem of threaded collectors is generally still present to a limited extent. Sophisticated marking algorithms are necessary. Read barriers may be necessary.

As the matrix indicates, LLVM's garbage collection infrastructure is already suitable for a wide variety of collectors, but does not currently extend to multithreaded programs. This will be added in the future as there is interest.

for (iterator I = begin(), E = end(); I != E; ++I) {
  GCFunctionInfo *FI = *I;
  unsigned FrameSize = FI->getFrameSize();
  size_t RootCount = FI->roots_size();

  for (GCFunctionInfo::roots_iterator RI = FI->roots_begin(),
                                      RE = FI->roots_end();
                                      RI != RE; ++RI) {
    int RootNum = RI->Num;
    int RootStackOffset = RI->StackOffset;
    Constant *RootMetadata = RI->Metadata;
  }
}

LLVM automatically computes a stack map. All a GCStrategy needs to do is access it using GCFunctionMetadata::roots_begin() and -end(). If the llvm.gcroot intrinsic is eliminated before code generation by a custom lowering pass, LLVM's stack map will be empty.

MyGC::MyGC() {
  InitRoots = true;
}

When set, LLVM will automatically initialize each root to null upon entry to the function. This prevents the GC's sweep phase from visiting uninitialized pointers, which will almost certainly cause it to crash. This initialization occurs before custom lowering, so the two may be used together.

Since LLVM does not yet compute liveness information, there is no means of distinguishing an uninitialized stack root from an initialized one. Therefore, this feature should be used by all GC plugins. It is enabled by default.

For GCs which use barriers or unusual treatment of stack roots, these flags allow the collector to perform arbitrary transformations of the LLVM IR:

class MyGC : public GCStrategy {
public:
  MyGC() {
    CustomRoots = true;
    CustomReadBarriers = true;
    CustomWriteBarriers = true;
  }
  
  virtual bool initializeCustomLowering(Module &M);
  virtual bool performCustomLowering(Function &F);
};

If any of these flags are set, then LLVM suppresses its default lowering for the corresponding intrinsics and instead calls performCustomLowering.

LLVM's default action for each intrinsic is as follows:

llvm.gcroot: Pass through to the code generator to generate a stack map.
llvm.gcread: Substitute a load instruction.
llvm.gcwrite: Substitute a store instruction.

If CustomReadBarriers or CustomWriteBarriers are specified, then performCustomLowering must eliminate the corresponding barriers.

performCustomLowering must comply with the same restrictions as FunctionPass::runOnFunction. Likewise, initializeCustomLowering has the same semantics as Pass::doInitialization(Module&).

The following can be used as a template:

#include "llvm/Module.h"
#include "llvm/IntrinsicInst.h"

bool MyGC::initializeCustomLowering(Module &M) {
  return false;
}

bool MyGC::performCustomLowering(Function &F) {
  bool MadeChange = false;
  
  for (Function::iterator BB = F.begin(), E = F.end(); BB != E; ++BB)
    for (BasicBlock::iterator II = BB->begin(), E = BB->end(); II != E; )
      if (IntrinsicInst *CI = dyn_cast<IntrinsicInst>(II++))
        if (Function *F = CI->getCalledFunction())
          switch (F->getIntrinsicID()) {
          case Intrinsic::gcwrite:
            // Handle llvm.gcwrite.
            CI->eraseFromParent();
            MadeChange = true;
            break;
          case Intrinsic::gcread:
            // Handle llvm.gcread.
            CI->eraseFromParent();
            MadeChange = true;
            break;
          case Intrinsic::gcroot:
            // Handle llvm.gcroot.
            CI->eraseFromParent();
            MadeChange = true;
            break;
          }
  
  return MadeChange;
}

LLVM can compute four kinds of safe points:

namespace GC {
  /// PointKind - The type of a collector-safe point.
  /// 
  enum PointKind {
    Loop,    //< Instr is a loop (backwards branch).
    Return,  //< Instr is a return instruction.
    PreCall, //< Instr is a call instruction.
    PostCall //< Instr is the return address of a call.
  };
}

A collector can request any combination of the four by setting the NeededSafePoints mask:

MyGC::MyGC() {
  NeededSafePoints = 1 << GC::Loop
                   | 1 << GC::Return
                   | 1 << GC::PreCall
                   | 1 << GC::PostCall;
}

It can then use the following routines to access safe points.

for (iterator I = begin(), E = end(); I != E; ++I) {
  GCFunctionInfo *MD = *I;
  size_t PointCount = MD->size();

  for (GCFunctionInfo::iterator PI = MD->begin(),
                                PE = MD->end(); PI != PE; ++PI) {
    GC::PointKind PointKind = PI->Kind;
    unsigned PointNum = PI->Num;
  }
}

Almost every collector requires PostCall safe points, since these correspond to the moments when the function is suspended during a call to a subroutine.

Threaded programs generally require Loop safe points to guarantee that the application will reach a safe point within a bounded amount of time, even if it is executing a long-running loop which contains no function calls.

Threaded collectors may also require Return and PreCall safe points to implement "stop the world" techniques using self-modifying code, where it is important that the program not exit the function without reaching a safe point (because only the topmost function has been patched).

LLVM allows a GC to print arbitrary assembly code before and after the rest of a module's assembly code. At the end of the module, the GC can print stack maps built by the code generator. (At the beginning, this information is not yet computed.)

Since AsmWriter and CodeGen are separate components of LLVM, a separate abstract base class and registry is provided for printing assembly code, the GCMetadaPrinter and GCMetadaPrinterRegistry. The AsmWriter will look for such a subclass if the GCStrategy sets UsesMetadata:

MyGC::MyGC() {
  UsesMetadata = true;
}

Note that LLVM does not currently have analogous APIs to support code generation in the JIT, nor using the object writers.

// lib/MyGC/MyGCPrinter.cpp - Example LLVM GC printer

#include "llvm/CodeGen/GCMetadataPrinter.h"
#include "llvm/Support/Compiler.h"

using namespace llvm;

namespace {
  class VISIBILITY_HIDDEN MyGCPrinter : public GCMetadataPrinter {
  public:
    virtual void beginAssembly(std::ostream &OS, AsmPrinter &AP,
                               const TargetAsmInfo &TAI);
  
    virtual void finishAssembly(std::ostream &OS, AsmPrinter &AP,
                                const TargetAsmInfo &TAI);
  };
  
  GCMetadataPrinterRegistry::Add<MyGCPrinter>
  X("mygc", "My bespoke garbage collector.");
}

The collector should use AsmPrinter and TargetAsmInfo to print portable assembly code to the std::ostream. The collector itself contains the stack map for the entire module, and may access the GCFunctionInfo using its own begin() and end() methods. Here's a realistic example:

#include "llvm/CodeGen/AsmPrinter.h"
#include "llvm/Function.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Target/TargetData.h"
#include "llvm/Target/TargetAsmInfo.h"

void MyGCPrinter::beginAssembly(std::ostream &OS, AsmPrinter &AP,
                                const TargetAsmInfo &TAI) {
  // Nothing to do.
}

void MyGCPrinter::finishAssembly(std::ostream &OS, AsmPrinter &AP,
                                 const TargetAsmInfo &TAI) {
  // Set up for emitting addresses.
  const char *AddressDirective;
  int AddressAlignLog;
  if (AP.TM.getTargetData()->getPointerSize() == sizeof(int32_t)) {
    AddressDirective = TAI.getData32bitsDirective();
    AddressAlignLog = 2;
  } else {
    AddressDirective = TAI.getData64bitsDirective();
    AddressAlignLog = 3;
  }
  
  // Put this in the data section.
  AP.SwitchToDataSection(TAI.getDataSection());
  
  // For each function...
  for (iterator FI = begin(), FE = end(); FI != FE; ++FI) {
    GCFunctionInfo &MD = **FI;
    
    // Emit this data structure:
    // 
    // struct {
    //   int32_t PointCount;
    //   struct {
    //     void *SafePointAddress;
    //     int32_t LiveCount;
    //     int32_t LiveOffsets[LiveCount];
    //   } Points[PointCount];
    // } __gcmap_<FUNCTIONNAME>;
    
    // Align to address width.
    AP.EmitAlignment(AddressAlignLog);
    
    // Emit the symbol by which the stack map can be found.
    std::string Symbol;
    Symbol += TAI.getGlobalPrefix();
    Symbol += "__gcmap_";
    Symbol += MD.getFunction().getName();
    if (const char *GlobalDirective = TAI.getGlobalDirective())
      OS << GlobalDirective << Symbol << "\n";
    OS << TAI.getGlobalPrefix() << Symbol << ":\n";
    
    // Emit PointCount.
    AP.EmitInt32(MD.size());
    AP.EOL("safe point count");
    
    // And each safe point...
    for (GCFunctionInfo::iterator PI = MD.begin(),
                                     PE = MD.end(); PI != PE; ++PI) {
      // Align to address width.
      AP.EmitAlignment(AddressAlignLog);
      
      // Emit the address of the safe point.
      OS << AddressDirective
         << TAI.getPrivateGlobalPrefix() << "label" << PI->Num;
      AP.EOL("safe point address");
      
      // Emit the stack frame size.
      AP.EmitInt32(MD.getFrameSize());
      AP.EOL("stack frame size");
      
      // Emit the number of live roots in the function.
      AP.EmitInt32(MD.live_size(PI));
      AP.EOL("live root count");
      
      // And for each live root...
      for (GCFunctionInfo::live_iterator LI = MD.live_begin(PI),
                                            LE = MD.live_end(PI);
                                            LI != LE; ++LI) {
        // Print its offset within the stack frame.
        AP.EmitInt32(LI->StackOffset);
        AP.EOL("stack offset");
      }
    }
  }
}

Implementing a garbage collector for LLVM is fairly straightforward. The LLVM garbage collectors are provided in a form that makes them easy to link into the language-specific runtime that a language front-end would use. They require functionality from the language-specific runtime to get information about where pointers are located in heap objects.

The implementation must include the llvm_gc_allocate and llvm_gc_collect functions. To do this, it will probably have to trace through the roots from the stack and understand the GC descriptors for heap objects. Luckily, there are some example implementations available.

The three most common ways to keep track of where pointers live in heap objects are (listed in order of space overhead required):

In languages with polymorphic objects, pointers from an object header are usually used to identify the GC pointers in the heap object. This is common for object-oriented languages like Self, Smalltalk, Java, or C#.
If heap objects are not polymorphic, often the "shape" of the heap can be determined from the roots of the heap or from some other meta-data [Appel89, Goldberg91, Tolmach94]. In this case, the garbage collector can propagate the information around from meta data stored with the roots. This often eliminates the need to have a header on objects in the heap. This is common in the ML family.
If all heap objects have pointers in the same locations, or pointers can be distinguished just by looking at them (e.g., the low order bit is clear), no book-keeping is needed at all. This is common for Lisp-like languages.

The LLVM garbage collectors are capable of supporting all of these styles of language, including ones that mix various implementations. To do this, it allows the source-language to associate meta-data with the stack roots, and the heap tracing routines can propagate the information. In addition, LLVM allows the front-end to extract GC information in any form from a specific object pointer (this supports situations #1 and #3).