XRay Flight Data Recorder Trace Format

Version:1 as of 2017-07-20

Introduction

When gathering XRay traces in Flight Data Recorder mode, each thread of an application will claim buffers to fill with trace data, which at some point is finalized and flushed.

A goal of the profiler is to minimize overhead, the flushed data directly corresponds to the buffer.

This document describes the format of a trace file.

General

Each trace file corresponds to a sequence of events in a particular thread.

The file has a header followed by a sequence of discriminated record types.

The endianness of byte fields matches the endianess of the platform which produced the trace file.

Header Section

A trace file begins with a 32 byte header.

Field	Size (bytes)	Description
version	2	Anticipates versioned readers. This document describes the format when version == 1
type	2	An enumeration encoding the type of trace. Flight Data Recorder mode traces have type == 1
bitfield	4	Holds parameters that are not aligned to bytes. Further described below.
cycle_frequency	8	The frequency in hertz of the CPU oscillator used to measure duration of events in ticks.
buffer_size	8	The size in bytes of the data portion of the trace following the header.
reserved	8	Reserved for future use.

The bitfield parameter of the file header is composed of the following fields.

Field	Size (bits)	Description
constant_tsc	1	Whether the platform’s timestamp counter used to record ticks between events ticks at a constant frequency despite CPU frequency changes. 0 == non-constant. 1 == constant.
nonstop_tsc	1	Whether the tsc continues to count despite whether the CPU is in a low power state. 0 == stop. 1 == non-stop.
reserved	30	Not meaningful.

Data Section

Following the header in a trace is a data section with size matching the buffer_size field in the header.

The data section is a stream of elements of different types.

There are a few categories of data in the sequence.

• Function Records: Function Records contain the timing of entry into and exit from function execution. Function Records have 8 bytes each.
• Metadata Records: Metadata records serve many purposes. Mostly, they capture information that may be too costly to record for each function, but that is required to contextualize the fine-grained timings. They also are used as markers for user-defined Event Data payloads. Metadata records have 16 bytes each.
• Event Data: Free form data may be associated with events that are traced by the binary and encode data defined by a handler function. Event data is always preceded with a marker record which indicates how large it is.
• Function Arguments: The arguments to some functions are included in the trace. These are either pointer addresses or primitives that are read and logged independently of their types in a high level language. To the tracer, they are all numbers. Function Records that have attached arguments will indicate their presence on the function entry record. We only support logging contiguous function argument sequences starting with argument zero, which will be the “this” pointer for member function invocations. For example, we don’t support logging the first and third argument.

A reader of the memory format must maintain a state machine. The format makes no attempt to pad for alignment, and it is not seekable.

Function Records

Function Records have an 8 byte layout. This layout encodes information to reconstruct a call stack of instrumented function and their durations.

Field	Size (bits)	Description
discriminant	1	Indicates whether a reader should read a Function or Metadata record. Set to 0 for Function records.
action	3	Specifies whether the function is being entered, exited, or is a non-standard entry or exit produced by optimizations.
function_id	28	A numeric ID for the function. Resolved to a name via the xray instrumentation map. The instrumentation map is built by xray at compile time into an object file and pairs the function ids to addresses. It is used for patching and as a lookup into the binary’s symbols to obtain names.
tsc_delta	32	The number of ticks of the timestamp counter since a previous record recorded a delta or other TSC resetting event.

On little-endian machines, the bitfields are ordered from least significant bit bit to most significant bit. A reader can read an 8 bit value and apply the mask 0x01 for the discriminant. Similarly, they can read 32 bits and unsigned shift right by 0x04 to obtain the function_id field.

On big-endian machine, the bitfields are written in order from most significant bit to least significant bit. A reader would read an 8 bit value and unsigned shift right by 7 bits for the discriminant. The function_id field could be obtained by reading a 32 bit value and applying the mask 0x0FFFFFFF.

Function action types are as follows.

Type	Number	Description
Entry	0	Typical function entry.
Exit	1	Typical function exit.
Tail_Exit	2	An exit from a function due to tail call optimization.
Entry_Args	3	A function entry that records arguments.

Entry_Args records do not contain the arguments themselves. Instead, metadata records for each of the logged args follow the function record in the stream.

Metadata Records

Interspersed throughout the buffer are 16 byte Metadata records. For typically instrumented binaries, they will be sparser than Function records, and they provide a fuller picture of the binary execution state.

Metadata record layout is partially record dependent, but they share a common structure.

The same bit field rules described for function records apply to the first byte of MetadataRecords. Within this byte, little endian machines use lsb to msb ordering and big endian machines use msb to lsb ordering.

Field	Size	Description
discriminant	1 bit	Indicates whether a reader should read a Function or Metadata record. Set to 1 for Metadata records.
record_kind	7 bits	The type of Metadata record.
data	15 bytes	A data field used differently for each record type.

Here is a table of the enumerated record kinds.

Number	Type
0	NewBuffer
1	EndOfBuffer
2	NewCPUId
3	TSCWrap
4	WallTimeMarker
5	CustomEventMarker
6	CallArgument

NewBuffer Records

Each buffer begins with a NewBuffer record immediately after the header. It records the thread ID of the thread that the trace belongs to.

Its data segment is as follows.

Field	Size (bytes)	Description
thread_Id	2	Thread ID for buffer.
reserved	13	Unused.

WallClockTime Records

Following the NewBuffer record, each buffer records an absolute time as a frame of reference for the durations recorded by timestamp counter deltas.

Its data segment is as follows.

Field	Size (bytes)	Description
seconds	8	Seconds on absolute timescale. The starting point is unspecified and depends on the implementation and platform configured by the tracer.
microseconds	4	The microsecond component of the time.
reserved	3	Unused.

NewCpuId Records

Each function entry invokes a routine to determine what CPU is executing. Typically, this is done with readtscp, which reads the timestamp counter at the same time.

If the tracing detects that the execution has switched CPUs or if this is the first instrumented entry point, the tracer will output a NewCpuId record.

Its data segment is as follows.

Field	Size (bytes)	Description
cpu_id	2	CPU Id.
absolute_tsc	8	The absolute value of the timestamp counter.
reserved	5	Unused.

TSCWrap Records

Since each function record uses a 32 bit value to represent the number of ticks of the timestamp counter since the last reference, it is possible for this value to overflow, particularly for sparsely instrumented binaries.

When this delta would not fit into a 32 bit representation, a reference absolute timestamp counter record is written in the form of a TSCWrap record.

Its data segment is as follows.

Field	Size (bytes)	Description
absolute_tsc	8	Timestamp counter value.
reserved	7	Unused.

CallArgument Records

Immediately following an Entry_Args type function record, there may be one or more CallArgument records that contain the traced function’s parameter values.

The order of the CallArgument Record sequency corresponds one to one with the order of the function parameters.

CallArgument data segment:

Field	Size (bytes)	Description
argument	8	Numeric argument (may be pointer address).
reserved	7	Unused.

CustomEventMarker Records

XRay provides the feature of logging custom events. This may be leveraged to record tracing info for RPCs or similarly trace data that is application specific.

Custom Events themselves are an unstructured (application defined) segment of memory with arbitrary size within the buffer. They are preceded by CustomEventMarkers to indicate their presence and size.

CustomEventMarker data segment:

Field	Size (bytes)	Description
event_size	4	Size of preceded event.
absolute_tsc	8	A timestamp counter of the event.
reserved	3	Unused.

EndOfBuffer Records

An EndOfBuffer record type indicates that there is no more trace data in this buffer. The reader is expected to seek past the remaining buffer_size expressed before the start of buffer and look for either another header or EOF.

Format Grammar and Invariants

Not all sequences of Metadata records and Function records are valid data. A sequence should be parsed as a state machine. The expectations for a valid format can be expressed as a context free grammar.

This is an attempt to explain the format with statements in EBNF format.

• Format := Header ThreadBuffer* EOF
• ThreadBuffer := NewBuffer WallClockTime NewCPUId BodySequence* End
• BodySequence := NewCPUId | TSCWrap | Function | CustomEvent
• Function := (Function_Entry_Args CallArgument*) | Function_Other_Type
• CustomEvent := CustomEventMarker CustomEventUnstructuredMemory
• End := EndOfBuffer RemainingBufferSizeToSkip

Function Record Order

There are a few clarifications that may help understand what is expected of Function records.

• Functions with an Exit are expected to have a corresponding Entry or Entry_Args function record precede them in the trace.
• Tail_Exit Function records record the Function ID of the function whose return address the program counter will take. In other words, the final function that would be popped off of the call stack if tail call optimization was not used.
• Not all functions marked for instrumentation are necessarily in the trace. The tracer uses heuristics to preserve the trace for non-trivial functions.
• Not every entry must have a traced Exit or Tail Exit. The buffer may run out of space or the program may request for the tracer to finalize toreturn the buffer before an instrumented function exits.