vti(5)vti(5)NAMEvti - Visual Threads Instrumentation tool; detects potential race con‐
ditions and inconsistent lock ordering
SYNOPSIS
atom appl_prog -toolvti [-toolargs="arg1 arg2 ..."] [atom_flags...]
PARAMETERS
File name of a fully linked non-static executable to be examined. This
program should be compiled with the -g, -g2, or -g3 flag to obtain more
complete debugging information. If the default symbol table level
(-g0) is used, line number information, static procedure names, and
file names are unavailable. If -g1 is used, the variable name symbolic
information is not available.
Other restrictions: the program cannot be linked with the non_shared
option, the program cannot be compiled with the -pg option (profiled),
and the program cannot be stripped of symbols (strip).
FLAGS
Identifies the Visual Threads Instrumentation (VTI) tool to the atom
command. Passes arguments to VTI's instrumentation routines. Use
whitespace characters to separate arguments from their parameters (if
any) and from other arguments. Specifies flags to the atom command.
See the atom(1) reference page for descriptions of other flags accepted
by the atom command, such as those that enable instrumentation of
shared libraries, specify the names of instrumented objects, and
request debugging information.
Note that the -excobj Atom flag behaves as described except that cer‐
tain libraries (such as libc.so, libpthreads.so, and libpthread.so) are
partially instrumented, as needed by VTI, even if they are specified in
the -excobj flag.
Consequently, after you have instrumented an application that uses
shared libraries, you must set the LD_LIBRARY_PATH environment variable
to point to the directory containing the instrumented shared libraries.
Typically, this would be the current directory or the directory speci‐
fied by the -shlibdir flag. (You may leave LD_LIBRARY_PATH pointing to
this directory while running other, uninstrumented applications.)
The VTI tool allows the following flags to be passed in the -toolargs
flag for VTI's instrumentation routine to use when instrumenting
appl_prog. Specifies an additional filename to be used for customiza‐
tion directives that control VTI instrumentation. By default, VTI
checks for a customization file in the current directory or in the home
directory. See "VTI Customization File". Multiple -customize options
can be specified. VTI will not check for inconsistent lock ordering;
this might be useful if you only want to see reports on unguarded data
race conditions. VTI will not check for unguarded data race condi‐
tions; useful if you want to see only reports about inconsistent lock
ordering.
DESCRIPTION
VTI is an Atom tool that instruments your program to include analysis
checks for "unguarded" data which can result in race conditions, and
checks for deadlocks of multithreaded C and C++ programs at run-time.
The algorithm for detecting "unguarded" data is as follows. For each
32-bit word that is not part of a thread stack, the tool allocates an
additional 32-bit word ('shadow memory'), which is used to keep track
of a candidate set of locks that might conceivably protect the memory
location. When a thread accesses the memory location, the location's
candidate set is intersected with the set of locks held by the thread.
If the candidate set becomes empty, the tool deduces that there is no
lock that protects the location, and reports an error.
This basic algorithm is augmented in a few ways to deal with memory
allocation, initialization, read-only data, read-write locks, error
suppression, and hardware atomic instructions, as follows: When memory
is newly allocated (for example by malloc()), the tool marks it as
"fresh"---not yet accessed by any thread. When a location is first
accessed, the tool does not keep track of the candidate set, but
instead remembers that the accessing thread "owns" the location, and
may access it without holding any locks. Maintenance of the candidate
set begins once the location is acessed by a second thread, at which
point the first thread loses ownership of the location. The tool keeps
track of whether a location has been written since it started maintain‐
ing the candidate set for the location. It will not report an error on
a location that has not been written. To accommodate read-write locks,
the tool keeps track of what locks are held in write (exclusive) mode,
and what locks are held in any mode. On read accesses, the intersec‐
tion of the candidate set is performed with the set of locks held in
any mode. On write accesses, the set of locks held in write (exclu‐
sive) mode is used. Once an error has been reported at a given loca‐
tion, further accesses of that location are ignored by the tool, until
the memory is reallocated (becomes fresh). The tool also keeps track
of all accesses to data via hardware atomic instructions (load-
locked/store_conditional instructions). If a subsequent access is made
to the same location without using a protected instruction, an error is
reported. This can be useful in detecting alignment-related race con‐
ditions. For example, consider a quadword structure which consists of
multiple fields, each accessed independently by multiple threads.
Depending on compiler alignment and exact usage by the program,
accesses to individual parts of the quadword may in fact be accesses to
the entire quadword; this can result in one thread inadvertently writ‐
ing the entire quadword instead of the intended field access.
More details of the algorithm can be found in the paper cited in
"Related Information" and in the sections of this description titled
"VTI False Alarms" and "VTI Test Coverage".
VTI also checks for potential deadlock conditions by checking for any
inconsistent use of lock ordering. For example, if mutex xm is first
locked before mutex ym , then any subsequent reversal of this lock
order will be reported. Inconsistent lock ordering is a common cause
of program deadlock.
You can use VTI for the following types of applications: For C or C++
applications using the Posix Threads Library that allocate memory by
using the malloc, calloc, realloc, valloc, alloca, sbrk, and new (C++
only) functions. (Applications that use either the -pthread or -threads
compilation option.) The application cannot use cancelability type of
PTHREAD_CANCEL_ASYNCHRONOUS. If your application uses a custom locking
package or memory allocator, you may still be able to use VTI by using
customization directives, or by using source code annotation direc‐
tives.
You can access VTI in two general ways: Invoking an atom command that
specifies the VTI atom tool and creates an instrumented version of your
application. Then, you set the environment variable LD_LIBRARY_PATH to
point to any instrumented libraries, and invoke the instrumented image
with any parameters required by the program. The VTI tool reports race
conditions and inconsistent lock order violations in a log file with
the filename appl_prog.vtilog. The graphical user interface of Visual
Threads (dxthreads) automatically invokes VTI when you choose Run from
the File menu, and you have enabled checking for certain rules
(UnguardedData or UnsafeFunction/UnsafeLibrary rules) Visual Threads
establishes the necessary environment for VTI, and after the instrumen‐
tation is complete, it invokes the instrumented image for you. Viola‐
tions are reported as popup Alarm boxes; additional information is pro‐
vided in the log file appl_prog.vtilog as well.
The next sections describe the customization and annotation mechanisms
of VTI and the optional use of VTI environmental variables, which are
not necessary for the majority of users or applications. See the
"Examples" section for a straightforward commandline usage example.
VTI Customization File
VTI can take as input one or more user-specified customization file(s).
A customization file is used during instrumentation of your application
and allows you to refine and control the reporting of violations, and
to describe your own synchronization mechanisms and memory allocation
routines. A VTI customization file is generally not necessary if you
are using a Posix Threads Library interface (-pthread or -threads com‐
pilation option), and standard memory allocation routines from within a
C or C++ program.
VTI reads a system-wide .vti file for directives, and then looks for a
user-specific If the file is not in the current directory, VTI looks
for one in the home directory. If you are using VTI automatically from
Visual Threads and wish to use a customization file, you must ensure
that the customization file is created before invoking Visual Threads;
it should be in the working directory you specify to Visual Threads or
in the home directory.
The following general syntax rules apply to the .vti file: VTI treats
lines beginning with a pound-sign character (#) as comments. Procedure
names and file names may be specified within double-quotes. Procedure
names should match those printed by the nm command. In particular, C++
names must include the list of argument types, for example, han‐
dler(int). Inlined procedures must be compiled with the cxx -noinline
command (or similar mechanism) in order to be independantly instru‐
mented or referenced. Some .vti file directives include source loca‐
tion specifications with the following syntax:
[[library^]source_file^]procedure [line num]
Each of the components (library,source_file,procedure) may contain
shell-style pattern matching characters: * ? []
The library and source_file will be full matches if they contain
slashes, otherwise they will match only on the specified components of
the pathnames.
For example, the following directive disables instrumentation for race
conditions of all routines named foobar in the application: ignore foo‐
bar
The following directive disables instrumentation of all routines named
str in the library libc.so. ignore libc.so^*^str*
The following directive disables instrumentation at line 17 in the pro‐
cedure bar in the source file ../wombat.c ignore ../wombat.c^barline 17
Some .vti file directives include register specifications with the fol‐
lowing syntax: the procedure's (non-floating point) return value one of
the procedure's first six integer arguments an arbitrary integer regis‐
ter
Registers other than r0 are sampled at the start of the procedure. r0
(retval) is sampled at the end of the procedure. See below for exam‐
ples of the register specification.
Use the following memory-related options in the .vti file: Declare that
the specified procedure causes some memory to be freed or allocated.
The memory is set to the "just allocated" state, and locking informa‐
tion previously gathered for it is forgotten. Location must specify a
procedure; the optional line number specification is not allowed for
this directive. Register must be an integer register. Len is the
length in bytes of the memory.
For example, the following directive declares that procedure foo
returns a pointer to a 32-byte region that is freshly allocated: fresh‐
mem retval 32 foo
The following declares that procedure bar_free in library libbar.so
takes a pointer to a 64-byte block of memory that is freshly allocated:
freshmem a0 64 libbar.so^*^bar_free
This directive is equivalent to the source annotation vti_set (regis‐
ter, VTI_FRESH, 0, len) at the appropriate point in the source. Spec‐
ify file as another .vti file. If the filename starts with a slash
("/"), it is treated as a full pathname. Otherwise, it is interpreted
as a pathname relative to the directory containing the file that
includes it. Don't instrument for race conditions at the given loca‐
tion. Turn off error reporting for each thread as it enters the speci‐
fied procedure(s) and turn it on again when the thread exits the proce‐
dure. While the thread has error reporting turned off, no unguarded
data or inconsistent lock errors will be reported. In addition, no
lock orderings will be remembered. This is equivalent to the use of
the vti_ignore(1) source annotation at the start of the procedure and
vti_ignore(0) at the end.
Location must specify a procedure; the optional line number specifica‐
tion is not allowed for this directive. Results may be confusing if one
procedure specified in the ignoreall directive calls another procedure
for which ignoreall has been specified, because error reporting will be
turned on again by the exit from the inner procedure. Declare that
memory returned from, or passed to a given procedure is to be ignored
for the purposes of unguarded data checks. Violations will not be
reported on the memory unless its state is modified by a freshmem
directive, or a vti_set source annotation.
Location must specify a procedure; the optional line number specifica‐
tion is not allowed for this directive. The location is instrumented,
but any detected race conditions for that location will be suppressed
and will not appear in the log file.
Use the following lock-related options in the .vti file: Indicates that
procedure is a procedure that destroys the mutex or read-write lock
that is its first argument. procedure must be a simple procedure name.
This is equivalent to putting the source annotation
vti_destroy_lock(lock) at the beginning of the procedure. Indicates
that procedure is a procedure that acquires the mutex or read-write
lock that is its first argument. procedure must be a simple procedure
name. This is equivalent to putting the source annotation
vti_write_lock(lock) at the beginning of the procedure. Indicates that
procedure is a procedure that acquires the read lock that is its first
argument. procedure must be a simple procedure name. This is equiva‐
lent to putting the source annotation vti_read_lock(lock) at the begin‐
ning of the procedure. Indicates that procedure is a procedure that
releases the read lock that is its first argument. procedure must be a
simple procedure name. This is equivalent to putting the source anno‐
tation vti_read_unlock(lock) at the beginning of the procedure. Indi‐
cates that procedure is a procedure that attempts to acquire the mutex
or read-write lock that is its first argument, but without blocking.
It returns 0 on success, and another value on failure. procedure must
be a simple procedure name. This is equivalent to calling the source
annotation vti_write_lock(lock) in the procedure when the call suc‐
ceeds. Indicates that procedure is a procedure that attempts to
acquire the read lock that is its first argument, but without blocking.
It returns 0 on success, and another value on failure. procedure must
be a simple procedure name. This is equivalent to calling the source
annotation vti_read_lock(lock) in the procedure when the call succeeds.
Indicates that procedure is a procedure that releases the mutex or
read-write lock that is its first argument. procedure must be a simple
procedure name. This is equivalent to putting the source annotation
vti_write_unlock(lock) at the beginning of the procedure.
VTI Source Code Annotations
In some cases, it may be easier to directly annotate your source code
to teach VTI about your own synchronization mechanisms and/or memory
allocation routines. Annotations provide functionality similar to the
VTI customization file directives for memory and locking routines.
Some examples of source code annotations include: Tell VTI to ignore
errors on variable locations. Tell VTI that variable locations is
freshly allocated. Tell VTI that a write lock or mutex at address
mutex has been acquired. Tell VTI that a write lock or mutex at
address mutex has been released.
To use source code annotations, reference the header file
/usr/include/vti.user.h in your modules, and link your application with
the archive library /lib/libvti.a as in this example:
cc -g -o t t.c -pthread -lvti
The file /usr/include/vti.user.h has complete details about source code
annotations.
VTI Environment Variables
VTI also provides environment variables that control and provide more
information about VTI at runtime. The customization file(s) and source
directives can be used to control VTI behavior at instrumentation time,
while the environment variables are used when the instrumented applica‐
tion is run.
If you are using VTI automatically from Visual Threads, you must define
any VTI environment variables you want to use before invoking Visual
Threads.
The environment variable VTI_ARGS can have the following values: When
used in conjunction with the environment variable VTI_TRACEMUTEX,
report all acquisitions and release of traced locks, not just the first
acquisitions which define the lock order. Log all VTI messages to file
instead of the default appl_prog.vtilog. "-" means stdout and "--"
means stderr. Don't shadow the data segments of shared libraries.
Don't suppress duplicates of unguarded data violations by PC. Nor‐
mally, if there are multiple unguarded data violations for a given pro‐
gram PC, VTI suppresses those duplicate reports. Don't suppress dupli‐
cates of unguarded data violations, by data word. Normally, each data
word in the program can produce only one unguarded data violation
report (unless the data is freed and reallocated). The shadow heap is
normally allocated starting at address 0x28000000000. For most pro‐
grams this is a good choice; however, if your application uses memory
in a less-typical manner which happens to conflict with this region,
you may see an error message from VTI:
drd_shadow_allocate: no space.
In this case, you should specify a shadow heap start address that does
not conflict with your application's use of memory. Shadow writable
user mmaped regions, in order to do race detection on such regions.
This is not done by default. Log all regions shadowed, and the loading
of instrumented libraries.
Use the following environment variables for more information about
locking: Include a whitespace-separated list of hex addresses to trace
VTI state transitions. This is useful to get more information about an
unguarded data race condition at a particular address. For example:
setenv VTI_TRACE 0x140001bec Include a whitespace-separated list of
decimal mutex identifiers. This is used to get more information about
a deadlock involving multiple mutexes. For example:
setenv VTI_TRACEMUTEX "11 13"
You would typically determine the list of mutex identifiers by first
running the VTI tool and noting the inconsistent order violation and
its associated mutexes. Then you would define VTI_TRACE_MUTEX based on
those associated mutex identifiers, and rerun the VTI tool.
VTI Log File
VTI places its output in a log file that includes the a description of
each unguarded data and inconsistent lock usage violation. If the
application does a fork or vfork operation, VTI creates another log
file for analysis of the child process, including the child pid as part
of the log file name.
Note that if the application does a vfork, VTI replaces this operation
with a fork in the instrumented program. This is necessary because VTI
requires initialization of system and pthread libraries which does not
occur on a vfork operation.
VTI False Alarms
When testing for unguarded data, the VTI tool assumes that the program‐
mer has followed a particular programming discipline: all shared
writable variables are protected by mutexes or read-write locks. The
tool allows for unlocked initialization of such variables as a special
case.
In general, the tool does not understand synchronization mechanisms
other than locks and hardware atomic instructions. As a special case,
the tool generates no error messages when exclusive access to a vari‐
able is ensured only by the synchronization that is implicit in thread
creation/join operations. The tool does not understand more general
uses of thread creation/join synchronization, nor other synchronization
mechanisms, such as semaphores.
In addition, the tool does not automatically notice if the program
starts to protect a memory location with a different lock from one it
used previously, unless the memory location has passed through a known
memory allocator. For example, if a program maintains its own free
lists of memory regions, it may be able to reuse memory for program
data locations, but the tool is not aware that a particular location
has been reallocated for a new purpose. Therefore, the tool is likely
to give spurious error reports (false alarms) when applied to a program
that uses alternative synchronization mechanisms or application-spe‐
cific memory allocation routines.
It is usually possible to suppress these false alarms with a small num‐
ber of source code annotations or directives in the .vti file. Typi‐
cally, one would declare memory as "fresh" when passing it from one
thread to another, or when it becomes protected by a new lock, such as
when it is reallocated after spending time on a free list. This can be
done by using the vti_set annotation with the VTI_FRESH option, or by
using the freshmem directive. In some cases, it is best to have the
tool "ignore" such memory, by using the VTI_IGNORE option to a vti_set
source annotation, or by using the ignoremem directive.
VTI Test Coverage
The algorithms used by the VTI tool can detect potential race condi‐
tions and deadlocks that did not actually occur when the instrumented
program was run. Moreover, the algorithms are largely independent of
timing and scheduling variations: if the same threads execute the same
statements, the same locations will be reported as unguarded and the
same lock orderings will be reported as inconsistent, regardless of the
order in which the threads ran.
However, the timing of the program can alter when the errors are
reported, and where in the program they are first observed. As one
would expect, VTI may report different sets of errors on different runs
if those runs execute different parts of the code, either because the
computation itself is timing dependent, or because different input data
were used.
VTI cannot find errors in code that was not executed, so it is impor‐
tant to exercise all the parts of the instrumented program that you
wish to check. In addition, to find unguarded data, the memory must be
accessed by at least two threads or VTI will not know that the memory
is shared.
VTI cannot detect errors between threads of different address spaces
(for example, two processes using shared memory).
There is one circumstance in which VTI may miss unguarded data as a
result of variations in scheduling. This occurs when a thread allo‐
cates some memory, initializes it, makes it available to other threads,
but then accesses the memory again before any other thread has accessed
it, and without holding an appropriate lock. If this happens, the tool
will erroneously assume that the last access is part of the initializa‐
tion of the memory, and will not report an error. It is possible for
the user to eliminate this error by explicitly marking the end of the
initialization code with a source annotation that declares the memory
as being protected by the appropriate lock. For example:
vti_set (&variable, 0, &lock, sizeof (variable));
See /usr/include/vti.user.h for more details about source annotations.
EXAMPLES
Invokes vti to instrument program, and produces the output executable
program.vti. Make sure that the loader can find any instrumented
shared libraries. Invokes the instrumented program, creating the log
file program.vtilog. Note that if program required any input parame‐
ters, the same input parameters should be specified for program.vti.
Typical output from the program.vtilog file might be something like:
data race by thread 3 on stl t0, 0(v0) accessing 0x140000300 at: rand
test1.c:109 (test1:0x120001ccc test1.vti:0x12003f394) thread_fun
test1.c:121 (test1:0x120001d40 test1.vti:0x12003f4e8) thdBase
?file?:? (libpthread.so:0x3ff805812e4
libpthread.so.test1.vti:0x3ff808420c4)
location was previously accessed readonly by multiple threads thread 3
holds locks: <none>
This log file output indicates that there was an unguarded data viola‐
tion (race condition) at source line number 109 in the function rand,
source file test1.c. (The function rand was called by function
thread_fun, which was created by the thdBase function in the pthreads
library.) The program location that was "unguarded" was at address
0x140000300. This location was previously accessed by multiple
threads, but the current thread doesn't hold any locks and is modifying
the program location.
FILES
Instrumented version of appl_prog. Log of race conditions, deadlocks,
and other reports. The name of this log file can be overriden by the
settings of the VTI_ARGS environment variable. Customization file for
VTI. Header file for compilation of an application which includes VTI
source annotations. Archive library to link with an application which
includes VTI source annotations.
RELATED INFORMATIONatom(1)
Programmer's Guide
"Eraser: A dynamic race detector for multithreaded programs", by S.Sav‐
age, M.Burrows, C.G.Nelson, P.G.Sobalvarro, T.E.Anderson in ACM Trans‐
actions of Computer Systems, Vol 15, No. 4, Nov 1997, pp. 391-412.
dxthreads(1)pthread(3)vti(5)
