LOCK_PROFILING
—
kernel lock profiling support
The LOCK_PROFILING
kernel option adds support for
measuring and reporting lock use and contention statistics. These statistics
are collated by “acquisition point”. Acquisition points are
distinct places in the kernel source code (identified by source file name and
line number) where a lock is acquired.
For each acquisition point, the following statistics are
accumulated:
- The longest time the lock was ever continuously held after being acquired
at this point.
- The total time the lock was held after being acquired at this point.
- The total time that threads have spent waiting to acquire the lock.
- The total number of non-recursive acquisitions.
- The total number of times the lock was already held by another thread when
this point was reached, requiring a spin or a sleep.
- The total number of times another thread tried to acquire the lock while
it was held after having been acquired at this point.
In addition, the average hold time and average wait time are
derived from the total hold time and total wait time respectively and the
number of acquisitions.
The LOCK_PROFILING
kernel option also adds
the following
sysctl(8)
variables to control and monitor the profiling code:
- debug.lock.prof.enable
- Enable or disable the lock profiling code. This defaults to 0 (off).
- debug.lock.prof.reset
- Reset the current lock profiling buffers.
- debug.lock.prof.stats
- The actual profiling statistics in plain text. The columns are as follows,
from left to right:
- max
- The longest continuous hold time in microseconds.
- wait_max
- The longest continuous wait time in microseconds.
- total
- The total (accumulated) hold time in microseconds.
- wait_total
- The total (accumulated) wait time in microseconds.
- count
- The total number of acquisitions.
- avg
- The average hold time in microseconds, derived from the total hold
time and the number of acquisitions.
- wait_avg
- The average wait time in microseconds, derived from the total wait
time and the number of acquisitions.
- cnt_hold
- The number of times the lock was held and another thread attempted to
acquire the lock.
- cnt_lock
- The number of times the lock was already held when this point was
reached.
- name
- The name of the acquisition point, derived from the source file name
and line number, followed by the name of the lock in parentheses.
- debug.lock.prof.rejected
- The number of acquisition points that were ignored after the table filled
up.
- debug.lock.prof.skipspin
- Disable or enable the lock profiling code for the spin locks. This
defaults to 0 (do profiling for the spin locks).
- debug.lock.prof.skipcount
- Do sampling approximately every N lock acquisitions.
Mutex profiling support appeared in FreeBSD 5.0.
Generalized lock profiling support appeared in FreeBSD
7.0.
The LOCK_PROFILING
option increases the size of
struct lock_object, so a kernel built with that option
will not work with modules built without it.
The LOCK_PROFILING
option also prevents
inlining of the mutex code, which can result in a fairly severe performance
penalty. This is, however, not always the case.
LOCK_PROFILING
can introduce a substantial
performance overhead that is easily monitorable using other profiling tools,
so combining profiling tools with LOCK_PROFILING
is
not recommended.
Measurements are made and stored in nanoseconds using
nanotime(9),
(on architectures without a synchronized TSC) but are presented in
microseconds. This should still be sufficient for the locks one would be
most interested in profiling (those that are held long and/or acquired
often).
LOCK_PROFILING
should generally not be
used in combination with other debugging options, as the results may be
strongly affected by interactions between the features. In particular,
LOCK_PROFILING
will report higher than normal
uma(9)
lock contention when run with INVARIANTS
due to
extra locking that occurs when INVARIANTS
is
present; likewise, using it in combination with
WITNESS
will lead to much higher lock hold times and
contention in profiling output.