Profiling with OProfile
More Information
Installation
Install using yum:
sudo yum -y install oprofile
Start the profiling daemon
Wait until you are ready to run your target program before you do this.
I think you need to run these commands as root.
"--reset" cleans out old data.
"--no-vmlinux" expresses that you are too lazy to find a kernel in the right format (w/ symbols, uncompressed)
sudo opcontrol --reset
sudo opcontrol --no-vmlinux
sudo opcontrol --start
The following command was used by someone else to get kernel symbols:
sudo opcontrol --start \
--vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux \
--separate=kernel
kernel-debuginfo might be a package you need.
Run your target program
Longer and harder is better.
Dump the profile data
You don't have to stop your program to dump the data.
You don't even have to stop profiling, but it will keep the results static for this HOWTO.
I think these commands need to be run as root too.
sudo opcontrol --dump
sudo opcontrol --shutdown
Generate a basic system-wide report
We'll add the "-f" flag so we get long filenames in the report.
sudo opreport -f
We get output that looks like this:
CPU: PIII, speed 800 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
samples| %|
------------------
861982 29.6616 /lib/libc-2.4.so
716161 24.6437 /usr/lib/libstdc++.so.6.0.8
504902 17.3741 /usr/lib/libbz2.so.1.0.3
461460 15.8792 /home/ksedgwic/skyler/tam.trunk/modules/MDLoader/Linux.DBGOBJ/RTQE-MDLoader.so
101235 3.4836 /no-vmlinux
99956 3.4396 /home/ksedgwic/skyler/tam.trunk/modules/MemDataWindow/Linux.DBGOBJ/RTQE-MDW.so
73765 2.5383 /usr/lib/libboost_filesystem.so.1.33.1
15907 0.5474 /home/ksedgwic/skyler/tam.trunk/channel/src/Linux.DBGOBJ/librtqe-channel.so
11904 0.4096 /usr/bin/oprofiled
Detailed listing
Run the opreport again, we add: * '-l' to list all symbols * '--demangle=smart' to demangle C++ symbols * '--debug-info' to list source code locations
sudo opreport -l --demangle=smart --debug-info
Detailed listing for a specific application or object
Run the opreport again, we add: * '-l' to list all symbols * '--demangle=smart' to demangle C++ symbols * '--debug-info' to list source code locations
We also add the full path of the object of interest
# Kernel
sudo opreport -l /usr/lib/debug/lib/modules/`uname -r`/vmlinux
# User module
sudo opreport -l --demangle=smart --debug-info \
/home/ksedgwic/skyler/tam.trunk/modules/MDLoader/Linux.DBGOBJ/RTQE-MDLoader.so
We get the following output:
samples % symbol name
32489 7.0405 MDL::FileLoader::readNext()
30441 6.5967 MDL::Field::Field(MDL::Field const&)
29384 6.3676 MDL::FileLoader::getMsgFromFile(string const&)
24771 5.3680 MDL::Field::~Field()
19645 4.2571 __i686.get_pc_thunk.bx
11751 2.5465 _Rb_tree<string, pair<string const, MDL::Field>, string const>
::insert_equal(pair<string const, MDL::Field> const&)
9908 2.1471 MDL::Allocator<double>::~Allocator()
9600 2.0804 vector<MDL::Field>::_M_insert_aux(vector<MDL::Field>::iterator
, MDL::Field const&)
9147 1.9822 MDL::MessageImpl::addField(MDL::Field const&)
9057 1.9627 MDL::Allocator<string>::~Allocator()
8985 1.9471 .plt
7595 1.6459 MDL::Allocator<double>::copy() const
7448 1.6140 MDL::IAlloc::~IAlloc()
...
Call graph listing for specific application or object
Similar command to the previous section, but use "-c" for call graph output.
sudo opreport -c --demangle=smart \
/home/ksedgwic/skyler/tam.trunk/modules/MDLoader/Linux.DBGOBJ/RTQE-MDLoader.so
We get the following output:
samples % symbol name
-------------------------------------------------------------------------------
32489 7.0405 MDL::FileLoader::readNext()
32489 100.000 MDL::FileLoader::readNext() [self]
-------------------------------------------------------------------------------
30441 6.5967 MDL::Field::Field(MDL::Field const&)
30441 100.000 MDL::Field::Field(MDL::Field const&) [self]
-------------------------------------------------------------------------------
29384 6.3676 MDL::FileLoader::getMsgFromFile(string const&)
29384 100.000 MDL::FileLoader::getMsgFromFile(string const&) [self]
-------------------------------------------------------------------------------
24771 5.3680 MDL::Field::~Field()
24771 100.000 MDL::Field::~Field() [self]
-------------------------------------------------------------------------------
19645 4.2571 __i686.get_pc_thunk.bx
19645 100.000 __i686.get_pc_thunk.bx [self]
-------------------------------------------------------------------------------
11751 2.5465 _Rb_tree<string, pair<string const, MDL::Field>, string const>::insert_equal(pair<string const, MDL::Field> const&)
11751 100.000 _Rb_tree<string, pair<string const, MDL::Field>, string const>::insert_equal(pair<string const, MDL::Field> const&) [self]
-------------------------------------------------------------------------------
9908 2.1471 MDL::Allocator<double>::~Allocator()
9908 100.000 MDL::Allocator<double>::~Allocator() [self]
-------------------------------------------------------------------------------
9600 2.0804 vector<MDL::Field>::_M_insert_aux(vector<MDL::Field>::iterator, MDL::Field const&)
9600 100.000 vector<MDL::Field>::_M_insert_aux(vector<MDL::Field>::iterator, MDL::Field const&) [self]
-------------------------------------------------------------------------------
9147 1.9822 MDL::MessageImpl::addField(MDL::Field const&)
9147 100.000 MDL::MessageImpl::addField(MDL::Field const&) [self]
-------------------------------------------------------------------------------
This output is a little suspicious (why are all the callee vectors 100%)?
Annotating source code
sudo opannotate --source --output-dir=/tmp/annotated \
/home/ksedgwic/skyler/tam.trunk/modules/MDLoader/Linux.DBGOBJ/RTQE-MDLoader.so
Excerpt from /tmp/annotated/home/ksedgwic/skyler/tam.trunk/modules/MDLoader/FileLoader.cpp:
: // remove trailing newline character
24703 5.3532 : size_t length = strlen(line) ;
1246 0.2700 : if (line[length-1] == '\n')
: {
593 0.1285 : line[--length] = '\0' ;
: }
516 0.1118 : sline = line ;
:
1152 0.2496 : switch (state)
: {
: case MSBegin:
: // read info, change to MSBody state
71 0.0154 : m_msgheadline = sline ;
20 0.0043 : state = MSBody ;
1 2.2e-04 : break ;
: case MSBody:
: // if blank line, save and
: // change to MSComplete state to terminate loop
1160 0.2514 : if (sline.length() == 0 || sline.find("\n") == 0)
: {
14 0.0030 : state = MSComplete ;
: break ;
: }
:
1432 0.3103 : s += sline + "\n" ;
: break ;
: default:
: abort() ;
: }
: }
:
Annotating assembly code
This is perhaps the most useful mode of all. All of the output appears in one file of interleaved source and assembly.
The file is sorted with the most significant routines at the top.
Best of all STL classes and other inlined code are placed where they are used, and fully accounted.
sudo opannotate --source --assembly \
/home/ksedgwic/skyler/tam.trunk/modules/MDLoader/Linux.DBGOBJ/RTQE-MDLoader.so \
> /tmp/annotated/assem