文件操作 - guide.html

文件: /usr/share/doc/pcp-doc/html/guide.html

编辑文件内容

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML>
<HEAD>
	<meta http-equiv="content-type" content="text/html; charset=utf-8">
	<meta http-equiv="content-style-type" content="text/css">
	<link href="pcpdoc.css" rel="stylesheet" type="text/css">
	<link href="images/pcp.ico" rel="icon" type="image/ico">
        <TITLE>PCP Quick Reference Guide</TITLE>
</HEAD>
<BODY LANG="en-AU" TEXT="#000060" DIR="LTR">
<TABLE WIDTH="100%" BORDER=0 CELLPADDING=0 CELLSPACING=0 STYLE="page-break-before: always">
	<TR> <TD WIDTH=64 HEIGHT=64><FONT COLOR="#000080"><A HREF="https://pcp.io/"><IMG SRC="images/pcpicon.png" ALT="pcpicon" ALIGN=TOP WIDTH=64 HEIGHT=64 BORDER=0></A></FONT></TD>
	<TD WIDTH=1><P>&nbsp;&nbsp;&nbsp;&nbsp;</P></TD>
	<TD WIDTH=500><P ALIGN=LEFT><A HREF="index.html"><FONT COLOR="#cc0000">Home</FONT></A>&nbsp;&nbsp;&middot;&nbsp;<A HREF="lab.pmchart.html"><FONT COLOR="#cc0000">Charts</FONT></A>&nbsp;&nbsp;&middot;&nbsp;<A HREF="timecontrol.html"><FONT COLOR="#cc0000">Time Control</FONT></A></P></TD>
	</TR>
</TABLE>
<H1 ALIGN=CENTER STYLE="margin-top: 0.48cm; margin-bottom: 0.32cm"><FONT SIZE=7>PCP Quick Reference Guide</FONT></H1>
<UL>
  <LI><A HREF="#intro">Introduction</A>
  <LI><A HREF="#install">Installation</A>
  <UL>
    <LI><A HREF="#collectors">Installing Collector Hosts</A>
    <LI><A HREF="#monitors">Installing Monitor Hosts</A>
    <LI><A HREF="#discovery">Dynamic Host Discovery</A>
    <LI><A HREF="#healthcheck">Installation Health Check</A>
  </UL>
  <LI><A HREF="#systemlevel">System Level Performance Monitoring</A>
  <UL>
    <LI><A HREF="#live">Monitoring Live Performance Metrics</A>
    <LI><A HREF="#retro">Retrospective Performance Analysis</A>
    <LI><A HREF="#visual">Visualizing iostat and sar Data</A>
  </UL>
  <LI><A HREF="#processes">Process Level Performance Monitoring</A>
  <UL>
    <LI><A HREF="#processmon">Live and Retrospective Process Monitoring</A>
    <LI><A HREF="#hotproc">Monitoring “Hot” Processes with Hotproc</A>
    <LI><A HREF="#instrument">Application Instrumentation</A>
  </UL>
  <LI><A HREF="#derived">Derived Metrics</A>
  <LI><A HREF="#pmie">Performance Metrics Inference</A>
  <LI><A HREF="#web">Web Services</A>
  <UL>
    <LI><A HREF="#pmwebd">Performance Metrics Web Daemon</A>
    <LI><A HREF="#browse">Web Interface for Performance Metrics</A>
  </UL>
  <LI><A HREF="#custom">Customizing and Extending PCP</A>
  <LI><A HREF="#next">Additional Information</A>
</UL>

<a name="intro"></a>
<H1>Introduction</H1>

<P><A HREF="https://pcp.io/">Performance Co-Pilot</A> (PCP) is an open source framework and toolkit for monitoring, analyzing, and responding to details of live and historical system performance. PCP has a fully distributed, plug-in based architecture making it particularly well suited to centralized analysis of complex environments and systems. Custom performance metrics can be added using the C, C++, Perl, and Python interfaces.

<P>This page provides quick instructions how to install and use PCP on a set of hosts of which one (a monitor host) will be used for monitoring and analyzing itself and other hosts (collector hosts).

<a name="install"></a>
<H1>Installation</H1>

<P>PCP is available on all recent Linux distribution releases, including Debian/Fedora/RHEL/SUSE/Ubuntu. For other operating systems and distributions you might want to consider installation <A HREF="https://pcp.io/source.html">from sources</A>.

<a name="collectors"></a>
<H3>Installing Collector Hosts</H3>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
       <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;To install basic PCP tools and services and enable collecting performance data on systemd based distributions, run:<br><B>
<br> # yum install pcp # or apt-get or dnf or zypper
<br> # systemctl enable pmcd
<br> # systemctl start pmcd
<br> # systemctl enable pmlogger
<br> # systemctl start pmlogger
</B></TD></TR>
</TABLE>

<P>Here we enable the Performance Metrics Collector Daemon (<a href="http://man7.org/linux/man-pages/man1/pmcd.1.html">pmcd(1)</a>) on the host which then in turn will control and request metrics on behalf of clients from various Performance Metrics Domain Agents (PMDAs). The PMDAs provide the actual data from different components (domains) in the system, for example from the Linux Kernel PMDA or the NFS Client PMDA. The default configuration includes over 1000 metrics with negligible overall overhead when queried. If no queries for metrics are sent to the agent, it doesn't do anything at all. Local PCP archive logs will also be enabled on the host for convenience with <a href="http://man7.org/linux/man-pages/man1/pmlogger.1.html">pmlogger(1)</a>.

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;To enable PMDAs which are not enabled by default, for example the PostgreSQL database PMDA, run the corresponding Install script:<br><B>
<br> # cd /var/lib/pcp/pmdas/postgresql
<BR> # ./Install
</B></TD></TR>
</TABLE>

<P>The client tools will contact local or remote PMCDs as needed, communication with PMCD over the network uses TCP port 44321 by default.

<a name="monitors"></a>
<H3>Installing Monitor Host</H3>

<P>The following additional packages can be optionally installed on the monitoring host to extend the set of monitoring tools from the base pcp package.

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;Install various system monitoring tools, graphical analysis tools, and documentation:<BR><B>
<BR># yum install pcp-doc pcp-gui pcp-system-tools # or apt-get or dnf or zypper
</B></TD></TR>
</TABLE>

<P>To enable centralized archive log collection on the monitoring host, its pmlogger is configured to fetch performance metrics from collector hosts. Add each collector host to the pmlogger configuration file <TT>/etc/pcp/pmlogger/control</TT> and then restart the pmlogger service on the monitoring host.

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;Enable recording of metrics from remote host <B>acme.com</B>:<B><BR>
<BR> # echo acme.com n n PCP_LOG_DIR/pmlogger/acme.com -r -T24h10m -c config.acme.com &gt;&gt; /etc/pcp/pmlogger/control
<BR>
<BR> # systemctl restart pmlogger
</B></TD></TR>
</TABLE>

<P>The health of the remote log collector will be done every half an hour. You can also run <KBD>/usr/libexec/pcp/bin/pmlogger_check -V -C</KBD> (on Fedora/RHEL) or <KBD>/usr/lib/pcp/bin/pmlogger_check -V -C</KBD> (on Debian/Ubuntu) manually to do a health check.

<P>Note that a default configuration file (config.acme.com above) will be generated if it does not exist already. This process is optional (a custom configuration for each host can be provided instead), see the <a href="http://man7.org/linux/man-pages/man1/pmlogconf.1.html">pmlogconf(1) manual page</a> for details on this.

<a name="discovery"></a>
<H3>Dynamic Host Discovery</H3>

<P>In dynamic environments manually configuring every host is not feasible, perhaps even impossible. PCP Manager (<a href="http://man7.org/linux/man-pages/man1/pmmgr.1.html">pmmgr(1)</a>, from the pcp-manager package) can be used instead of directly invoking pmlogger to auto-discover and auto-configure new collector hosts.

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
	<TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;To install the PMMGR daemon and begin monitoring either statically or dynamically configured hosts, run:<br><B>
<br> # yum install pcp-manager # or apt-get or dnf or zypper
<br> # systemctl enable pmmgr
<br> # echo acme.com >> /etc/pcp/pmmgr/target-host
<br> # echo avahi >> /etc/pcp/pmmgr/target-discovery
<br> # echo probe=<FONT COLOR="#cc0000">ip.addr.tup.le/netmask</FONT> >> /etc/pcp/pmmgr/target-discovery
<br> # systemctl restart pmmgr
<br> # find /var/log/pcp/pmmgr
</B></TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Discover use of the PCP pmcd service on the local network:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmfind.1.html">pmfind</a> -s pmcd</B>
</TD></TR>
</TABLE>

<a name="healthcheck"></a>
<H3>Installation Health Check</H3>

<P>Basic health check for running services, network connectivity between hosts, and enabled PMDAs can be done simply as follows.

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;Check PCP services on remote host <B><FONT COLOR="#cc0000">munch</FONT></B> and historically, from a local archive for host <B><FONT COLOR="#cc0000">smash</FONT></B>:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pcp.1.html">pcp</a> -h <FONT COLOR="#cc0000">munch</FONT></B><PRE>
Performance Co-Pilot configuration on munch:
  platform: SunOS munch 5.11 oi_151a8 i86pc
  hardware: 4 cpus, 3 disks, 4087MB RAM
  timezone: EST-10
  services: pmcd pmproxy
      pmcd: Version 3.8.9-1, 3 agents
      pmda: pmcd mmv solaris
      pmie: /var/log/pcp/pmie/munch/pmie.log
</PRE>
<B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pcp.1.html">pcp</a> -a /var/log/pcp/pmlogger/<FONT COLOR="#cc0000">smash</FONT>/20140729</B><PRE>
Performance Co-Pilot configuration on smash:
   archive: /var/log/pcp/pmlogger/smash/20140729
  platform: Linux smash 2.6.32-279.46.1.el6.x86_64 #1 SMP Mon May 19 16:16:00 EDT 2014 x86_64
  hardware: 8 cpus, 2 disks, 1 node, 23960MB RAM
  timezone: EST-10
  services: pmcd pmproxy pmwebd
      pmcd: Version 3.9.8-1, 8 agents
      pmda: pmcd proc xfs linux mmv nvidia dmcache postgresql
  pmlogger: primary logger: /var/log/pcp/pmlogger/smash/20140729.00.10
      pmie: /var/log/pcp/pmie/smash/pmie.log
</PRE></TD></TR>
</TABLE>

<a name="systemlevel"></a>
<H1>System Level Performance Monitoring</H1>

<P>PCP comes with a wide range of command line utilities for accessing live performance metrics via PMCDs or historical data using archive logs. The following examples illustrate some of the most useful use cases, please see the corresponding manual pages for each command for additional information. In the examples below <B>-h &lt;host&gt;</B> could be used to query a remote host, the default is the local host. Shell completion support for Bash and especially for Zsh allows completing available metrics, metricsets (with <a href="http://man7.org/linux/man-pages/man1/pmrep.1.html">pmrep</a>), and available command line options.

<a name="live"></a>
<H3>Monitoring Live Performance Metrics</H3>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Display all the enabled performance metrics on a host with a short description:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pminfo.1.html">pminfo</a> -t</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Display detailed information about a performance metric and its current values:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pminfo.1.html">pminfo</a> -dfmtT disk.partitions.read</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Monitor live disk write operations per partition with two second interval using fixed point notation (use <EM>-i</EM> instance to list only certain metrics and <EM>-r</EM> for raw values):<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmval.1.html">pmval</a> -t 2sec -f 3 disk.partitions.write</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Monitor live CPU load, memory usage, and disk write operations per partition with two second interval using fixed width columns on the remote host acme:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmdumptext.1.html">pmdumptext</a> -Xlimu -t 2sec 'kernel.all.load[1]' mem.util.used disk.partitions.write -h acme.com</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Monitor live process creation rate and free/used memory with two second interval printing timestamps and using GBs for output values in CSV format:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmrep.1.html">pmrep</a> -p -b GB -t 2sec -o csv kernel.all.sysfork mem.util.free mem.util.used</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Monitor system metrics in a top-like window:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pcp-atop.1.html">pcp atop</a></B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Monitor system metrics in a sar-like (System Activity Report) manner:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pcp-atopsar.1.html">pcp atopsar</a></B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20> 
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Monitor system metrics in a sar like fashion with two second interval from two different hosts:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmstat.1.html">pmstat</a> -t 2sec -h <FONT COLOR="#cc0000">acme1.com</FONT> -h <FONT COLOR="#cc0000">acme2.com</FONT></B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Monitor system metrics in an iostat like fashion with two second interval:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmiostat.1.html">pmiostat</a> -t 2sec</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20> 
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Monitor performance metrics with a GUI application with two second default interval from two different hosts. Use <EM>File-&gt;New Chart</EM> to select metrics to be included in a new view and use <EM>File-&gt;Open View</EM> to use a predefined view:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmchart.1.html">pmchart</a> -t 2sec -h <FONT COLOR="#cc0000">acme1.com</FONT> -h <FONT COLOR="#cc0000">acme2.com</FONT></B>
</TD></TR>
</TABLE>

<a name="retro"></a>
<H3>Retrospective Performance Analysis</H3>

<P>PCP archive logs are located under <TT>/var/log/pcp/pmlogger/</TT><B><FONT COLOR="#cc0000">hostname</FONT></B>, and the archive names indicate the time they cover. Archives are self-contained, and machine- and version-independent so they can be transfered to any machine for offline analysis.

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Check the host, timezone and the time period an archive covers:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmdumplog.1.html">pmdumplog</a> -L acme.com/20140902</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Check PCP configuration at the time when an archive was created:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pcp.1.html">pcp</a> -a acme.com/20140902</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Display all enabled performance metrics at the time when an archive was created:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pminfo.1.html">pminfo</a> -a acme.com/20140902</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Display detailed information about a performance metric at the time when an archive was created:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pminfo.1.html">pminfo</a> -df mem.freemem -a acme.com/20140902</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Dump past disk write operations per partition in an archive using fixed point notation (use <EM>-i</EM> instance to list only certain metrics and <EM>-r</EM> for raw values):<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmval.1.html">pmval</a> -f 3 disk.partitions.write -a acme.com/20140902</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Replay past disk write operations per partition in an archive with two second interval using fixed point notation between 9 AM and 10 AM (use full dates with syntax like <EM>@"2014-08-20 14:00:00"</EM>):<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmval.1.html">pmval</a> -d -t 2sec -f 3 disk.partitions.write -S @09:00 -T @10:00 -a acme.com/20140902</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Calculate average values of performance metrics in an archive between 9 AM / 10 AM using table like formatting including the time of minimum/maximum value and the actual minimum/maximum value:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmlogsummary.1.html">pmlogsummary</a> -HlfiImM -S @09:00 -T @10:00 acme.com/20140902 disk.partitions.write mem.freemem</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Dump past CPU load, memory usage, and disk write operations per partition in an archive averaged over 10 minute interval with fixed columns between 9 AM and 10 AM:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmdumptext.1.html">pmdumptext</a> -Xlimu -t 10m -S @09:00 -T @10:00 'kernel.all.load[1]' 'mem.util.used' 'disk.partitions.write' -a acme.com/20140902</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Replay <a href="http://man7.org/linux/man-pages/man8/vmstat.8.html">vmstat</a> like metrics (using a customizable metricset definition from the <a href="http://man7.org/linux/man-pages/man5/pmrep.conf.5.html">pmrep.conf</a> configuration file) from an archive on every full 5 minutes using UTC as timezone:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmrep.1.html">pmrep</a> -a acme.com/20140902 -A 5min -t 5min -Z UTC :vmstat</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Summarize differences in past performance metrics between two archives, comparing 2 AM / 3 AM in the first archive to 9 AM / 10 AM in the second archive (grep for <EM>'+'</EM> to quickly see values which were zero during the first period):<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmdiff.1.html">pmdiff</a> -S @02:00 -T @03:00 -B @09:00 -E @10:00 acme.com/20140902 acme.com/20140901</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Replay past system metrics in an archive in a top-like window starting 9 AM:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pcp-atop.1.html">pcp atop</a> -b 09:00 -r acme.com/20140902
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pcp-atop.1.html">pcp</a> -S @09:00 -a acme.com/20140902 atop</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Dump past system metrics in a sar like fashion averaged over 10 minute interval in an archive between 9 AM and 10 AM:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmstat.1.html">pmstat</a> -t 10m -S @09:00 -T @10:00 -a acme.com/20140902</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Dump past system metrics in an <a href="http://man7.org/linux/man-pages/man1/iostat.1.html">iostat(1)</a> like fashion averaged over one hour interval in an archive:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmiostat.1.html">pmiostat</a> -t 1h -a acme.com/20140902</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Dump past system metrics in a <a href="http://man7.org/linux/man-pages/man1/free.1.html">free(1)</a> like fashion at a specific historical time offset:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pcp.1.html">pcp</a> -a acme.com/20140902 -O @10:02 free</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Replay performance metrics with a GUI application with two second default interval in an archive between 9 AM and 10 AM. Use <EM>File-&gt;New Chart</EM> to select metrics to be included in a new view and use <EM>File-&gt;Open View</EM> to use a predefined view:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmchart.1.html">pmchart</a> -t 2sec -S @09:00 -T @10:00 -a acme.com/20140902</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Merge several archives as a new combined archive (see the manual page how to write configuration file to collect only certain metrics):<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmlogextract.1.html">pmlogextract</a> &lt;archive1&gt; &lt;archive2&gt; &lt;newarchive&gt;</B>
</TD></TR>
</TABLE>

<a name="visual"></a>
<H3>Visualizing iostat and sar Data</H3>

<P><a href="http://man7.org/linux/man-pages/man1/iostat.1.html">iostat</a> and <a href="http://man7.org/linux/man-pages/man1/sar.1.html">sar</a> data can be imported as PCP archives which then allows inspecting and visualizing the data with PCP tools. The <a href="http://man7.org/linux/man-pages/man1/iostat2pcp.1.html">iostat2pcp(1)</a> importer is in the <EM>pcp-import-iostat2pcp</EM> package and the <a href="http://man7.org/linux/man-pages/man1/sar2pcp.1.html">sar2pcp(1)</a> importer is in the <EM>pcp-import-sar2pcp</EM> package.

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Import iostat data to a new PCP archive and visualize it:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/iostat.1.html">iostat</a> -t -x 2 > iostat.out
<BR> $ <a href="http://man7.org/linux/man-pages/man1/iostat2pcp.1.html">iostat2pcp</a> iostat.out iostat.pcp
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmchart.1.html">pmchart</a> -t 2sec -a iostat.pcp </B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Import sar data from an existing sar archive to a new PCP archive and visualize it (sar logs are under <TT>/var/log/sysstat</TT> on Debian/Ubuntu):<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/sar2pcp.1.html">sar2pcp</a> /var/log/sa/sa15 sar.pcp
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmchart.1.html">pmchart</a> -t 2sec -a sar.pcp </B>
</TD></TR>
</TABLE>

<a name="processes"></a>
<H1>Process Level Performance Monitoring</H1>

<P>PCP provides details of each running process via the standard PCP interfaces and tools on the localhost but due to security and performance considerations, most of the process related information is not stored in archive logs by default. Also for security reasons, only root can access some details of running processes of other users.
<P>Custom application instrumentation is possible with the Memory Mapped Value (MMV) PMDA.

<a name="processmon"></a>
<H3>Live and Retrospective Process Monitoring</H3>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Display all the available process related metrics:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pminfo.1.html">pminfo</a> proc</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Monitor the number of open file descriptors of the process 1234:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmval.1.html">pmval</a> -t 2sec 'proc.fd.count[1234]'</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Monitor the CPU time, memory usage (RSS), and the number of threads of the process 1234:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmdumptext.1.html">pmdumptext</a> -Xlimu -t 2sec 'proc.psinfo.utime[1234]' 'proc.memory.rss[1234]' 'proc.psinfo.threads[1234]'</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Monitor all outgoing network metrics for the wlan0 interface:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmrep.1.html">pmrep</a> -i wlan0 -v network.interface.out</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Display all the available process related metrics in an archive:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pminfo.1.html">pminfo</a> proc -a acme.com/20140902</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Display the number of running processes on 2014-08-20 14:00:<BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmval.1.html">pmval</a> -s 1 -S @"2014-08-20 14:00" proc.nprocs -a acme.com/20140820</B>
</TD></TR>
</TABLE>

<a name="hotproc"></a>
<H3>Monitoring “Hot” Processes with Hotproc</H3>

<P>It is also possible to monitor “hot” or “interesting” processes by name, for example all processes of which command name is <TT>java</TT> or <TT>python</TT>. This monitoring of “hot” processes can also be enabled or disabled based on certain criterias or from the command line on the fly. The metrics will be available under the namespace <EM>hotproc</EM>.
<P>Configuring processes to be monitored constantly using the <EM>hotproc</EM> namespace can be done using the configuration file <TT>/var/lib/pcp/pmdas/proc/hotproc.conf</TT> - see the <a href="http://man7.org/linux/man-pages/man1/pmdaproc.1.html">pmdaproc(1)</a> manual page for details. This allows monitoring these processes regardless of their PIDs and also logging the metrics easily.

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Enable monitoring of all Java instances on the fly and display all the collected metrics:<BR><B>
<BR> # <a href="http://man7.org/linux/man-pages/man1/pmstore.1.html">pmstore</a> hotproc.control.config 'fname == "java"'
<BR> # <a href="http://man7.org/linux/man-pages/man1/pminfo.1.html">pminfo</a> -f hotproc</B>
</TD></TR>
</TABLE>

<a name="instrument"></a>
<H3>Application Instrumentation</H3>

<P>Applications can be instrumented in the PCP world by using Memory Mapped Values (MMVs). <a href="http://man7.org/linux/man-pages/man1/pmdammv.1.html">pmdammv</a> is a PMDA which exports application level performance metrics using memory mapped files. It offers an extremely low overhead instrumentation facility that is well-suited to long running, mission critical applications where it is desirable to have performance metrics and availability information permanently enabled.

<P>Application to be instrumented with MMV need to be PCP MMV aware, APIs are available for several languages including C, C++, Perl, and Python. Java applications may use the separate <A HREF="https://github.com/performancecopilot/parfait">Parfait</A> class library for enabling MMV.

<P>See the <A HREF="https://pcp.io/doc/pcp-programmers-guide.pdf">Performance Co-Pilot Programmer's Guide PDF</A> for more information about application instrumentation.

<a name="derived"></a>
<H1>Derived Metrics</H1>

<P>PCP provides a wide range of performance metrics but still in some cases the readily available metrics may not exactly provide what is needed. Derived metrics (see <a href="http://man7.org/linux/man-pages/man3/pmloadderivedconfig.3.html">pmLoadDerivedConfig(3)</a>) may be used to extend the available metrics with new (derived) metrics by using simple arithmetic expressions (see <a href="http://man7.org/linux/man-pages/man3/pmregisterderived.3.html">pmRegisterDerived(3)</a>).

<P>The following example illustrates how to define corresponding metrics which are displayed by <KBD>sar -d</KBD> but are not provided by default by PCP:

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Create a file containing definitions of derived metrics and point PCP_DERIVED_CONFIG to it when running PCP utilities:</BR><B>
<BR> $ cat ./pcp-deriv-metrics.conf
<BR> disk.dev.avqsz = disk.dev.read_rawactive + disk.dev.write_rawactive
<BR> disk.dev.avrqsz = 2 * rate(disk.dev.total_bytes) / rate(disk.dev.total)
<BR> disk.dev.await = 1000 * (rate(disk.dev.read_rawactive) + rate(disk.dev.write_rawactive)) / rate(disk.dev.total)
<BR> $ export PCP_DERIVED_CONFIG=./pcp-deriv-metrics.conf
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmval.1.html">pmval</a> -t 2sec -f 3 disk.dev.avqsz
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmval.1.html">pmval</a> -t 2sec -f 3 disk.dev.avrqsz -h acme.com
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmval.1.html">pmval</a> -t 2sec -f 3 disk.dev.await -a acme.com/20140902</B>
</TD></TR>
</TABLE>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
Define a derived metric on the command line and monitor it with standard metrics:</BR><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmrep.1.html">pmrep</a> -t 2sec -p -b MB -e "mem.util.allcache = mem.util.bufmem + mem.util.cached + mem.util.slab" mem.util.free mem.util.allcache mem.util.used
</TD></TR>
</TABLE>

<a name="pmie"></a>
<H1>Performance Metrics Inference</H1>

<P>Performance Metrics Inference Engine (<a href="http://man7.org/linux/man-pages/man1/pmie.1.html">pmie(1)</a>) can evaluate rules and generate alarms, run scripts, or automate system management tasks based on live or past performance metrics.

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;To enable and start PMIE:<br><B>
<br> # systemctl enable pmie
<br> # systemctl start pmie
</B></TD></TR>
</TABLE>

<P>To enable the monitoring host to run PMIE for collector hosts, add each host to the <TT>/etc/pcp/pmie/control</TT> configuration file.

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;Enable monitoring of metrics from remote host <B>acme.com</B>:<B>
<BR> # echo acme.com n PCP_LOG_DIR/pmie/acme.com -c config.acme.com
<BR>
<BR> # systemctl restart pmie
</B></TD></TR>
</TABLE>

<P>Some examples in plain English describing what could be done with PMIE:
<UL>
<LI>If the number of IP received packets exceeds a threshold run a script to adjust firewall rules to limit the incoming traffic
<LI>If 3 out of 4 consecutive samples taken every minute of disk operations exceeds a threshold between 9 AM and 5 PM send an email and write a system log message
<LI>If all hosts in a group have CPU load over a threshold for more than 10 minutes or they have more application processes running than a threshold limit generate an alarm and run a script to tune the application
</UL>

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;
This example shows a PMIE script, checks its syntax, runs it against an archive, and prints a simple message if more than 5 GB of memory was in use between 9 AM and 10 AM using one minute sampling interval:<BR><B>
<BR> $ cat pmie.ex</B>
<PRE><B>bloated = ( mem.util.used > 5 Gbyte )
      -> print "%v memory used on %h!"</B></PRE><B>
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmie.1.html">pmie</a> -C pmie.ex
<BR> $ <a href="http://man7.org/linux/man-pages/man1/pmie.1.html">pmie</a> -t 1min -c pmie.ex -S @09:00 -T @10:00 -a acme.com/20140820</B>
</TD></TR>
</TABLE>

<a name="web"></a>
<H1>PCP Web Services</H1>

<a name="pmwebd"></a>
<H3>Performance Metrics Web Daemon</H3>

<P>Performance Metrics Web Daemon (<a href="http://man7.org/linux/man-pages/man1/pmwebd.1.html">pmwebd(1)</a>) is a front-end to both PMCD and PCP archives, providing a REST web service (over HTTP/JSON) suitable for use by web-based tools wishing to access performance data over HTTP. Custom applications can access all the available PCP information using this method, including custom metrics generated by custom PMDAs.

<TABLE WIDTH="100%" BORDER=0 CELLPADDING=10 CELLSPACING=20>
        <TR><TD BGCOLOR="#e2e2e2" WIDTH="70%"><BR><IMG SRC="images/stepfwd_on.png" ALT="" WIDTH=16 HEIGHT=16 BORDER=0>&nbsp;&nbsp;&nbsp;To install the PCP web service:<br><B>
<br> # yum install pcp-webapi # or apt-get or dnf or zypper
<br> # systemctl enable pmwebd
<br> # systemctl start pmwebd
</B></TD></TR>
</TABLE>

<a name="browse"></a>
<H3>User Web Interface for Performance Metrics</H3>

<P>Several browser interfaces for accessing PCP performance metrics are also available. These web interfaces make PCP metrics available via your choice of <A HREF="http://grafana.org/">Grafana</A> or <A HREF="http://graphite.wikidot.com/">Graphite</A>.

<P>After installing the PCP web services daemon as described above, install the pcp-webjs package and then just point a browser toward <a href="http://localhost:44323">http://localhost:44323</a>.

<a name="custom"></a>
<H1>Customizing and Extending PCP</H1>

<P>PCP PMDAs offer a way for administrators and developers to customize and extend the default PCP installation. The pcp-libs-devel package contains all the needed development related examples, headers, and libraries. New PMDAs can easily be added, below is a quick list of references for starting development:

<UL>
<LI>Some examples exist below <TT>/var/lib/pcp/pmdas/</TT> - the simple, sample, and txmon PMDAs are easy to read PMDAs.
  <UL>
    <LI>The simple PMDA provides implementations in C, Perl and Python.</LI>
  </UL></LI>
<LI>A simple command line monitor tool is <TT>/usr/share/pcp/demos/pmclient</TT> (C language).</LI>
<LI>Good initial Python monitor examples are <TT>/usr/libexec/pcp/bin/pcp/pcp-*</TT> (Fedora/RHEL) or <TT>/usr/lib/pcp/bin/pcp-*</TT> (Debian/Ubuntu).
  <UL>
    <LI>Slightly more complex examples are the pcp-free, pmiostat, pmcollectl commands.</LI>
  </UL></LI>
<LI>The applications in the <A HREF="https://www.sourceware.org/git/gitweb.cgi?p=pcpfans.git;a=shortlog;h=refs/heads/webjs">pcp-webjs</A> source tree are helpful when developing new web applications.</LI>
</UL>

<a name="next"></a>
<H1>Additional Information</H1>

<UL>
<LI><A HREF="https://pcp.io/">https://pcp.io/</A> - PCP home page
<LI><A HREF="https://pcp.io/presentations.html">https://pcp.io/presentations.html</A> - PCP Presentations
<LI><A HREF="https://pcp.io/doc/pcp-users-and-administrators-guide.pdf">https://pcp.io/doc/pcp-users-and-administrators-guide.pdf</A> - Performance Co-Pilot User's and Administrator's Guide PDF
<LI><A HREF="https://pcp.io/doc/pcp-programmers-guide.pdf">https://pcp.io/doc/pcp-programmers-guide.pdf</A> - Performance Co-Pilot Programmer's Guide PDF
</UL>

<P><BR></P>
<HR>
<CENTER>
<TABLE WIDTH="100%" BORDER=0 CELLPADDING=0 CELLSPACING=0>
	<TR> <TD WIDTH="50%"><P>Copyright &copy; 2007-2010 <A HREF="https://www.aconex.com/"><FONT COLOR="#000060">Aconex</FONT></A><BR>Copyright &copy; 2000-2004 <A HREF="https://www.sgi.com/"><FONT COLOR="#000060">Silicon Graphics Inc</FONT></A></P></TD>
       <TD WIDTH="50%"><P ALIGN=RIGHT><A HREF="https://pcp.io/"><FONT COLOR="#000060">PCP Site</FONT></A><BR>Copyright &copy; 2012-2018 <A HREF="https://www.redhat.com/"><FONT COLOR="#000060">Red Hat</FONT></A></P></TD> </TR>
</TABLE>
</CENTER>
</BODY>
</HTML>

文件操作 - guide.html

文件: /usr/share/doc/pcp-doc/html/guide.html

编辑文件内容

修改文件时间

删除文件