Implementing Performance Co-Pilot (PCP) on Enterprise Linux Distributions

-

 

Overview

Performance Co-Pilot (PCP) is an open-source, distributed framework for monitoring, visualising, storing, and analysing system-level performance metrics. PCP uses a collector–monitor architecture where Performance Metrics Domain Agents (PMDAs) gather data from different system subsystems and the Performance Metrics Collector Daemon (pmcd) coordinates collection. Metrics are archived by pmlogger for retrospective analysis.

PCP replaces the legacy Oracle OSWatcher Black Box (OSWbb) and covers the same ground — CPU, memory, disk I/O, network, and process statistics — while offering binary archive storage, extensible PMDAs, conditional alerting via PMIE, and native Grafana integration.

Distribution Compatibility Matrix

PCP is included in the base repositories of all major RHEL-family distributions. Since RHEL 8, 9, and 10 all ship PCP in their BaseOS or AppStream repos, and Rocky Linux, AlmaLinux, and Oracle Linux maintain binary or ABI compatibility with RHEL, the installation commands are identical across all of them.

Distribution

Version 8

Version 9

Version 10

Notes

Oracle Linux

Supported

Supported

Supported

OEL 9/10: PCP is the default monitoring tool, replacing OSWatcher. pcp-oracle-conf available in ol9/ol10_addons.

RHEL

Supported

Supported

Supported

PCP included in base repos since RHEL 7.4. pcp-zeroconf recommended by Red Hat for support environments.

Rocky Linux

Supported

Supported

Supported

1-to-1 RHEL binary compatible. PCP packages mirror RHEL versions in BaseOS/AppStream.

AlmaLinux

Supported

Supported

Supported

ABI-compatible with RHEL. PCP and Grafana included in AlmaLinux 10.1 release notes as monitoring utilities.

Tip: The installation steps in this document apply equally to all four distributions. Oracle Linux-specific extras (such as pcp-oracle-conf) are called out separately where applicable.


Quick Install (All Distributions)

The pcp-zeroconf package is the easiest way to get a fully functional PCP setup. It installs all dependencies, enables pmcd and pmlogger, and starts collecting data immediately with no further configuration.

sudo dnf install pcp-zeroconf -y

This single command handles everything: package installation, service enablement, and a sensible default logging configuration (10-second sampling interval, 14-day retention).

Manual Install (Granular Control)

If you prefer more control over what gets installed:

# Core PCP + command-line monitoring tools

sudo dnf install pcp pcp-system-tools -y

 # Optional: GUI charting tool (requires X11/Wayland)

sudo dnf install pcp-gui -y

 # Optional: Grafana integration for web dashboards

sudo dnf install pcp-pmda-openmetrics grafana -y

Oracle Linux Extras

Oracle provides a pcp-oracle-conf package in the OEL 9 and 10 addons repository. This package ships an Oracle-tuned pmlogger configuration (config.ora) that logs a curated set of metrics relevant to Oracle database and cloud workloads.

# OEL 9

sudo dnf install --enablerepo=ol9_addons pcp-oracle-conf -y

 # OEL 10

sudo dnf install --enablerepo=ol10_addons pcp-oracle-conf -y

Note: The pcp-oracle-conf package overrides the default pmlogger configuration. After installation, the active config file is /var/lib/pcp/config/pmlogger/config.ora instead of config.default.

Enable and Start PCP Services

If you used pcp-zeroconf, services are already running. For manual installs, enable the core daemons:

# Enable and start the collector daemon and archive logger

sudo systemctl enable --now pmcd pmlogger

 # Optional: enable the inference engine (alerting / rule evaluation)

sudo systemctl enable --now pmie

 # Optional: enable the REST API / time series proxy

sudo systemctl enable --now pmproxy

Verify Installation

pcp

Expected output (example from an OEL 9 host):

[root@pcp-demo opc]# pcp

Performance Co-Pilot configuration on pcp-demo:

 platform: Linux pcp-demo 6.12.0-203.76.7.3.el9uek.x86_64 #2 SMP PREEMPT_DYNAMIC Fri Jun 5 00:02:29 PDT 2026 x86_64

 hardware: 2 cpus, 1 disk, 1 node, 11496MB RAM

 timezone: IST-5:30

 services: pmcd

     pmcd: Version 6.3.7-5, 12 agents, 6 clients

     pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm jbd2

           dm openmetrics

 pmlogger: primary logger: /var/oled/pcp/pmlogger/pcp-demo/20260614.10.55

     pmie: primary engine: /var/oled/pcp/pmie/pcp-demo/pmie.log

Firewall Configuration (Remote Monitoring)

If you need to monitor this host from a remote PCP monitor, open TCP port 44321 (pmcd) and optionally 44322 (pmproxy):

sudo firewall-cmd --permanent --add-port=44321/tcp

sudo firewall-cmd --permanent --add-port=44322/tcp   # if using pmproxy

    sudo firewall-cmd --reload

Configuration

Logging Interval

The default logging interval depends on the install method: pcp-zeroconf uses 10 seconds, while a base pcp install uses 60 seconds. To change it, edit the control file:

sudo vi /etc/pcp/pmlogger/control.d/local

Add -t <interval> to the arguments. For example, to log every 30 seconds:

LOCALHOSTNAME   y   n   PCP_ARCHIVE_DIR/LOCALHOSTNAME   -r -T24h10m -t 30s -c config.default

Then restart pmlogger:

sudo systemctl restart pmlogger

Metric Selection

Use pmlogconf to interactively configure which metric groups are logged:

sudo pmlogconf /var/lib/pcp/config/pmlogger/config.default

This walks through each metric group (CPU, disk, memory, network, filesystem, process, etc.) and lets you enable/disable and set per-group intervals.

Archive Retention

By default, PCP retains archives for 14 days. To change retention to 30 days:

sudo vi /etc/sysconfig/pmlogger_timers

 

# Add or modify:

PMLOGGER_DAILY_PARAMS="-E -k 30"

The -k 30 flag tells pmlogger_daily to keep archives for 30 days. Compressed archives use approximately 10:1 compression, so storage requirements are modest — roughly 10–15 MB/day uncompressed per typical server.

Important: Ensure sufficient space on /var (or the mount containing /var/log/pcp). Red Hat recommends at least 5 GB for small servers, 10 GB for larger systems, on an ongoing basis.

Using PCP — Common Commands

Task

Command

System overview

pcp

Live vmstat equivalent

pmrep :vmstat  or  pcp vmstat

Live iostat equivalent

pmrep :iostat  or  pcp iostat

Live mpstat (per-CPU)

pmrep :mpstat

Live top equivalent

pcp htop  or  pcp atop

Live dstat equivalent

pcp dstat

Network interface stats

pmrep network.interface

Free memory / swap

pmrep mem.util.free mem.util.swapFree

List all available metrics

pminfo | less

Describe a specific metric

pminfo -dt kernel.all.load

Query a metric value live

pmval -s 5 -t 2 kernel.all.load

Replay archived vmstat

pmrep -a /var/log/pcp/pmlogger/$(hostname)/<archive> :vmstat

Compare two time periods

pmdiff <archive1> <archive2>

Graph archived data (GUI)

pmchart -a <archive>

Check PCP service status

systemctl status pmcd pmlogger pmie

Archive Management

Archive Location

Archives are stored under /var/log/pcp/pmlogger/<hostname>/. Each day produces a set of files (index, metadata, data volumes) named by timestamp.

ls -lh /var/log/pcp/pmlogger/$(hostname)/

On-Demand Capture

To capture a one-off performance snapshot for a specific duration (e.g., during a load test):

# Capture 10 minutes of data at 5-second intervals

pmlogger -T 10m -t 5s /tmp/loadtest-$(date +%Y%m%d)

Merge Archives

To merge multiple archive files for continuous analysis:

pmlogextract archive1 archive2 archive3 merged_archive

Export to CSV

For offline analysis in spreadsheets or external tools:

pmrep -a <archive> -o csv -F output.csv kernel.all.cpu.user kernel.all.cpu.sys disk.all.read disk.all.write

Alerting with PMIE (Performance Metrics Inference Engine)

PMIE evaluates rules against live or archived metrics and can trigger actions (syslog, email, script execution) when conditions are met.

Enable PMIE

sudo systemctl enable --now pmie

Example Rules

Create a custom rule file at /etc/pcp/pmie/config.d/custom.pmie:

// Alert if CPU idle drops below 10% for 5 consecutive samples

kernel.all.cpu.idle < 10

    -> syslog "[PCP] CRITICAL: CPU idle below 10% - value: %v";

 // Alert if free memory drops below 500 MB

mem.util.free < 512000

    -> syslog "[PCP] WARNING: Free memory below 500 MB - value: %v KB";

 // Alert if disk utilisation exceeds 90%

some_inst disk.dev.avactive > 900000

    -> syslog "[PCP] WARNING: Disk utilisation high on %i - value: %v";

Restart PMIE to apply:

sudo systemctl restart pmie


Comments

Post a Comment

Popular posts from this blog

Access Oracle OCI Object Storage through GUI Client

-
Image
 As we already know that there are several GUI clients available for Amazon AWS S3 Object storage like - S3 Browser (https://s3browser.com/), CloudBerry Explorer (https://www.msp360.com/explorer/windows/amazon-s3.aspx), CyberDuck (https://cyberduck.io/) etc. Using these tools we can easily connect to the Object Storage service, upload & download "objects" i.e files and folders and also do some advance management like Bucket Policies, Replication, Versioning, lifecycle policies etc. Now these tools can also be used with Oracle OCI Object storage, but there are some prerequisites to get the proper credentials. You need to get following information before you can connect to you OCI Object storage - Object Storage Namespace: REST Endpoint: Access Key: Secret Key ID: How to get these details - Login to your OCI Account, head over to - Hamburger Icon on left hand top corner -> Storage -> Under Object Storage & Archive -> Buckets. It will list all the buckets in thi...
Read more

Instance OS Baseline Configuration Runbook

-
Overview & Scope This runbook defines the standard baseline configuration procedure for new OCI compute instances across five supported operating system families. It is intended for use by Cloud SRE and operations personnel during initial instance provisioning. Objectives Each OS configuration procedure achieves the following baseline state: •        Set system timezone to Asia/Kolkata (IST, UTC+05:30) •        Apply all available OS security and feature updates •        Enable and configure the host-based firewall to permit SSH (Linux) or RDP (Windows) ingress exclusively from the VCN CIDR block •        Enable EPEL repository where applicable (Oracle Linux, Rocky Linux, Alma Linux) •        Create a named administrative user account separate from the default OCI-provisioned user (opc / Administrator) •   ...
Read more

Accessing OCI Compute Instances Using VNC Console (Instance Console Connection)

-
Image
  Overview In Oracle Cloud Infrastructure (OCI), administrators occasionally face scenarios where SSH access to a compute instance is unavailable —for example, due to network misconfiguration, firewall issues, or OS-level boot problems. To handle such situations, OCI provides Instance Console Connection , which enables VNC-based graphical console access and serial console access to compute instances. In this post, I demonstrate how to access an OCI Compute instance using the VNC console , explain when to use it, and highlight best practices for secure and effective troubleshooting. ⚠️ This demonstration is performed in a personal OCI tenancy using test instances. No customer environments or confidential information are exposed. Oracle Products Used Oracle Cloud Infrastructure OCI Compute Instance Console Connection OCI IAM Virtual Cloud Network (VCN) What Is Instance Console Connection? Instance Console Connection provides ou...
Read more