# process-exporter
Prometheus exporter that mines /proc to report on selected processes.
[release]: https://github.com/ncabatoff/process-exporter/releases/latest
[![Release](https://img.shields.io/github/release/ncabatoff/process-exporter.svg?style=flat-square")][release]
[![Powered By: GoReleaser](https://img.shields.io/badge/powered%20by-goreleaser-green.svg?branch=master)](https://github.com/goreleaser)
![Build](https://github.com/ncabatoff/process-exporter/actions/workflows/build.yml/badge.svg)
Some apps are impractical to instrument directly, either because you
don't control the code or they're written in a language that isn't easy to
instrument with Prometheus. We must instead resort to mining /proc.
## Installation
Either grab a package for your OS from the [Releases][release] page, or
install via [docker](https://hub.docker.com/r/ncabatoff/process-exporter/).
## Running
Usage:
```
process-exporter [options] -config.path filename.yml
```
or via docker:
```
docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v `pwd`:/config ncabatoff/process-exporter --procfs /host/proc -config.path /config/filename.yml
```
Important options (run process-exporter --help for full list):
-children (default:true) makes it so that any process that otherwise
isn't part of its own group becomes part of the first group found (if any) when
walking the process tree upwards. In other words, resource usage of
subprocesses is added to their parent's usage unless the subprocess identifies
as a different group name.
-threads (default:true) means that metrics will be broken down by thread name
as well as group name.
-recheck (default:false) means that on each scrape the process names are
re-evaluated. This is disabled by default as an optimization, but since
processes can choose to change their names, this may result in a process
falling into the wrong group if we happen to see it for the first time before
it's assumed its proper name.
-procnames is intended as a quick alternative to using a config file. Details
in the following section.
To disable any of these options, use the `-option=false`.
## Configuration and group naming
To select and group the processes to monitor, either provide command-line
arguments or use a YAML configuration file.
The recommended option is to use a config file via -config.path, but for
convenience and backwards compatibility the -procnames/-namemapping options
exist as an alternative.
### Using a config file
The general format of the -config.path YAML file is a top-level
`process_names` section, containing a list of name matchers:
```
process_names:
- matcher1
- matcher2
...
- matcherN
```
The default config shipped with the deb/rpm packages is:
```
process_names:
- name: "{{.Comm}}"
cmdline:
- '.+'
```
A process may only belong to one group: even if multiple items would match, the
first one listed in the file wins.
(Side note: to avoid confusion with the cmdline YAML element, we'll refer to
the command-line arguments of a process `/proc/<pid>/cmdline` as the array
`argv[]`.)
#### Using a config file: group name
Each item in `process_names` gives a recipe for identifying and naming
processes. The optional `name` tag defines a template to use to name
matching processes; if not specified, `name` defaults to `{{.ExeBase}}`.
Template variables available:
- `{{.Comm}}` contains the basename of the original executable, i.e. 2nd field in `/proc/<pid>/stat`
- `{{.ExeBase}}` contains the basename of the executable
- `{{.ExeFull}}` contains the fully qualified path of the executable
- `{{.Username}}` contains the username of the effective user
- `{{.Matches}}` map contains all the matches resulting from applying cmdline regexps
- `{{.PID}}` contains the PID of the process. Note that using PID means the group
will only contain a single process.
- `{{.StartTime}}` contains the start time of the process. This can be useful
in conjunction with PID because PIDs get reused over time.
- `{{.Cgroups}}` contains (if supported) the cgroups of the process
(`/proc/self/cgroup`). This is particularly useful for identifying to which container
a process belongs.
Using `PID` or `StartTime` is discouraged: this is almost never what you want,
and is likely to result in high cardinality metrics which Prometheus will have
trouble with.
#### Using a config file: process selectors
Each item in `process_names` must contain one or more selectors (`comm`, `exe`
or `cmdline`); if more than one selector is present, they must all match. Each
selector is a list of strings to match against a process's `comm`, `argv[0]`,
or in the case of `cmdline`, a regexp to apply to the command line. The cmdline
regexp uses the [Go syntax](https://golang.org/pkg/regexp).
For `comm` and `exe`, the list of strings is an OR, meaning any process
matching any of the strings will be added to the item's group.
For `cmdline`, the list of regexes is an AND, meaning they all must match. Any
capturing groups in a regexp must use the `?P<name>` option to assign a name to
the capture, which is used to populate `.Matches`.
Performance tip: give an exe or comm clause in addition to any cmdline
clause, so you avoid executing the regexp when the executable name doesn't
match.
```
process_names:
# comm is the second field of /proc/<pid>/stat minus parens.
# It is the base executable name, truncated at 15 chars.
# It cannot be modified by the program, unlike exe.
- comm:
- bash
# exe is argv[0]. If no slashes, only basename of argv[0] need match.
# If exe contains slashes, argv[0] must match exactly.
- exe:
- postgres
- /usr/local/bin/prometheus
# cmdline is a list of regexps applied to argv.
# Each must match, and any captures are added to the .Matches map.
- name: "{{.ExeFull}}:{{.Matches.Cfgfile}}"
exe:
- /usr/local/bin/process-exporter
cmdline:
- -config.path\s+(?P<Cfgfile>\S+)
```
Here's the config I use on my home machine:
```
process_names:
- comm:
- chromium-browse
- bash
- prometheus
- gvim
- exe:
- /sbin/upstart
cmdline:
- --user
name: upstart:-user
```
### Using -procnames/-namemapping instead of config.path
Every name in the procnames list becomes a process group. The default name of
a process is the value found in the second field of /proc/<pid>/stat
("comm"), which is truncated at 15 chars. Usually this is the same as the
name of the executable.
If -namemapping isn't provided, every process with a comm value present
in -procnames is assigned to a group based on that name, and any other
processes are ignored.
The -namemapping option is a comma-separated list of alternating
name,regexp values. It allows assigning a name to a process based on a
combination of the process name and command line. For example, using
-namemapping "python2,([^/]+)\.py,java,-jar\s+([^/]+).jar"
will make it so that each different python2 and java -jar invocation will be
tracked with distinct metrics. Processes whose remapped name is absent from
the procnames list will be ignored. On a Ubuntu Xenian machine being used as
a workstation, here's a good way of tracking resource usage for a few
different key user apps:
process-exporter -namemapping "upstart,(--user)" \
-procnames chromium-browse,bash,gvim,prometheus,process-exporter,upstart:-user
Since upstart --user is the parent process of the X11 session, this will
make all apps started by the user fall into the group named "upstart:-user",
unless they're one of the others named explicitly with -procnames, like gvim.
## Group Metrics
There's no meaningful way to name a process that will only ever name a single process, so process-exporter assumes that every metric will be attached
to a group of processes - not a
[process group](https://en.wikipedia.org/wiki/Process_group) in the technical
sense, just one or more processes that meet a configuration's specification
of what should be monit