Chapter 5: Running `stealth'

Now that stealth has been compiled, the construction of a policy file has been covered, and a service-account on the client has been defined, what must be done to run stealth in practice?

Here's what remains to be done:

Install stealth at a proper location
Construct one or more policy files
Learn to interpret stealth's output.
Optionally, automate the removal of old log-files.
Determine a schedule for running stealth automatically, e.g. using cron(1) or ssh-cron(1)

In this chapter, these topics are discussed.

5.1: Installing `stealth'

As stealth is mainly a system administrator's tool, it could be installed in /usr/bin. In that case, do (as root) from the directory where stealth was compiled/unpacked:

    ./build install program

Alternatively, another default location may be specified in the INSTALL.im file or may be provided to the build script. E.g.,

    ./build install program /usr/local/bin/stealth

installing the binary program as /usr/local/bin/stealth.

5.2: Stealth command-line and policy file options

Short options are provided between parentheses, immediately following their long option equivalents. Option descriptions starting with (C) can only be used on the command-line, and are ignored when specified in the second section of the policy file.

--daemon (-d) <path>: (C) run as background (daemon) process. tt<path> specifies the absolute filename of the pid-file used for communication with the daemon process;
--dry-run: (C) no integrity scans or reloads are performed, but are assumed OK. Remaining tasks are normally performed;
--help (-h): (C) Display help information and exit;
--log (-L) <path>: log messages are appended to `path'. If path does not exist, it is first created;
--logmail: mail sent by stealth is logged (requires --log or --syslog);
--max-size <size>[BKMG]: files retrieved by GET commands may at most have <size> bytes (B), KBytes (K), MBytes (M), GBytes (G). The default size is 10M, the default unit is B.
--no-mail: mail is not sent. By default mail is sent as configured in the policy-file (--logmail can be specified independently from --no-mail);
--parse-policy-file (-p): (C) parse the policy file, after which stealth ends.
Specify once to see the numbered commands;
twice to see the policy file parsing steps as well.
Results are written to the std. output.
--random-interval (-i) <interval>[m]>: start the scan a random interval of <interval> seconds (or minutes if an `m' is appended (no blanks) to <interval>) following the delay specified at --repeat (see below). This option requires specification of the --repeat option;
--reload <pid-file>: (C) reloads the configuration and skip-files and restarts the scan of the stealth daemon process.
--repeat <seconds>: wake up and perform an integrity scan at interrupts or after <seconds> seconds (or minutes if an `m' is appended (no blanks) to <seconds>) after completing the previous integrity scan. The option --random-interval can be used to add a random delay to <seconds> until the next integrity scan is performed.
--rerun <pid-file>: start executing the integrity scan commands that are specifed in the stealth daemon process's policy file;
--resume <pid-file>: (C) resume a suspended stealth process, implies --rerun;
--run-command (-r) <nr>: (C) Only execute command number <nr> (natural number). Command numbers are shown by stealth ---parse-policy-file;
--skip-files (-s) <skippath>: all entries in skippath (specified using an absolute path) are skipped. Their integrity is not monitored. If an entry is already present in a log file then stealth once generates an IGNORING message in the mail sent to the address specified at EMAIL in the policy file. Each entry mentioned in filepath must be on a line of its own and must be specified using absolute paths. Entries ending in a slash are assumed to be directories whose full contents must be skipped. Other entries are interpreted as the path names of files to skip. Initial and trailing blanks, empty lines and lines having a # as their 1st non blank character are ignored.
--stdout (-o): messages are (also) written to the std. output stream (not available when for option --daemon);
--suspend <pid-file>: (C) suspends a currently active stealth process. Use --resume to re-activate an stealth daemon or --terminate to end an stealth daemon;
--syslog: write syslog messages;
--syslog-facility <facility>: syslog facility to use. By default facility DAEMON is used;
--syslog-priority <priority>: syslog priority to use. By default priority NOTICE is used;
--syslog-tag <tag>: <tag> specifies the identifier that is prefixed to syslog messages. By default the tag `STEALTH' is used, see also the next section;
--terminate <pid-file>: (C) terminate a currently active stealth process;
--time-stamp (-t) <type>: the time-stamps to use. By default UTC. To use the local time specify --time-stamp LT. The --time-stamp option does not apply to time-stamps generated by syslog (see also the next section);
--usage: (C) Display help information and exit;
--verbosity <value>: determines the amount of logged information. Requires options --log or --syslog. Possible values are:
0: nothing is logged
1: mode reports and policy commands
2: also: ipc commands and actions
3: also: integrity scan informative messages
--version (-v): (C) Display stealth's version information and terminate;

<pid-file>: absolute filename of a file that is used for communication with a stealth daemon process;
policy: path to the policy file;

Only one of the options --daemon, --reload, --rerun, --resume, --suspend, and --terminate can be specified. The options --reload, --rerun, --resume, --suspend, and --terminate ignore any other options.

The following options are still recognized for backward compatibility with stealth pre-3.00 versions and will be removed in future versions. They generate error messages suggesting alternatives:

--echo-commands (-e): echo commands to std error when they are processed; use --log instead.
--keep-alive: run as a daemon; use --daemon instead.
--only-stdout: scan report is written to stdout; use --stdout instead.
--quiet (-q): suppresses progress messages written to stderr; use --verbosity 0 instead.
--suppress <pid-file>: suppresses a currently active stealth process; use --suspend instead.

The following options were discontinued starting with stealth version 3.00.00:

--debug (option --verbosity or --dry-run could be used instead);
--no-child-processes;
--parse-config-file.

When specifying long options in policy files the initial hyphens should be omitted. Here are some examples:

    %%
    log /tmp/stealth.log
    verbosity 3

5.2.1: Rsyslog filtering

When using rsyslogd(1) property based filters may be used to filter syslog messages and write them to a file of your choice. E.g., to filter messages starting with the syslog message tag (e.g., STEALTH) use


:syslogtag, isequal, "STEALTH:"   /var/log/stealth.log
:syslogtag, isequal, "STEALTH:"   ~

Note that the colon is part of the tag, but is not specified with the syslog-tag option.

This causes all messages having the STEALTH: tag to be written on /var/log/stealth.log after which they are discarded. More extensive filtering is also supported, see, e.g., http://www.rsyslog.com/doc/rsyslog_conf_filter.html and http://www.rsyslog.com/doc/property_replacer.html

Time stamps written by rsyslogd are not controlled by stealth's --time-stamp option, but, e.g., by a TZ specification in /etc/default/rsyslog. Simply add the line

    export TZ=UTC

to /etc/default/rsyslog, followed by restarting rsyslogd configures rsyslogd to generate time stamps using UTC.

5.3: Construct one or more policy files

Here we assume that stealth is run by root, and that root wants to store information about the host client under the subdirectory /root/stealth/client.

Stealth reports should be sent to the user admin@elsewhere, who is only interested in a short notice of changes, as the full report can always be read elsewhere. For this a support-script was developed filtering the report generated by stealth down to its essentials.

As the sha1sum program on the client may be compromised, it is a good idea to transfer the client's sha1sum program to the controller first, verifying the integrity of that program at the controller, before trusting it to compute the sha1sums of the client's files. The same holds true for any libraries and support programs (like find) that are used intensively during integrity scans.

Sha1sum checks should be performed on all setuid and setgid files on the client, and in order to be able reach all files on client, root@controller is allowed to login to the root@client account using an ssh connection.

Furthermore, sha1sum checks should be performed on all configuration files, living under /etc and on the file /usr/bin/find which is used intensively to perform the checks.

The required policy file is now constructed according to the abovementioned requirements.

5.3.1: DEFINE directives

First we write some DEFINE directives simplifying complex command specifications:

    DEFINE  SSHCMD  /usr/bin/ssh root@client -T -q exec /bin/bash --noprofile
    DEFINE  EXECSHA1 -xdev -perm +u+s,g+s \( -user root -or -group root \) \ 
                    -type f -exec /usr/bin/sha1sum {} \;

The first DEFINE defines the ssh command to use: an ssh-connection will be made to the root account at the client.

The second DEFINE shows the arguments for find(1) when looking for all root setuid or setgid normal files. For all these files the sha1sum(1) program should be run.

5.3.2: USE directives

Next we specify some USE directives, to fit our specific, local, situation:

    USE BASE        /root/stealth/client
    USE EMAIL       admin@elswhere
    USE MAILER      /root/bin/stealthmail
    USE MAILARGS    "Client STEALTH report"
    USE SSH         ${SSHCMD}

All output is written under the /root/stealth/client directory;
Mail is sent to the user admin@elsewhere;
As mail program we use a filtering script (stealthmail), which is installed in /root/bin;
The script handles its own argument. As it can be used by stealth performing integrity scans on other clients as well, it is given an argument which can be used as e-mail subject, identifying the client-computer that has been integrity-scanned;
The ssh-command is defined by the SSHCMD. It's definition is used at the USE SSH specification;

Default values of all remaining USE directives are OK, and thus were not explicitly specified. They are:

    USE DD          /bin/dd
    USE DIFF        /usr/bin/diff
    USE REPORT      report
    USE SH          /bin/sh

5.3.3: Commands

First, we copy the client's sha1sum program to the controller. In practice, this should also include the shared object libraries that are used by sha1sum, as they might have become corrupted as well.

5.3.3.1: Obtaining the client's sha1sum program

First, the sha1sum program is copied to a local directory:

    GET /usr/bin/sha1sum /root/tmp

This command must succeed.

5.3.3.2: Checking the integrity of the client's sha1sum program

Next, we check the received sha1sum program, using our own:


    LABEL \nCheck the client's sha1sum program
    LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum

The LABEL command writes the label to the report file just before writing the sha1sum program's output.

The LOCAL command checks the sha1sum of the program copied from the client. The report is written on the file /root/stealth/client/local/sha1. If this fails, stealth terminates, alerting admin@elsewhere that the check failed. This is a serious event, as it indicates that either the controller's sha1sum is behaving unexpectedly or that the client's sha1sum program has unexpectedly changed.

The sha1sum program may have changed due to a normal upgrade. If so, admin@elsewhere will know this, and can (probably) ignore the warning. The next time stealth is run, the (now updated) SHA1 value is used, and it again compares the obtained SHA1 value to the one obtained for the downloaded sha1sum program.

5.3.3.3: Checking the client's /usr/bin/find program

The client normally uses its find command intensively: find is a great tool for producing reports about almost any conceivable combination of characteristics of sets of files. Of course, the client's find command must itself be OK, as well as the client's sha1sum program. Now that we know that the client's sha1sum program is OK, we can use it to check the client's /usr/bin/find program.

Note that the controller itself will not suffer any processing load here: only the client itself is taxed for checking the intergrity of its own files:

    LABEL \nchecking the client's /usr/bin/find program
    CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find

5.3.3.4: Checking the client's setuid/setgid files

Having checked the client's sha1sum and find programs, sha1 checksum checks should be performed on all setuid and setgid files on the client. For this we activate the sha1sum program on the client. In order to check the setuid/setgid files, the following command is added to the policy file:

    LABEL \nsuid/sgid/executable files uid or gid root on the / partition     
    CHECK LOG = remote/setuidgid /usr/bin/find / ${EXECSHA1}

5.3.3.5: Checking the configuration files in the client's /etc/ directory

Finally, the client's configuration files are checked. Some of these files change so frequently that we don't want them to be checked. E.g., /etc/adjtime, /etc/mtab. To check the configuration file, do:

    LABEL \nconfiguration files under /etc
    CHECK LOG = remote/etcfiles                         \ 
          /usr/bin/find /etc -type f -not -perm /6111   \ 
            -not -regex "/etc/\(adjtime\|mtab\)"        \ 
            -exec /usr/bin/sha1sum {} \;

5.3.4: The complete `policy' file

Here is the complete policy file we've constructed so far:

    DEFINE  SSHCMD  /usr/bin/ssh root@client -T -q exec /bin/bash --noprofile
    DEFINE  EXECSHA1 -xdev -perm +u+s,g+s \( -user root -or -group root \) \ 
                    -type f -exec /usr/bin/sha1sum {} \;

    USE BASE        /root/stealth/client
    USE EMAIL       admin@elswhere
    USE MAILER      /root/bin/stealthmail
    USE MAILARGS    "Client STEALTH report"
    USE SSH         ${SSHCMD}

    USE DD          /bin/dd
    USE DIFF        /usr/bin/diff
    USE REPORT      report
    USE SH          /bin/sh

    GET /usr/bin/sha1sum /root/tmp

    LABEL \nCheck the client's sha1sum program
    LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum

    LABEL \nchecking the client's /usr/bin/find program
    CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find
 
    LABEL \nsuid/sgid/executable files uid or gid root on the / partition     
    CHECK LOG = remote/setuidgid /usr/bin/find / ${EXECSHA1}

    LABEL \nconfiguration files under /etc
    CHECK LOG = remote/etcfiles                         \ 
          /usr/bin/find /etc -type f -not -perm /6111   \ 
            -not -regex "/etc/\(adjtime\|mtab\)"        \ 
            -exec /usr/bin/sha1sum {} \;

5.4: Running `stealth' for the first time

When stealth is now run, it creates its initial report files under root/stealth/client.

The first time stealth is run, it is usually run `by hand'. The initial run by hand probably benefits from the --stdout option, as it shows all executed commands on the standard output:

    stealth --stdout policy

Furthermore, the reports are initialized. Running stealth this way for the policy file constructed in the previous sections produces the following output (lines were wrapped to improve readability):

    GET /usr/bin/sha1sum /root/tmp
    LABEL \nCheck the client's sha1sum program
    LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum
    LABEL \nchecking the client's /usr/bin/find program
    CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find
    LABEL \nsuid/sgid/executable files uid or gid root on the / partition     
    CHECK LOG = remote/setuidgid /usr/bin/find / -xdev -perm +u+s,g+s 
            \( -user root -or -group root \) -type f 
            -exec /usr/bin/sha1sum {} \;
    LABEL \nconfiguration files under /etc
    CHECK LOG = remote/etcfiles                         /usr/bin/find /etc 
            -type f -not -perm /6111   -not -regex "/etc/\(adjtime\|mtab\)"
            -exec /usr/bin/sha1sum {} \;
    LOCAL /usr/bin/scp -q root@client:/usr/bin/sha1sum /root/tmp
    LABEL \nCheck the client's sha1sum program
    LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum
    LABEL \nchecking the client's /usr/bin/find program
    CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find
    LABEL \nsuid/sgid/executable files uid or gid root on the / partition     
    CHECK LOG = remote/setuidgid /usr/bin/find / -xdev -perm +u+s,g+s 
            \( -user root -or -group root \) -type f 
            -exec /usr/bin/sha1sum {} \;
    LABEL \nconfiguration files under /etc
    CHECK LOG = remote/etcfiles                        /usr/bin/find /etc 
            -type f -not -perm /6111   -not -regex "/etc/\(adjtime\|mtab\)"
            -exec /usr/bin/sha1sum {} \;

5.4.1: The mailed report

The /root/bin/stealthmail script is called with the following arguments:

    "Client STEALTH report" admin@elswhere

The mailed report contains information comparable to this:

    STEALTH (3.00.00) started at Wed, 20 Aug 2014 11:06:50 +0000
        
    Check the client's sha1sum program
    Initialized report on local/sha1
    
    checking the client's /usr/bin/find program
    Initialized report on remote/binfind
    
    suid/sgid/executable files uid or gid root on the / partition     
    Initialized report on remote/setuidgid
    
    configuration files under /etc
    Initialized report on remote/etcfiles

5.4.2: Files under /root/stealth/client

Under /root/stealth/client the following entries are now available:

local: below this directory the reports of the locally performed checks are found. Using our demo policy file, only one logfile is found here: sha1, containing the client's SHA1 checksum of its /usr/bin/sha1sum program:
```
    45251e259bfaf1951658a7b66c328c52  /root/tmp/sha1sum
```

remote: at this directory the reports of the remotely performed checks are found. Using our demo policy file, three files were created:

The file binfind, containing the checksum of the client's /usr/bin/find program:

    fc62fc774999584f1e29e0f94279a652  /usr/bin/find

The file etcfiles, containing the checksums of the client's configuration files under /etc (shown only partially):

    ced739ecb2c43a20053a9f0eb308b2b0  /etc/modutils/aliases
    a2322d7e2f95317b2ddf3543eb4c74c0  /etc/modutils/paths
    f9e3eac60200d41dd5569eeabb4eddff  /etc/modutils/arch/i386
    f07da2ebf00c6ed6649bae5501b84c4f  /etc/modutils/arch/m68k.amiga
    2893201cc7f7556160fa9cd1fb5ba56a  /etc/modutils/arch/m68k.atari
        ...
    bf73b4e76066381cd3caf80369ce1d0e  /etc/deluser.conf
    4cd70d9aee333307a09caa4ef003501d  /etc/adduser.conf.dpkg-save
    8c749353c5027d0065359562d4383b8d  /etc/gimp/1.2/gtkrc_user
    3ec404ec597ef5460600cccf0192f4d6  /etc/gimp/1.2/unitrc
    8c740345b891179228e3d1066291167b  /etc/gimp/1.2/gtkrc

The file setuidgid, containing the checksums of the client's setuid/setgid root files (shown only partially):

    030f3f84ec76a8181cca087c4ba655ea  /bin/login
    b6c0209547d88928f391d2bf88af34aa  /bin/ping
    5d324ad212b2ff8f767637ac1a8071ec  /bin/su
    344dbedc398d5114966914419ef53fcc  /usr/bin/wall
    27b045bd7306001f9ea31bc18712d8b7  /usr/bin/rxvt-xpm
    ...
    3567b18ffc39c2dc6ec0c0d0fc483f4f  /usr/lib/ssh-keysign
    3383a7955ac2406311e9aa51c6ac9c2c  /usr/X11R6/bin/X
    3c99ea0425c6e0278039e16478d2fb57  /usr/X11R6/bin/xterm
    d590f7f5b4d6ae61680692a52235d342  /usr/local/bin/setuidcall
    4c17203d7d91ec4946dea2f0ae365d5b  /sbin/unix_chkpwd

Of course, the checksums and the filenames shown are only for documentation purposes. At other systems different files and/or checksums will be reported.

The file /root/client/report New lines are always appended to the /root/client/report file. It will never shorten, unless shorten by the systems administrator at `controller'.

This file contains the following:

    STEALTH (3.00.00) started at Wed, 20 Aug 2014 11:06:50 +0000

    Check the client's sha1sum program
    Initialized report on local/sha1
    
    checking the client's /usr/bin/find program
    Initialized report on remote/binfind
    
    suid/sgid/executable files uid or gid root on the / partition     
    Initialized report on remote/setuidgid
    
    configuration files under /etc
    Initialized report on remote/etcfiles

This completes the information generated by stealth during its first run.

5.5: Subsequent `stealth' runs

5.5.1: All files unaltered

When stealth is subsequently run, it updates its report files under root/stealth/client. If nothing has changed, the log-files remain unaltered. Subsequent runs will, however, add some new info to the file /root/client/report:

    STEALTH (3.00.00) started at Wed, 20 Aug 2014 11:06:50 +0000
    
    Check the client's sha1sum program
    Initialized report on local/sha1
    
    checking the client's /usr/bin/find program
    Initialized report on remote/binfind
    
    suid/sgid/executable files uid or gid root on the / partition     
    Initialized report on remote/setuidgid
    
    configuration files under /etc
    Initialized report on remote/etcfiles
    
    STEALTH (3.00.00) started at Wed, 20 Aug 2014 11:13:38 +0000

Note that just one extra line was added: a timestamp showing the date/time of the last run. The systems administrator may rotate the report file every once in a while to reclaim some disk space.

5.5.2: Modifications occurred

Basically, three kinds of modifications are possible: additions, modifications, and removals. Here we'll show the effects all these changes have on stealth's output.

For illustrative purposes, the following changes were made to the client's files:

/etc/motd was changed
the file timezone~ was removed
the file /etc/motd.org was created

Next, stealth was again run, producing the following output:

The following new info is now added to file /root/client/report:

    STEALTH (3.00.00) started at Wed, 20 Aug 2014 11:13:38 +0000
    
    configuration files under /etc
    ADDED: /etc/motd.org
        < 945d0b8208e9861b8f9f2de155e619f9  /etc/motd.org
    MODIFIED: /etc/motd
        < 7f96195d5f051375fe7b523d29e379c1  /etc/motd
        > 945d0b8208e9861b8f9f2de155e619f9  /etc/motd
    REMOVED: /etc/timezone~
        > 6322bc8cb3ec53f5eea33201b434b74b  /etc/timezone~

Note that all changes were properly detected and logged in the file /root/client/report.

Furthermore, a matching report was sent by mail:

    STEALTH (3.00.00) started at Wed, 20 Aug 2014 11:13:38 +0000

    configuration files under /etc
    ADDED: /etc/motd.org
        < 945d0b8208e9861b8f9f2de155e619f9  /etc/motd.org
    MODIFIED: /etc/motd
        < 7f96195d5f051375fe7b523d29e379c1  /etc/motd
        > 945d0b8208e9861b8f9f2de155e619f9  /etc/motd
    REMOVED: /etc/timezone~
        > 6322bc8cb3ec53f5eea33201b434b74b  /etc/timezone~

Note that the report only shows the info that was added to the /root/client/report file.

The report itself could be beautified further. E.g., I use the following script to mail the report to the addressee:


    #!/bin/bash
    
    NAME=`basename $0`
    
    tee /root/stealth/lastreport/$NAME | egrep -v '^([[:space:]]|[[:space:]]*$)' |
        sort | uniq | mail -s $1 $2

For the client computer, this little script writes the mailed report on a file /root/stealth/lastreport/client, overwriting its previous contents, removes all lines beginning with blanks (thus trimming away the diff-generated lines), and e-mails the sorted and uniqed lines using mail. The addressee (admin@elsewhere) then receives the following information:


    ADDED: /etc/motd.org
    MODIFIED: /etc/motd
    REMOVED: /etc/timezone~
    STEALTH (3.00.00) started at Wed, 20 Aug 2014 11:13:38 +0000
    configuration files under /etc

In practice this provides me with all the information I need if something out of the ordinary has happened.

Finally, the file
```
    /root/stealth/client/remote/etcfiles
```
was recreated, saving the old file as
```
    /root/stealth/client/remote/etcfiles.20021028-112851
```
As remarked earlier (see section 4.3), many logfile.YYMMDD-HHMMSS files could eventually accumulate. As discussed in section 4.3, it might be considered to remove old log files every now and then.

5.5.3: Failing LOCAL commands

If the client's sha1sum program itself is altered, a serious situation has developed. In that case, further actions by stealth would be suspect, as their results might easily be currupted. Additional checks will be performed, but a warning is generated on the report file (and in the mail sent to admin@elsewhere:

    STEALTH (3.00.00) started at Wed, 20 Aug 2014 11:13:38 +0000

    Check the client's sha1sum program
    MODIFIED: /root/tmp/sha1sum
        < fc62fc774999584f1e29e0f94279a652  /root/tmp/sha1sum
        > 45251e259bfaf1951658a7b66c328c52  /root/tmp/sha1sum
    
    *** BE CAREFUL *** REMAINING RESULTS MAY BE FORGED
    
    configuration files under /etc
    REMOVED: /etc/motd.org
        > 945d0b8208e9861b8f9f2de155e619f9  /etc/motd.org
    MODIFIED: /etc/motd
        < 945d0b8208e9861b8f9f2de155e619f9  /etc/motd
        > 7f96195d5f051375fe7b523d29e379c1  /etc/motd

(The report shows the removal of the previously added file motd.org, and the modification of motd. These are real, as the original motd file, modified earlier, was restored at this point).

5.5.4: Skipping (some) integrity checks

Some files or directories may not require integrity checks. Automated processes may modify files which are not threatening the proper functioning of running programs or processes. In those cases a file can be prepared holding the absolute paths of entries to be skipped. Each entry should appear on a line of its own without any additional information.

Stealth can be informed about this file using the --skip-files skippath option. The file holding the paths of the entries to be skipped should be specified using absolute paths, using one entry per line. Initial and trailing blanks, empty lines and lines having a # as their first non blank character are ignored.

Here is an example:

    stealth -e --skip-files /root/stealth/remote/skipping remote.pol

If an entry /etc/skipme appears in the current logs which is thereafter added to the skippath file then the mail generated by stealth once contains a line like the following:

    SKIPPING: /etc/skipme
        > a7695bb2d019e60988e757a4b692acfe  /etc/skipme

The reported hash-value is the hash-value at the time of the stealth-run reporting the SKIPPING message.

Entries ending in a slash are assumed to be directories whose contents must (recursively) be skipped.

5.6: Automating repeated `stealth' runs

To automate stealth's integrity scans, a file /etc/cron.d/stealth could be created, containing a line like


    2,17,32,47 * * * *  root    test -x /usr/bin/stealth && \ 
                                /usr/bin/stealth -q /root/stealth/client.pol

This starts stealth 2 minutes after every hour. In this example the ssh-key must not require a passphrase, as crontab(1) cannot provide passphrases of ssh-keys. Ssh-keys requiring passphrases can, however, be used if repeated stealth runs are controlled by a program like ssh-cron(1).

In general, randomized events are harder to notice. For this stealth offers the --repeat and --random-interval options. Both options expect an argument in seconds (or in minutes, if an m is appended to the specification). After each integrity scan the next integrity scan starts after the time interval specified by the --repeat option plus a random time value selected from the time interval specified by the --random-interval option. For example, the stealth daemon started by the following command repeatedly performs integrity scans between two and five minutes after the last integrity scan completed:

    stealth -d /var/run/client.pid -r 2m -i 3m /root/stealth/client.pol

5.7: Report File Rotation

When stealth performs integrity scans it appends information to the report file. This file therefore eventually grows to a large size, and the systems manager controlling stealth might want to rotate the report file every once in a while (e.g., using a program like logrotate(1), also see the upcoming section 5.7.2). To ensure that no log-rotation takes place while stealth is busy performing integrity scans (thus modifying the report file) the options --suspend and --resume were implemented. Both options require the process-ID file of currently active stealth process as their argument.

For example, if a stealth process was once started using the command

    stealth --daemon /var/run/small.pid --repeat 900 \ 
                    /var/stealth/policies/small.pol

then the --suspend and --resume commands for this process should be called as:

    stealth --suspend /var/run/small.pid
    stealth --resume /var/run/small.pid

The stealth process identified in the files provided as arguments to the --suspend and --resume options is called the daemon stealth process below.

The --suspend option has the following effect:

If the daemon stealth process is currently processing its policy file, performing an integrity scan, then the currently executing policy file command is completed, whereafter further commands are ignored, except for --resume (see below) and --terminate.
Any scheduled integrity scans following the --suspend command are ignored by the daemon stealth process;
The daemon stealth process writes a message that it is being suspended to the report file and then processes the report file as usual.

Now that the report file will no longer be affected by the daemon stealth process, log-rotation may take place. E.g., a program like logrotate(1) allows its users to specify a command or script just before log-rotation takes place, and `stealth --suspend pidfile' could be specified nicely in such a pre-rotation section.

The --resume option has the following effect:

The daemon stealth process resumes its activities by performing another integrity scan. Thus, --resume implies --rerun.
Any scheduled integrity scans following the --resume command are again honored by the daemon stealth process.

Note that, once --suspend has been issued, all commands except --resume and --terminate are ignored by the daemon stealth process. While suspended, the --terminate command is acknowledged as a `emergency exit' which may or may not interfere with, e.g., an ongoing log-rotation process. The daemon stealth process should not normally be terminated while it is in its suspended mode. The normal way to terminate a stealth process running in the background is:

Wait for the daemon stealth process to complete an ongoing series of integrity scan commands;
Issue the `stealth --terminate pidfile' command.

5.7.1: Status file cleanup

Whenever stealth is run and it encounters a modified situation the already existing status file summarizing that particular situation is saved and a new status file is created. Eventually, this will result in many status files. While report files can be rotated, it is pointless to rotate old status files, since they are never modified. Instead, status files exceeding a certain age could be removed and more recent files might be zipped to conserve space. In stealth's binary distribution the file /usr/share/doc/stealth/usr/bin/stealthcleanup is provided which can be used to perform such a cleanup. The script expects one argument: a resource file defining the following shell variables:

directories: the directories below which the status files are found;
gzdays: the number of days a status file must exist before it is compressed using gzip(1);
rmdays: the maximum age (in days) of compressed status files. Files exceeding this age are removed using rm(1).

Here is the stealthcleanup script as contained in the binary distribution's /usr/share/doc/stealth/usr/bin directory:

#!/bin/bash

usage()
{
    echo "
Usage: $0 [-v] rc-file
Where:
    -v: Show the actions that are performed
    rc-file: resource file defining:
            \`directories' - one or more directories containing status files
            \`gzdays'      - number of days status files may exist before they
                             are compressed
            \`rmdays'      - number of days gzipped status files may exist
                             before they are removed. 
"
    exit 1
}


error()
{
    echo "$*" >&2
    exit 1
}

if [ "$1" == "-v" ]
then
    verbose=1
    shift 1
else
    verbose=0
fi

[ $# == 1 ] || usage

# now source the configuration file
. $1           

for x in $directories
do
    cd $x || error "\`$x' must be a directory"
    if [ $verbose -eq 1 ]
    then
        echo "
cd $x"
    fi

    if [ $verbose -eq 1 ]
    then
        echo \
    /usr/bin/find ./ -mtime +$rmdays -type f -regex '.*[0-9]+-[0-9]+\.gz' \
        -exec /bin/rm {} \;
    fi
    /usr/bin/find ./ -mtime +$rmdays -type f -regex '.*[0-9]+-[0-9]+\.gz' \
        -exec /bin/rm {} \;

    if [ $verbose -eq 1 ]
    then
        echo \
    /usr/bin/find ./ -mtime +$gzdays -type f -regex '.*[0-9]+-[0-9]+' \
        -exec /bin/gzip {} \;
    fi 
    /usr/bin/find ./ -mtime +$gzdays -type f -regex '.*[0-9]+-[0-9]+' \
        -exec /bin/gzip {} \;
done

exit 0

Assuming that the status files are written in /var/stealth/target/local and /var/stealth/target/remote; that status file should be compressed when older than 2 days and removed after 30 days, the resource file is:

directories="
    /var/stealth/target/local
    /var/stealth/target/remote
    "

rmdays=30
gzdays=3

Furthermore assuming that the resourcefile is installed in /etc/stealth/cleanup.rc and the stealthcleanup script itself in /usr/bin/stealthcleanup, the stealthcleanup script could be called as follows:

    /usr/bin/stealthcleanup /etc/stealth/cleanup.rc

Note that stealthcleanup may be called whether or not there are active stealth processes, as stealth does not use status files anymore once they have been written.

5.7.2: Using `logrotate' to control report- and status files

A program like logrotate(1) allows its users to specify a command or script immediately following log-rotation, and `stealth

--resume
pidfile

' could be specified nicely in such a post-rotation section.

Here is an example of a specification that can be used with logrotate(1). Logrotate (on Debian systems) keeps its configuration files in /etc/logrotate.d, and assuming there is a host target, whose report file is /var/stealth/target/report, the required logrotate(1) specification file (e.g., /etc/logrotate.d/target could be:

/var/stealth/target/report {
    weekly
    rotate 12
    compress
    missingok
    prerotate
        /usr/bin/stealth --suppress /var/run/stealth.target
    endscript
    postrotate
        /usr/bin/stealth --resume /var/run/stealth.target
    endscript 
}

Using this specification file, logrotate(1) will

perform weekly rotations of the report file;
keep up to 12 rotated files, compressing them using gzip(1);
suspend the stealth daemon, before rotating its report file; suppressed;
following the rotation, stealth's actions are resumed.

Note thet stealth --resume xxx always initiates another file integrity scan.