Watching a server for modified files with Wazuh

Imagine logging into your own or your client’s server and finding a PHP file in the document root that neither you nor your colleagues ever uploaded. You open it in a text editor, and it is a webshell: a script an attacker drops in to get full control of the website, or, even worse, the whole server. Your existing file monitoring tool tells you the file appeared at 04:12. It does not tell you what put it there, so you are left working out whether it was a deploy or an intrusion by hand, out of SSH logs and deploy history, hours after it mattered.

This post builds the file integrity monitor that answers this question for you. It watches every file in every folder you specify on the server, records which process made each change, and emails you within seconds when the web server process writes something executable. It can also summarize every other less critical change, on a schedule, if needed.

We will use Wazuh (opens in a new tab), an open-source security platform that includes a file integrity monitor, and configure it to watch the web server’s document root. When a file is modified, Wazuh will log the event, including the process that made the change, and send an alert email to you.

A complete Wazuh setup consists of installing the Wazuh manager, installing the Wazuh agent on the server to be monitored, and configuring the file integrity monitoring rules. The Wazuh agent will monitor the specified directories and report any changes to the Wazuh manager, which will then trigger an alert if a critical change is detected. There is also a Wazuh dashboard that can be used to view the alerts and logs in a user-friendly interface, but it requires a heftier server to run, so we will focus on the manager and agent setup in this post.

Our setup will consist of 2 VMs:

I tested everything below on Wazuh 4.14.x and Ubuntu 22.04. You will need root access on both machines and an SMTP account that can send email. The alert rules also make two assumptions about how the website is run: that new code arrives on the server over SFTP as a dedicated deploy user, and that PHP runs through php-fpm. If you are not sure whether that is true for your server, do not worry, section 5 shows you how to check.

Throughout the post, /home/user/webapps/appname stands in for the document root of your website, so swap it for your own path. <MANAGER_IP>, <SITE> and the mail settings are placeholders too.

1. Install the manager

The manager is the piece that receives reports from the agents, decides which changes matter, and sends the emails. It lives on its own VM, and without the indexer and dashboard it is a light workload.

If you have looked at the Wazuh website, you probably saw a one-line installer called wazuh-install.sh -a. Do not use it here. It installs the indexer and the dashboard as well, and neither of them is part of this setup.

apt-get install gnupg apt-transport-https
curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/wazuh.gpg --import && chmod 644 /usr/share/keyrings/wazuh.gpg
echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" | tee -a /etc/apt/sources.list.d/wazuh.list
apt-get update
apt-get install wazuh-manager
systemctl daemon-reload && systemctl enable wazuh-manager && systemctl start wazuh-manager
systemctl status wazuh-manager

There is one rule about versions that is worth knowing right away: the manager has to be at the same version as every agent reporting to it, or newer. If an automatic package update pushes an agent ahead of the manager one night, the agent stops being able to talk to it. To avoid that, we disable the Wazuh repository and put the package on hold, so nothing gets upgraded unless you decide to do it yourself:

sed -i "s/^deb /#deb/" /etc/apt/sources.list.d/wazuh.list
apt-get update
echo "wazuh-manager hold" | dpkg --set-selections

Next, the firewall. Agents send their data to the manager on port 1514/tcp and register themselves on port 1515/tcp, so both ports have to be open, inbound, from each agent’s IP address. If your manager sits behind a hosting control panel, add these rules in the panel rather than with ufw directly, because the panel tends to overwrite raw firewall rules on its own. Remember that every agent you enroll later on needs its IP added here as well.

ss -tlnp | grep -E '1514|1515'

The last step on the manager is to switch off two features that are enabled by default but need the indexer we did not install: vulnerability detection and the indexer connection. If you leave them on, they keep trying to reach an indexer that does not exist, which wastes CPU in wazuh-modulesd and floods ossec.log with lines like indexer-connector: WARNING ... retrying. Open /var/ossec/etc/ossec.conf and set both to no:

<vulnerability-detection>
  <enabled>no</enabled>
</vulnerability-detection>

<indexer>
  <enabled>no</enabled>
</indexer>

systemctl restart wazuh-manager

2. Install an agent

The agent is the part that actually sits on the server you want to protect and watches the files. On that server, run the same repository commands from section 1 first, then install the agent package. Notice that the manager’s IP address is passed in as an environment variable on the install line itself, which registers the agent against the manager in one go:

apt-get install gnupg apt-transport-https
curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/wazuh.gpg --import && chmod 644 /usr/share/keyrings/wazuh.gpg
echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" | tee -a /etc/apt/sources.list.d/wazuh.list
apt-get update

# installs and registers against the manager in one step
WAZUH_MANAGER="<MANAGER_IP>" apt-get install wazuh-agent

# pin it, same as the manager
sed -i "s/^deb /#deb/" /etc/apt/sources.list.d/wazuh.list
apt-get update
echo "wazuh-agent hold" | dpkg --set-selections

systemctl daemon-reload && systemctl enable wazuh-agent && systemctl start wazuh-agent
systemctl status wazuh-agent

A small trap here: WAZUH_MANAGER is only read during the installation. Setting it afterwards does nothing, so if you forgot it, reinstall the package rather than trying to export the variable later.

Once the agent is running, go back to the manager and ask it which agents it knows about:

/var/ossec/bin/agent_control -l

Your new agent should be listed as Active. If it says anything else, check the manager’s firewall first, since that is the usual cause, and then look at /var/ossec/logs/ossec.log on the agent. Do not go any further until that line reads Active, because nothing in the rest of this post will work without it.

3. auditd and the start order

Before we tell Wazuh what to watch, we need to talk about how it watches, because this is where the “which process did it” part comes from.

Wazuh can watch files live in two ways. The first, called realtime, uses a Linux feature called inotify. It is simple and reliable, and it tells you that a file changed, but nothing more. The second, called whodata, reads from the Linux audit subsystem instead. The audit subsystem sits inside the kernel and can record, for every write to a watched directory, which user and which process performed it. That is the information we want, so we will use whodata, and that means the audit daemon, auditd, has to be installed and running on the agent.

apt-get update && apt-get install -y auditd audispd-plugins
systemctl status auditd --no-pager | head -3   # active (running)
auditctl -l                                    # "No rules" is correct here

Seeing No rules is fine at this point. Wazuh adds its own audit rules when it starts watching directories. The performance overhead is small, because those rules only audit writes to the application directories, not all the read traffic the website generates while serving pages.

There is one fragile moment in all of this. When the agent starts, it attaches itself to auditd over a socket. If auditd is restarting, or in a stale state, at that exact moment, the attachment fails. Wazuh then quietly drops back to realtime mode, without any obvious error, and you are back to knowing that a file changed but not what changed it. To prevent that, we add a small systemd drop-in file that forces auditd to be restarted right before the agent, every time. After this, systemctl restart wazuh-agent is always safe:

mkdir -p /etc/systemd/system/wazuh-agent.service.d
cat > /etc/systemd/system/wazuh-agent.service.d/auditd-first.conf << 'EOF'
[Unit]
After=auditd.service
Wants=auditd.service

[Service]
ExecStartPre=-/usr/bin/systemctl restart auditd.service
EOF
systemctl daemon-reload

The - in front of the ExecStartPre command tells systemd to carry on even if the auditd restart fails. That is deliberate: an agent that runs without attribution is still better than no agent at all. What the drop-in cannot help with is auditd restarting on its own, for example during a package update. Whenever that happens, run systemctl restart wazuh-agent afterwards and confirm that Whodata engine started shows up in /var/ossec/logs/ossec.log.

4. Choose what to watch

The file integrity module inside Wazuh is called syscheck, and it is configured through a <syscheck> block that already exists in /var/ossec/etc/ossec.conf on the agent. Edit that existing block rather than adding a second one, because two <syscheck> blocks in the same file lead to confusing results.

Here is the block I use. Read the comments, because they explain why each part is there:

<syscheck>
    <disabled>no</disabled>
    <!-- Daily full scan every 86400 seconds (24 hours). -->
    <frequency>86400</frequency>
    <scan_on_start>yes</scan_on_start>
    <alert_new_files>yes</alert_new_files>

    <!-- The OS directories Wazuh watches by default, left off here: this install is about the webapp roots, and /etc churns on its own.
    <directories>/etc,/usr/bin,/usr/sbin</directories>
    <directories>/bin,/sbin,/boot</directories> -->

    <!-- Each webapp root gets its OWN directive. Do NOT watch a shared parent covering multiple sites. See the warning below. -->
    <directories check_all="yes" whodata="yes" report_changes="yes">/home/user/webapps/appname</directories>

    <!-- Bulky media and generated dirs: fully monitored, but NO report_changes, so no content copies are stored. Adjust to your site. -->
    <directories check_all="yes" whodata="yes">/home/user/webapps/appname/public/products</directories>
    <directories check_all="yes" whodata="yes">/home/user/webapps/appname/public/cpresources</directories>
    <directories check_all="yes" whodata="yes">/home/user/webapps/appname/public/images</directories>

    <!-- runtime noise excluded entirely -->
    <ignore>/home/user/webapps/appname/storage</ignore>
    <ignore>/home/user/webapps/appname/vendor</ignore>
    <ignore type="sregex">.log$|.swp$</ignore>

    <!-- binaries: never store or diff their content -->
    <nodiff type="sregex">.png$|.PNG$|.jpg$|.JPG$|.jpeg$|.webp$|.gif$|.ico$|.svg$|.woff$|.woff2$|.ttf$|.eot$|.mp4$|.webm$|.pdf$|.zip$|.gz$</nodiff>

    <!-- cap stored copies at 1MB per file, the 50MB default fills the 1GB quota fast -->
    <diff>
      <file_size>
        <enabled>yes</enabled>
        <limit>1MB</limit>
      </file_size>
    </diff>
</syscheck>

A quick glossary for the attributes, since they are the heart of the whole thing. whodata="yes" is what gives you the process name behind each change. report_changes="yes" tells Wazuh to keep a copy of each watched file, so that when the file changes, the alert can include a diff showing exactly what changed. <ignore> removes a path from monitoring completely, while <nodiff> keeps watching a file but never prints its contents in an alert, which is what you want for images and archives.

To apply the configuration, restart the agent. At the same time, delete the stored file copies, so they are rebuilt under the rules you just wrote instead of being kept from the old ones:

systemctl stop wazuh-agent
rm -rf /var/ossec/queue/diff/*
systemctl start wazuh-agent

If you would like to prove that file watching works before adding attribution on top of it, you can use realtime="yes" on those <directories> lines instead of whodata="yes". Realtime does not need auditd and alerts on the same changes, it just cannot tell you which process made them. Once you are happy that alerts arrive, switch the attribute back to whodata="yes" and restart.

Before you adapt this block to your own layout, there are six things you should know:

• Give every webapp root its own <directories> line, and never watch the parent folder that contains several sites. This one bit me. When I watched the parent, changes made from a shell were audited normally, but writes made by php-fpm in the very same tree produced no kernel audit events at all: no alerts, and nothing in the logs to say something was wrong. Giving each site its own line fixed it immediately, and as a bonus it reduces audit volume, because a parent watch also audits every site’s cache churn.

  <directories check_all="yes" whodata="yes" report_changes="yes">/home/user/webapps/site-one</directories>
  <directories check_all="yes" whodata="yes" report_changes="yes">/home/user/webapps/site-two</directories>

• alert_new_files defaults to no, which means brand new files would never alert. A new file is exactly what a webshell is, so this has to be yes.

• A more specific <directories> entry wins over a broader one for the paths beneath it. That is how the media directories above stay fully monitored while dropping the stored copies: they are inside the app root, but their own line does not carry report_changes.

• <nodiff> does not stop copies from being stored. It only hides the diff in the alert. If you want to stop storing a copy of a file, you have to remove report_changes from the directory it lives in.

• Every restart has a blind window. When the agent starts, it first runs a full scan of every watched file and records the result as the baseline, which is Wazuh’s stored fingerprint of what “normal” looks like. Whodata only starts once that scan is finished. Anything created in between gets recorded into the baseline as if it had always been there, and will never alert. So after every restart, wait until you see Whodata engine started in ossec.log before you test anything.

• Restart after every config change, and check systemctl status wazuh-agent afterwards. A single typo in the XML stops the agent from starting at all.

Confirm it came up clean

tail -f /var/ossec/logs/ossec.log

What you are waiting for is the initial scan finishing, followed by the watcher starting. With realtime="yes" it looks like this:

2026/07/23 01:42:07 wazuh-syscheckd: INFO: (6009): File integrity monitoring scan ended.
2026/07/23 01:42:07 wazuh-syscheckd: INFO: FIM sync module started.
2026/07/23 01:42:09 wazuh-syscheckd: INFO: (6012): Real-time file integrity monitoring started.

With whodata="yes" the third line should read (6019) Whodata engine started instead. If you asked for whodata but got the realtime line, the agent could not attach to auditd, and section 11 covers how to fix that.

Two more checks on the agent:

grep -iE "whodata|who-data|audit" /var/ossec/logs/ossec.log | tail -15
auditctl -l

auditctl -l should now list one wazuh_fim rule per watched directory. That is the kernel itself confirming it is watching the paths you asked for.

Keep the diff store small

The stored copies of files live in /var/ossec/queue/diff, and there is a 1GB quota on that directory. When it fills up, alerts still arrive, but with the message Unable to calculate diff due to 'disk_quota' limit, meaning you still learn that a file changed but you no longer see what changed inside it. A few commands show you how much space is used and by what:

du -sh /var/ossec/queue/diff
du -h --max-depth=3 /var/ossec/queue/diff | sort -rh | head -15
ls /var/ossec/queue/diff/file | wc -l

Ideally every stored copy is under 1MB, and the number of stored files is roughly the number of code and config files your site has, so a few thousand. If you see something much larger, it is almost always a bulky directory that still has report_changes on it. Remove the attribute for that directory, then purge and restart as shown above.

If you want to look inside the baseline database, note that it is locked while the agent runs, so copy it first:

cp /var/ossec/queue/fim/db/fim.db /tmp/f.db && sqlite3 /tmp/f.db "SELECT count(*) FROM file_entry WHERE path LIKE '/home/user/webapps/appname%';"

5. Fingerprint your own stack

The alert rules we write in section 6 match on the name of the process that wrote a file, and that name differs from one server to another. On my servers it is php-fpm; on yours it might be apache2, or node, or something else entirely. If you guess wrong, the rules will load without complaint and simply never match anything. So before writing any rules, spend ten minutes watching what your own server actually reports. Do this once per server, and once for every PHP-serving site on it.

On the manager, watch the alert log:

tail -f /var/ossec/logs/alerts/alerts.log

Now upload a file to the site over SFTP, as the deploy user. An alert should appear naming /usr/lib/openssh/sftp-server as the process, with /usr/sbin/sshd as its parent. That is what a legitimate deploy looks like.

Next, simulate the attack. We want a file written through the website, by the web server process, rather than from your shell. The easiest way is to drop a tiny PHP script that writes another file when you visit it in a browser:

cat > /home/user/webapps/appname/public/writetest.php << 'EOF'
<?php
file_put_contents(__DIR__ . '/web-written-test.php', "<?php // simulated webshell " . date('c') . "\n");
echo "done";
EOF
chown user:user /home/user/webapps/appname/public/writetest.php
curl -s https://<SITE>/writetest.php     # must return "done"

This alert should name a process ending in php-fpm, or whatever your stack uses. Write that name down, you will need it in the next section. If the alert arrives with no audit block and no process name at all, attribution is not working, and none of the rules below can match. Fix that with section 11 before going on.

Compare the user field between the two tests as well. On panel-managed hosting you will usually see the same application user for both the SFTP upload and the web write, which is exactly why these rules key on the process name and not on the user.

Finally, delete both test files. The deletions will fire rule 553, which is fine. More importantly, leaving a script in your document root that can write files on request hands an attacker the very thing you are trying to detect.

6. Write the rules

Now we teach the manager what to care about. Wazuh rules live on the manager, and custom ones go in /var/ossec/etc/rules/local_rules.xml.

A little background helps here. Every alert in Wazuh has a level from 0 to 15, which is how important it is. Wazuh ships with rules that already fire on file changes: rule 550 for a modified file, 553 for a deleted one, and 554 for a new one, all at low levels. Our three rules build on top of those using if_sid, which means “only consider this rule if one of these rules already matched”, and they add the attribution on top. The last rule that matches an event is the one that decides its final level.

<group name="syscheck,site_fim,">

  <!-- Base: web server process wrote a file (add, modify or delete). Extend the process list when enrolling servers with other stacks, e.g. php-fpm|apache2 -->
  <rule id="100100" level="7">
    <if_sid>550,553,554</if_sid>
    <field name="process_name">php-fpm</field>
    <description>Web process (php-fpm) wrote file: $(file)</description>
  </rule>

  <!-- Escalation: executable or config content is the webshell signature -->
  <rule id="100101" level="12">
    <if_sid>100100</if_sid>
    <field name="file">.php$|.phtml$|.phar$|.htaccess$|.sh$|.cgi$|.pl$|.py$|.env$|.js$</field>
    <description>CRITICAL: Web process wrote executable file $(file) - possible webshell</description>
  </rule>

  <!-- Downgrade: CMS regenerating its own assets through php-fpm, which is expected. Adjust the path fragment to your CMS. -->
  <rule id="100102" level="7">
    <if_sid>100101</if_sid>
    <field name="file">cpresources</field>
    <description>CMS assets regenerated by web process (expected)</description>
  </rule>

</group>

systemctl restart wazuh-manager

Reading them top to bottom: rule 100100 catches any file that php-fpm wrote, whatever it is. Rule 100101 raises that to level 12 when the filename looks executable or like a config file, because that is the webshell signature, and it is the only rule that will send an instant email. Rule 100102 then takes one known-good case, the CMS regenerating its own assets through php-fpm, and drops it back down to level 7, since that is normal behaviour and should not wake anyone up.

One thing that cost me time: match on the decoded field names, which are process_name, user_name and file. If you look in alerts.json you will see nested names like audit.process.name and path, and it is tempting to use those. They fail silently: the rules load and never match. The Wazuh documentation describes this under “creating custom FIM rules” and “mapping FIM fields to Wazuh alerts”, in the file integrity monitoring section.

Now repeat the two tests from section 5. The web write should produce rule 100101 at level 12, and the SFTP upload should still be a plain 554 at level 5. If the web write still lands on 554, the process name in rule 100100 does not match what section 5 showed you.

7. The instant email

Everything so far ends up in a log file on the manager. This section makes the critical case reach your inbox within seconds.

To send email from the command line we use swaks, a small tool that talks to an SMTP server directly, so nothing else needs a mail setup. Install it on the manager:

apt-get install -y swaks

I recommend sending one email by hand with swaks first, using your SMTP details, before wiring it into Wazuh. If the automated emails go quiet later, you will then know whether it is the mail account or the Wazuh side that broke.

The SMTP credentials and the recipient addresses go in one file that only root can read:

cat > /etc/wazuh-mail.env << 'EOF'
# Default recipient, also used for agents with no per-agent mapping
MAIL_TO="[email protected]"
MAIL_FROM="wazuh@<MAIL_DOMAIN>"
SMTP_SERVER="smtp.example.net:587"
SMTP_USER="<SMTP_LOGIN>"
SMTP_PASS="<SMTP_PASSWORD>"

# Per-agent digest recipients: hostname with - and . replaced by _
# Comma-separate for multiple recipients.
MAIL_TO_vm_agent_one="[email protected]"
MAIL_TO_vm_agent_two="[email protected],[email protected]"
EOF
chmod 600 /etc/wazuh-mail.env

A word of caution: never debug a script that sources this file with bash -x, because the trace output prints your SMTP password.

Wazuh has a feature called active response, which runs a script of your choosing whenever a particular rule fires. It hands the script the full alert as JSON on standard input. Our script pulls out the useful fields and mails them:

cat > /var/ossec/active-response/bin/critical-mail.sh << 'EOF'
#!/bin/bash
source /etc/wazuh-mail.env
mkdir -p /var/log/wazuh
read -r INPUT
ALERT=$(echo "$INPUT" | python3 -c "
import sys, json
d = json.load(sys.stdin)
a = d.get('parameters', {}).get('alert', {})
s = a.get('syscheck', {})
au = s.get('audit', {})
print(a.get('agent',{}).get('name','unknown'))
print(f\"Rule: {a.get('rule',{}).get('description','?')}\")
print(f\"File: {s.get('path','?')}\")
print(f\"Event: {s.get('event','?')}\")
print(f\"User: {au.get('user',{}).get('name','?')}\")
print(f\"Process: {au.get('process',{}).get('name','?')}\")
print(f\"Agent: {a.get('agent',{}).get('name','?')}\")
print(f\"Time: {a.get('timestamp','?')}\")
" 2>/dev/null)
AGENT=$(echo "$ALERT" | head -1)
BODY=$(echo "$ALERT" | tail -n +2)
swaks --to "$MAIL_TO" --from "$MAIL_FROM" \
      --server "$SMTP_SERVER" --auth LOGIN \
      --auth-user "$SMTP_USER" --auth-password "$SMTP_PASS" --tls \
      --header "Subject: [CRITICAL][$AGENT] Wazuh: possible webshell detected" \
      --body "$BODY" >> /var/log/wazuh/critical-mail.log 2>&1
EOF
chmod 750 /var/ossec/active-response/bin/critical-mail.sh
chown root:wazuh /var/ossec/active-response/bin/critical-mail.sh

The script prints the agent name on its own first line and then splits it off into AGENT, which is how the hostname ends up in the subject line. That matters once several servers report to one manager. Pay attention to the two permission lines at the end as well: if active response cannot execute the script, you get no mail, no error, and not even a log file to look in.

If you would like the email to include the actual diff, add this inside the Python block, right after the Time line:

diff = s.get('diff')
if diff:
    if len(diff) > 4000:
        diff = diff[:4000] + '\n... [diff truncated]'
    print(f"\nWhat changed:\n{diff}")
else:
    print("\nWhat changed: (no diff available for this event)")

Keep in mind that new files have no diff, since there is nothing to compare them against. Also, real attacker code in an email body can trip the malware scanner at your mail provider. If critical emails stop arriving after you enable the diff, retest with a harmless change before assuming something broke.

Two shell traps I ran into: redirecting to >> /path/file.log silently kills the command when the directory does not exist, which is what the mkdir -p at the top is for. And the AGENT= and BODY= lines have to stay outside the python3 -c "…" block, because bash pasted into the Python string fails quietly while stderr is being discarded.

Now register the script in /var/ossec/etc/ossec.conf on the manager, inside the <ossec_config> block. The first part gives the script a name, the second says which rule should trigger it:

<command>
  <name>critical-mail</name>
  <executable>critical-mail.sh</executable>
  <timeout_allowed>no</timeout_allowed>
</command>

<active-response>
  <command>critical-mail</command>
  <location>server</location>
  <rules_id>100101</rules_id>
</active-response>

systemctl restart wazuh-manager

<location>server</location> means the script runs on the manager, not on the agent, which keeps your SMTP credentials on a single host.

Repeat the web write test from section 5 one more time. A [CRITICAL][<agent>] email should land within seconds. If it does not, /var/log/wazuh/critical-mail.log will contain the swaks error. If that file does not exist at all, the script never ran: check the 750 root:wazuh permissions and grep ossec.log for active-response.

8. The digest email

The instant email only covers the one case that really cannot wait. The digest is there for everything else. Every thirty minutes, a single cron script on the manager sends one email per agent listing what changed and which process changed it. If nothing changed, no email is sent.

The same script also sends a second kind of email, for changes discovered by the daily scan that whodata never saw happen. That is how you find out about changes made during an outage or while the agent was restarting.

Both emails are routed by hostname through the MAIL_TO_<host> map in /etc/wazuh-mail.env, falling back to the default MAIL_TO when there is no mapping. And both are limited to the security-relevant extensions, matching rule 100101: .php .phtml .phar .js .sh .cgi .pl .py .env .ini .htaccess. Every other file is still monitored and still shows up in alerts.log and alerts.json, it just does not get emailed.

cat > /usr/local/bin/wazuh-fim-digest.sh << 'EOF'
#!/bin/bash
source /etc/wazuh-mail.env
LOG=/var/log/wazuh/fim-digest.log
mkdir -p "$(dirname "$LOG")"
# LOCAL time, not UTC: alerts.json timestamps carry the local offset and
# the comparison below is a string comparison.
SINCE=$(date -d '30 minutes ago' +%Y-%m-%dT%H:%M:%S)

TMPDIR=$(mktemp -d /tmp/fim-digest.XXXXXX)
trap 'rm -rf "$TMPDIR"' EXIT

python3 - "$SINCE" "$TMPDIR" << 'PYEOF'
import sys, json, os, re
from collections import defaultdict
since, tmpdir = sys.argv[1], sys.argv[2]
byagent = defaultdict(list)
byscan = defaultdict(list)
try:
    with open('/var/ossec/logs/alerts/alerts.json') as f:
        for line in f:
            try:
                a = json.loads(line)
            except Exception:
                continue
            if a.get('timestamp', '') < since:
                continue
            r = a.get('rule', {})
            if 'syscheck' not in r.get('groups', []):
                continue
            if r.get('id') == '100101':
                continue  # already emailed via Tier 1
            s = a.get('syscheck', {})
            path = s.get('path', '?')
            if not re.search(r'\.(php|phtml|phar|js|sh|cgi|pl|py|env|ini)$|\.htaccess$', path):
                continue
            au = s.get('audit', {})
            name = au.get('process', {}).get('name')
            agent = a.get('agent', {}).get('name', 'unknown')
            if name:
                proc = name.split('/')[-1]
                byagent[agent].append(
                    f"{a.get('timestamp','')[:19]}  {s.get('event','?'):9} {proc:12} {path}")
            else:
                byscan[agent].append(
                    f"{a.get('timestamp','')[:19]}  {s.get('event','?'):9} {path}")
except FileNotFoundError:
    pass
for agent, lines in byagent.items():
    with open(os.path.join(tmpdir, agent), 'w') as out:
        out.write(f"{len(lines)} executable change(s) in the last 30 minutes\n\n")
        out.write("\n".join(lines) + "\n")
for agent, lines in byscan.items():
    with open(os.path.join(tmpdir, agent + '.scan'), 'w') as out:
        out.write(f"{len(lines)} executable file(s) changed within the last 24 hours\n")
        out.write("(found by the daily integrity scan; not witnessed live)\n\n")
        out.write("\n".join(lines) + "\n")
PYEOF

shopt -s nullglob
for f in "$TMPDIR"/*; do
    AGENT=$(basename "$f" .scan)
    case "$f" in
      *.scan) TAG="FIM daily scan" ;;
      *)      TAG="FIM digest" ;;
    esac
    SAFE=$(echo "$AGENT" | tr '.-' '__')
    VAR="MAIL_TO_${SAFE}"
    RECIPIENT="${!VAR:-$MAIL_TO}"
    echo "$(date -Is) sending $TAG for $AGENT to $RECIPIENT" >> "$LOG"
    swaks --to "$RECIPIENT" --from "$MAIL_FROM" \
          --server "$SMTP_SERVER" --auth LOGIN \
          --auth-user "$SMTP_USER" --auth-password "$SMTP_PASS" --tls \
          --header "Subject: [$TAG][$AGENT] $(head -1 "$f")" \
          --body "$(cat "$f")" >> "$LOG" 2>&1
    echo "$(date -Is) swaks exit=$? for $AGENT" >> "$LOG"
done
if [ -z "$(ls -A "$TMPDIR")" ]; then
    echo "$(date -Is) nothing to send" >> "$LOG"
fi
EOF
chmod 700 /usr/local/bin/wazuh-fim-digest.sh
echo "*/30 * * * * root /usr/local/bin/wazuh-fim-digest.sh" > /etc/cron.d/wazuh-fim-digest

The script has two halves. The Python half reads the last thirty minutes of syscheck alerts, skips anything the instant email already sent, and writes one temporary file per agent: <agent> for changes that have a process attached, and <agent>.scan for changes that do not, which are the ones the daily scan found. The bash half then mails each file to the right recipient and logs the result, or logs nothing to send when the window was quiet.

One note: the “last 24 hours” wording in the scan email is only accurate while the agents scan once a day, as set by frequency or scan_time in section 4. You may be tempted to turn off that scheduled scan to avoid the occasional duplicate entry. Do not. It is the only layer that catches what happens while whodata is not watching.

9. Verify a new agent

Run through this table on every agent you enroll, and for every site on it. I cannot stress this enough: a misconfigured agent fails quietly, so silence tells you nothing until every row below passes.

On top of the table, auditctl -l should list -w <app-root> -p wa -k wazuh_fim for each watched directory, which proves the kernel is watching what you think it is.

Do not treat the shell test and the web test as interchangeable. A write from your shell proves that the pipeline works end to end. Only a write triggered through the website proves that the web server process is covered, and that is the case this whole setup exists for.

10. What to expect once it runs

Once everything is in place, this is what your inbox will look like:

If a daily scan email shows up at an odd time, it is usually because an agent was restarted and scan_on_start caught up on the changes that happened during the restart’s blind window.

Expect the critical email to stay silent for long stretches. On my servers it has never fired outside of a test, and that is exactly what a tripwire is for. The digest is the email you will actually read: a short list of what changed on which server, with the process that did it, arriving whether or not you happened to be paying attention that day.

11. Troubleshooting

Whodata fell back to realtime

This is the most common problem, and the most dangerous one because it is silent. In the agent’s ossec.log, right after startup, you will see:

ERROR: (6642): Audit health check couldn't be completed correctly.
WARNING: (6913): Who-data engine could not start. Switching who-data to real-time.

When this happens, your digests keep arriving and changes keep being detected, so everything looks fine. But attribution is gone, which means rules 100100 and 100101 can never match, and the critical email can never fire. Nothing else warns you about it.

systemctl restart auditd && systemctl restart wazuh-agent
grep -E "6642|6913|6019" /var/ossec/logs/ossec.log | tail -3

You want to see (6019) ... Whodata engine started and no 6642 or 6913. Then confirm with one web-triggered write, which should give you rule 100101 and the email. If the problem keeps coming back, check systemctl status auditd first, then auditctl -s for enabled 1 with a nonzero pid and lost 0, then ls -la /var/ossec/queue/sockets/audit to confirm the socket exists, and finally cat /etc/audit/plugins.d/af_wazuh.conf for active = yes.

Web writes produce no alerts

The symptom: writes from a shell alert with full attribution, but php-fpm writes in the same tree produce nothing at all. Work through these steps in order, because each one isolates a different segment of the pipeline.

1. Agent Active? /var/ossec/bin/agent_control -l on the manager. If not, the manager firewall is probably missing the agent’s IP.
2. Does a shell write alert? If not, check for Whodata engine started since the last restart, the <directories> paths, and whether the test path sits under an <ignore>.
3. Check the kernel level with the raw log, not ausearch, whose windowing hides records that are present: grep "<test-filename>" /var/log/audit/audit.log | tail -3
4. Audit rules loaded? auditctl -l should list -w <app-root> -p wa -k wazuh_fim per watched directory. Present now does not prove present during the test, since rule state flaps around auditd restarts.
5. Ordered restart, wait for Whodata engine started, retest.
6. Shell writes audited but web writes invisible at kernel level? That is the shared-parent problem from section 4. Replace the parent <directories> entry with one per webapp root.
7. Web writes alerting as 554 or 550 but never 100101? Read the (Audit) Process name: line. If it is not php-fpm, extend rule 100100 and restart the manager. Mode: realtime instead of Mode: whodata means attribution is down, so fix that first.

It helps to keep the two kinds of failure apart in your head. No alert at all means the problem is somewhere in steps 1 to 6, which is agent and kernel territory. An alert that arrives with the wrong rule is step 7, which is manager territory.

If none of that helps, there are two upstream causes worth checking. Ubuntu ships /etc/audit/rules.d/audit.rules with a -a task,never line that suppresses syscall auditing entirely. Remove it, run augenrules --load, restart auditd and the agent in order, then restart the web service so its worker processes are re-forked under the new rules. Pre-existing audit rules from another tool can also conflict with Wazuh’s own.

And check what your curl actually returned during the test. A 404 or a 500 means the write never happened, so there was nothing to detect in the first place.

No daily scan email

Most of the time this is correct behaviour, not a fault. The scan email only carries changes that whodata did not see happen. Anything caught live updates the baseline immediately, which leaves the scan with nothing to report. To prove the path works, stop the agent, modify a monitored .php file, and start it again. You can also confirm the scan interval with grep -E "frequency|scan_time" /var/ossec/etc/ossec.conf and look for scan ended in ossec.log.

Diff missing from an alert

Either the file is new, in which case there is nothing to compare against, or the diff store has hit its quota, which section 4 explains how to fix.

12. Every file in one place

For reference, here is everything this post touched, where it lives, and which section explains it:

A few files you will never edit by hand but should know about: /var/ossec/logs/alerts/alerts.log and alerts.json hold every alert on the manager, /var/ossec/logs/ossec.log is the daemon log on both hosts, /var/ossec/queue/diff/ holds the stored file copies on the agent, and /var/ossec/queue/fim/db/fim.db is the baseline database.

The three commands you will end up using most, after any edit on the agent:

vi /var/ossec/etc/ossec.conf
systemctl restart auditd && systemctl restart wazuh-agent
tail -f /var/ossec/logs/ossec.log

The two mail scripts are long, so they stay in sections 7 and 8. Here are the short files, in full, without the explanatory comments:

<syscheck>
    <disabled>no</disabled>
    <frequency>86400</frequency>
    <scan_on_start>yes</scan_on_start>
    <alert_new_files>yes</alert_new_files>

    <directories check_all="yes" whodata="yes" report_changes="yes">/home/user/webapps/appname</directories>

    <directories check_all="yes" whodata="yes">/home/user/webapps/appname/public/products</directories>
    <directories check_all="yes" whodata="yes">/home/user/webapps/appname/public/cpresources</directories>
    <directories check_all="yes" whodata="yes">/home/user/webapps/appname/public/images</directories>

    <ignore>/home/user/webapps/appname/storage</ignore>
    <ignore>/home/user/webapps/appname/vendor</ignore>
    <ignore type="sregex">.log$|.swp$</ignore>

    <nodiff type="sregex">.png$|.PNG$|.jpg$|.JPG$|.jpeg$|.webp$|.gif$|.ico$|.svg$|.woff$|.woff2$|.ttf$|.eot$|.mp4$|.webm$|.pdf$|.zip$|.gz$</nodiff>

    <diff>
      <file_size>
        <enabled>yes</enabled>
        <limit>1MB</limit>
      </file_size>
    </diff>
</syscheck>

<group name="syscheck,site_fim,">

  <rule id="100100" level="7">
    <if_sid>550,553,554</if_sid>
    <field name="process_name">php-fpm</field>
    <description>Web process (php-fpm) wrote file: $(file)</description>
  </rule>

  <rule id="100101" level="12">
    <if_sid>100100</if_sid>
    <field name="file">.php$|.phtml$|.phar$|.htaccess$|.sh$|.cgi$|.pl$|.py$|.env$|.js$</field>
    <description>CRITICAL: Web process wrote executable file $(file) - possible webshell</description>
  </rule>

  <rule id="100102" level="7">
    <if_sid>100101</if_sid>
    <field name="file">cpresources</field>
    <description>CMS assets regenerated by web process (expected)</description>
  </rule>

</group>

Active response registration, inside <ossec_config>:

<command>
  <name>critical-mail</name>
  <executable>critical-mail.sh</executable>
  <timeout_allowed>no</timeout_allowed>
</command>

<active-response>
  <command>critical-mail</command>
  <location>server</location>
  <rules_id>100101</rules_id>
</active-response>

[Unit]
After=auditd.service
Wants=auditd.service

[Service]
ExecStartPre=-/usr/bin/systemctl restart auditd.service

MAIL_TO="[email protected]"
MAIL_FROM="wazuh@<MAIL_DOMAIN>"
SMTP_SERVER="smtp.example.net:587"
SMTP_USER="<SMTP_LOGIN>"
SMTP_PASS="<SMTP_PASSWORD>"

# Per-agent digest recipients: hostname with - and . replaced by _
MAIL_TO_vm_agent_one="[email protected]"
MAIL_TO_vm_agent_two="[email protected],[email protected]"

*/30 * * * * root /usr/local/bin/wazuh-fim-digest.sh

13. Still on the list

This setup does its job, but there are a few things I have not gotten to yet, and you might want to consider them for your own servers:

• A whodata watchdog. The fallback to realtime is silent, and it really should send an email. A cron job on the manager could read the recent alerts.json entries per agent and complain when an agent that should be producing Mode: whodata events has gone entirely realtime. I have not built this yet.
• Centralised agent config. Once you pass three or so agents, editing every ossec.conf by hand gets old. Wazuh supports agent groups and a centrally pushed agent.conf, which is the better way to do it at that scale.
• Upgrades. When it is time to upgrade, re-enable the repository, upgrade the manager first and the agents second, put both back on hold, and then run systemctl daemon-reload so the drop-in from section 3 survives the upgrade.