Watching a server for modified files with Wazuh
A file integrity monitor that catches a webshell within seconds, names the process that wrote it, and emails me about it without burying me in noise.

Imagine logging into a server and finding a PHP file in the document root that you and your team never uploaded. It is a webshell: a script an attacker drops in so that the site itself becomes their command line. File integrity monitoring tells you the file appeared at 04:12. It does not tell you what put it there, so you are left reconstructing “deploy or intrusion” out of SSH logs and deploy history by hand, hours after it mattered.
This post builds the version that answers the question for you. It watches every file in every site on the server, records which process made each change, emails you within seconds when the web server process writes something executable, and summarises everything else on a schedule.
Four terms first, because the rest of the post leans on them. The agent runs on the server you want watched and does the file watching. The manager is a separate host that collects events from every agent, applies the rules, and sends the mail. Whodata is Wazuh’s name for watching through the Linux audit subsystem, and it is the piece that tells you which process wrote a file rather than only that the file changed. Without it, you are back to the situation in the first paragraph. Every alert also carries a level from 0 to 15, which is Wazuh’s own measure of how much it should worry you; the setup below emails instantly at level 12 and digests everything below that.
| Host | Runs |
|---|---|
vm-manager |
Manager only, no indexer and no dashboard. Rules and mail. |
vm-agent (one or more) |
Agent. Hosts the websites, watches each webapp root. |
Tested on Wazuh 4.14.x and Ubuntu 22.04. You need root on both hosts and an SMTP account. The rules assume deploys arrive over SFTP as a dedicated user and that PHP runs through php-fpm; section 6 shows how to confirm that on your own server.
Paths below use /home/user/webapps/appname as the document root, so swap that
for your own. <MANAGER_IP>, <SITE> and the mail values are placeholders
too.
1. Install the manager
Skip wazuh-install.sh -a. It installs the indexer and dashboard too, and
neither is used here.
apt-get install gnupg apt-transport-httpscurl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/wazuh.gpg --import && chmod 644 /usr/share/keyrings/wazuh.gpgecho "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" | tee -a /etc/apt/sources.list.d/wazuh.listapt-get updateapt-get install wazuh-managersystemctl daemon-reload && systemctl enable wazuh-manager && systemctl start wazuh-managersystemctl status wazuh-managerYour manager has to stay at or above the version of every agent reporting to it, so disable the repository and hold the package. Otherwise the host’s package management can push the manager ahead of an agent, or an agent ahead of the manager, on a night you were not watching:
sed -i "s/^deb /#deb/" /etc/apt/sources.list.d/wazuh.listapt-get updateecho "wazuh-manager hold" | dpkg --set-selectionsOpen 1514/tcp for agent data and 1515/tcp for enrollment, inbound from each
agent’s IP. Behind a hosting control panel, add those rules in the panel, since
it overwrites raw ufw. Every agent enrolled later needs its address added too.
ss -tlnp | grep -E '1514|1515'Turn off vulnerability detection and the indexer connection in
/var/ossec/etc/ossec.conf. Both are on by default, both expect an indexer that
this install does not have, and left alone they burn wazuh-modulesd CPU and
fill ossec.log with indexer-connector: WARNING ... retrying.
<vulnerability-detection> <enabled>no</enabled></vulnerability-detection>
<indexer> <enabled>no</enabled></indexer>systemctl restart wazuh-manager2. Install an agent
On the server you want watched, run the same repository commands as section 1, then:
# installs and registers against the manager in one stepWAZUH_MANAGER="<MANAGER_IP>" apt-get install wazuh-agent
# pin it, same as the managersed -i "s/^deb /#deb/" /etc/apt/sources.list.d/wazuh.listapt-get updateecho "wazuh-agent hold" | dpkg --set-selections
systemctl daemon-reload && systemctl enable wazuh-agent && systemctl start wazuh-agentsystemctl status wazuh-agentWAZUH_MANAGER is read at install time only, so setting it afterwards does
nothing. With the agent running, check from the manager:
/var/ossec/bin/agent_control -lYou want the agent listed as Active. If it is not, check the manager
firewall first, then /var/ossec/logs/ossec.log on the agent. Nothing below
works until that line reads Active.
3. auditd and the start order
Whodata reads from the Linux audit subsystem, so auditd has to be running on the agent.
apt-get update && apt-get install -y auditd audispd-pluginssystemctl status auditd --no-pager | head -3 # active (running)auditctl -l # "No rules" is correct hereNo rules is expected; Wazuh installs its own once it starts watching. Overhead
is small, because those rules audit writes to application directories, not the
read traffic the site serves.
The agent attaches to auditd over a socket at startup. If auditd is restarted or
stale at that moment the attachment fails, whodata drops to realtime without
saying so, and you are back to seeing that a file changed without seeing what
changed it. This drop-in enforces the order, so systemctl restart wazuh-agent
is always safe afterwards:
mkdir -p /etc/systemd/system/wazuh-agent.service.dcat > /etc/systemd/system/wazuh-agent.service.d/auditd-first.conf << 'EOF'[Unit]After=auditd.serviceWants=auditd.service
[Service]ExecStartPre=-/usr/bin/systemctl restart auditd.serviceEOFsystemctl daemon-reloadUntil that file exists, restart the two together by hand, in this order:
systemctl restart auditd && systemctl restart wazuh-agenttail -f /var/ossec/logs/ossec.logThe - prefix lets the agent start even when auditd is broken, since degraded
detection beats none. The drop-in does not cover auditd restarting on its own,
during a package update for instance, so after any auditd restart run
systemctl restart wazuh-agent and confirm Whodata engine started in
/var/ossec/logs/ossec.log.
4. Choose what to watch
Syscheck is Wazuh’s file integrity module, and its configuration is a
<syscheck> block that already exists in /var/ossec/etc/ossec.conf on the
agent. Edit that block, and never add a second one.
<syscheck> <disabled>no</disabled> <!-- Daily full scan. Alternative: <scan_time>03:30</scan_time> pins it to a clock time so the daily-scan email, when there is one, lands predictably. --> <frequency>86400</frequency> <scan_on_start>yes</scan_on_start> <alert_new_files>yes</alert_new_files>
<!-- The OS directories Wazuh watches by default, left off here: this install is about the webapp roots, and /etc churns on its own. <directories>/etc,/usr/bin,/usr/sbin</directories> <directories>/bin,/sbin,/boot</directories> -->
<!-- Each webapp root gets its OWN directive. Do NOT watch a shared parent covering multiple sites. See the warning below. --> <directories check_all="yes" whodata="yes" report_changes="yes">/home/user/webapps/appname</directories>
<!-- Bulky media and generated dirs: fully monitored, but NO report_changes, so no content copies are stored. Adjust to your site, find the culprits with: du -h --max-depth=2 /home/user/webapps/appname | sort -rh | head -20 --> <directories check_all="yes" whodata="yes">/home/user/webapps/appname/public/products</directories> <directories check_all="yes" whodata="yes">/home/user/webapps/appname/public/cpresources</directories> <directories check_all="yes" whodata="yes">/home/user/webapps/appname/public/images</directories>
<!-- runtime noise excluded entirely --> <ignore>/home/user/webapps/appname/storage</ignore> <ignore>/home/user/webapps/appname/vendor</ignore> <ignore type="sregex">.log$|.swp$</ignore>
<!-- binaries: never store or diff their content --> <nodiff type="sregex">.png$|.PNG$|.jpg$|.JPG$|.jpeg$|.webp$|.gif$|.ico$|.svg$|.woff$|.woff2$|.ttf$|.eot$|.mp4$|.webm$|.pdf$|.zip$|.gz$</nodiff>
<!-- cap stored copies at 1MB per file, the 50MB default fills the 1GB quota fast --> <diff> <file_size> <enabled>yes</enabled> <limit>1MB</limit> </file_size> </diff></syscheck>Apply it by restarting the agent. Purge the stored copies at the same time, so they are rebuilt under the rules you just changed rather than kept from the old ones:
systemctl stop wazuh-agentrm -rf /var/ossec/queue/diff/*systemctl start wazuh-agentwhodata="yes" gives attribution. report_changes="yes" stores a copy of the
file so the alert can carry a diff. <ignore> drops a path entirely, <nodiff>
keeps watching but never prints contents.
If you would rather prove the file watching works before adding attribution to
it, put realtime="yes" on those directives instead. Realtime uses inotify,
needs no auditd, and alerts on the same changes; it just cannot name the
process. Switch the attribute to whodata="yes" and restart once you are
satisfied.
Six things to know before you adapt it to your own layout:
-
Give every webapp root its own
<directories>line, never the parent that contains them. Watch the parent and you can lose half your coverage without being told. On my own servers, shell writes were audited normally while php-fpm writes in the same tree produced no kernel audit events at all, with no alerts and nothing logged to say so. Per-root directives fixed it immediately, and they cut audit volume, since a parent watch also audits every site’s cache churn.<directories check_all="yes" whodata="yes" report_changes="yes">/home/user/webapps/site-one</directories><directories check_all="yes" whodata="yes" report_changes="yes">/home/user/webapps/site-two</directories> -
alert_new_filesdefaults tono, so new files never alert. A new file is the webshell case. -
A more specific
<directories>entry overrides the app-root one for paths beneath it, which is how the media directories above keep monitoring while dropping stored copies. -
<nodiff>does not stop copies being stored, it only redacts the diff in the alert. Only droppingreport_changesstops the copy. -
Every restart has a blind window. The full scan runs first and whodata starts after it. Anything created in between is recorded into the baseline, the stored fingerprint of every watched file, which means it counts as normal from then on and never alerts. Wait for
Whodata engine startedinossec.logbefore you test anything. -
Restart after every config change, and check
systemctl status wazuh-agentafterwards, because malformed XML stops the agent starting.
Confirm it came up clean
tail -f /var/ossec/logs/ossec.logThe startup sequence you are waiting for is the scan finishing and then the
watcher starting. With realtime="yes" it reads:
2026/07/23 01:42:07 wazuh-syscheckd: INFO: (6009): File integrity monitoring scan ended.2026/07/23 01:42:07 wazuh-syscheckd: INFO: FIM sync module started.2026/07/23 01:42:09 wazuh-syscheckd: INFO: (6012): Real-time file integrity monitoring started.With whodata="yes" the third line becomes (6019) Whodata engine started
instead. Getting the realtime line when you asked for whodata means the audit
attachment failed, which is section 11.
Two more checks on the agent:
grep -iE "whodata|who-data|audit" /var/ossec/logs/ossec.log | tail -15auditctl -lauditctl -l should now list a wazuh_fim rule per watched directory, which is
the kernel’s own confirmation that it is watching what you asked for.
Keep the diff store small
Stored copies live in /var/ossec/queue/diff under a 1GB quota. Once it fills,
alerts arrive carrying Unable to calculate diff due to 'disk_quota' limit,
which means you still learn that a file changed but no longer see what changed
in it.
du -sh /var/ossec/queue/diffdu -h --max-depth=3 /var/ossec/queue/diff | sort -rh | head -15ls /var/ossec/queue/diff/file | wc -lIdeally every stored copy is under 1MB and the count is roughly your number of
code and config files, a few thousand. Anything bulkier is a directory that
still has report_changes on it: drop the attribute for that directory, then
purge and restart as above.
The baseline database is locked while the agent runs, so copy it before querying:
cp /var/ossec/queue/fim/db/fim.db /tmp/f.db && sqlite3 /tmp/f.db "SELECT count(*) FROM file_entry WHERE path LIKE '/home/user/webapps/appname%';"5. Fingerprint your own stack
The rules in section 6 match on a process name, and that name differs per stack:
php-fpm here, apache2 or node elsewhere. Guess it and you get rules that
load cleanly and never match anything, so spend ten minutes watching what your
own server reports. Do this once per server and once per PHP-serving site on
it.
Watch alerts on the manager:
tail -f /var/ossec/logs/alerts/alerts.logUpload a file over SFTP as the deploy user. The alert should name
/usr/lib/openssh/sftp-server, parent /usr/sbin/sshd.
Then simulate the attack from the agent, writing a file through the website rather than from your shell:
cat > /home/user/webapps/appname/public/writetest.php << 'EOF'<?phpfile_put_contents(__DIR__ . '/web-written-test.php', "<?php // simulated webshell " . date('c') . "\n");echo "done";EOFchown user:user /home/user/webapps/appname/public/writetest.phpcurl -s https://<SITE>/writetest.php # must return "done"That alert should name a process ending in php-fpm. If yours arrives with no
audit block and no process at all, attribution is down and nothing below can
match, so fix that with section 11 before going on.
Compare the user field across both tests. On panel-managed hosting you will usually see the same application user either way, which is why these rules key on the process instead. Delete both test files afterwards: the deletions fire rule 553, and leaving a live write-capable script in a document root hands an attacker the thing you are trying to detect.
6. Write the rules
On the manager, in /var/ossec/etc/rules/local_rules.xml. Wazuh’s defaults
already alert on a modified file (550), a deleted one (553) and a new one (554)
at low levels. These three inherit from those through if_sid and add the
attribution.
<group name="syscheck,site_fim,">
<!-- Base: web server process wrote a file (add, modify or delete). Extend the process list when enrolling servers with other stacks, e.g. php-fpm|apache2 --> <rule id="100100" level="7"> <if_sid>550,553,554</if_sid> <field name="process_name">php-fpm</field> <description>Web process (php-fpm) wrote file: $(file)</description> </rule>
<!-- Escalation: executable or config content is the webshell signature --> <rule id="100101" level="12"> <if_sid>100100</if_sid> <field name="file">.php$|.phtml$|.phar$|.htaccess$|.sh$|.cgi$|.pl$|.py$|.env$|.js$</field> <description>CRITICAL: Web process wrote executable file $(file) - possible webshell</description> </rule>
<!-- Downgrade: CMS regenerating its own assets through php-fpm, which is expected. Adjust the path fragment to your CMS. --> <rule id="100102" level="7"> <if_sid>100101</if_sid> <field name="file">cpresources</field> <description>CMS assets regenerated by web process (expected)</description> </rule>
</group>systemctl restart wazuh-manager100100 catches any file php-fpm wrote. 100101 raises that to level 12 when the filename looks executable, which is the webshell signature and the only rule that emails instantly. 100102 drops the CMS asset path back to level 7, because php-fpm writing there is normal. The last matching rule wins.
Match on decoded field names: process_name, user_name, file.
alerts.json shows nested names such as audit.process.name and path, and
those fail silently in <field name>: the rules load and never match. The Wazuh
documentation covers this under file integrity monitoring, creating custom FIM
rules, mapping FIM fields to Wazuh alerts.
Repeat the two tests from section 5 to confirm. The web write should now produce 100101 at level 12, and the SFTP upload should stay 554 at level 5. If the web write still lands on 554, the process name in rule 100100 does not match what section 5 showed you.
7. The instant email
On the manager, and it needs swaks:
apt-get install -y swaksSend one message by hand with it first, so a later silence tells you which half broke. Credentials and routing go in a root-only file:
cat > /etc/wazuh-mail.env << 'EOF'# Default recipient, also used for agents with no per-agent mappingMAIL_FROM="wazuh@<MAIL_DOMAIN>"SMTP_SERVER="smtp.example.net:587"SMTP_USER="<SMTP_LOGIN>"SMTP_PASS="<SMTP_PASSWORD>"
# Per-agent digest recipients: hostname with - and . replaced by _# Comma-separate for multiple recipients.EOFchmod 600 /etc/wazuh-mail.envNever run bash -x on a script that sources this file, because the trace prints
the password.
Wazuh’s active response runs a script when a chosen rule fires. This one reads the alert on standard input and mails the useful fields:
cat > /var/ossec/active-response/bin/critical-mail.sh << 'EOF'#!/bin/bashsource /etc/wazuh-mail.envmkdir -p /var/log/wazuhread -r INPUTALERT=$(echo "$INPUT" | python3 -c "import sys, jsond = json.load(sys.stdin)a = d.get('parameters', {}).get('alert', {})s = a.get('syscheck', {})au = s.get('audit', {})print(a.get('agent',{}).get('name','unknown'))print(f\"Rule: {a.get('rule',{}).get('description','?')}\")print(f\"File: {s.get('path','?')}\")print(f\"Event: {s.get('event','?')}\")print(f\"User: {au.get('user',{}).get('name','?')}\")print(f\"Process: {au.get('process',{}).get('name','?')}\")print(f\"Agent: {a.get('agent',{}).get('name','?')}\")print(f\"Time: {a.get('timestamp','?')}\")" 2>/dev/null)AGENT=$(echo "$ALERT" | head -1)BODY=$(echo "$ALERT" | tail -n +2)swaks --to "$MAIL_TO" --from "$MAIL_FROM" \ --server "$SMTP_SERVER" --auth LOGIN \ --auth-user "$SMTP_USER" --auth-password "$SMTP_PASS" --tls \ --header "Subject: [CRITICAL][$AGENT] Wazuh: possible webshell detected" \ --body "$BODY" >> /var/log/wazuh/critical-mail.log 2>&1EOFchmod 750 /var/ossec/active-response/bin/critical-mail.shchown root:wazuh /var/ossec/active-response/bin/critical-mail.shThe agent name is printed first and split off into AGENT, which puts the
hostname in the subject when several servers report to one manager. The
permissions on the last two lines matter as much as the script does: if active
response cannot run it you get no mail, no error, and no log file to look in.
To include the diff in the body, append this inside the Python block, after the Time line:
diff = s.get('diff')if diff: if len(diff) > 4000: diff = diff[:4000] + '\n... [diff truncated]' print(f"\nWhat changed:\n{diff}")else: print("\nWhat changed: (no diff available for this event)")New files have no diff, since there is nothing to compare against. Attacker content in a mail body can also trip provider malware scanning, so if critical mail stops arriving after enabling this, retest with a benign change.
Two shell traps: >> /path/file.log silently kills the command when the
directory does not exist, which the mkdir -p covers, and the AGENT= and
BODY= lines have to stay outside the python3 -c "…" block, since bash pasted
into the Python string fails silently while stderr is discarded.
Register the script in /var/ossec/etc/ossec.conf, inside <ossec_config>:
<command> <name>critical-mail</name> <executable>critical-mail.sh</executable> <timeout_allowed>no</timeout_allowed></command>
<active-response> <command>critical-mail</command> <location>server</location> <rules_id>100101</rules_id></active-response>systemctl restart wazuh-manager<location>server</location> runs it on the manager, keeping the SMTP
credentials on one host.
Repeat the web write from section 5. A [CRITICAL][<agent>] email should arrive
within seconds. If it does not, /var/log/wazuh/critical-mail.log has the swaks
error; if that file is missing entirely, the script never ran, so check
750 root:wazuh and grep ossec.log for active-response.
8. The digest email
The digest is there so that everything the instant email does not cover still reaches you. Every thirty minutes you get one email per agent, listing what changed and which process changed it, sent by a single cron script on the manager.
The same script sends a second email for changes found by the daily scan that
whodata never witnessed, which is how an outage or an agent restart shows up.
Both are routed by hostname through the map in /etc/wazuh-mail.env, falling
back to MAIL_TO.
Both emails cover security-relevant extensions only, matching rule 100101:
.php .phtml .phar .js .sh .cgi .pl .py .env .ini .htaccess. Everything else is
still monitored and still alerts into alerts.log and alerts.json.
cat > /usr/local/bin/wazuh-fim-digest.sh << 'EOF'#!/bin/bashsource /etc/wazuh-mail.envLOG=/var/log/wazuh/fim-digest.logmkdir -p "$(dirname "$LOG")"# LOCAL time, not UTC: alerts.json timestamps carry the local offset and# the comparison below is a string comparison.SINCE=$(date -d '30 minutes ago' +%Y-%m-%dT%H:%M:%S)
TMPDIR=$(mktemp -d /tmp/fim-digest.XXXXXX)trap 'rm -rf "$TMPDIR"' EXIT
python3 - "$SINCE" "$TMPDIR" << 'PYEOF'import sys, json, os, refrom collections import defaultdictsince, tmpdir = sys.argv[1], sys.argv[2]byagent = defaultdict(list)byscan = defaultdict(list)try: with open('/var/ossec/logs/alerts/alerts.json') as f: for line in f: try: a = json.loads(line) except Exception: continue if a.get('timestamp', '') < since: continue r = a.get('rule', {}) if 'syscheck' not in r.get('groups', []): continue if r.get('id') == '100101': continue # already emailed via Tier 1 s = a.get('syscheck', {}) path = s.get('path', '?') if not re.search(r'\.(php|phtml|phar|js|sh|cgi|pl|py|env|ini)$|\.htaccess$', path): continue au = s.get('audit', {}) name = au.get('process', {}).get('name') agent = a.get('agent', {}).get('name', 'unknown') if name: proc = name.split('/')[-1] byagent[agent].append( f"{a.get('timestamp','')[:19]} {s.get('event','?'):9} {proc:12} {path}") else: byscan[agent].append( f"{a.get('timestamp','')[:19]} {s.get('event','?'):9} {path}")except FileNotFoundError: passfor agent, lines in byagent.items(): with open(os.path.join(tmpdir, agent), 'w') as out: out.write(f"{len(lines)} executable change(s) in the last 30 minutes\n\n") out.write("\n".join(lines) + "\n")for agent, lines in byscan.items(): with open(os.path.join(tmpdir, agent + '.scan'), 'w') as out: out.write(f"{len(lines)} executable file(s) changed within the last 24 hours\n") out.write("(found by the daily integrity scan; not witnessed live)\n\n") out.write("\n".join(lines) + "\n")PYEOF
shopt -s nullglobfor f in "$TMPDIR"/*; do AGENT=$(basename "$f" .scan) case "$f" in *.scan) TAG="FIM daily scan" ;; *) TAG="FIM digest" ;; esac SAFE=$(echo "$AGENT" | tr '.-' '__') VAR="MAIL_TO_${SAFE}" RECIPIENT="${!VAR:-$MAIL_TO}" echo "$(date -Is) sending $TAG for $AGENT to $RECIPIENT" >> "$LOG" swaks --to "$RECIPIENT" --from "$MAIL_FROM" \ --server "$SMTP_SERVER" --auth LOGIN \ --auth-user "$SMTP_USER" --auth-password "$SMTP_PASS" --tls \ --header "Subject: [$TAG][$AGENT] $(head -1 "$f")" \ --body "$(cat "$f")" >> "$LOG" 2>&1 echo "$(date -Is) swaks exit=$? for $AGENT" >> "$LOG"doneif [ -z "$(ls -A "$TMPDIR")" ]; then echo "$(date -Is) nothing to send" >> "$LOG"fiEOFchmod 700 /usr/local/bin/wazuh-fim-digest.shecho "*/30 * * * * root /usr/local/bin/wazuh-fim-digest.sh" > /etc/cron.d/wazuh-fim-digestThe Python half reads the last thirty minutes of syscheck alerts, skips what the
instant email already sent, and writes one file per agent: <agent> for changes
with a process attached, <agent>.scan for changes without one. The bash half
mails each file to the mapped recipient and logs the result, or logs
nothing to send when the window was quiet.
The “last 24 hours” wording holds only while the agents scan daily, per
frequency or scan_time in section 4. Do not disable that scheduled scan to
avoid the occasional duplicate: it is the only layer that covers a whodata
outage.
9. Verify a new agent
Run these on every agent you enroll and every site on it. A misconfigured agent fails quietly, so silence tells you nothing until this table passes.
| Test | Expected |
|---|---|
| Modify existing file in app root, from the shell | 550, Mode: whodata, audit block, real diff, no quota error |
| Create new file from the shell, after “Whodata engine started” | 554 |
SFTP upload of a .php |
554 level 5, sftp-server process, no critical email, appears in next digest to the mapped recipient |
Web-triggered write of a .php, per PHP-serving site |
100101 level 12 plus [CRITICAL][<agent>] email within seconds |
Stop agent, shell-modify a .php, start agent |
[FIM daily scan] email after the next cron tick, entry unattributed |
| A quiet 30 minute window | no digest email, nothing to send in the digest log |
New agent with no MAIL_TO_<host> mapping |
digest falls back to the default MAIL_TO, then add the mapping |
| Agent log after any restart | (6019) Whodata engine started, and no 6642 or 6913 |
auditctl -l should also list -w <app-root> -p wa -k wazuh_fim for each
watched directory, which proves the kernel is watching what you think.
The shell test and the web test are not interchangeable. A shell write proves the pipeline works; only a web-triggered write proves the daemon is covered, and the daemon is the case that matters.
10. What to expect once it runs
| When | What it means | |
|---|---|---|
[CRITICAL][host] |
Within seconds | php-fpm wrote an executable or config file. Treat as an incident until proven otherwise. |
[FIM digest][host] |
Every 30 minutes, only if something changed | Attributed changes to risky file types. Deploys land here. |
[FIM daily scan][host] |
After a scan finds unwitnessed changes | Something changed while whodata was not watching. Worth reading closely. |
| Nothing at all | Most of the time | Nothing matched. This is the normal state. |
A daily scan email arriving off schedule usually follows an agent restart, when
scan_on_start catches up on the restart’s blind window.
Expect the critical email to stay silent for long stretches. On my servers it has never fired outside a test, which is what a tripwire is for. The digest is the one you will actually read: a list of what changed on which server, attributed to a process, arriving whether or not you were paying attention.
11. Troubleshooting
Whodata fell back to realtime
In the agent’s ossec.log at startup:
ERROR: (6642): Audit health check couldn't be completed correctly.WARNING: (6913): Who-data engine could not start. Switching who-data to real-time.Your digests keep arriving and changes keep being detected, but attribution is gone, so rules 100100 and 100101 cannot match and the critical email cannot fire. Nothing else tells you.
systemctl restart auditd && systemctl restart wazuh-agentgrep -E "6642|6913|6019" /var/ossec/logs/ossec.log | tail -3You want (6019) ... Whodata engine started and no 6642 or 6913. Confirm with
one web-triggered write, which should give 100101 and the email. If it persists,
check systemctl status auditd, then auditctl -s for enabled 1 with a
nonzero pid and lost 0, then ls -la /var/ossec/queue/sockets/audit for the
socket, then cat /etc/audit/plugins.d/af_wazuh.conf for active = yes.
Web writes produce no alerts
You see shell writes alerting with full attribution while php-fpm writes in the same tree produce nothing at all. Work through these in order, since each step isolates one segment of the pipeline.
- Agent Active?
/var/ossec/bin/agent_control -lon the manager. If not, the manager firewall is probably missing the agent’s IP. - Does a shell write alert? If not, check for
Whodata engine startedsince the last restart, the<directories>paths, and whether the test path sits under an<ignore>. - Check the kernel level with the raw log, not
ausearch, whose windowing hides records that are present:grep "<test-filename>" /var/log/audit/audit.log | tail -3 - Audit rules loaded?
auditctl -lshould list-w <app-root> -p wa -k wazuh_fimper watched directory. Present now does not prove present during the test, since rule state flaps around auditd restarts. - Ordered restart, wait for
Whodata engine started, retest. - Shell writes audited but web writes invisible at kernel level? That is the
shared-parent problem from section 4. Replace the parent
<directories>entry with one per webapp root. - Web writes alerting as 554 or 550 but never 100101? Read the
(Audit) Process name:line. If it is notphp-fpm, extend rule 100100 and restart the manager.Mode: realtimeinstead ofMode: whodatameans attribution is down, so fix that first.
Keep the two failure classes apart as you go. No alert at all is steps 1 to 6, which is agent and kernel territory, while an alert carrying the wrong rule is step 7, which is manager territory.
Two upstream causes are worth checking if none of that helps. A -a task,never
line in /etc/audit/rules.d/audit.rules ships by default on Ubuntu and
suppresses syscall auditing: remove it, run augenrules --load, restart in
order, then restart the web service so its workers re-fork. Pre-existing audit
rules can also conflict with Wazuh’s own.
Check what your curl actually returned, too. A 404 or a 500 means the write
never happened and there was nothing to detect.
No daily scan email
Usually that is correct behaviour rather than a fault. The email only carries
changes whodata did not witness, and anything caught live updates the baseline
immediately, leaving the scan nothing to report. To prove the path works, stop
the agent, modify a monitored .php, then start it again. Also confirm the
interval with grep -E "frequency|scan_time" /var/ossec/etc/ossec.conf and look
for scan ended in ossec.log.
Diff missing from an alert
Either the file is new, in which case there is nothing to compare against, or the diff store hit its quota, which section 4 covers.
12. Every file in one place
| File | Host | Purpose | Section |
|---|---|---|---|
/etc/apt/sources.list.d/wazuh.list |
both | Package repository, commented out to pin versions | 1 |
/var/ossec/etc/ossec.conf |
manager | Vulnerability detection off, active response registration | 1, 7 |
/var/ossec/etc/ossec.conf |
agent | <syscheck>: what is watched, ignored and diffed |
4 |
/etc/systemd/system/wazuh-agent.service.d/auditd-first.conf |
agent | Restarts auditd before the agent | 3 |
/var/ossec/etc/rules/local_rules.xml |
manager | Rules 100100 to 100102 | 6 |
/etc/wazuh-mail.env |
manager | SMTP credentials and per-agent recipients. Mode 0600 | 7 |
/var/ossec/active-response/bin/critical-mail.sh |
manager | Instant email. Mode 750, owner root:wazuh |
7 |
/usr/local/bin/wazuh-fim-digest.sh |
manager | Digest and daily scan email. Mode 700 | 8 |
/etc/cron.d/wazuh-fim-digest |
manager | Runs the digest every 30 minutes | 8 |
/var/log/wazuh/critical-mail.log |
manager | Instant email output and swaks errors | 7 |
/var/log/wazuh/fim-digest.log |
manager | Digest output and exit codes | 8 |
Not edited by hand, but worth knowing:
/var/ossec/logs/alerts/alerts.log and alerts.json hold every alert on the
manager, /var/ossec/logs/ossec.log is the daemon log on both hosts,
/var/ossec/queue/diff/ holds stored copies on the agent, and
/var/ossec/queue/fim/db/fim.db is the baseline database.
The sequence you will use most, after any edit on the agent:
vi /var/ossec/etc/ossec.confsystemctl restart auditd && systemctl restart wazuh-agenttail -f /var/ossec/logs/ossec.logThe two mail scripts are long and stay in sections 7 and 8. The short files, in full:
<syscheck> <disabled>no</disabled> <frequency>86400</frequency> <scan_on_start>yes</scan_on_start> <alert_new_files>yes</alert_new_files>
<directories check_all="yes" whodata="yes" report_changes="yes">/home/user/webapps/appname</directories>
<directories check_all="yes" whodata="yes">/home/user/webapps/appname/public/products</directories> <directories check_all="yes" whodata="yes">/home/user/webapps/appname/public/cpresources</directories> <directories check_all="yes" whodata="yes">/home/user/webapps/appname/public/images</directories>
<ignore>/home/user/webapps/appname/storage</ignore> <ignore>/home/user/webapps/appname/vendor</ignore> <ignore type="sregex">.log$|.swp$</ignore>
<nodiff type="sregex">.png$|.PNG$|.jpg$|.JPG$|.jpeg$|.webp$|.gif$|.ico$|.svg$|.woff$|.woff2$|.ttf$|.eot$|.mp4$|.webm$|.pdf$|.zip$|.gz$</nodiff>
<diff> <file_size> <enabled>yes</enabled> <limit>1MB</limit> </file_size> </diff></syscheck><group name="syscheck,site_fim,">
<rule id="100100" level="7"> <if_sid>550,553,554</if_sid> <field name="process_name">php-fpm</field> <description>Web process (php-fpm) wrote file: $(file)</description> </rule>
<rule id="100101" level="12"> <if_sid>100100</if_sid> <field name="file">.php$|.phtml$|.phar$|.htaccess$|.sh$|.cgi$|.pl$|.py$|.env$|.js$</field> <description>CRITICAL: Web process wrote executable file $(file) - possible webshell</description> </rule>
<rule id="100102" level="7"> <if_sid>100101</if_sid> <field name="file">cpresources</field> <description>CMS assets regenerated by web process (expected)</description> </rule>
</group>Active response registration, inside <ossec_config>:
<command> <name>critical-mail</name> <executable>critical-mail.sh</executable> <timeout_allowed>no</timeout_allowed></command>
<active-response> <command>critical-mail</command> <location>server</location> <rules_id>100101</rules_id></active-response>[Unit]After=auditd.serviceWants=auditd.service
[Service]ExecStartPre=-/usr/bin/systemctl restart auditd.serviceMAIL_FROM="wazuh@<MAIL_DOMAIN>"SMTP_SERVER="smtp.example.net:587"SMTP_USER="<SMTP_LOGIN>"SMTP_PASS="<SMTP_PASSWORD>"
# Per-agent digest recipients: hostname with - and . replaced by _*/30 * * * * root /usr/local/bin/wazuh-fim-digest.sh13. Still on the list
- Least privilege. Take write access to code-serving directories away from the php-fpm user and leave it only where the CMS needs it. The webshell then cannot land, and rule 100101 becomes a tripwire that should never fire.
- A whodata watchdog. The fallback to realtime is silent and should send
mail. A cron job on the manager could read recent
alerts.jsonper agent and complain when an agent that should be producingMode: whodataevents has gone entirelyrealtime. Not built yet. - Centralised agent config. Past three agents, Wazuh agent groups and a
pushed
agent.confbeat editing eachossec.confby hand. - Upgrades. Re-enable the repository, upgrade the manager first and the
agent second, re-hold both, then run
systemctl daemon-reloadso the drop-in from section 3 survives.