Skip to content
ProHomeLab
RU

Traefik

Installing Traefik in an LXC Container on Proxmox as a Systemd Service - Part 3 - CrowdSec

Stilicho2011Updated 12 min read44 views

Also available in: RU

Part 5 of 6 in the series Traefik
  1. Traefik in Docker - Setting Up a Reverse Proxy from Scratch
  2. Installing Traefik in a Proxmox LXC Container as a Systemd Service - Part 1
  3. Middlewares in Traefik - What They Are, Why You Need Them, and the Full List for a Homelab
  4. Installing Traefik in an LXC Container on Proxmox as a Systemd Service - Part 2
  5. Installing Traefik in an LXC Container on Proxmox as a Systemd Service - Part 3 - CrowdSec
  6. Traefik on Its Own VLAN - Configuring the Firewall in OPNsense
Contents

Introduction

In part 2, in the "what's left" section, I put off installing CrowdSec for a separate pass. This is that pass. CrowdSec is installed in the same LXC as Traefik itself, rather than as a separate Docker container like before. This simplifies connectivity (no Docker DNS names, everything talks over localhost) and lets CrowdSec read access.log directly from disk, without shuttling the file between different environments.

Note

Updated October 9, 2026. CrowdSec 1.8 brought bot protection to the WAF via a challenge page (bot challenge) - it gets its own new section, "Bot protection," near the end of the article. The AppSec source has also been fixed: the appsec-crs collection used to get installed in the earlier version of this article, but its rules were never actually loaded.


Installation

CrowdSec is installed via the official repository-add script, then through regular apt:

Terminal window
curl -s https://install.crowdsec.net | sudo sh
apt update
apt list crowdsec
apt install crowdsec

apt list crowdsec is an optional step, just to confirm the package actually shows up in the list after adding the repository, before installing it.


Collections

A collection is a ready-made set of detection scenarios for a particular type of workload. Before installing, I update the hub index to make sure I'm installing current versions rather than whatever was baked into the package at build time:

Terminal window
cscli hub update

I install the package for the HTTP perimeter (Traefik) right away, plus the base scenarios for the OS itself:

Terminal window
cscli collections install crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/base-http-scenarios crowdsecurity/sshd crowdsecurity/linux crowdsecurity/appsec-generic-rules crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-crs
  • crowdsecurity/traefik - access-log parser specifically for the Traefik format;
  • crowdsecurity/http-cve + base-http-scenarios - detection of known CVE exploitation attempts and generic HTTP attacks;
  • crowdsecurity/sshd + linux - since CrowdSec now lives in the same LXC as Traefik, I'm covering SSH access to the container itself at the same time;
  • appsec-generic-rules, appsec-virtual-patching, appsec-crs - for the AppSec component (WAF-like protection at the request level, before it reaches the backend) - more on that separately below. The first two are wired up by the shared crowdsecurity/appsec-default config, while the OWASP CRS rules from appsec-crs come with their own crowdsecurity/crs config, which has to be listed explicitly in the AppSec source (more on that below too).
Terminal window
systemctl reload crowdsec

Automatic hub updates on a schedule

Scenarios and parsers in the hub are updated periodically (new CVEs, fixes to existing detections), but manually remembering to run cscli hub update is rare. I put it in a cron file so the index update and upgrade of installed collections happen on their own once a day, at midnight:

Terminal window
nano /etc/cron.d/crowdsec-hub-update
cron
0 0 * * * root /usr/bin/cscli hub update && /usr/bin/cscli hub upgrade

A file in /etc/cron.d/ doesn't require crontab -e and isn't tied to a user crontab - it's picked up automatically by the system cron once saved, without an additional daemon reload. hub upgrade pulls in new versions of already-installed collections/parsers/scenarios (it doesn't install new ones - only updates what's already enabled). If the upgrade actually changed something, cscli will itself prompt for systemctl reload crowdsec - you can either add that as a third command on the same cron line, or, as I prefer, leave it under manual control, so you don't reload the production service's configuration without reviewing what actually got updated.


Remediation profile

The profile determines exactly what to do when CrowdSec decides to block something - not just "ban it," but by what logic and where to notify.

nano /etc/crowdsec/profiles.yaml:

yaml
name: default_ip_remediation
filters:
- Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
- type: ban
duration: 4h
notifications:
- http_default
on_success: break
---
name: default_range_remediation
filters:
- Alert.Remediation == true && Alert.GetScope() == "Range"
decisions:
- type: ban
duration: 4h
notifications:
- http_default
on_success: break

Two profiles - one for banning a specific IP, another for banning an entire range (if the attack is distributed across a subnet). Both ban for 4 hours and send a notification through the http_default plugin, which is configured separately.


Notifications via Gotify

Since I already have Gotify in my stack (push notifications to phone/desktop), it makes sense to route CrowdSec alerts there too, rather than setting up an email integration from scratch (other options don't officially work in Russia as of this writing).

nano /etc/crowdsec/notifications/http.yaml:

yaml
type: http
name: http_default
log_level: info
format: |
{{ range . -}}
{{ $alert := . -}}
{
"extras": {
"client::display": {
"contentType": "text/markdown"
}
},
"priority": 3,
{{range .Decisions -}}
"title": "{{.Type }} {{ .Value }} for {{.Duration}}",
"message": "{{.Scenario}} \n\n[crowdsec cti](https://app.crowdsec.net/cti/{{.Value -}}) \n\n[shodan](https://shodan.io/host/{{.Value -}})"
{{end -}}
}
{{ end -}}
url: https://gotify.domain.ru/message
method: POST
headers:
X-Gotify-Key: your_application_token
Content-Type: application/json

The template produces a Gotify-compatible JSON: a title with the decision type and value (IP/range, ban duration), a message body with the scenario that triggered, plus ready-made links to CrowdSec CTI and Shodan for a quick manual check of the address.

X-Gotify-Key is the application token for a specific application in Gotify (created in its own UI), not a user token.


Log sources (acquisition)

The CrowdSec version we're installing uses the /etc/crowdsec/acquis.d/ directory - a separate file per source, instead of one monolithic acquis.yaml. Same logic as the split of dynamic/ for Traefik itself: sources are easier to read and maintain when they're not lumped together in one pile.

Traefik access logs

Terminal window
touch /etc/crowdsec/acquis.d/traefik.yaml
yaml
poll_without_inotify: false
filenames:
- /var/log/traefik/*.log
labels:
type: traefik

poll_without_inotify: false - use inotify (more efficient than polling the file on a timer) to track new lines in the log.

Through the *.log mask, this source picks up not just access.log but also traefik.log (Traefik's own operational log - startup, config reload, TLS certificate issuance). The crowdsecurity/traefik-logs parser is built for the access-log format, so lines from traefik.log will honestly show up as "unparsed" in cscli metrics - this is expected, not a config error, and there's no need to remove traefik.log from the source.

System logs

Terminal window
touch /etc/crowdsec/acquis.d/syslog.yaml
yaml
filenames:
- /var/log/auth.log
- /var/log/syslog
labels:
type: syslog

This is what we installed the sshd/linux collections for - since CrowdSec is now in the same LXC that receives external traffic, it makes sense to also keep an eye on SSH brute-force attempts and suspicious OS-level activity.

Debian 13 (trixie) - rsyslog isn't installed by default

On Debian 13, this source won't work on its own. Starting with bookworm, rsyslog is no longer installed by default - /var/log/auth.log and /var/log/syslog simply don't exist or aren't written to unless the package is explicitly installed. Check with:

Terminal window
ls -la /var/log/auth.log /var/log/syslog

If auth.log is missing and syslog exists but is zero bytes - rsyslog isn't installed:

Terminal window
apt install rsyslog
systemctl enable --now rsyslog

Next comes a trap specific to trixie: even after installing the package, it does not create the routing rules file /etc/rsyslog.d/50-default.conf, which used to come "out of the box" before. The /etc/rsyslog.d/ directory is empty after installation (except possibly postfix.conf, if Postfix is installed too). Without these rules, rsyslogd listens to the journal but doesn't know where to write auth/authpriv messages - the files stay empty. Create it manually:

Terminal window
nano /etc/rsyslog.d/50-default.conf

The minimum CrowdSec needs:

auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
Terminal window
systemctl restart rsyslog

Verification - any action via sudo (for example sudo -k && sudo whoami, to clear the cache and go through PAM again) should immediately show up in auth.log:

Terminal window
ls -la /var/log/auth.log /var/log/syslog

Both files should exist and grow. After that, systemctl restart crowdsec and cscli metrics - Acquisition Metrics should show entries for file:/var/log/auth.log and file:/var/log/syslog with a non-zero number of lines read.

Terminal window
systemctl reload crowdsec
cscli metrics

cscli metrics shows whether CrowdSec is seeing any lines at all from the connected sources - if the counters are zero, there's no point starting on AppSec yet; acquisition needs to be sorted out first.


AppSec component

AppSec is WAF-like protection at the level of the request itself (attack signatures, virtual patching for known CVEs), operating separately from the classic anti-brute-force log analysis. Traefik talks to it directly through the bouncer plugin, before the request reaches the actual backend.

Terminal window
touch /etc/crowdsec/acquis.d/appsec.yaml
yaml
listen_addr: 127.0.0.1:7422
appsec_configs:
- crowdsecurity/appsec-default
- crowdsecurity/crs
name: myAppSecComponent
source: appsec
labels:
type: appsec

There are two configs here, and that matters. crowdsecurity/appsec-default loads the rules from the appsec-virtual-patching and appsec-generic-rules collections - they run in-band, meaning they inspect the request and block it before it ever reaches the backend. The OWASP CRS rules from the appsec-crs collection, on the other hand, only load through their own crowdsecurity/crs config: leave it out, and the collection is installed but its rules simply never load. CRS also runs out-of-band - it doesn't block the request, it just raises an alert that the regular CrowdSec scenarios then act on. That's a sensible default: CRS rules cast a wide net, they often produce false positives on real-world apps, and blocking requests on them right away would be risky.

Important to check before starting: the address must be 127.0.0.1, not 127.0.0.0 - the latter is the address of the 127.0.0.0/8 network itself, not a host, and isn't guaranteed to bind correctly. Traefik's bouncer plugin below uses localhost:7422, which resolves to 127.0.0.1 - both addresses must match literally, otherwise AppSec will be unreachable, and crowdsecAppsecUnreachableBlock: true in the plugin config will block all traffic through Traefik if the component isn't responding.

Terminal window
systemctl reload crowdsec
systemctl reload traefik
cscli metrics show acquisition

Registering the bouncer and connecting it to Traefik

Terminal window
cscli bouncers add traefik-bouncer

The command outputs the API key once - save it immediately, you won't be able to view it again (same logic as with the Cloudflare token in part 2), only recreate it.

Middleware in dynamic/middlewares/crowdsec.yaml:

yaml
http:
middlewares:
crowdsec:
plugin:
bouncer:
enabled: true
logLevel: INFO
updateIntervalSeconds: 15
updateMaxFailure: 0
defaultDecisionSeconds: 15
httpTimeoutSeconds: 10
crowdsecMode: stream
crowdsecAppsecEnabled: true
crowdsecAppsecHost: localhost:7422
crowdsecAppsecFailureBlock: true
crowdsecAppsecUnreachableBlock: true
crowdsecLapiKey: your_key_from_cscli_bouncers_add
crowdsecLapiHost: localhost:8080
crowdsecLapiScheme: http
forwardedHeadersTrustedIPs:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
clientTrustedIPs:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16

localhost instead of a Docker name like crowdsec:8080, which is what the old Docker-based setup would have used - both services are now in the same LXC, so no additional network connectivity is needed at all.

Last step - uncomment the middleware in the static config (it was commented out in part 2):

yaml
entryPoints:
web:
http:
middlewares:
- crowdsec@file # now can be uncommented
- rate-limit@file
websecure:
http:
middlewares:
- crowdsec@file # and here too
- rate-limit@file
Terminal window
systemctl restart traefik

Bot protection - AppSec bot challenge

Section added October 9, 2026.

CrowdSec 1.8 gave the WAF a new mode: bot detection. Instead of letting a request through or blocking it outright, AppSec serves the client a lightweight challenge page: the browser solves a small computational puzzle (proof-of-work) and collects a "fingerprint" of its environment. CrowdSec then scores that fingerprint against a set of automation tells - a headless browser, traces of Selenium, Playwright, or WebDriver, mismatches between what the browser claims to be and what it can actually do - and adds them up. Score above the threshold, and you're a bot; below it, and you get a cookie and browse the site without further checks. Essentially your own take on Cloudflare's challenge, running entirely on your own server.

The hub now has collections for this, and you install exactly one of them:

Collection Rejection threshold When to pick it
crowdsecurity/appsec-bot-challenge 75 points The recommended default
crowdsecurity/appsec-bot-challenge-strict 45 points Catches more bots, but also gets real people wrong more often
crowdsecurity/appsec-bot-challenge-permissive 100 points Only stops near-certain automation

All three already include exclusions: verified "good" bots (search engines, AI crawlers, ad and social crawlers, monitoring) and machine-facing paths where a challenge page would definitely break things - robots.txt and similar crawler files, static assets, APIs, RSS feeds, and webhooks. Don't install two of them at once: the documentation warns explicitly that the thresholds stack.

What it takes to make it work

  • CrowdSec 1.8.1 or newer. The feature landed in 1.8.0, and 1.8.1 already fixed a false positive on Brave with Shields enabled. If you installed CrowdSec from the official repository, as at the start of this article, a regular apt upgrade does the trick; cscli version shows what you've got.
  • A bouncer that can serve the challenge page. The Traefik plugin only learned to do this in v1.8.0, while part two of this series pins v1.7.1. As of this update, v1.8.0 only exists as the pre-release v1.8.0-alpha.
Warning

Don't enable the bot challenge with the old plugin. The CrowdSec documentation warns that behind a bouncer without support for it, the feature will most likely start silently rejecting every single client. And with crowdsecAppsecFailureBlock: true from our config, that means every service behind Traefik goes down at once.

On top of that, CrowdSec itself flags bot detection as an Alpha feature. So as of this update, I'd treat it as an experiment: trying it on one non-critical service is fine, turning it on for the whole homelab is premature.

Why you shouldn't just flip it on for everything

In my setup, the crowdsec@file middleware sits on the entrypoint, so requests to every service behind Traefik pass through AppSec. And the bot challenge is a JavaScript challenge page that only a browser can pass. Mobile apps and other programmatic clients will get an HTML page instead of the server's response, and they won't know what to do with it. The built-in exclusions only cover the usual suspects like /api/ and /webhooks/, while plenty of self-hosted services have clients hitting other paths - the mobile Vaultwarden app, for instance, authenticates via /identity/, and Gotify clients work with /message and /stream. There are browser-side limits as well: solving the puzzle requires SSE4.1 support and executable memory, so very old or deliberately hardened browsers won't get through.

So services that apps talk to need to be excluded from the challenge with a config of your own. Don't edit the hub files - they say so themselves - create your own instead, for example exempting the subdomains of those services entirely:

Terminal window
nano /etc/crowdsec/appsec-configs/my-bot-exclusions.yaml
yaml
name: custom/my-bot-exclusions
inband:
pre_eval:
- filter: "req.Host in ['vault.domain.ru', 'gotify.domain.ru']"
apply:
- ExemptFromChallenge("my-apps")

The Traefik access log is the best place to see what else needs excluding: whatever paths your apps hit, add those.

Turning it on

Once the plugin and CrowdSec are on the right versions, here's the order. Install the collection:

Terminal window
cscli collections install crowdsecurity/appsec-bot-challenge

Add the bot detection configs and your own exclusions to the AppSec source, /etc/crowdsec/acquis.d/appsec.yaml:

yaml
listen_addr: 127.0.0.1:7422
appsec_configs:
- crowdsecurity/appsec-default
- crowdsecurity/crs
- crowdsecurity/appsec-bot-*
- custom/my-bot-exclusions
name: myAppSecComponent
source: appsec
labels:
type: appsec

The crowdsecurity/appsec-bot-* wildcard picks up both the challenge engine (appsec-bot-challenge-scoring) and all the exclusions from the collection. That matters: the -balanced config on its own only carries the rejection threshold, and the challenge page is actually served by appsec-bot-challenge-scoring - without it, AppSec keeps processing requests but never challenges anyone.

In Traefik's static config, bump the plugin version:

yaml
experimental:
plugins:
bouncer:
moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
version: v1.8.0-alpha

And restart both services:

Terminal window
systemctl reload crowdsec
systemctl restart traefik

A few places where things can break without a single error in the logs:

  • The router has to let the /crowdsec-internal path through. The challenge page loads its script and puzzle worker from the absolute path /crowdsec-internal/challenge/.... Routers with a plain Host(...) rule let any path through, so they're fine. But if a router's rule includes a PathPrefix, you have to add that path explicitly, or the script won't load and the client stays stuck on the challenge page.
  • crowdsecAppsecBodyLimit must be greater than zero - otherwise the solved puzzle never reaches AppSec, and the browser loops on the challenge forever. The plugin defaults it to 10 MB and our config doesn't override it, so you're fine unless you set it to 0 yourself.
  • The Content-Security-Policy header. If your header middlewares set a strict CSP, it may block the challenge page from working. If the challenge won't pass, start there.

Checking bot detection

After the restart, the CrowdSec logs should show WAF challenge runtime initialized - that means the configuration got picked up. Then:

Terminal window
cscli metrics show appsec

The Ch. Requested counter should climb along with requests. If Processed is growing while Ch. Requested sits at zero, requests are reaching AppSec but nobody's being challenged, and the problem is in your appsec_configs list, not the bouncer. The simplest external test: open the site in a regular browser (the challenge page flashes by and lets you through), then request it with curl (which stays stuck on the challenge page because it can't solve the puzzle - exactly what you want).


Verification

Terminal window
cscli decisions list # currently active bans
cscli metrics # overall stats on scenarios and bouncers
cscli bouncers list # confirms that traefik-bouncer connected and is polling LAPI

If cscli bouncers list shows the bouncer but with a stale last-request time - check the connectivity between Traefik and localhost:8080 (the mere fact that both are in the same LXC doesn't guarantee the plugin is pointed at the right port/scheme).

From here the focus shifts from Traefik itself to the network around it - in the next part I move Traefik onto its own VLAN and set up firewall rules in OPNsense.