CrowdSec Docker Part 3: Traefik Bouncer for Additional Security

In Parts 1 and 2 of this series, we saw CrowdSec + Firewall Bouncer and Cloudflare Bouncer for blocking malicious traffic before it reaches the origin server by Cloudflare WAF and after it reaches the origin server by the Firewall Bouncer.

Unfortunately, Cloudflare has a limit of 10,000 and the approximate number of banned IPs in the CrowdSec community blocklist is about 19,000. So, you are only being protected from about half the bad IPs.

This is where CrowdSec Traefik Bouncer fits in. For any attacker that accesses your server using the fully qualified domain name and gets through Cloudflare WAF, Traefik Bouncer can act as the defense. If you have another reserve proxy, then the concept is still the same. I will try to cover Nginx Reverse Proxy bouncer in a separate guide.

For now, let's see how to install Traefik Bouncer for CrowdSec to close the gap.

CrowdSec Traefik Bouncer

Traefik Bouncer is one of the easiest bouncers to setup. But we are going to do some extra steps to get the best out of Traefik and CrowdSec.

In my previous post on CrowdSec, I mentioned I was blocking up to 60 attacks per hour on my server. And, Cloudflare bouncer blocks about 750 attacks in 24 hours.

So, I rarely see any bad traffic reach Traefik. But if it does, we need protection. Hence why I added it to my Docker-Traefik stack.

Or you may be on the Cloudflare free plan with only one list allowed and you may already be using it. In this situation, you can ignore Cloudflare bouncer and just setup Traefik bouncer.

How does the Traefik bouncer work?

Bouncers are responsible for dropping traffic from IPs that are in the blocklists. They work with the CrowdSec API to access the decisions and take action.

The Traefik Bouncer works in the same way. It accesses recent decisions using the CrowdSec API and blocks those IPs from accessing services behind Traefik. As simple as that.

 Loading ...

Setup Traefik Bouncer

We are going to add Traefik bouncer as a docker container to our existing stack.

But before we added the Traefik bouncer docker container, let's do some recommended/optional tweaks.

1. Enable Proper Logging in Traefik

In Part 1, we setup CrowdSec with support for Traefik scenarios. CrowdSec can parse Traefik logs and identify malicious behaviors. Let's ensure proper logging is enabled on Traefik. I have the following CLI arguments defined in the Traefik compose:

      - --log.filePath=/logs/traefik.log
      - --log.level=INFO # (Default: error) DEBUG, INFO, WARN, ERROR, FATAL, PANIC
      - --accessLog=true
      - --accessLog.filePath=/logs/access.log
      - --accessLog.bufferingSize=100 # Configuring a buffer of 100 lines
      - --accessLog.filters.statusCodes=204-299,400-499,500-599

I am logging both Traefik container logs and service access logs. For access logs, I have defined a few access codes that I want to log.

While this is not necessary for the Traefik bouncer to work, feeding Traefik logs will ensure that attacks on Traefik are being detected and processed along with other points of attack to build the blocklists.

2. Create a CrowdSec API Key for Traefik Bouncer

Before we start configuring the Traefik Bouncer, let's generate an API key to allow it to connect to CrowdSec API (aka LAPI). Use the following command to generate the key for traefik-bouncer-dshb (name it whatever you want). Remember to specify the correct path of your docker-compose file.

sudo docker compose -f /home/USERNAME/docker/docker-compose-t2-web.yml exec -t crowdsec cscli bouncers add traefik-bouncer-dshb

Your API key should be generated and displayed as shown below.

Note it down.

Be the 1 in 200,000. Help us sustain what we do.

123 / 150 by Dec 31, 2024

You will gain benefits such as Deployarr access, discord roles, exclusive content, ad-free browsing, and more.

🔥 Holiday Sale! 25% Off Platinum Membership $399.99 $299.99 (ends December 31).

Join the Geek Army (starting from just $1.67/month)

Why did we start Memberships?

3. Traefik Bouncer Docker Compose

Next, let's add the Traefik Bouncer docker-compose to our stack:

  # CrowdSec Bouncer - Traefik
  # sudo docker exec crowdsec cscli bouncer add traefik-bouncer
  traefik-bouncer:
    <<: *common-keys-core # See EXTENSION FIELDS at the top
    image: fbonalair/traefik-crowdsec-bouncer
    container_name: traefik-bouncer
    environment:
      GIN_MODE: release # default is debug (more logs)
      CROWDSEC_BOUNCER_API_KEY: $CROWDSEC_BOUNCER_TRAEFIK_API_KEY # sudo docker exec crowdsec cscli bouncers add traefik-bouncer
      CROWDSEC_AGENT_HOST: crowdsec:8080 # CrowdSec host and port
      CROWDSEC_BOUNCER_LOG_LEVEL: 2 # https://pkg.go.dev/github.com/rs/zerolog#readme-leveled-logging

If you have been following my guides, understanding the above code should be easy. But here is a brief explanation with relevant links:

• *common-keys-core - Extension field as described my Docker guide. Extension fields minimize the repetition of compose statements. common-keys-core sets the restart policy to always and makes the container a part of the t2_proxy network.
• $CROWDSEC_BOUNCER_TRAEFIK_API_KEY - Environmental variable that needs to be created with the API key generated in the previous step.
• CROWDSEC_AGENT_HOST - URL for CrowdSec agent. Since both CrowdSec and Traefik bouncer are on the same network (t2_proxy), we can reach CrowdSec using the hostname (crowdsec).

Save, exit, and start the container. If the container starts and does not exit with errors, then you are good. Unfortunately, little to nothing is output to Docker container logs. So we won't be able to see any confirmations.

What we could do is check the bouncers list using the command:

sudo docker compose -f /home/USERNAME/docker/docker-compose-t2-web.yml exec -t crowdsec cscli bouncers list

You should see Traefik bouncer listed with a check mark for a valid API key. You will not see any version information until we start using the bouncer in the steps below.

4. Create Traefik Forward Auth Middleware

The bouncer is running but we haven't setup our Docker services to use it yet. To do that, we have to add Traefik bouncer as a middleware to route all requests through it.

Open middlewares.yaml file inside /home/USER/docker/appdata/traefik2/rules/ folder (if you are following the folder structure recommended in my guides) and add the following middleware:

    middlewares-traefik-bouncer:
      forwardAuth:
        address: "http://traefik-bouncer:8080/api/v1/forwardAuth" 
        trustForwardHeader: true

Note that formatting is important in YAML. If you have any doubts about where to add this, check my GitHub Repo.

This middleware should be picked up on the fly by Traefik. But we are still not using it.

5. Add Traefik Bouncer to Middlewares Chain

In my Traefik guides, I use middlewares-chain.yml to define a sequence of middlewares to be used for services.

Add middlewares-traefik-bouncer as the first middleware in all of the chains. For example, the chain for Google OAuth should look like this:

    chain-oauth:
      chain:
        middlewares:
          - middlewares-traefik-bouncer 
          - middlewares-rate-limit
          - middlewares-https-redirectscheme
          - middlewares-secure-headers
          - middlewares-oauth
          - middlewares-compress

Again, check middlewares-chain.yml in my GitHub Repo if you have any questions.

That's it. All the services with chain-oauth@file middleware defined in Docker labels should now be protected by CrowdSec.

6. Verify Traefik Bouncer

Unfortunately, I do not know an easy way to check if the Traefik bouncer is working. We can export the metrics to Prometheus, feed them to Grafana, and visualize it there. But that is a whole other topic.

One option is to fake a ban as described in the Traefik Bouncer GitHub page.

But a lazy way to check is to see if Traefik Bouncer is connected and talking to the CrowdSec API using the following command:

sudo docker compose -f /home/USERNAME/docker/docker-compose-t2-web.yml exec -t crowdsec cscli bouncers list

When we ran the above command previously, we did not see any IP information or version information for the Traefik bouncer. This time around we should see the Traefik Bouncer connected to the API as shown below.

That's it. Traefik Bouncer should start doing its job and protecting you from attacks.

 Loading ...

Limitations and Things to Know

CrowdSec Traefik Bouncer works great. However, there is one limitation.

Traefik Bouncer only works when your server is accessed through a fully qualified domain name. If the attacker has your IP address and reaches your server using IP then CrowdSec Traefik Bouncer will not protect you.

That responsibility will fall on your Host's Firewall Bouncer.

Concluding Remarks

There you have it. Now we have Traefik Bouncer to take some of the load off of the Firewall Bouncer and catch anything missed by the Cloudflare Bouncer.

Since Traefik version 2.3, plugins are supported in Traefik. The same effect can also be achieved using the CrowdSec Bouncer Traefik Plugin. Maybe I will cover that in a separate post.

Setting up CrowdSec Traefik Bouncer was much easier than the Cloudflare Bouncer. I strongly recommend adding it to catch any attempts missed by the Cloudflare bouncer or if you cannot use the Cloudflare Bouncer for some reason.