Yelp created a repository to aid in the processing of notifications via ELK logs. This repo went dormant, but another fork of it ElastAlert2 has replaced it. For me, this is personally an amazing application. I can get notifications, without having to buy into a commercial license.

Setup

Source: https://elastalert2.readthedocs.io/en/latest/running_elastalert.html#as-a-python-package

It requires Python 3.11+ If you don’t want to disrupt your current Python version, you can do an alt install of python. This is described in detail via: https://tecadmin.net/how-to-install-python-3-11-on-debian/

Once Python 3.11 is ready:

$ pip install elastalert2
$ elastalert-create-index

Configuration

Create a folder to house the notification rules, and config.yaml that the code requires. You can grab an example config.yaml from the elastalert2 GitHub repo, along with sample rule yaml files.

Inside the folder, you’ll have something like:

• config.yaml
• /rules/somerule.yaml

The config.yaml needs to be modified to point to the rules location, the ELK location, and access method to ELK.

Rule Example

Sample rules from the GitHub repo can be added into a rules folder and modified to your own fields you want to set up as a threshold.

Rules can filter on any field within the system. For example:

- filter
   http.response.status_code: 500
- filter
   agent.hostname: my-server123

In the above example, two filters are applied, one to filter on 500 response codes, and the other to filter on a specific server.

Email Notifications

Email notifications can be added to the bottom of the rule:

email:
- "me@mysite.com"

smtp_host: smtp.mysite.com
smtp_port: 25

I’ve been able to use gmail with a gmail account using a Google App API to set a secure token and pass that into config. If you don’t need to do that though… well it’s easier to just use your smtp host that you have access to.

Emails can be formatted with Jita:

alert_text_type: alert_text_jinja

alert_text: |
  Alert triggered! *({{num_hits}} Matches!)*
  500 level errors on {{related.ip}} ({{source.as.organization.name}})
  Endpoint: {{http.request.raw_request_line}}

Running ElastAlert2

ElastAlert2 can be run as a simple test of your rules by running the following example (replace with your pathing):

python -m elastalert.elastalert --verbose --rule /[path to your rules]/[your rule].yaml --config /[path to your config]/config.yaml 

If all works out you should see 1 rule processed and see the results.

Once you have several rules, you can omit the –rule param, so rules are run as configured in the –config param.

Creating a systemd task

You can create a simple systemd service on Debian, by creating a file in /etc/systemd/system/elastalert.service

It’s contents would be similar to:

[Unit]
Description=Email Notifications from ELK
After=multi-user.target

[Service]
Type=simple
Restart=always
User=ADD YOUR USERNAME
ExecStart=/usr/local/bin/python -m elastalert.elastalert --verbose --config ADD YOUR PATH TO CONFIG.YAML/config.yaml

[Install]
WantedBy=multi-user.target

Once done, start it in the normal fashion (sudo systemctl start elastalert2) and you can check status on it with sudo systemctl status elastalert2, or stop it with sudo systemctl stop elastalert2.

Documentation on Rules

For more on rules, see the official documentation at:

https://elastalert2.readthedocs.io/en/latest/