• Background
• Features
  • Database
  • Data Report
  • Visitor Map
• Installation
  • Docker
  • Parameters
  • Data Sources
  • Installation Extras +
• Setup
  • Setup Guide +
• Development
  • Feedback
  • In Progress
  • Pre-Release Notes
    • Detailed Changelog +
  • Local Installation +

BeatLog parses NGINX reverse proxy and fail2ban logs into readable tables and reports. Use BeatLog to assess server traffic and tailor fail2ban filters.

BeatLog is a Python based web application, and utilizes a PostgreSQL database for storage. Log files are parsed line-by-line using regex. BeatLog provides default patterns for parsing, but each log's regex scheme can be customized.

In addition to parsing data from log files, it categorizes each IP address as coming from "Home" (same IP as server) or "Outside" (anywhere else). Location information (coordinates, city, country) can be added to all "Outside" entries, using MaxMind's GeoLite2 database (SWAG users, see: mod installation). If only coordinates are provided from the MaxMind database, locations can be ascertained using the Nominatim reverse geocoding API, based on OpenStreetMap data.

See the BeatLog Documentation for a full description of features. The Database, Report, and Visitor Map are briefly highlighted here.

Data is saved in a PostgreSQL database and can be used for your own purposes. See the Processed Data section in the Setup Guide to see the table and field schema used for parsed log data, and the Database Explorer page to see how data can be queried and viewed within BeatLog. A simple JSON API provides daily summaries and bandwidth statistics.

Adminer can be installed to facilitate interaction with the database.

A report synthesizes all log data from the previous few days or a custom date range. Charts are integrated using CanvasJS, and Bootstrap is used for tables and styling. Within the report, you can:

• Analyze home and outside connections against fail2ban finds, bans, and ignores to assess efficacy of fail2ban filters.
• Scrutinize traffic from frequent visitors and monitor popular client requests
• Known Devices can be identified and separated from other outside connections

Documentation

Visitor locations are plotted onto an interactive map using LeafletJS and OpenStreetMap tiles.

• Tool tips show location names and total connections or unique visitors (IPs) over the selected time range
• Location marker sizes are scaled by total connections or unique visitors
• Tabular data is presented beneath the map

Documentation

BeatLog docker images are created via workflows with two different tags. "Stable" images are built and pushed with each release, and "Latest" images are built and pushed with each commit. Therefore, the stable or arm32v7-stable tags are recommended, unless there are pending updates that are desired.

Current release: alpha-0.1.7, In development: alpha-0.1.8

A PostgreSQL database is required, and can be included in the same docker deployment, as shown below. Or, connect to an existing database, by providing connection settings under environment:.

Logs and other files are added to the container via volumes. See the data sources section for specific files in the mounted directories and their usage. In the example below, the specified directories, and their contents, will be available within a created /import/ directory.

It is important to mount files that may change (log turnover, changed fail2ban settings, MaxMindDB updates) indirectly via their parent directories. Directly mounted files will not update within the container.

The compose parameters are detailed in the next section. Optional healthcheck and adminer additions are shown on the Installation Extras page.

With an existing, connectable database, depends_on: and the following lines are not needed. Ensure a database with the name specified in BeatLog's environment exists.

---
version: "2.1"
services:
  beatlog:
    image: nbpub/beatlog:stable
    container_name: beatlog
    user: 1000:1000 # optional	
    ports:
      - 5000:8000 # access from 5000 instead of 8000, for demonstration
    environment:
      - TZ=Pacific/Galapagos
      - db_host=<IP>
      - db_password=changeme	  
      - FLASK_SECRET_KEY=<secretkey>
      - check_IP=12
      - check_Log=3	  
    volumes:
      - /path_to/swag_config/log:/import/log # NGINX and fail2ban logs
      - /path_to/swag_config/fail2ban:/import/fail2ban # fail2ban jail.local
      - /path_to/swag_config/geoip2db:/import/geoip2db # MaxMindDB
    depends_on:
      - db
  db:
    image: postgres:15
    container_name: beatlog_db
    ports:
      - 5432:5432
    environment:
      - POSTGRES_USER=beatlog
      - POSTGRES_PASSWORD=changeme
      - TZ=Pacific/Galapagos	  
    volumes:
      - /path_to/beatlog_conifg/db:/var/lib/postgresql/data # recommended
    restart: unless-stopped   	

Container images are configured using parameters passed at runtime (such as those above). These parameters are separated by a colon and indicate <external>:<internal> respectively. For example, setting ports to 5433:5432 would expose port 5432 from inside the container to be accessible from the host's IP on port 5433 outside the container.

Sensitive data can be passed to compose using secrets, if desired.

See the Installation Extras page for more, including how to add healthchecks to the docker containers and how to update the database.

BeatLog reads the following files for the information described. See the Parsing and Processed Data sections of the Guide for more information.

• NGINX reverse proxy
  • access.log - client requests to the server
  • error.log - client request errors and associated severity levels
• fail2ban
  • fail2ban.log - all activity of fail2ban service, relevant information is parsed and the rest ignored
  • jail.local - fail2ban settings and activated filters, checks ignored IPs
• MaxMindDB
  • GeoLite2-City.mmdb - database to match IP addressess to locations, updated twice monthly

Create the container, monitor logs for proper startup, and then navigate to the WebUI at the port specified http://<your-ip>:5000.
If a database connection error is presented, check the parameters provided in your compose file and consult the container logs for more information.

Get the most information from BeatLog following these steps (detailed in Setup Guide):

1. Specify Log File locations */Logs/add/
2. Load default regex methods or create and test your own */Logs/add_regex/
3. Associate regex methods with Log Files to prepare for parsing */Logs/<log_file>/regex/
4. Specify fail2ban jail.local location */jail/
5. Specify MaxMind GeoLite2-City database location, in geography settings */settings#geography
6. Add a user-agent for the Nominatim API to fill unnamed locations, in geography settings
7. Parse Logs! Schedule parsing!

Details, Previous Versions

• fixed links on some BeatLog pages

Version-by-version details

• Local Installation
• Null Connection Pool
• Releases