Proxylizer/Introduction

From MikroTik Wiki
Jump to navigation Jump to search

What is Proxylizer

Mikrotik Proxylizer is a convenient system with a web interface, for web-proxy log storage in database and report generation from stored logs.

Mikrotik web-proxy is able to send log entries to a remote location using syslog protocol. Remote host must process the log entries to get required statistics. Mikrotik Proxylizer is designed to accomplish this task in a convenient way.

With Mikrotik Proxylizer, received web-proxy logs are stored in a database for further processing. Using web interface system administrators define reports which are sent to a certain email address, what data must be collected, and when the report has to be generated. Periodic reports are available (daily, weekly and monthly).

Mikrotik Proxylizer can be used to:

  • Collect statistics about company staff member visited sites;
  • Detect spyware which sends information to remote web sites.

Features

Mikrotik Proxylizer features:

  • Web-proxy logs stored in SQL database (MySQL supported at the time);
  • Log filtering in reports based on host IP address and requested domain;
  • Reports include:
    • Overall user report - list of IP addresses with time spent on browsing web;
    • Specific user report - list of domains with time spent for specified IP address;
    • Domain report - list of the most popular domains with IP count and spent time;
  • Reports contain day of the week and time of the day restrictions;
  • Scheduled report generation: daily, weekly, monthly;
  • Reports are generated automatically without any user intervention;
  • Reports sent in emails as attached PDF (portable document format, platform independent);
  • All generated report history accessible in the web interface;
  • Reports are generated in background process;
  • Multiple reports can be generated in parallel to utilize multi-core processors efficiently;

Architecture

Mikrotik Proxylizer consists of multiple parts:

  • SQL database - storage of all log entries. MySQL supported at the time, other SQL database support planned in future;
  • Log reader - script responsible for log entry transfer from syslog to SQL database. Syslog-ng is used to listen on incoming syslog entries and write them to pipe file. PHP script is used to read the pipe and insert entries in database afterwards;
  • Report generator - every minute a script is started which takes the first report which must be generated and collects the required data from database. The result is sent to email and stored in database for later access in the web;
  • Web interface - configuration, statistics and history user interface for the system administrator.

The Proxylizer is interconnected with other system components: syslog entry source, SQL database, scheduler, SMTP server, web browser. Proxylizer contains scripts for database table structures and periodic tasks, therefore database and scheduler are treated as part of the Proxylizer. However both SQL database server and scheduler service are third party applications: MySQL and Cron in the current version. The collaboration is shown in the following diagram:

Proxylizer Collaboration

Requirements

Hardware requirements

Recommended requirements :

  • CPU: 1 GHz
  • RAM: at least the size of database,

We have tested it on a server with the following hardware:

  • CPU: Intel(R) Pentium(R) Dual Core 2.80GHz
  • RAM:1 GB

This server accepted insert of approximately 500 records per second. 1GB of disk space was used by approximately 4.5 million records.

Operating system and software requirements

Currently only the Linux distributions are supported. Proxylizer is tested on Ubuntu platform but there should be no problem running it on other distros, including Debian, SUSE, and others. Technically it is possible to install it on a MacOS machine, however it is not tested yet.

On the server side Proxylizer requires the following third-party software:

On the client side Proxylizer requires only a modern Web-browser. The following browsers are supported:

Network infrastructure requirements

Mikrotik Proxylizer utilizes the following communications (shown as arrows in the collaboration picture):

Mikrotik router to Proxylizer server

Mikrotik Router needs access to Proxylizer server to send syslog entries to it. It is sufficient to allow only communication through one port. Syslog uses port 514 by default.

Between Proxylizer server and database server

Both web scripts and data collector/report scripts need access to MySQL database. Usually MySQL database and web scripts are located on the same host machine without any additional infrastructure requirements. But it is also possible to use database server on remote host. In that case two requirements must be met:

  1. Proxylizer server needs access to the remote hosts database port, 3306 by default.
  2. MySQL must be configured to allow remote connections. This can be done in MySQL config files. On Ubuntu distribution they are located in /etc/mysql.

Between Proxylizer server and user browser

Proxylizer has Web-interface for configuration and report history review. To deliver the interface to users browser, HTTP traffic has to be sent over TCP connection. Ie., TCP connection between the user machine and Proxylizer server's HTTP port (80 by default or 443 for secure HTTPS connection) must be enabled. If the Proxylizer server is behind a firewall router, the HTTP traffic must be forwarded from the firewall to Proxylizer server, using DST-NAT. For example, if the firewall is Mikrotik router and Proxylizer has IP address 10.1.1.2, the NAT is enabled by typing the following command in the firewall router's console: 

/ip firewall nat add action=dst-nat protocol=tcp dst-port=80 to-addresses=10.1.1.2 to-ports=80

Proxylizer server to SMTP server

Proxylizer can send automatic email reports with statistics. To send email, it requires SMTP server. If the SMTP server and Proxylizer are running on the same host machine no additional infrastructure is required. If the SMTP server is located on a remote host, access to it for Proxylizer must be enabled. The default SMTP port is 25.

Retrieved from "https://wiki.mikrotik.com/index.php?title=Proxylizer/Introduction&oldid=10690"