Proxylizer/Concepts

Collected Log Data

Received logs from MikroTik router Web Proxy-server contain various information about web requests. Information stored from these logs is domain , IP addresses, event time and is the site loaded from proxy server cache or not. All URL info that is behind '?' is dropped off and not stored in data base, because this info is too large and it is unnecessary for monitoring Web Proxy server request statistics.

Received domain is stored in other table then logs and divided in 3 parts sub domain, domain and top domain. If domain has more than 3 parts, for example abc.def.ghi.mikrotik.com, it is divided in 'abc.def.ghi' + 'mikrotik' + 'com'.

It takes approximately 200KB of space to store 1000 log records in the data base. You can see data base statistic in the Status section on the Proxylizer web page.

Reports

TODO

• Aprakstīt reportu tipus: domēnu un useru
• Freqency
• Kā once atšķiras no pārējiem
• reportu var nosūtīt uz epastu. ja nesūta uz epastu - to var apskatīt webā
• date interval
• day interval
• weekdays
• time interval, var pielikt vairākus
• IP - var vai nu vienai konkrētai IP, vai visām IP. kā no tā mainās reportētie dati
• domain - var filtrēt pēc trim domēna daļām
• top - var paņemt populārākos ierakstus
• generate time - var nborādīt, kad ģenerēt
• reportu var uzģenerēt tikai tad, kad ir savākti visi dati. t.i., nedēļas reportu var uzģenerēt tikai nākošajā 1dienā. tomēr, ja reports ir 1dien-5dien un sūtīts tiek 6dienā, var nosūtīt tajā pašā nedēļā. princips - sūtīt, cik ātri vien iespējams

Created Reports

Reports consist of two parts: 1. Report configuration - contains information about what data has to be collected, when the reports will be generated and who will receive them; 2. The generated report - contains the actual data based on configuration rules.

The configuration is created in the report section on the Proxylizer web page.

Reports are generated automatically by a background script which is run by the scheduler.

When a configuration is deleted, all generated reports for that configuration are deleted as well.

Report Generation

TODO

• katru minūti tiek palaists skripts, kas nosaka, kādi reporti jāģenerē
• reporti ģenerējas "as fast s possible" - tiklīdz ir vajadzīgie dati un ir sasniegts ģenerēšanas laiks
• piemērs: šodien uztaisu reportu, kas sāk darboties mēneša sākumā, tiks uzģenerēti visi reporti no mēneša pirmās dienas līdz vakardienai
• ko nozīmē "vienlaikus ģenerējamo reportu skaits". links uz referencei, kur norādīts, kā to mainīt
• norādīt, kā var diagnosticēt reportu ģenerēšanos problēmas - ka tiek nosūtīts epasts un kur var atrast logus

Report History

The whole history of generated reports are available in the report section on the Proxylizer web page. Each report in the table has the (history) icon in the action column. Clicking on it reveals the list of generated reports for this configuration. A particular report can be accessed then by clicking on the (View) icon.

Inactive Reports

If a report is temporary not needed, it can be inactivated instead of deleting. It has the following advantages:

• The history is saved;
• Report can be activated later without creating new configuration.

During the time when a report is inactive, it is not generated. If it is activated afterwards, report will be generated again starting from the activation point. For example, if a daily report is inactive for a period of 20.jan.2009.-25.jan.2009. the next report generated after 19.jan.2009. will be the one about 26.jan.2009., reports about the inactivity period will be skipped.

Reports can be activated/inactivated in the Reports section on the Proxylizer Web page.

Report Editing

Report configurations can be edited after creating. But some restrictions apply:

• Reports with frequency "once" can be modified only before they are generated. When the report is processed, the configuration is read-only, because this is what "once" means;
• To other report types only the fields, which do not change the semantics of the report, can be edited. For example: date interval, weekdays and generate time can be changed for daily report as they all do not affect the contents of daily report - they only specify some properties of when the report must be generated. Report type, frequency, time interval, IP address and domain are read only, because changing these values leads to a completely different report.

IP Users

Each Web-proxy request has a source IP address - address of the host, which generated the web request. Reports can be filtered using this IP address.

Usually one static IP address corresponds to a specific staff member. To make the filtering by IP address easy, Proxylizer has the ability to assign real persons to IP addresses. This can be done in the IP Users section on the Proxylizer web page.

IP users are used only to assign person names and email addresses to IP addresses. IP users do not have access to Proxylizer web page.

Each IP User (person) has the following attributes:

• IP - address of the person's computer;
• Name;
• E-mail - person's address to which report emails can be sent;
• Admin - when checked, this user is treated as administrator. Some Proxylizer functions are designed for admins, for example, carbon copy (CC) of email reports can be sent to all admin email addresses;
• User receives empty reports - when checked, this user receives email reports even when they are empty (for example, reports of weekend data when no web requests are generated). This is useful for administrators to identify report generation problems - the administrator is sure that report will be sent anyway, and when it is not received, it happens only because of some system or infrastructure failure.

Database Statistics

Database (DB) statistics are shown in Status section on the Proxylize Web Page. The following statistics are available:

• HIT-MISS ratio: shows the web-proxy hit/miss ration - what part of all requests are found in the proxy's cache;
• Total domain count: total count of different domains stored in the database. Note that, for example, www.mikrotik.com and wiki.mikrotik.com are counted as two different domains;
• Total hit count: total count of requests logged by the web-proxy;
• Oldest record - oldest request stored in the DB right now. This determines the oldest point in time we have data about;
• Latest record - most recent web request logged in the DB. If this field contains an old value (let's say, more than one hour ago) this means one of the following: either the clients are not using web-proxy or requests are not logged to Proxylizer DB. This value can be used to detect system errors;
• Data base size: how big is the database currently. This can be used to identify how much the size of the DB grows daily, monthly etc.

As you could notice, these values are loaded not immediately after logging in the web interface, rather several seconds later. The reason is simple - calculating the statistics takes some time and to not force the web user to wait, they are loaded in background. Once the values are calculated, they are cached for the whole web session. To get the actual statistics, Refresh button can be used later.