Testwiki/Proxy (Cache) services
Contents
Proxy (Cache) services
Web proxy (/ip proxy)
MikroTik RouterOS performs proxying of HTTP and HTTP-proxy (for FTP, HTTP and HTTPS protocols) requests. Proxy server performs Internet object cache function by storing requested Internet objects, i.e., data available via HTTP and FTP protocols on a system positioned closer to the recipient in the form of speeding up customer browsing by delivering them requested file copies from proxy cache at local network speed. MikroTik RouterOS implements the following proxy server features:
- Regular HTTP proxy – customer (itself) specify what is proxy server for him
- Transparent proxy – customer does not know about the proxy being enabled and there isn’t need any additional configuration for web browser of client.
- Access list by source, destination, URL and requested method (HTTP firewall)
- Cache access list to specify which objects to cache, and which not.
- Direct Access List – to specify which resources should be accessed directly, and which - through another proxy server
- Logging facility – allows to get and to store information about proxy operation
- Parent proxy support – allows to specify other proxy server, ('if they don’t have the requested object ask their parents, or to the original server.)
A proxy server usually is placed at various points between users and the destination server (also known as origin server) on the Internet. (see Figure 10.1).
A Web proxy (cache) watches requests coming from client, saving copies of the responses for itself. Then, if there is another request for the same URL, it can use the response that it has, instead of asking the origin server for it again. If proxy has not requested file, it downloads that from the original server.
There can be many potential purpose of proxy server:
- To decrease access speed to resources (it takes less time for the client to get the object).
- Works as HTTP firewall (deny access undesired web pages),
Allows to filter web content (by specific parameters, like source address, destination address and port, URL, HTTP request method) scan outbound content, e.g., for data leak protection.
Note that it may be useful to have Web proxy running even with no cache when you want to use it only as something like HTTP and FTP firewall (for example, denying access undesired web pages or deny specific type of files e.g. .mp3 files) or to redirect requests to external proxy (possibly, to a proxy with caching functions) transparently.
Proxy configuration example
In MikroTik RouterOS proxy configuration are organized under /ip proxy menu. See below how to enable the proxy on port 8080 and set up 195.10.10.1 as proxy source address:
[admin@MikroTik] ip proxy> set enabled=yes port=8080 src-address=195.10.10.1
[admin@MikroTik] ip proxy> print
enabled: yes
src-address: 195.10.10.1
port: 8080
parent-proxy: 0.0.0.0:0
cache-drive: system
cache-administrator: "admin@mikrotik.com"
max-disk-cache-size: none
max-ram-cache-size: 100000KiB
cache-only-on-disk: yes
maximal-client-connections: 1000
maximal-server-connections: 1000
max-object-size: 2000KiB
max-fresh-time: 3d
When setting up regular proxy service, make sure it serves only your clients and prevent unauthorized access to it by creating firewall that allow only your clients to use proxy, otherwise it may be used as an open proxy.
Remember that regular proxy require also to configure client’s web browser settings.
For example:
|
Explorer 8.x |
Firefox 3.x |
Opera 10.x |
|
Select Tools>Internet options. Click the Connections tab. Select the necessary connection and choose Settings button. Configure proxy address and port. |
Select Tools>Options. Click the Advanced tab. Open the Network tab. Click the Connection/Settings Select Manual proxy configuration' |
Select Tool>Preferences. Open the Advanced tab/Network. Click the Proxy servers. Enter proxy address and port. |
Transparent proxy configuration example
RouterOS can also act as a Transparent Caching server, with no configuration required in the customer’s web browser. Transparent proxy does not modify requested URL or response. RouterOS will take all HTTP requests and redirect them to the local proxy service. This process will be entirely transparent to the user (users may not know anything about proxy server that is located between them and original server), and the only difference to them will be the increased browsing speed.
To enable the transparent mode, firewall rule in destination NAT has to be added, specifying which connections (to which ports) should be transparently redirected to the proxy. Check proxy settings above and redirect us users (192.168.1.0/24) to proxy server.
[admin@MikroTik] ip firewall nat> add chain=dstnat protocol=tcp src-address=192.168.1.0/24 dst-port=80 \ action=redirect to-ports=8080 [admin@MikroTik] ip firewall nat> print Flags: X - disabled, I - invalid, D - dynamic 0 chain=dstnat protocol=tcp dst-port=80 action=redirect to-ports=8000 [admin@MikroTik] ip firewall nat>
The web proxy can be used as transparent and normal web proxy at the same time. In transparent mode it is possible to use it as standard web proxy, too. However, in this case, proxy users may have trouble to reach web pages which are accessed transparently.
Proxy based firewall – Access List (/ip proxy access)
Access list is implemented in the same way as MikroTik firewall rules. Rules are processed from the top to the bottom. First matching rule specifies decision of what to do with this connection. Connections can be matched by its source address, destination address, destination port, substring of requested URL (Uniform Resource Locator) or request method. If none of these parameters is specified, every connection will match this rule.
If connection is matched by a rule, action property of this rule specifies whether connection will be allowed or not (deny). If connection does not match any rule, it will be allowed.
In this example assume that we have configured transparent proxy server as given example above. Argument description:
src-address - source address of the request
dst-address - destination address of the request
dst-port - destination port of the request
path - the URL of the request. Can be regular expression.
action - (allow / deny) action to take.
method - (any | connect | delete | get | head | options | post | put | trace) - HTTP method used in the request
Block particular Websites.
/ip proxy access add dst-host=www.facebook.com action=deny
It will block website http://www.facebook.com, we can always block the same for different networks by giving src-address.
/ip proxy access add src-address=192.168.1.0/24 dst-host=www.facebook.com action=deny
Users from network 192.168.1.0/24 not be able to access to website www.facebook.com.
You can block also websites that contain specific words in URL:
/ip proxy access add dst-host=:mail action=deny
This statement will block all websites which contain word “mail” in URL. Like www.mail.com, www.hotmail.com, mail.yahoo.com etc.
We can also stop downloading specific types of files like .flv, .avi, .mp4, .mp3, .exe, .dat, …etc.
/ip proxy access add path=*.flv action=deny add path=*.avi action=deny add path=*.mp4 action=deny add path=*.mp3 action=deny add path=*.zip action=deny add path=*.rar action=deny.
Here are available also different wildcard characters, to creating specific conditions and to match it by proxy access list.
Wildcard properties (dst-host and dst-path) match a complete string (i.e., they will not match "example.com" if they are set to "example"). Available wildcards are '*' (match any number of any characters) and '?' (match any one character).
Regular expressions are also accepted here, but if the property should be treated as a regular expression, it should start with a colon (':').
To show that no symbols are allowed before the given pattern, we use ^ symbol at the beginning of the pattern.
To specify that no symbols are allowed after the given pattern, we use $ symbol at the end of the pattern.
Direct access (/ip proxy direct)
If parent-proxy property is specified, it is possible to tell proxy server whether to try to pass the request to the parent proxy or to resolve it connecting to the requested server directly. Direct Access List is managed just like Proxy Access List and determines access to proxy server.
Unlike the access list, the direct proxy access list has default action equal to deny. It takes place when no rules are specified or a particular request did not match any rule.
Cache management (/ip proxy cache)
In the same way as you implemented proxy access-list you can define also access-list for proxy that filter proxy content. Cache access list specifies, which requests (domains, servers, pages) have to be cached locally by web proxy, and which not. Default action (if no matching rule is found) is to store object into cache memory.
More information and /ip proxy command references are found here:
http://wiki.mikrotik.com/wiki/Manual:IP/Proxy
DNS proxy (/ip dns)
DNS (Domain Name System) cache facility on MikroTik router to provide domain name resolution for router itself as well as for the connected clients. This facility allows router works as default DNS resolver for clients on the LAN.
Router (resolver) receives local client DNS requests and forwards to public DNS only those requests that are not yet in the local DNS cache.
Domain name are useful as argument to a local resolver, which retrieves information associated with the domain name (basically IP address). A user may ask for the host IP address associated with particular domain name (For example: What is IP address of domain name “www.mikrotik.com”?' -> Resolver return answer that IP address for www.mikrotik.com is 159.148.147.196).
DNS query and response from “Wireshark” traffic analyzer:
Domain Name System (query) No. Time Source Destination Protocol Info 614 80.581710 172.16.16.243 172.16.16.1 DNS(53) Standard query A www.mikrotik.com Domain Name System (response) No. Time Source Destination Protocol Info 615 80.592878 172.16.16.1 172.16.16.243 DNS(53) Standard query response A 159.148.147.196
A special type of request is sent to the resolver on a port 53. If you want to receive IP addresses of all public domain names your local resolver must know at least one public name server. When the resolver processes a user query that is not in local cache it ask to public name server for required information. In this way resolver learns content of name server.
DNS cache is used to minimize DNS requests to an external DNS server as well as to minimize DNS resolution time.
DNS proxy (cache) setup
A MikroTik router with DNS feature enabled can be set as a DNS server for any DNS-compliant client. When the remote requests are enabled (allow-remote-requests: yes), the MikroTik router responds to TCP and UDP DNS requests on port 53.
DNS cache management can be accessed under the /ip dns submenu. To enable DNS cache, use the set command, for example:
[admin@MikroTik] /ip dns> set servers=10.5.8.1,159.148.60.20
allow-remote-requests=yes
[admin@MikroTik] /ip dns> print
servers: 10.5.8.1,159.148.60.20
allow-remote-requests: yes
max-udp-packet-size: 512
cache-size: 2048KiB
cache-max-ttl: 1w
cache-used: 144KiB
[admin@MikroTik] /ip dns>
Moreover, MikroTik router can be specified as a primary DNS server under its dhcp-server settings.
Property Description
|
|
|
|
servers (IP address list; default: 0.0.0.0) |
comma separated list of DNS server IP addresses |
|
allow-remote-requests (yes | no; default: no) |
specifies whether to allow network requests |
|
cache-size (integer: 512..10240; default: 2048KiB) |
specifies the size of DNS cache in KiB |
|
cache-max-ttl (time; default: 1w) |
Specifies maximum time-to-live for cache records. In other words, cache records will expire unconditionally after cache-max-ttl time. Shorter TTL received from DNS servers are respected |
|
cache-used (read-only: integer) |
displays the current cache size in KiB |
Static DNS entries
MikroTik RouterOS allows to add static DNS entries. It allows you to link the particular domain names with the respective IP addresses and advertize this address to the DNS clients. Two main properties that must be specify are address and domain name:
- address – (IP address) IP address to resolve domain name with
- name – (text) DNS name to be resolved to a given IP address. May be a regular expression
[admin@MikroTik] /ip dns static> add name=admin.example.com address=1.1.1.2
Remember that the meaning of a dot (.) in regular expressions is any character, so the expression should be escaped properly. For example, if you need to match anything within example.com domain but not all the domains that just end with example.com, use name=".example.com" and this condition accept domain names like www.another-example.com, ns1.another-exampe.com.
For example:
[admin@MikroTik] /ip dns static> print detail Flags: D - dynamic, X - disabled, R - regexp 0 name="admin.example.com" address=1.1.1.2 ttl=1d 1 R name=".example.com" address=1.2.2.3 ttl=1d
MikroTik /ip dns command references you can find:
http://wiki.mikrotik.com/wiki/Manual:IP/DNS
ARP proxy
Main purpose of ARP protocol and operation were discussed at the beginning of this course material.
As we remember then IP addresses are used to determine destination of packet and which direction to forward packet next but hardware addresses must be used to actually transport data from one host to another. Router has a table of currently used ARP entries. Normally the table is built dynamically (ARP request and response), but to increase network security, it can be partially or completely built statically by adding static entries.
Before we speak about proxy ARP we need to know how ARP table looks and how create static ARP entries. When you create static entry you need to specify three parameters:
- Address – IP address to be mapped
- MAC-address – MAC address to be mapped to
- Interface – Interface on which to forward frames with that MAC address
Next example is showed how to add static MAC entries:
[admin@MikroTik] /ip arp> add address=10.10.10.3 mac-address=00:11:22:33:44:55 interface=ether2 [admin@MikroTik] /ip arp> print Flags: X - disabled, I - invalid, H - DHCP, D - dynamic # ADDRESS MAC-ADDRESS INTERFACE 0 D 10.0.1.14 00:0C:42:21:16:49 ether3 1 D 10.0.1.6 00:0C:42:43:39:00 ether1 2 10.10.10.3 00:11:22:33:44:55 ether2 [admin@MikroTik] /ip arp>
In RouterOS allows to set several ARP modes under interfaces configuration:
[admin@MikroTik] /interface ethernet> set 0 arp=? Arp ::= disabled | enabled | proxy-arp | reply-only [admin@MikroTik] /interface vlan> set 0 arp=? Arp ::= disabled | enabled | proxy-arp | reply-only [admin@MikroTik] /interface bridge> set 2 arp=? Arp ::= disabled | enabled | proxy-arp | reply-only
As we can see one of mode is proxy-arp that provides router to work as ARP proxy on specified interface.
Disabled – this means that arp on specific interface is turned off and ARP requests from clients behind this interface are not answered. Therefore, static arp entry should be added on the client machines. For example, the router's IP and MAC addresses should be added to the Windows workstations using the arp –s command:
C:\Windows\system32>arp -s 173.16.16.231 00-11-22-33-44-55
- Enabled – this mode is by default on all interfaces. It determines that router answered to client’s requests and new dynamic entries will be added to ARP table.
- Reply Only – in this mode ARP entries can be added only statically. Router is allowed to reply only to those static ARP entries found in the ARP table. In such case we can to increase network security.
- Proxy ARP – a router with properly configured proxy ARP feature acts like a transparent ARP proxy between directly connected networks. Proxy ARP is the technique in which one host, usually a router, answers to ARP requests intended for another host. Proxy ARP can help host on a subnet reach remote host in other subnet without the need to configure routing or a default gateway.
Example below is explained how proxy ARP works.
For Example, the Host A (172.16.1.2) on Subnet A want to send packets to Host D (172.16.2.3) on Subnet B. As shown in the figure 10.2, Host A has a /16 subnet mask this means that Host A believes that it is directly connected to all of network 172.16.0.0 (In the same LAN). Since the Host A believes that is directly connected it sends an ARP request to the destination to clarify MAC address of Host D. (in case when Host A finds that destination IP address is not from the same subnet it send packet to default gateway.)
Host A broadcasts an ARP request on Subnet A, as shown:
Info from packet analyzer software:
No. Time Source Destination Protocol Info
12 5.133205 00:1b:38:24:fc:13 ff:ff:ff:ff:ff:ff ARP Who has 173.16.2.3? Tell 173.16.1.2
Packet details:
Ethernet II, Src: (00:1b:38:24:fc:13), Dst: (ff:ff:ff:ff:ff:ff)
Destination: Broadcast (ff:ff:ff:ff:ff:ff)
Source: (00:1b:38:24:fc:13)
Type: ARP (0x0806)
Address Resolution Protocol (request)
Hardware type: Ethernet (0x0001)
Protocol type: IP (0x0800)
Hardware size: 6
Protocol size: 4
Opcode: request (0x0001)
[Is gratuitous: False]
Sender MAC address: 00:1b:38:24:fc:13
Sender IP address: 173.16.1.2
Target MAC address: 00:00:00:00:00:00
Target IP address: 173.16.2.3
With this ARP request, Host A (172.16.1.2) asks that Host D (172.16.2.3) send its MAC address. The ARP request packet is then encapsulated in an Ethernet frame with the MAC address of Host A as the source address and a broadcast (FF:FF:FF:FF:FF:FF) as the destination address. Layer 2 broadcast means that frame will be sent to all hosts in the same layer 2 broadcast domain which includes the ether0 interface of the router, but does not reach Host D, because router by default does not forward layer 2 broadcast.
Since the router knows that the target address (172.16.2.3) is on another subnet but it can reach Host D, it replies with its own MAC address to Host A.
No. Time Source Destination Protocol Info 13 5.133378 00:0c:42:52:2e:cf 00:1b:38:24:fc:13 ARP 172.16.2.3 is at 00:0c:42:52:2e:cf Packet details: Ethernet II, Src: 00:0c:42:52:2e:cf, Dst: 00:1b:38:24:fc:13 Destination: 00:1b:38:24:fc:13 Source: 00:0c:42:52:2e:cf Type: ARP (0x0806) Address Resolution Protocol (reply) Hardware type: Ethernet (0x0001) Protocol type: IP (0x0800) Hardware size: 6 Protocol size: 4 Opcode: reply (0x0002) [Is gratuitous: False] Sender MAC address: 00:0c:42:52:2e:cf Sender IP address: 172.16.1.254 Target MAC address: 00:1b:38:24:fc:13 Target IP address: 172.16.1.2
This is the Proxy ARP reply that the router sends to Host A. Router sends back unicast proxy ARP reply with its own MAC address as the source address and the MAC address of Host A as the destination address, by saying "send these packets to me, and I'll get it to where it needs to go."
When Host A receives ARP response it updates its ARP table, as shown:
C:\Users\And>arp -a Interface: 173.16.2.1 --- 0x8 Internet Address Physical Address Type 173.16.1.254 00-0c-42-52-2e-cf dynamic 173.16.2.3 00-0c-42-52-2e-cf dynamic 173.16.2.2 00-0c-42-52-2e-cf dynamic
After MAC table update, Host A forwards all the packets intended for Host D (172.16.2.3) directly to router interface ether0 (00:0c:42:52:2e:cf) and the router forwards packets to Host D. The ARP cache on the hosts in Subnet A is populated with the MAC address of the router for all the hosts on Subnet B. Hence, all packets destined to Subnet B are sent to the router. The router forwards those packets to the hosts in Subnet B.
When proxy ARP is used multiple IP addresses by host are mapped to a single MAC address (the MAC address of this router) which indicates that proxy ARP is in used.
Proxy ARP can be enabled on each interface individually with command arp=proxy-arp, as shown:
Setup proxy ARP:
[admin@MikroTik] /interface ethernet> set 1 arp=proxy-arp [admin@MikroTik] /interface ethernet> print Flags: X - disabled, R - running # NAME MTU MAC-ADDRESS ARP 0 R ether1 1500 00:30:4F:0B:7B:C1 enabled 1 R ether2 1500 00:30:4F:06:62:12 proxy-arp [admin@MikroTik] interface ethernet>
This behavior can be useful, for example, if you want to assign dial-in (ppp, pppoe, pptp) clients IP addresses from the same address space as used on the connected LAN.
