Difference between revisions of "NetworkPro on Quality of Service"

From MikroTik Wiki
Jump to: navigation, search
(will finish this a bit later)
m (will edit later, this article can take indefinite amount of info)
Line 3: Line 3:
 
==Theory==
 
==Theory==
  
Let's begin with some theory that I've accumulated over the years. It should save your life when dealing with bandwidth management and QoS. ''(Italic text are my comments)''
+
Let's begin with some theory that I've accumulated over the years.<br/>It should save your life when dealing with '''Bandwidth Management and QoS'''.
  
===TCP rate control===
+
===TCP control for QoS===
  
 
[http://en.wikipedia.org/wiki/Bandwidth_management wikipedia.org/wiki/Bandwidth_management] TCP rate control - artificially adjusting TCP window size as well as controlling the rate of ACKs being returned to the sender.  
 
[http://en.wikipedia.org/wiki/Bandwidth_management wikipedia.org/wiki/Bandwidth_management] TCP rate control - artificially adjusting TCP window size as well as controlling the rate of ACKs being returned to the sender.  
  
''RouterOS HTB can adjust TCP window by dropping packets out of the set bandwidth limit, but (small) ACKs should never be dropped!''
+
{{ Note | The information at The Wikipedia.org website is considered inaccurate, biased, too general, or insecure }}
  
''More on TCP and the ACK packet:'' [http://en.wikipedia.org/wiki/ACK_(TCP) wikipedia.org/wiki/ACK_(TCP)]
+
RouterOS HTB can adjust the TCP window by dropping packets out of the set bandwidth limit.
  
===TCP Window===
+
{{ Note | The TCP ACKs should never be dropped! (refer to the example at the end of the article)}}
 +
 
 +
''More general information on TCP and the ACK packet:'' [http://en.wikipedia.org/wiki/ACK_(TCP) wikipedia.org/wiki/ACK_(TCP)]
 +
 
 +
===The TCP Window===
  
 
A TCP window is the amount of data a sender can send on a particular TCP connection before it gets an acknowledgment (ACK packet) back from the receiver that it has gotten some of it.
 
A TCP window is the amount of data a sender can send on a particular TCP connection before it gets an acknowledgment (ACK packet) back from the receiver that it has gotten some of it.
Line 21: Line 25:
 
The primary reason for the window is congestion control. The whole network connection, which consists of the hosts at both ends, the routers in between and the actual physical connections themselves (be they fiber, copper, satellite or whatever) will have a bottleneck somewhere that can only handle data so fast. Unless the bottleneck is the sending speed of the transmitting host, then if the transmitting occurs too fast the bottleneck will be surpassed resulting in lost data. The TCP window throttles the transmission speed down to a level where congestion and data loss do not occur.
 
The primary reason for the window is congestion control. The whole network connection, which consists of the hosts at both ends, the routers in between and the actual physical connections themselves (be they fiber, copper, satellite or whatever) will have a bottleneck somewhere that can only handle data so fast. Unless the bottleneck is the sending speed of the transmitting host, then if the transmitting occurs too fast the bottleneck will be surpassed resulting in lost data. The TCP window throttles the transmission speed down to a level where congestion and data loss do not occur.
  
===Control is over what is Sent Out===
+
===Control is Outbound===
  
 
Inbound traffic for router - traffic that hits routers' interfaces, no matter from what side - Internet or local - it will be received by interface no matter what, even malformed packets, and you cannot do anything with these.
 
Inbound traffic for router - traffic that hits routers' interfaces, no matter from what side - Internet or local - it will be received by interface no matter what, even malformed packets, and you cannot do anything with these.
 
Outbound traffic for router - traffic that goes out of routers' interfaces, no matter of direction, to your network or out of it. This is where you can set up queues and prioritize, and limit!
 
Outbound traffic for router - traffic that goes out of routers' interfaces, no matter of direction, to your network or out of it. This is where you can set up queues and prioritize, and limit!
  
HTB allows to order and/or shape outgoing traffic (traffic that is leaving router via any interface).  
+
HTB allows us to work with the Outgoing traffic (the traffic that is leaving the router via any interface).  
  
 
Example1: Client sends out  10Mbps UDP traffic - this traffic will get to the routers local interface, but in one of the HTBs (global-in, global-total, global-out or outgoing interface) it will be shaped to (let’s say) 1Mbps. So only 1Mbps will leave the router. But in the next second the client can send 10Mbps once again and we will shape them again.
 
Example1: Client sends out  10Mbps UDP traffic - this traffic will get to the routers local interface, but in one of the HTBs (global-in, global-total, global-out or outgoing interface) it will be shaped to (let’s say) 1Mbps. So only 1Mbps will leave the router. But in the next second the client can send 10Mbps once again and we will shape them again.
Line 41: Line 45:
 
2) and 4) - is Outbound traffic
 
2) and 4) - is Outbound traffic
  
HTB can control 1) into 2) and 3) into 4)
+
HTB can control 1) when its sent out as 2) and 3) when its sent out as 4)
  
 
{{Note | Connections can't be upload or download, packets are. For example TCP connections have traffic in both directions.}}
 
{{Note | Connections can't be upload or download, packets are. For example TCP connections have traffic in both directions.}}
  
''Visual representation:''
+
===QoS Packet Flow & Double Control===
  
 
[[Image:QoS_Packet_Flow.gif]]
 
[[Image:QoS_Packet_Flow.gif]]
  
QoS includes several facilities, in the following order:
+
Working with packets for bandwidth management is done in this order:
  
 
1. Mangle chain prerouting<br/>
 
1. Mangle chain prerouting<br/>
Line 58: Line 62:
 
6. HTB out interface<br/>
 
6. HTB out interface<br/>
  
So, in one router, you can shape twice if you use:
+
So, in one router, you can do:
 +
 
 +
a) #1 +#2 - first marking & shaping, #3+#5 - second marking & shaping<br/>
 +
b) #1 +#2 - first marking & shaping, #3+#6 - second marking & shaping<br/>
 +
c) #1 +#2 - first marking & shaping, #4+#5 - second marking & shaping<br/>
 +
d) #1 +#2 - first marking & shaping, #4+#6 - second marking & shaping<br/>
  
a) #1 and #2 for first marking and shaping, and #3+#5 for second<br/>
+
Double Control is achieved with '''Queue Tree'''
b) #1 and #2 for first marking and shaping, and #3+#6 for second<br/>
 
c) #1 and #2 for first marking and shaping, and #4+#5 for second<br/>
 
d) #1 and #2 for first marking and shaping, and #4+#6 for second<br/>
 
  
 
===HTB Queue Tree===
 
===HTB Queue Tree===
  
Queue tree creates an unidirectional queue in one of the HTBs. It is also The way to add a queue on a separate interface. This way (setting up queue tree on the interface HTB: parent:ether1/pppoe1…) it is possible to ease mangle configuration - you don't need separate marks for download and upload - only upload will get to Private interface and only download will get to Public interface. (If you are marking connections + their packets)
+
Queue Tree is an unidirectional queue in one of the HTBs. It is also The way to add a queue on a separate interface (parent:ether1/pppoe1..) - this way it is possible to ease mangle configuration - you don't need separate marks per outgoing interface.<br/>
Also it is possible to have double queuing (example: prioritization of traffic in global-in or global-out, limitation per client on the outgoing interface). If you have simple queues and queue tree in the same HTB - simple queues will get traffic first.  
+
Also it is possible to have double queuing, for example: prioritization of traffic in global-in or global-out, and limitation per client on the outgoing interface.
Queue tree is not ordered - all traffic passes it together (unlike simple queue where the order matters).
+
{{ Note | If you have simple queues and queue tree in the same HTB - simple queues will get the traffic first, so the Queue Tree after that will not work. }}
 +
{{ Note | Queue Tree is not ordered - all traffic passes it together, where as with Queue Simple - traffic is evaluated by each Simple Queue, one by one, from top to bottom. If the traffic matches the Simple Queue - then its managed by it, otherwise its passed down. }}
 +
 
 +
===Guaranteeing Bandwidth with the HTB "Priority" Feature===
  
priority (1..8) : Prioritize one child queue over other child queue. Does not work on parent queues (if queue has at least one child). One is the highest, eight is the lowest priority. Child queue with higher priority will have chance to reach its limit-at before child with lower priority and after that child queue with higher priority will have chance to reach its max-limit before child with lower priority. Priority has nothing to do with bursts.
+
See the official article http://wiki.mikrotik.com/wiki/Manual:HTB
  
Large queue sizes can increase latency, but utilize channel better.
+
We already know that '''limit-at''' ('''CIR''') to all queues will be given out no matter what.  
?Parent queue have to have match all the traffic that will go for child queues.?
 
  
as for TCP, you may increase queue size if you have drops
+
Priority is responsible for distribution of remaining parent queues traffic to child queues so that they are able to reach '''max-limit'''
  
in real-time UDP traffic queues should be smaller to drop packets out of bandwidth limit, because they almost useless when arrived with delay
+
Queue with higher priority will reach its '''max-limit''' before the queue with lower priority. 8 is the lowest priority, 1 is the highest.
 +
 +
Make a note that priority only works:
 +
*  for '''leaf''' queues - priority in '''inner''' queue have no meaning.
 +
*  if '''max-limit''' is specified (not 0)
  
so, see what exactly you need
+
"Priority" feature (1..8) : Prioritize one child queue over other child queue. Does not work on parent queues (if queue has at least one child). One is the highest, eight is the lowest priority. Child queue with higher priority will have chance to reach its limit-at before child with lower priority (''contradiction - to be confirmed'') and after that child queue with higher priority will have chance to reach its max-limit before child with lower priority.
  
'''Chupaka''' http://forum.mikrotik.com/viewtopic.php?f=2&t=12870
+
{{ Note | The "Priority" feature has nothing to do with the [http://wiki.mikrotik.com/wiki/Manual:Queues_-_Burst "bursts" HTB feature].}}
  
Because PCQ does not affect child queues, it really does not matter, what type you set for parent: parent is just to see total traffic. (type applies only to Child queues)
+
{{ Note | Queue type applies only to Child queues. It doesn't matter what queue type you set for the parent. The parent is only used to set the max-limit and to group the leaf queues. Without setting the max-limit properly, the HTB Queue Tree will not drop enough low-priority packets, so the bandwidth control would lost. In order to have Control - we must set the max-limit to a lower value - 99.99% to 85% of the tested throuput of the bottleneck }}
  
 
'''Janis'''
 
'''Janis'''

Revision as of 18:28, 10 May 2010

Version.png

Applies to RouterOS: v2.9, v3, v4, v5

Theory

Let's begin with some theory that I've accumulated over the years.
It should save your life when dealing with Bandwidth Management and QoS.

TCP control for QoS

wikipedia.org/wiki/Bandwidth_management TCP rate control - artificially adjusting TCP window size as well as controlling the rate of ACKs being returned to the sender.

Icon-note.png

Note: The information at The Wikipedia.org website is considered inaccurate, biased, too general, or insecure


RouterOS HTB can adjust the TCP window by dropping packets out of the set bandwidth limit.

Icon-note.png

Note: The TCP ACKs should never be dropped! (refer to the example at the end of the article)


More general information on TCP and the ACK packet: wikipedia.org/wiki/ACK_(TCP)

The TCP Window

A TCP window is the amount of data a sender can send on a particular TCP connection before it gets an acknowledgment (ACK packet) back from the receiver that it has gotten some of it.

For example if a pair of hosts are talking over a TCP connection that has a TCP window size of 64 KB (kilobytes), the sender can only send 64 KB of data and then it must stop and wait for an acknowledgment from the receiver that some or all of the data has been received. If the receiver acknowledges that all the data has been received then the sender is free to send another 64 KB. If the sender gets back an acknowledgment from the receiver that it received the first 32 KB (which could happen if the second 32 KB was still in transit or it could happen if the second 32 KB got lost or dropped, shaped), then the sender could only send another 32 KB since it can't have more than 64 KB of unacknowledged data.

The primary reason for the window is congestion control. The whole network connection, which consists of the hosts at both ends, the routers in between and the actual physical connections themselves (be they fiber, copper, satellite or whatever) will have a bottleneck somewhere that can only handle data so fast. Unless the bottleneck is the sending speed of the transmitting host, then if the transmitting occurs too fast the bottleneck will be surpassed resulting in lost data. The TCP window throttles the transmission speed down to a level where congestion and data loss do not occur.

Control is Outbound

Inbound traffic for router - traffic that hits routers' interfaces, no matter from what side - Internet or local - it will be received by interface no matter what, even malformed packets, and you cannot do anything with these. Outbound traffic for router - traffic that goes out of routers' interfaces, no matter of direction, to your network or out of it. This is where you can set up queues and prioritize, and limit!

HTB allows us to work with the Outgoing traffic (the traffic that is leaving the router via any interface).

Example1: Client sends out 10Mbps UDP traffic - this traffic will get to the routers local interface, but in one of the HTBs (global-in, global-total, global-out or outgoing interface) it will be shaped to (let’s say) 1Mbps. So only 1Mbps will leave the router. But in the next second the client can send 10Mbps once again and we will shape them again.

Example2: Client sends out 10Mbps TCP traffic - this traffic will get to the routers local interface, but in one of the HTBs (global-in, global-total, global-out or outgoing interface) it will be shaped to (let’s say) 1Mbps. So only 1Mbps will leave the router. Source gets ACK replies only for 1Mbps of 10Mbps, so source, in the next second, will send a little more than 1Mbps of TCP traffic. (due to TCP Window adjusting to our shaped bandwidth)

There are 4 ways we can look at a flow:
1) client upload that router receives on the local interface
2) client upload that router sends out to the Internet
3) client download that router receives on the public interface
4) client download that router sends out to the customer

1) and 3) - is Inbound traffic
2) and 4) - is Outbound traffic

HTB can control 1) when its sent out as 2) and 3) when its sent out as 4)

Icon-note.png

Note: Connections can't be upload or download, packets are. For example TCP connections have traffic in both directions.


QoS Packet Flow & Double Control

QoS Packet Flow.gif

Working with packets for bandwidth management is done in this order:

1. Mangle chain prerouting
2. HTB global-in
3. Mangle chain forward
4. Mangle chain postrouting
5. HTB global-out
6. HTB out interface

So, in one router, you can do:

a) #1 +#2 - first marking & shaping, #3+#5 - second marking & shaping
b) #1 +#2 - first marking & shaping, #3+#6 - second marking & shaping
c) #1 +#2 - first marking & shaping, #4+#5 - second marking & shaping
d) #1 +#2 - first marking & shaping, #4+#6 - second marking & shaping

Double Control is achieved with Queue Tree

HTB Queue Tree

Queue Tree is an unidirectional queue in one of the HTBs. It is also The way to add a queue on a separate interface (parent:ether1/pppoe1..) - this way it is possible to ease mangle configuration - you don't need separate marks per outgoing interface.
Also it is possible to have double queuing, for example: prioritization of traffic in global-in or global-out, and limitation per client on the outgoing interface.

Icon-note.png

Note: If you have simple queues and queue tree in the same HTB - simple queues will get the traffic first, so the Queue Tree after that will not work.


Icon-note.png

Note: Queue Tree is not ordered - all traffic passes it together, where as with Queue Simple - traffic is evaluated by each Simple Queue, one by one, from top to bottom. If the traffic matches the Simple Queue - then its managed by it, otherwise its passed down.


Guaranteeing Bandwidth with the HTB "Priority" Feature

See the official article http://wiki.mikrotik.com/wiki/Manual:HTB

We already know that limit-at (CIR) to all queues will be given out no matter what.

Priority is responsible for distribution of remaining parent queues traffic to child queues so that they are able to reach max-limit

Queue with higher priority will reach its max-limit before the queue with lower priority. 8 is the lowest priority, 1 is the highest.

Make a note that priority only works:

  • for leaf queues - priority in inner queue have no meaning.
  • if max-limit is specified (not 0)

"Priority" feature (1..8) : Prioritize one child queue over other child queue. Does not work on parent queues (if queue has at least one child). One is the highest, eight is the lowest priority. Child queue with higher priority will have chance to reach its limit-at before child with lower priority (contradiction - to be confirmed) and after that child queue with higher priority will have chance to reach its max-limit before child with lower priority.

Icon-note.png

Note: The "Priority" feature has nothing to do with the "bursts" HTB feature.


Icon-note.png

Note: Queue type applies only to Child queues. It doesn't matter what queue type you set for the parent. The parent is only used to set the max-limit and to group the leaf queues. Without setting the max-limit properly, the HTB Queue Tree will not drop enough low-priority packets, so the bandwidth control would lost. In order to have Control - we must set the max-limit to a lower value - 99.99% to 85% of the tested throuput of the bottleneck


Janis In my presentation I told that creating priorities separately for each client is suicide - there is no hardware that can handle small queue tree for every user (if you have 1000 of them). So in my presentation I discuss next best thing, which is as close as possible to desired behavior.

The main Idea of the setup is to have two separate QoS steps:

1) In the first step we prioritize traffic, we are making sure that traffic with higher priority has more chance to get to the customers than traffic with lower priority.

Example: We have total of 100Mbps available, but clients at this particular moment would like to receive 10Mbps of Priority=1 traffic, 20Mbps of Priority=4 and 150Mbps of Priority=8. Of course after our prioritization and limitation 80Mbps of priority=8 will be dropped. And only 100Mbps total will get to the next step.

2) Next step is per-user limitation, we already have only higher priority traffic, but now we must make sure that some user will not overuse it, so we have PCQ with limits.

This way we get virtually the same behavior as "per user prioritization".

Macgaiver So the plan for you might be to mark by traffic type in prerouting and limit by traffic type in global-in. Then remark traffic by IP addresses in forward and limit them on the outgoing interface.

1) you need to mark all traffic at the same place (prerouting) (that would be managed by one particular Queue) 2) you must mark upload and download for every type of traffic separately (if you use global-total/in/out or Queue Simple or if you use Queue Tree and you do not mark connections first or if you do not set up the Queue in the interface HTB) 4) you must have a parent queue, that has max-limit and (let’s say) parent=global-in - all other queues parent=<parent> (for proper Queue Tree or in case simple PCQ – just one Queue can manage all) 5) you need 2 sets of those queues - one for upload, one for download

Priority doesn't work without limitation.

1) HTTP browsing connection usually is not more than 0,5MB (or 4Mb) 2) Mangle facility can mark only part of connection - you must use mark-packet directly without mark-connection (and mark separately upload and download traffic) 3) create two marks "first_bytes" and "last_bytes" 4) create queue structure in queue tree (one for download on local interface, one for upload on public interface) queue structure must have 3 queues: a) parent with max-limit b) queue for "first_bytes" with priority=1 and limit-at and max-limit specified c) queue for "last_bytes" wint priority=8 and limit-at and max-limit specified

That is it - I use it every day to prioritize normal HTTP over other traffic on port 80 (like downloads and any other large transfer that would use port 80)

Janis Each simple queue creates 3 separate queues: One in global-in (“direct” part) One in Global-out (“reverse” part) One in Global-total (“total” part) Simple queues are ordered - similar to firewall rules further down = longer packet processing further down = smaller chance to get traffic (so it’s necessary to reduce number of queues)

Queuing Placement

Limitation for in mangle chain “forward” marked traffic can be placed in the “global-out” or interface queue (see packet flow diagram)

If queues will be placed in the interface queues queues on the public interface will capture only client upload queues on the local interface will capture only client's download

If queues will be placed in global-out download and upload will be limited together (separate marks needed)

dot-bot In the case of Simple Queues, the order is for 'catching traffic' (mangle) and the priority is for packet queue management in the HTB.


Create packet marks in the mangle chain “Prerouting” for traffic prioritization in the global-in Queue

Example

Upload QoS for ADSL, tested and seems working good enough. Can be easily adapted and upgraded for all needs - up/down, ADSL or dedicated optic.

/ip firewall mangle
add action=mark-packet chain=postrouting out-interface=ADSL1 passthrough=no
    new-packet-mark=QoS_1_Up dst-port=80,443 packet-size=0-666 protocol=tcp tcp-flags=syn comment=QoS
add action=mark-packet chain=postrouting out-interface=ADSL1 passthrough=no
    new-packet-mark=QoS_1_Up dst-port=80,443 packet-size=0-123 protocol=tcp tcp-flags=ack
add action=mark-packet chain=postrouting out-interface=ADSL1 passthrough=no
    new-packet-mark=QoS_1_Up dst-port=53,123 protocol=udp
add action=mark-packet chain=postrouting out-interface=ADSL1 passthrough=no
    new-packet-mark=QoS_2_Up dst-port=80,443 connection-bytes=0-1000000 protocol=tcp
add action=mark-packet chain=postrouting out-interface=ADSL1 passthrough=no
    new-packet-mark=QoS_2_Up dst-port=110,995,143,993,25,20,21 packet-size=0-666 protocol=tcp tcp-flags=syn
add action=mark-packet chain=postrouting out-interface=ADSL1 passthrough=no
    new-packet-mark=QoS_2_Up dst-port=110,995,143,993,25,20,21 packet-size=0-123 protocol=tcp tcp-flags=ack
add action=mark-packet chain=postrouting out-interface=ADSL1 passthrough=no
    new-packet-mark=QoS_3_Up packet-size=0-666 protocol=tcp tcp-flags=syn
add action=mark-packet chain=postrouting out-interface=ADSL1 passthrough=no
    new-packet-mark=QoS_3_Up packet-size=0-123 protocol=tcp tcp-flags=ack
add action=mark-packet chain=postrouting out-interface=ADSL1 passthrough=no
    new-packet-mark=QoS_4_Up dst-port=110,995,143,993,25,20,21 protocol=tcp
add action=mark-packet chain=postrouting out-interface=ADSL1 passthrough=no
    new-packet-mark=QoS_4_Up dst-port=80,443 connection-bytes=1000000-0 protocol=tcp
add action=mark-packet chain=postrouting out-interface=ADSL1 passthrough=no
    new-packet-mark=QoS_8_Up p2p=all-p2p
add action=mark-packet chain=postrouting out-interface=ADSL1 passthrough=no
    new-packet-mark=QoS_7_Up
add action=mark-packet chain=postrouting out-interface=ADSL1 passthrough=no
    new-packet-mark=QoS_2_Up src-port=8291 comment=WinBox


/queue tree
add max-limit=666K name=QoS_ADSL1_Up parent=ADSL1
add name=QoS_1 packet-mark=QoS_1_Up parent=QoS_ADSL1_Up priority=1
add name=QoS_2 packet-mark=QoS_2_Up parent=QoS_ADSL1_Up priority=2
add name=QoS_3 packet-mark=QoS_3_Up parent=QoS_ADSL1_Up priority=3
add name=QoS_7 packet-mark=QoS_7_Up parent=QoS_ADSL1_Up priority=7
add name=QoS_8 packet-mark=QoS_8_Up parent=QoS_ADSL1_Up priority=8
add name=QoS_4 packet-mark=QoS_4_Up parent=QoS_ADSL1_Up priority=4


Icon-note.png

Note: This HTB Queue Tree should be upgraded to a structure like Example 3 in the HTB article.


Icon-note.png

Note: To fix traceroute - ICMP needs to have priority, for example 4.


Icon-note.png

Note: Recommended queue sizes are small.