Manual:Xen: Difference between revisions
New page: == Virtualization Overview == Virtualization techonogies enable single physical device to execute multiple different operating systems and applications. Virtualization support in RouterOS... |
networking |
||
Line 1: | Line 1: | ||
== Virtualization Overview == | == Virtualization Overview == | ||
Virtualization techonogies enable single physical device to execute multiple different operating systems | Virtualization techonogies enable single physical device to execute multiple different operating systems. Virtualization support in RouterOS allows to run multiple copies of RouterOS sofware and even other supported operating systems. Note that virtualization support depends on system architecture, not all architectures that RouterOS supports allow virtualization. | ||
Ability to run non-RouterOS sofware allows user to run applications that are not included in RouterOS. | |||
== x86 Virtualization Support == | == x86 Virtualization Support == | ||
Line 21: | Line 23: | ||
The latter approach is more flexible because allows user to specify disk image size. | The latter approach is more flexible because allows user to specify disk image size. | ||
To be able to run non-RouterOS operating system in VM, you need Linux kernel, disk image and initial ram disk (if necessary) files. | |||
Note that one disk image at the same time can only be used by one VM. | Note that one disk image at the same time can only be used by one VM. | ||
Line 146: | Line 150: | ||
There are multiple ways to stop running VM: | There are multiple ways to stop running VM: | ||
* preferred way is to shut down <b>from</b> guest VM (e.g. by connecting, logging and | * preferred way is to shut down <b>from</b> guest VM (e.g. by connecting to guest VM, logging in and issuing "/system shutdown" command). | ||
* force shutdown from host RouterOS by using "/xen | * force shutdown from host RouterOS by using "/xen shutdown <VM name>" command; | ||
* simply by disabling VM entry in "/xen" menu, note that this is the most dangerous way of stopping running VM, because guest VM can leave its filesystem in corrupt state (disabling VM entry for VM is the same as unplugging power for physical device). | * simply by disabling VM entry in "/xen" menu, note that this is the most dangerous way of stopping running VM, because guest VM can leave its filesystem in corrupt state (disabling VM entry for VM is the same as unplugging power for physical device). | ||
Line 158: | Line 162: | ||
# NAME MEMORY WEIGHT STATE | # NAME MEMORY WEIGHT STATE | ||
0 ros1 64 256 shutdown | 0 ros1 64 256 shutdown | ||
</pre> | |||
In order to boot VM that is shut down, you must either disable and enable VM entry in "/xen" menu or use "/xen start <VM name>" command. | |||
== Configuring VM Networking == | |||
In order for guest VM to participate in network, virtual interfaces that connect guest VM with host must be created. Virtual network connection with guest VM can be thought of as point-to-point ethernet network connection, which terminates in guest VM as "/interface ethernet" type interface and in host as "/interface virtual-ethernet" interface. By configuring appropriate data forwarding (either by bridging or routing) to/from virtual-ethernet interface in host system, guest VM can be allowed to participate in real network. | |||
=== Configuring Network Interfaces for Guest VM === | |||
Network interfaces that will appear in guest VM as ethernet interfaces are configured in "/xen interface" menu: | |||
<pre> | |||
[admin@MikroTik] /xen interface> add virtual-machine=ros1 type=dynamic | |||
[admin@MikroTik] /xen interface> print detail | |||
Flags: X - disabled | |||
0 virtual-machine=ros1 vm-mac-addr=02:1C:AE:C1:B4:B2 type=dynamic static-interface=none dynamic-mac-addr=02:38:19:0C:F3:98 dynamic-bridge=none | |||
</pre> | |||
Above command creates interface for guest VM "ros1" with type "dynamic". | |||
There are 2 types of interfaces: | |||
* dynamic - endpoint of virtual network connection in host ("/interface virtual-ethernet") will be created dynamically when guest VM will be booted. By using this type of interface user avoids manually creating endpoint of virtual connection in host, at the expense of limited flexibility how this connection can be used (e.g. there is no way how to reliably assign IP address to dynamically created interface). Currently, it can only be automatically added to bridge specified in "dynamic-bridge" parameter. This behaviour is similar to dynamic WDS interfaces for wireless WDS links. | |||
* static - endpoint of virtual network connection in host ("/interface virtual-ethernet") must be manually created. This type of interface allows maximum flexibility because interface that will connect with guest VM is previously known (therefore IP addresses can be added, interface can be used in filter rules, etc.), at the expense of having to create "/interface virtual-ethernet" manually. | |||
VM interfaces have the following parameters: | |||
* virtual-machine - to which VM this interface belongs; | |||
* vm-mac-addr - MAC address of ethernet interface in <b>guest</b> system; | |||
* type - interface type as described above | |||
* static-interface - when "type=static", this parameter specifies which "/interface virtual-ethernet" in host system will be connected with guest; | |||
* dynamic-mac-addr - when "type=dynamic", automatically created "/interface virtual-ethernet" in <b>host</b> system will have this MAC address; | |||
* dynamic-bridge - when "type=dynamic", dynamically created "/interface virtual-ethernet" will automatically get added as bridge port to this bridge. | |||
=== Configuring Dynamic Interfaces === | |||
To create virtual connection that will have its endpoint in host dynamically made, use the following command: | |||
<pre> | |||
[admin@MikroTik] /xen interface> add virtual-machine=ros1 type=dynamic | |||
[admin@MikroTik] /xen interface> print detail | |||
Flags: X - disabled | |||
0 virtual-machine=ros1 vm-mac-addr=02:1C:AE:C1:B4:B2 type=dynamic static-interface=none dynamic-mac-addr=02:38:19:0C:F3:98 dynamic-bridge=none | |||
</pre> | |||
After enabling "ros1" VM, you can confirm that new virtual-ethernet interface is made with given dynamic-mac-addr: | |||
<pre> | |||
[admin@MikroTik] /xen> /interface virtual-ethernet print | |||
Flags: X - disabled, R - running | |||
# NAME MTU ARP MAC-ADDRESS | |||
0 R vif1 1500 enabled 02:38:19:0C:F3:98 | |||
</pre> | |||
And in guest VM ethernet interface is available with given vm-mac-addr: | |||
<pre> | |||
[admin@Guest] > int ethernet print | |||
Flags: X - disabled, R - running, S - slave | |||
# NAME MTU MAC-ADDRESS ARP | |||
0 R ether1 1500 02:1C:AE:C1:B4:B2 enabled | |||
</pre> | |||
By configuring "dynamic-bridge" setting, virtual-ethernet interface can be automatically added as bridge port to some bridge in host system. For example, if it is necessary to forward traffic between "ether1" interface on host and VM "ros1" ethernet interface, the following steps must be taken: | |||
Create bridge on host system and add "ether1" as bridge port: | |||
<pre> | |||
[admin@MikroTik] > /interface bridge add name=to-ros1 | |||
[admin@MikroTik] > /interface bridge port add bridge=to-ros1 interface=ether1 | |||
</pre> | |||
Next, specify that virtual-ethernet should automatically get added as bridge port: | |||
<pre> | |||
[admin@MikroTik] /xen interface> print detail | |||
Flags: X - disabled | |||
0 virtual-machine=ros1 vm-mac-addr=02:1C:AE:C1:B4:B2 type=dynamic static-interface=none dynamic-mac-addr=02:38:19:0C:F3:98 dynamic-bridge=none | |||
[admin@MikroTik] /xen interface> set 0 dynamic-bridge=to-ros1 | |||
</pre> | |||
After this virtual-ethernet interface is added as bridge port on host: | |||
<pre> | |||
[admin@MikroTik] /xen interface> /interface bridge port print | |||
Flags: X - disabled, I - inactive, D - dynamic | |||
# INTERFACE BRIDGE PRIORITY PATH-COST HORIZON | |||
0 ether1 to-ros1 0x80 10 none | |||
1 D vif1 to-ros1 0x80 10 none | |||
</pre> | |||
By using similar configuration, user can, for example, "pipe" all traffic through guest VM - if there are 2 physical interfaces in host, user can create 2 bridges and bridge all traffic through guest VM (assuming that operating system in guest is configured in such a way that ensures data forwarding between its interfaces). | |||
=== Configuring Static Interfaces === | |||
To create virtual connection whose endpoint in host system will be static interface, at first create static virtual-ethernet interface: | |||
<pre> | |||
[admin@MikroTik] /interface virtual-ethernet> add name=static-to-ros1 disabled=no | |||
[admin@MikroTik] /interface virtual-ethernet> print | |||
Flags: X - disabled, R - running | |||
# NAME MTU ARP MAC-ADDRESS | |||
0 R vif1 1500 enabled 02:38:19:0C:F3:98 | |||
1 static-to-ros1 1500 enabled 02:3A:1B:DB:FC:CF | |||
</pre> | |||
Next, create interface for guest VM: | |||
<pre> | |||
[admin@MikroTik] /xen interface> add virtual-machine=ros1 type=static static-interface=static-to-ros1 | |||
[admin@MikroTik] /xen interface> print | |||
Flags: X - disabled | |||
# VIRTUAL-MACHINE TYPE VM-MAC-ADDR | |||
0 ros1 dynamic 02:1C:AE:C1:B4:B2 | |||
1 ros1 static 02:DF:66:CD:E9:74 | |||
</pre> | |||
Now we can confirm that virtual-ethernet interface is active: | |||
<pre> | |||
[admin@MikroTik] /xen interface> /interface virtual-ethernet print | |||
Flags: X - disabled, R - running | |||
# NAME MTU ARP MAC-ADDRESS | |||
0 R static-to-ros1 1500 enabled 02:3A:1B:DB:FC:CF | |||
1 R vif1 1500 enabled 02:38:19:0C:F3:98 | |||
</pre> | |||
And in guest system: | |||
<pre> | |||
[admin@Guest] > /interface ethernet print | |||
Flags: X - disabled, R - running, S - slave | |||
# NAME MTU MAC-ADDRESS ARP | |||
0 R ether1 1500 02:1C:AE:C1:B4:B2 enabled | |||
1 R ether2 1500 02:DF:66:CD:E9:74 enabled | |||
</pre> | |||
Having static interface in host system allows to use interface in configuration wherever specifying interface is necessary, e.g. adding ip address: | |||
<pre> | |||
[admin@MikroTik] > ip address add interface=static-to-ros1 address=1.1.1.1/24 | |||
</pre> | |||
In similar way we add IP address to appropriate interface in guest system and confirm that routing is working: | |||
<pre> | |||
[admin@Guest] > /ip address add interface=ether2 address=1.1.1.2/24 | |||
[admin@Guest] > /ping 1.1.1.1 | |||
1.1.1.1 64 byte ping: ttl=64 time=5 ms | |||
1.1.1.1 64 byte ping: ttl=64 time<1 ms | |||
1.1.1.1 64 byte ping: ttl=64 time<1 ms | |||
3 packets transmitted, 3 packets received, 0% packet loss | |||
round-trip min/avg/max = 0/1.6/5 ms | |||
</pre> | </pre> |
Revision as of 12:31, 9 June 2008
Virtualization Overview
Virtualization techonogies enable single physical device to execute multiple different operating systems. Virtualization support in RouterOS allows to run multiple copies of RouterOS sofware and even other supported operating systems. Note that virtualization support depends on system architecture, not all architectures that RouterOS supports allow virtualization.
Ability to run non-RouterOS sofware allows user to run applications that are not included in RouterOS.
x86 Virtualization Support
Virtualization support on x86 architecture systems is implemented using Xen hypervisor (http://www.xen.org). This enables RouterOS to run other operating systems that support Xen paravirtualization in "virtual machines" (guests), controlled by RouterOS software (host).
Support for virtualization for x86 architecture systems is included in RouterOS software versions starting with 3.11. To enable virtualization support "xen" package must be installed.
Host RouterOS software sets up virtual machines such that they use file in RouterOS host file system as disk image(s). Additionally host RouterOS can set up virtual ethernet network interfaces between itself and virtual machine. This enables virtual machines to participate in network under control of host RouterOS software.
In order to execute operating system in virtual machine, you need:
- OS kernel that supports Xen paravirtualization
- OS disk image
- (optionally) initial ram disk to use while booting OS in VM
If RouterOS image is used for booting in VM, OS kernel and initial ram disk are not necessary - specifying RouterOS disk image is sufficient. RouterOS images for use by VMs can be created in 2 ways:
- either by taking image from existing RouterOS x86 installation that supports virtualization (version >= 3.11)
- or by using special RouterOS functions to create RouterOS image to use in VM (note that these functions do not produce RouterOS image that can be copied and successfully run from physical media!).
The latter approach is more flexible because allows user to specify disk image size.
To be able to run non-RouterOS operating system in VM, you need Linux kernel, disk image and initial ram disk (if necessary) files.
Note that one disk image at the same time can only be used by one VM.
Creating RouterOS image to use in VM
To create RouterOS image to use in VM use "/xen make-routeros-image" command:
[admin@MikroTik] /xen> make-routeros-image file-name=ros1.img file-size=32 [admin@MikroTik] /xen> /file print # NAME TYPE SIZE CREATION-TIME 0 ros1.img .img file 33554432 jun/06/2008 14:47:23
This produces 32MB RouterOS image that is ready to use in VM. New RouterOS image is based on host system sofware and therefore contains all sofware packages that are installed on host system, but does not contain host configuration.
VM Configuration
All virtualization for x86 architecture related functions are configured under "/xen" menu.
Memory Available to Host RouterOS
By default all the memory is available to host system, for example for system with 1GB of memory:
[admin@MikroTik] > /system resource print uptime: 2m4s version: "3.9" free-memory: 934116kB total-memory: 963780kB cpu: "Intel(R)" cpu-count: 2 cpu-frequency: 2813MHz cpu-load: 0 free-hdd-space: 77728884kB total-hdd-space: 79134596kB write-sect-since-reboot: 989 write-sect-total: 989 architecture-name: "x86" board-name: "x86" [admin@MikroTik] > /xen global-settings print memory-for-main: unlimited
In some cases this may limit ability to allocate necessary memory for running guest VMs, because host system may have used memory for e.g. filesystem caching purposes. Therefore it is advised to configure limit of memory available to host system (exact value for limit depends on what sofware features are used on host system - in general, the same rules as for choosing amount of physical memory for regular RouterOS installation apply):
[admin@MikroTik] > /system resource print uptime: 2m4s version: "3.9" free-memory: 934116kB total-memory: 963780kB cpu: "Intel(R)" cpu-count: 2 cpu-frequency: 2813MHz cpu-load: 0 free-hdd-space: 77728884kB total-hdd-space: 79134596kB write-sect-since-reboot: 989 write-sect-total: 989 architecture-name: "x86" board-name: "x86" [admin@MikroTik] > /xen global-settings print memory-for-main: unlimited [admin@MikroTik] > /xen global-settings set memory-for-main=128 [admin@MikroTik] > /system reboot Reboot, yes? [y/N]: y system will reboot shortly .... [admin@MikroTik] > /system resource print uptime: 1m5s version: "3.11" free-memory: 114440kB total-memory: 131272kB cpu: "Intel(R)" cpu-count: 2 cpu-frequency: 2813MHz cpu-load: 0 free-hdd-space: 77728884kB total-hdd-space: 79134596kB write-sect-since-reboot: 794 write-sect-total: 794 architecture-name: "x86" board-name: "x86"
Creating RouterOS VM
Assuming that RouterOS image "ros1.img" is previously made, new VM to run RouterOS can be created:
[admin@MikroTik] /xen> add name=ros1 disk=hda disk-image=ros1.img memory=64 console-telnet-port=64000 [admin@MikroTik] /xen> print detail Flags: X - disabled 0 X name="ros1" disk=hda disk-image="ros1.img" initrd="" kernel="" kernel-cmdline="" cpu-count=1 memory=64 weight=256 console-telnet-port=64000 state=disabled
The following parameters were passed to "add" command:
- disk=hda & disk-image=ros1.img - these parameters specify that file "ros1.img" in host filesystem will be set up as disk "hda" (IDE Primary Master) in guest system;
- memory=64 - this specifies amount of memory for guest VM;
- console-telnet-port=64000 - specifies that host system will listen on port 64000 and once telnetted to, will forward guests console output to telnet client and accept console input from telnet client.
There are few other settings:
- kernel & initrd - VM kernel file to boot and initial ram disk file to use (if specified), as noted before, specifying these is not necessary when booting RouterOS image;
- kernel-cmdline - command line to pass to Linux kernel
- cpu-count - how many CPUs should be made available to VM;
- weight - proportional "importance" of this VM when scheduling multiple VMs for execution. Taking into account that host operating system shares CPUs with all running guest VMs, weight parameter specifies proportional share of CPU(s) that guest operating system will get when multiple operating systems start competing for CPU resource. "Weight" of host operating system is 256. So, for example, if guest VM is also configured with weight 256, if both OSes will be running at 100% CPU usage, both will get equal share of CPU. If guest VM will be configured with weight 128, it will get only 1/3 of CPU.
Starting, Stopping and Connecting to RouterOS VM
To start booting guest VM, enable it:
[admin@MikroTik] /xen> enable ros1 [admin@MikroTik] /xen> print Flags: X - disabled # NAME MEMORY WEIGHT STATE 0 ros1 64 256 running
There are 2 (mutually exclusive, because there is just one virtual console provided for guest VM) ways to connect to console of running VM:
- by using "/xen console <VM name>" command, or
- by using telnet program and connecting to port specified in "console-telnet-port" parameter.
There are multiple ways to stop running VM:
- preferred way is to shut down from guest VM (e.g. by connecting to guest VM, logging in and issuing "/system shutdown" command).
- force shutdown from host RouterOS by using "/xen shutdown <VM name>" command;
- simply by disabling VM entry in "/xen" menu, note that this is the most dangerous way of stopping running VM, because guest VM can leave its filesystem in corrupt state (disabling VM entry for VM is the same as unplugging power for physical device).
VM shutdown state can be confirmed in "/xen" menu:
[admin@MikroTik] /xen> shutdown ros1 [admin@MikroTik] /xen> print Flags: X - disabled # NAME MEMORY WEIGHT STATE 0 ros1 64 256 shutdown
In order to boot VM that is shut down, you must either disable and enable VM entry in "/xen" menu or use "/xen start <VM name>" command.
Configuring VM Networking
In order for guest VM to participate in network, virtual interfaces that connect guest VM with host must be created. Virtual network connection with guest VM can be thought of as point-to-point ethernet network connection, which terminates in guest VM as "/interface ethernet" type interface and in host as "/interface virtual-ethernet" interface. By configuring appropriate data forwarding (either by bridging or routing) to/from virtual-ethernet interface in host system, guest VM can be allowed to participate in real network.
Configuring Network Interfaces for Guest VM
Network interfaces that will appear in guest VM as ethernet interfaces are configured in "/xen interface" menu:
[admin@MikroTik] /xen interface> add virtual-machine=ros1 type=dynamic [admin@MikroTik] /xen interface> print detail Flags: X - disabled 0 virtual-machine=ros1 vm-mac-addr=02:1C:AE:C1:B4:B2 type=dynamic static-interface=none dynamic-mac-addr=02:38:19:0C:F3:98 dynamic-bridge=none
Above command creates interface for guest VM "ros1" with type "dynamic".
There are 2 types of interfaces:
- dynamic - endpoint of virtual network connection in host ("/interface virtual-ethernet") will be created dynamically when guest VM will be booted. By using this type of interface user avoids manually creating endpoint of virtual connection in host, at the expense of limited flexibility how this connection can be used (e.g. there is no way how to reliably assign IP address to dynamically created interface). Currently, it can only be automatically added to bridge specified in "dynamic-bridge" parameter. This behaviour is similar to dynamic WDS interfaces for wireless WDS links.
- static - endpoint of virtual network connection in host ("/interface virtual-ethernet") must be manually created. This type of interface allows maximum flexibility because interface that will connect with guest VM is previously known (therefore IP addresses can be added, interface can be used in filter rules, etc.), at the expense of having to create "/interface virtual-ethernet" manually.
VM interfaces have the following parameters:
- virtual-machine - to which VM this interface belongs;
- vm-mac-addr - MAC address of ethernet interface in guest system;
- type - interface type as described above
- static-interface - when "type=static", this parameter specifies which "/interface virtual-ethernet" in host system will be connected with guest;
- dynamic-mac-addr - when "type=dynamic", automatically created "/interface virtual-ethernet" in host system will have this MAC address;
- dynamic-bridge - when "type=dynamic", dynamically created "/interface virtual-ethernet" will automatically get added as bridge port to this bridge.
Configuring Dynamic Interfaces
To create virtual connection that will have its endpoint in host dynamically made, use the following command:
[admin@MikroTik] /xen interface> add virtual-machine=ros1 type=dynamic [admin@MikroTik] /xen interface> print detail Flags: X - disabled 0 virtual-machine=ros1 vm-mac-addr=02:1C:AE:C1:B4:B2 type=dynamic static-interface=none dynamic-mac-addr=02:38:19:0C:F3:98 dynamic-bridge=none
After enabling "ros1" VM, you can confirm that new virtual-ethernet interface is made with given dynamic-mac-addr:
[admin@MikroTik] /xen> /interface virtual-ethernet print Flags: X - disabled, R - running # NAME MTU ARP MAC-ADDRESS 0 R vif1 1500 enabled 02:38:19:0C:F3:98
And in guest VM ethernet interface is available with given vm-mac-addr:
[admin@Guest] > int ethernet print Flags: X - disabled, R - running, S - slave # NAME MTU MAC-ADDRESS ARP 0 R ether1 1500 02:1C:AE:C1:B4:B2 enabled
By configuring "dynamic-bridge" setting, virtual-ethernet interface can be automatically added as bridge port to some bridge in host system. For example, if it is necessary to forward traffic between "ether1" interface on host and VM "ros1" ethernet interface, the following steps must be taken:
Create bridge on host system and add "ether1" as bridge port:
[admin@MikroTik] > /interface bridge add name=to-ros1 [admin@MikroTik] > /interface bridge port add bridge=to-ros1 interface=ether1
Next, specify that virtual-ethernet should automatically get added as bridge port:
[admin@MikroTik] /xen interface> print detail Flags: X - disabled 0 virtual-machine=ros1 vm-mac-addr=02:1C:AE:C1:B4:B2 type=dynamic static-interface=none dynamic-mac-addr=02:38:19:0C:F3:98 dynamic-bridge=none [admin@MikroTik] /xen interface> set 0 dynamic-bridge=to-ros1
After this virtual-ethernet interface is added as bridge port on host:
[admin@MikroTik] /xen interface> /interface bridge port print Flags: X - disabled, I - inactive, D - dynamic # INTERFACE BRIDGE PRIORITY PATH-COST HORIZON 0 ether1 to-ros1 0x80 10 none 1 D vif1 to-ros1 0x80 10 none
By using similar configuration, user can, for example, "pipe" all traffic through guest VM - if there are 2 physical interfaces in host, user can create 2 bridges and bridge all traffic through guest VM (assuming that operating system in guest is configured in such a way that ensures data forwarding between its interfaces).
Configuring Static Interfaces
To create virtual connection whose endpoint in host system will be static interface, at first create static virtual-ethernet interface:
[admin@MikroTik] /interface virtual-ethernet> add name=static-to-ros1 disabled=no [admin@MikroTik] /interface virtual-ethernet> print Flags: X - disabled, R - running # NAME MTU ARP MAC-ADDRESS 0 R vif1 1500 enabled 02:38:19:0C:F3:98 1 static-to-ros1 1500 enabled 02:3A:1B:DB:FC:CF
Next, create interface for guest VM:
[admin@MikroTik] /xen interface> add virtual-machine=ros1 type=static static-interface=static-to-ros1 [admin@MikroTik] /xen interface> print Flags: X - disabled # VIRTUAL-MACHINE TYPE VM-MAC-ADDR 0 ros1 dynamic 02:1C:AE:C1:B4:B2 1 ros1 static 02:DF:66:CD:E9:74
Now we can confirm that virtual-ethernet interface is active:
[admin@MikroTik] /xen interface> /interface virtual-ethernet print Flags: X - disabled, R - running # NAME MTU ARP MAC-ADDRESS 0 R static-to-ros1 1500 enabled 02:3A:1B:DB:FC:CF 1 R vif1 1500 enabled 02:38:19:0C:F3:98
And in guest system:
[admin@Guest] > /interface ethernet print Flags: X - disabled, R - running, S - slave # NAME MTU MAC-ADDRESS ARP 0 R ether1 1500 02:1C:AE:C1:B4:B2 enabled 1 R ether2 1500 02:DF:66:CD:E9:74 enabled
Having static interface in host system allows to use interface in configuration wherever specifying interface is necessary, e.g. adding ip address:
[admin@MikroTik] > ip address add interface=static-to-ros1 address=1.1.1.1/24
In similar way we add IP address to appropriate interface in guest system and confirm that routing is working:
[admin@Guest] > /ip address add interface=ether2 address=1.1.1.2/24 [admin@Guest] > /ping 1.1.1.1 1.1.1.1 64 byte ping: ttl=64 time=5 ms 1.1.1.1 64 byte ping: ttl=64 time<1 ms 1.1.1.1 64 byte ping: ttl=64 time<1 ms 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0/1.6/5 ms