Linux KVM - usage compendium

kvm
iptables
nat
linux
virtualization
Tags: #<Tag:0x00007f0ca73ec7a8> #<Tag:0x00007f0ca73ec668> #<Tag:0x00007f0ca73ec500> #<Tag:0x00007f0ca73ec370> #<Tag:0x00007f0ca73ec208>

#1

Documentation - the final frontier

… these are the adventures of me using KVM on Linux. These are not the most elegant solutions out there. But this works for me.


This is a growing compendium, not an article. Please research the validity of these information independently. If you spot an error, please let me know


##Summary:

List the IPs of the guests - dnsmasq

My setup is pretty simple. I use NATing with iptables and have dnsmasq on the machine. Now what are the IPs of the guest VMs? I want to SSH / RDP into my web server VM.

#!/bin/bash

LEASES=/var/lib/libvirt/dnsmasq/br1.leases

if [ -z $1 ]; then
        echo -e "\e[00;31mNo argument given, chose hostname from\e[00m:"
        more $LEASES | awk '{ print $3 " " $4 }'
        exit 2
fi

You can replace more with cat. On your system.

This script (more or less) will list you something like:

192.168.100.10 web
192.168.100.20 teamcity
192.168.100.30 gitlab
192.168.100.40 ansible-tower
192.168.100.50 docker-containers
...

This assumes:

  • br.1 is your interface
  • you use dnsmasq on Debian (paths can differ)

How to NAT out a KVM guest via iptables

By default my KVM guests get an internal IP (version 4). I use iptables NAT, and port forwarding to map the specific service to the external interface.

#!/bin/bash

# enable IP forwarding 
echo "1" > /proc/sys/net/ipv4/ip_forward
modprobe ip_conntrack

# we want to flush all rules
iptables -F 
# we want to flush prerouting as well
iptables -t nat -F 

# map web server port 80
iptables -t nat -I PREROUTING -p tcp -d 8.8.8.8 --dport 80 -j DNAT --to-destination  192.168.
100.10:80
# WAN for VMs behind NAT (no services routed)
iptables -I FORWARD -m state -d 192.168.100.0/24 --state NEW,RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

This assumes:

  • clean NAT tables
  • you use eth0 as the network interface alias on your KVM host
  • IP forwarding is not active by default on your distribution
  • you want to use stateful iptables rules
  • external IP is 8.8.8.8
  • internal web server IP is 192.168.100.10
  • the web server is on TCP port 80

Add what ever services you like.

A KVM guest needs direct network access - port mapping

If you host something like an OpenVPN Access Server as a KVM guest, you will realize that NATing can become complex. NATing isn’t needed in all scenarios.

iptables -A FORWARD -p tcp -i eth0 -d 192.168.100.123 --dport 1194 -j ACCEPT
iptables -A FORWARD -p udp -i eth0 -d 192.168.100.123 --dport 1194 -j ACCEPT

This assumes:

  • eth0 is the interface you need
  • you want to forward TCP and UDP port 1194 directly to the 192.168.100.123 IP

Enable Kernel Same-page Merging (KSM)

Kernel Same-page Merge (KSM) is a process that allows guests to share identical memory pages. By sharing pages, the combined memory usage of the guest is reduced. The savings are especially increased when multiple guests are running similar base operating system images.

(Source)

Many Linux distributions do not enable this by default, even if you compile the kernel feature. The ksmtuned daemon can dynamically adjust the frequency of the deduplication algorithms.

[email protected]:/home/marius/scripts# grep -H '' /sys/kernel/mm/ksm/pages_*
/sys/kernel/mm/ksm/pages_shared:0
/sys/kernel/mm/ksm/pages_sharing:0
/sys/kernel/mm/ksm/pages_to_scan:100
/sys/kernel/mm/ksm/pages_unshared:0
/sys/kernel/mm/ksm/pages_volatile:0
[email protected]:/home/marius/scripts# cat /sys/kernel/mm/ksm/run
0
[email protected]:/home/marius/scripts# echo 1 > /sys/kernel/mm/ksm/run
[email protected]:/home/marius/scripts# grep -H '' /sys/kernel/mm/ksm/pages_*
/sys/kernel/mm/ksm/pages_shared:0
/sys/kernel/mm/ksm/pages_sharing:0
/sys/kernel/mm/ksm/pages_to_scan:100
/sys/kernel/mm/ksm/pages_unshared:0
/sys/kernel/mm/ksm/pages_volatile:15900

What exactly the kernel will do with my 15900 volatile pages depends on the kernel version. If you set it this, the value will grow first, and then at some point residual pages can be shared. Here are more details about this.

pages_shared: The number of unswappable kernel pages that KSM is using
pages_sharing: An indication of memory savings
pages_unshared: The number of unique pages repeatedly checked for merging
pages_volatile: The number of pages that are changing too often

Bottom line is: KSM can share pages between guests, which do not change frequently. So if you check 15 minutes later it may look like this:

[email protected]:/home/marius/scripts# grep -H '' /sys/kernel/mm/ksm/pages_*
/sys/kernel/mm/ksm/pages_shared:145
/sys/kernel/mm/ksm/pages_sharing:402901
/sys/kernel/mm/ksm/pages_to_scan:100
/sys/kernel/mm/ksm/pages_unshared:5608
/sys/kernel/mm/ksm/pages_volatile:5709842

ESXi (6.5) - usage compendium for Road Warriors