Sunday, January 17, 2021

Network Bandwidth Monitoring/Analysis Using RaspberryPi II

Pi could be used to Monitor the Bandwidth Usage of entire Home Network. Time series data could be visualized in different types of Charts. Historic data visualization supported (e.g Monthly Bandwidth Usage). Each Network/IP/Machine level statistics could be analyzed and viewed.

The below configuration explain about setting up such an environment. The only prerequisite would be, Make all your traffic go through Pi. i.e Change your Default Gateway from Router to Pi. This could be easily done through, updating the Router LAN settings, and putting Raspberry Pi IP-Address, in the Default Gateway box.

Now update Raspberry Pi’s Default Gateway as Routers IP, and enable IP forwarding. This will make all traffic destined for Internet will be automatically routed to the Router from Pi. Finally setup ‘ntopng’, to churn the network packets, for analysis. 

Detailed instructions are given here.

image

image

Monday, January 11, 2021

Raspberry Pi, as Network Security Appliance/AD-Blocker

In this article, we will explain how to use RaspberryPi, as a Network Security Appliance for private network, which could be used to block all ADS, and secure the network by whitelisting/blacklisting DNS names.

It could effectively improve the overall network performance as all the AD requests going out of your private network will get blocked, and never goes out of your private network. It extends the protection to all computers in your network including mobile/other devices connects to your home network.


The below configuration shows, Pi-Hole deployed in Docker containers hosted on a RaspberryPi, and managed through Portainer (A Container orchestration platform, again running inside a Docker Container)

image

 

You could find the instructions on setting up the same here.

The only change we’d, is using a new version of Pi-Hole image (pihole/pihole:v5.0-armhf)

A single day operation statistics is given below: In a single day operation, Pi-Hole has blocked 31.7% of total requests, which are of type Ads.

Since the CPU/Memory (~4.5MB) footprints are pretty low, Pi runs smooth 24x7 and meets the network performance.

image

Bypass Qemu Network Stack with Virtio-Vhost, for Near Native Performance

For virtualized environments, primary performance bottleneck could happen with below resources:

a. CPU

b. Storage

c. Network

For CPU, direct passthrough option would give a performance near to bare metal. For Storage and Network, deploying VIRTIO drivers (paravirtualization) could provide a descent performance as well. These concepts have been explained here.

With network, one more performance tuning is possible, to completely bypass the Hypervisor (Qemu) network stack, and let the GUEST directly interact with HOST’s network stack. This may give almost similar network performance for guest applications as with application running on the host.

Implementing this requires, enabling ‘vhost_net’ driver on the host, and update the VM Settings to use this driver, which is detailed here.

In an optimal setting, you will also enable the same number of IO-Queues, as with the total number of CPU Cores allotted to the Virtual Machine.

Reference

image

 

 

Wednesday, January 15, 2020

Automate IE Proxy Setting Changes on Network Adapter Changes–Windows Powershell

If you need to turn on IE Proxy settings, say if the LAN cable is plugged in and reverse the setting, once Wireless got connected, you could rely on Powershell script to automate the same in Windows environment.

This is a two step process:

1. Create the powershell script, which detects the Network Adapter Change, and update IE Settings accordingly

Sample Script here:

2. Create a new Task in “Task Scheduler”, which will be triggered on Network Changes (NetworkProfile->Event ID:10000) and System Startup

Set the above script as the Action, for the task

Eg

Program: powershell.exe

Arguments: -windowstyle hidden "<PathTo>\SwitchProxyOnNetworkChange.ps1":

Note: This will run the script in hidden mode, so that user won’t notice the powershell window – for the best experience


From now on whenever your Network get changed, the script will be triggered automatically to set IE Proxy settings accordingly.

Wednesday, November 13, 2019

Configure Console Only Login On Ubuntu 16.04+ Desktop

Occasionally we may want to login to our Ubuntu Desktop as console only mode (for eg: running our Ubuntu desktop as a bare metal hypervisor, only to launch Virtual Machines which will be connected remotely).

This will help to run the desktop with bare minimum memory (120MB RAM on Ubuntu 16.04) and CPU usage.

By default Ubuntu Desktop 16.04+ does not provide (which has Systemd than the legacy init system) this option, and you’ve to configure the same. Once configured you will get a new Grub menu entry on startup, to login as Console Mode.

  1. Copy a sample menu entry from (/boot/grub/grub.cfg) to (/etc/grub.d/40_custom)

    Sample:

    menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-0e341ce4-5a12-4a6d-a997-5336d1670353' {

           recordfail

           load_video

           gfxmode $linux_gfx_mode

           insmod gzio

           if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi

           insmod part_msdos

           insmod ext2

           set root='hd2,msdos5'

           if [ x$feature_platform_search_hint = xy ]; then

             search --no-floppy --fs-uuid --set=root --hint-bios=hd2,msdos5 --hint-efi=hd2,msdos5 --hint-baremetal=ahci2,msdos5  0e341ce4-5a12-4a6d-a997-5336d1670353

           else

             search --no-floppy --fs-uuid --set=root 0e341ce4-5a12-4a6d-a997-5336d1670353

           fi

            linux /boot/vmlinuz-4.15.0-54-generic root=UUID=0e341ce4-5a12-4a6d-a997-5336d1670353 ro  ipv6.disable=1  scsi_mod.use_blk_mq=1

           initrd /boot/initrd.img-4.15.0-54-generic

    }

     

  2. Update the copied entry as per below (highlighted text). The crux is ‘systemd.unit’ property.

    Sample:

    menuentry 'Ubuntu-TextMode' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-0e341ce4-5a12-4a6d-a997-5336d1670353' {

           recordfail

           #load_video

           #gfxmode $linux_gfx_mode

           insmod gzio

           if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi

           insmod part_msdos

           insmod ext2

           set root='hd2,msdos5'

           if [ x$feature_platform_search_hint = xy ]; then

             search --no-floppy --fs-uuid --set=root --hint-bios=hd2,msdos5 --hint-efi=hd2,msdos5 --hint-baremetal=ahci2,msdos5  0e341ce4-5a12-4a6d-a997-5336d1670353

           else

             search --no-floppy --fs-uuid --set=root 0e341ce4-5a12-4a6d-a997-5336d1670353

           fi

            linux /vmlinuz root=UUID=0e341ce4-5a12-4a6d-a997-5336d1670353 systemd.unit=multi-user.target ro  ipv6.disable=1  scsi_mod.use_blk_mq=1

           initrd /initrd.img

    }


  3. Update grub using (Sudo update-grub2)

  4. Reboot

  5. Select the menu entry “Ubuntu-TextMode’

Thursday, October 31, 2019

Enable High Performance, Block Multi Queue-IO Scheduler under Ubuntu 16.04 Desktop

Compared to Single Queue-IO Schedulers (eg. cfq, deadline etc), we can gain a high IO performance improvment through MultiQueue IO Schedulers (eg bfq, kyber). These are the new IO Schedulers built in to Linux Kernal 4.12+, but disabled until 5.0. It uses multiple IO Queue (leveraging CPU cores) to provide a high responsive system. These are specificaly designed for Desktop OS. Its architecture has been detailed here and here.

Enabling the same On Ubuntu 16.04 has been mentiond below:

1. Add both mq-deadline, bfq and kyber-iosched, to /etc/modules, to load the modules on startup

2. Append, scsi_mod.use_blk_mq=1 to the GRUB_CMDLINE_LINUX parameter in /etc/default/grub file

3. Create /etc/udev/rules.d/60-scheduler.rules, to assign bfq and kyber to block devices

eg:

# set bfq scheduler for non-rotating disks | SDD
ACTION=="add|change", KERNEL=="sd[a-z]", TEST!="queue/rotational", ATTR{queue/scheduler}="bfq"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="bfq"

# set kyber scheduler for rotating disks | HDD
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="kyber"

4. Update grub using “sudo update-grub2” and reboot

5. Now your SSD would be using bfq scheduler, and HDD using kyber scheduler. Enjoy your high responsive system !


References:

https://www.omgubuntu.co.uk/2017/07/linux-kernel-4-12-released-bfq

https://www.thomas-krenn.com/en/wiki/Linux_Multi-Queue_Block_IO_Queueing_Mechanism_(blk-mq)

https://lwn.net/Articles/767987/

https://lwn.net/Articles/784267/

https://wiki.ubuntu.com/Kernel/Reference/IOSchedulers

https://www.stephenrlang.com/2018/01/io-scheduler-tuning/

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.2_release_notes/storage

https://www.hecticgeek.com/2016/09/supercharge-ubuntu-16-04-lts-xanmod-kernel/

https://www.cnx-software.com/2019/08/14/bfq-budget-fair-queuing-i-o-scheduler-improves-linux-systems-responsiveness-video/

https://unix.stackexchange.com/questions/375600/how-to-enable-and-use-the-bfq-scheduler

https://wiki.debian.org/SSDOptimization#Low-Latency_IO-Scheduler

Virtualization to outperform Bare Metal performance? Simulating a RAM Cache, RAID Controller in a Desktop System

In general you could get a near native performance on Qemu-KVM following the instructions, specified in this article. 

Now what if you could get performance levels that can outperform the host system?

In Qemu-KVM, there are options to enable the same, which will provide you performance, more than the Bare Metal system, by the use of aggressive caching.

Note: This inherently has the risk of data loss, in case of a host failure or crash, hence not recommended for production workloads.

This tweaks can be applied to a POC/Development VM, by which you can complete your tasks much faster than before. To overcome data loss/OS corruption, its always recommended that you snapshot your Virtual disks before enabling this mode. So in case of any issues, you could revert back to your previous working image.

Below are the performance level, we’ve achieved with/without the Virtual Disk performance options. We’ve seen 60-70% performance gain on an average, once the system is up and running.

Windows7 VM

Normal (Seconds)

With IO Tweaks (Seconds)

System Reboots

57

13

Opening Visual Studio 2013

8

3

System Shutdown

21

9

 

The tests are comleted on a Windows7 VM with the below configuration:

This would be a great option for development and Virtual machines for POCs, typical usecase under Desktops/Laptops. The settings are related to IO mode on the virtual disks, and which extensively use caching, and most of the writes to disk, will be kept in cache, until there is sufficient data. Once it has, it will write them to disk in one go. This is “Analogous to RAID controller with RAM cache”, and KVM brings this feature to a normal desktop/laptop without any major efforts

 


During the initial startup, performance levels remain at par, however the performance will steadily move up, once you’ve the system up and running and more data has been cached in host overtime.

This level of performance may not be achievable from a Windows Desktop (Running on Bare Metal), as it would periodically synchronize IO with the underlying disks, which will hinder the performance for high data reliability.

You could get additional preformance boost, if you use the “raw” storage format, instead of “qcow2”

To know more, read through the below posts:

QEMU Disk IO performance comparison: Native or threads?

VIRTUALIZATION TUNING AND OPTIMIZATION GUIDE

Virtual IO CACHING

How to improve Windows perfomance when running inside KVM

What is the memory module on a RAID card needed for?