Tuesday, November 18, 2014

Disabling Cell Broadcast/Service Messages in Latest Android Phones

I love my newly bought Moto G (2nd Gen), Android Phone. But every now and then, an annoying pop up floats on the screen, requesting to subscribe promotional offers. Pressing the 'OK' button will automatically subscribe to the offer, and we don't want that to happen, as it will cost money.

Googling, revealed that, these are 'Cell Broadcast' messages, broadcasted by your network operator. They also known by some other terms like 'Service messages'.

In earlier Android versions, you can disable it directly from "Message Options". These options are called by different names like, 'Cell Broadcast', 'Push Messages' or 'Service Messages'.

But with Moto G (And may be in Android 4.4 and above), we've not seen this option in the message settings and nothing similar in call/wireless and network menus.

Calling network operator gives the same answer like, "Please do it yourself, The option is in your phone. Please disable 'Service Messages' in your phone'. We've explored almost all menu options, but nothing sounds similar.

Finally after hours of struggling, we've found out the option.


"The menu option is there, but called by a different name! It is "Emergency Alerts"


It names differently (like threats to life, display child abduction emergency etc), but disabling these options, did the trick.

There are also other ways to disable 'Cell broadcast messages' and are given below.

Option-B [Use another device, temporarily, to deactivate 'Cell Broadcast']

As 'Cell Broadcast' is not related to an individual device, but tagged to your SIM in general, we can opt the below workaround.

a) Find an old phone which has an option to enable/disable Broadcast Channels and Messages (in my case its a Samsung D500).

b) Insert the first SIM in that old phone and go to the settings menu for Broadcast messages.

c) At first the Channels and Messages usually appear as disabled. I changed the setting for both Channels and Messages to Enabled in the old phone.

d) Switch-off the old phone and Switch-on

e) Go to the settings menu for Broadcast messages. Change the settings for both Channels and Messages to Disabled.

f) Switch-off the old phone and Switch-on

g) Switch-off the old phone and remove SIM card

h) Do the same steps (a-g) for the second SIM

i) Insert the SIM cards back in the new phone and switch on

Original source

Option-C [Subscribe to Do Not Disturb (DND) option of your network operator]

Subscribe or request Do Not Distrubt option from your network provider. Most of them does, buy simply sending an SMS to a toll free number, Or you can register for it in their website.

Do Not Disturb or (DND) function on most PBX or PABX systems prevents calls from ringing on an extension for which DND is activated. Wiki link here.

Note: For Tata Docomo, "simply call or SMS at 1909 (tollfree)" - SMS STOP to 1909.". Original source


Hope this might be helpful to someone facing a similar issue.

Friday, November 7, 2014

'Dynamic', to ease 'Reflection' - Microsoft.NET

One of the nice feature that, we're quite obsessive with Microsoft.NET languages (C#, VB.net) is


You might be already knowing, the capabilities of 'Reflection' in .NET. Most often, it is heavily used along with 'Custom Attributes'.

But we've found a very good use of it.

Reflection to achieve 'Late-Bindings' with our code

(Especially in scenarios, where you are integrating your application with external COM Servers, like Out Of Process Servers [EXE] or In Process Servers [DLL], which are written in a other native languages like Visual C++ [ATL or MFC])

This is not only limited to integration with 'COM Servers', but we can also leverage, 'Late-Bindings' with .NET types as well (which resides especially in other assemblies).

In other terms,

through 'Reflection' you can remove version dependency

to a referenced assembly or COM Object, in your project.

But implementing 'Version Independence' through Reflection API's is quite complex and less tidy. New 'dynamic' keyword in C# will specifically help you in that front, where you don't have to dig available methods/properties, through 'Reflection API's. It will be done under the hood.

Before coming into details, we should be aware of the below well known scenarios.;

Early Binding
Late Binding

Early Binding

To understand the scenario, lets take a case study, that was a real exercise for our team. We'd to create an application (.NET Windows), that should extend 'Office Outlook' notifications. i.e By default, Outlook does not support 'Desktop Notifications' for new mails, arriving at 'Secondary Mailbox's. So we're trying to add that functionality, which was long awaited by our support team, as they are having dedicated 'Secondary Mailboxes' for their support activity related mails. As of now, they need to periodically check their mailboxes for new mails. Quite tedious right? So if we can show the typical 'Outlook Mail Notifications' , for new mails arriving at 'Secondary Mailboxes', that will be a huge time saver for our support team.

So obviously, we need to create an application (.NET Windows), integrated with 'Office Outlook'. As you would know, Office Outlook is a 'COM Server' (Out Of Proc -EXE). At the time of development, we all had Office Outlook 2007 installation in our machines. So we just took the easy way out. We created the .NET project, and added the 'Office Outlook 2007' COM Reference from the COM tab. The implementation was quite easy and we completed the project in weeks.

Deployment commenced, and the support team was quite happy with the features. They don't have to periodically scan the mailbox anymore! A few more months passed, until the most inevitable happened!

IT Infrastructure Team, had rolled out a new version of Office (Office 2012). So Office Outlook 2007, has been upgraded to Office Outlook 2012. For our surprise, our application crashed.

Why? This is a

'Early Binding' scenario, where our application requires the dependencies (in this case Outlook) to be available with exact version, that had referenced during the development of the application.

So what could be the solution? Simple, Just recompile the project, by changing reference to Office Outlook 2012, instead of Office Outlook 2007. But what will happen, if a new version of Office (Office365) will be deployed again. Every time, we've to recompile and deploy our application.

Obviously this is not a real solution. The actual solution is 'Late Binding'

Late Binding

If we can access 'Office Outlook' at run time, without caring the version, that could be the most viable solution. We don't have to recompile the application every time, when a new upgrade of Office happens.

This 'Late Binding' is achieved through a technique called 'Reflection' in .NET.

The application we've just talked about is detailed in
this blog. It has been completely implemented through '.NET Reflection' and it will work with any version of Office Outlook (2007, 2012, 365 etc). If you're curious on the implementation, you can also download the source from the above link.

OK that's about the whole theory. Now if you explore '.NET Reflection' API's, it is quite difficult to implement (We will soon get into that with a real example). That means we are compromising 'Ease Of Implementation' for achieving 'Version Independence'. Can we've best of both the worlds?

i.e 'Ease of Implementation' and 'Version Independence' . Yes you can, with 'dynamic' feature of C#

We can understand the concepts with a simple example and code walk through.

Code Sample (Early Binding [Assembly Reference] ---> Late Binding [Reflection] ---> Late Binding [Dynamic] )

For this exercise, we will try to do the below with 'Office Word' (A COM Server written in Visual C++ !). A simple Office Word Automation from a .NET Program!

From our .NET application, we will perform the below,

a. Create the Word Application Object
b. Create a Blank Word Document Object
c. Make the 'Word' application visible
d. Close the Word application

Early Binding Implementation:


As we've already discussed, we've a static dependency to 'Interop.Word' (in our case it is Office Word 2007). But the implementation is quite straightforward and easy.

Late Binding Implementation (Reflection):


As you can see, the implementation is quite lengthy and tedious with 'Reflection'. But see the best part of it, You've not even referenced, 'Office Word Interop' in the 'References' section (See right hand side of the figure) and that is why it is working with every versions of 'Office Word'.

Late Binding Implementation (Dynamic):


As you can see with 'dynamic' keyword, you have best of both the worlds. You don't have the dependency to the 'Office Word Interop' (See right hand side of the fig) and at the same time, You've a easy implementation like 'Early Binding' scenario.


'Early Binding' (Static Dependencies) may be a problem while referring COM/External assemblies in your .NET program.

'Late Binding' can resolve this dependency issue, with 'Reflection'.
But 'Reflection' makes your code lengthier and complex.

The 'dynamic' keyword brings you best of both the world of 'Early Binding' and 'Version Independence'

Comments, Suggestions! Most Welcome!

Dynamic JSON parsers in Microsoft.NET

We are aware of typical, static Json parsers, those can parse a json string to a predefined POCO C# object. This is practical for scenarios, where you have a pre-defined json schema, before you design your C# classes. Also your json object structure (json schema) should not change, if it does your code breaks, while parsing dynamic json string to your static C# object.

But mostly,

JSON is dynamic in nature

That's why most, No-SQL databases uses it or at least it's derivatives (like BSON by MongoDB). We've a situation here, where we've to parse such dynamic Json objects, that can change over time or changes based on other elements in the json.

For eg:

If you represent a 'system event' as Json, you've a different Json for a 'KeyEvent', when compared to a 'MouseEvent'.
See the example below.

"Key Event" example:
eventType: "Key",
KeyCode: "35",
Special: "Shift"

"Mouse Event" example:
eventType: "Mouse",
Button: "Left",
X: "30",
Y: "45"

This is just a simple sample, that shows how dynamic Json, can change over time. Static Json parsers wont do well with such scenarios.

Our objective was to parse, such dynamic JSON, and put the values as key/value pairs to a RDBMS database.

Our research on Dynamic JSON parsers, provided two options

1. System.Json Namespace (From Microsoft)

2. Newtonsoft.Json Namespace (From Newtonsoft)

Both can be installed through Nuget in Visualstudio.

Install Newtonsoft JSON through package manager console
PM> Install-Package Newtonsoft.Json

Install System.Json through package manager console
PM> Install-Package System.Json -Version 4.0.20126.16343

System.Json Namespace

Microsoft way of doing it. But it has a major limitation. If you've multiple json objects, in the same string, it will not support parsing it, unless you split them by your own means.

The below shows a simple usage of System.Json, on parsing a json string.

string jsonStr = "{Name:'Test',Marks:['20','81']}";
JsonValue parsedJsonObject = JsonObject.Parse(jsonStr);
switch (parsedJsonObject.JsonType)
case JsonType.String:
case JsonType.Number:
case JsonType.Boolean:
//JSon properties, get the value by converting it to string
string value = Convert.ToString(parsedJsonObject);
case JsonType.Array:
JsonArray jArray = parsedJsonObject as JsonArray;
for (int index = 0; index < jArray.Count; ++index)
JsonValue jArrayItem = jArray[index];
//Now recursively parse, each array item. i.e jArrayItem
case JsonType.Object:
JsonObject jObject = parsedJsonObject as JsonObject;
foreach (string key in jObject.Keys)
JsonValue jSubObject = jObject[key];
//Now recursively parse, each usb item. i.e jSubObject

Newtonsoft.Json Namespace

Excellent Json parser. It has a good performance over other parsers, when you've a bulk of Json objects to parse.

The below shows a simple usage of Newtonsoft.Json, on parsing a json string.

string jsonStr = "{Name:'Test',Marks:['20','81']}";
JToken parsedJsonObject = JToken.Parse(jsonStr);
switch (parsedJsonObject.Type)
case JTokenType.Raw:
case JTokenType.Boolean:
case JTokenType.Bytes:
case JTokenType.Date:
case JTokenType.Float:
case JTokenType.Guid:
case JTokenType.String:
case JTokenType.TimeSpan:
case JTokenType.Integer:
case JTokenType.Uri:
//JSon properties, get the value by converting it to string
string value = Convert.ToString(parsedJsonObject);
case JTokenType.Array:
JArray jArray = parsedJsonObject as JArray;
for (int index = 0; index < jArray.Count; ++index)
JToken jArrayItem = jArray[index];
//Now recursively parse, each array item. i.e jArrayItem
case JTokenType.Object:
JObject jObject = parsedJsonObject as JObject;
foreach (JProperty key in jObject.Properties())
JToken jSubObject = jObject[key.Name];
//Now recursively parse, each usb item. i.e jSubObject

Also Newtonsoft, has one more advantage over System.Json, as

it can parse multiple json, included in the same string
For eg. It can parse a Json string something like this.
string jsonStr = "{hello:'there'} {goodbye:'you',Thanks:'you'}";

The below code snippet, shows how to parse a string that contains multiple 'Json' objects. It does read, the two different Json object, in two while loop iterations.

string jsonStr = "{hello:'there'} {goodbye:'you',Thanks:'you'}";
using (var stream = new MemoryStream(System.Text.Encoding.UTF8.GetBytes(jsonStr)))
var serializer = new JsonSerializer();
using (var reader = new StreamReader(stream))
using (var jsonReader = new JsonTextReader(reader))
//Setting, to read multiple json, if it exists
jsonReader.SupportMultipleContent = true;
while (jsonReader.Read())
var obj= serializer.Deserialize(jsonReader);


So in essence, 'System.Json' is an obvious option, for those who always wants packages provided by Microsoft. For client environments, where security is a major concern, this will be the most viable option.

But if you need more advanced options, like 'parsing a json string that contains multiple json objects', you can rely on 'Newtonsoft.Json' namespace.

If you've aware about, such products, mention them in comments.

High Performing Linux File System With EXT4 (Journal Off)

For virtualization purpose, we required a very high performing Linux file system (Host machine, Lubuntu 14.04 LTS Version). Earlier we'd tried with 'EXT2' file system, but not impressive enough.

Later we came to know that, 'EXT4' is the best file system for high performance and stability. As you would probably know, 'EXT4' is a journaling files system by default, which cost a bit on the performance front.

So if you' opt the below combination, you will get the ultimate file system performance, that will benefits your host machine, as well as guests (IO Performance)

1. Use EXT4 File System

2. Switch Off 'Journaling'

3. Enable "Write Back" mode

So, say you've your partition already formatted with 'EXT4' (It is with Journaling-On by default). Now from command line do the below operations to switch off 'Journal'.

Say for eg: Our EXT4 partition is '/dev/sda5'

# Enable writeback mode. Provides the best ext4 performance.
tune2fs -o journal_data_writeback /dev/sda5
# Delete journal option
tune2fs -O ^has_journal /dev/sda5

Now recheck the files system.

# Required fsck
e2fsck -f /dev/sda5

We're done! We can further optimize the performance with the below additional options with EXT4 (fstab opions: data=writeback, noatime, nodiratime)

/dev/sda5 /opt ext4 defaults,data=writeback,noatime,nodiratime 0 0

Ok. the above is applicable, if we already have a 'EXT4' files system.

What if we've an EXT2 file system, that we need to make it as a 'EXT4' with the above options? Just convert it to EXT4 first (using the below command) and perform the above.
# /dev/sda5 disk (Change EXT2 to ext4)
mkfs.ext4 /dev/sda5

Hope, this might be useful for someone having similar experiments!

Single Sign-On With Office365 (Windows Azure) from Corporate Domains

Recently we'd given an Office365 introduction, as part of our customer migrating their Office2007 environments to Office365 subscription based model.

The most surprising part of the training, was

The demonstration of corporate users, directly login to Office365 (Hosted in Windows Azure Cloud) without entering any credentials from their Windows7 laptops.

Wow! That's quite amazing.

But how it could be? Typically an organization have it's own domain (Active Directory/Domain Forest) and domain controllers. Lets say it is an On-Premise active directory (eg. was.int.mycompany1.org). Usually it is a well defined boundary, within which users access and authorization has been defined and restricted.

Normally two separate domains, will have no security context in common, So that they are logically separated. One user in first domain has nothing to do with or have access to resources in other domain. (say between for eg: was.int.mycompany1.org and was.int.mycompany2.org). But if situation demands, you can set a common security context with the two independent private domains using 'Domain Trusts'. In that case we say

"One domain Trusts another domain, and security context have been set between the using Trusted Domain Objects"

Using "Trusts", user in one domain can be authenticated and authorized in another domain. But that will be happening for 'Private' domains/organizations.

But "Windows Azure" is a public cloud that hosts Office365. So it is not a private domain and have to support multiple independent organizations any way. So how this really works? How it really separate one organization context from another, while validating users?

The answer in the fact that,

"Windows Azure Active Directory is not a normal AD, but it is Multi-Tenant Active Directory"

So it support multiple organizations and keeps them separate from each other. But how 'Windows Azure Active Directory' knows about the corporate users identities, that's privately stored with in the Organization's On-Premise active directory, that is private to the organization?

The user objects get synchronized/copied between the corporate active directory and the Windows Azure active directory.

This is done using a software component called 'DirSync' and called as,

"Active Directory Federation Services (ADFS)"

The below figure will give you a better idea on this.


As you can see, the bottom oval represents your organization's domain (Active directory), and its being synched to 'Windows Azure Active Directory' using 'DirSync' component.

That means, once you logged into to your 'On Premises Domain Controller', you need not login again, to access your 'Office365' applications from the 'Windows Azure Cloud'. As now 'Windows Azure AD' have your details, it allows you to login instantly and is 'Zero Sign-On'.

This is called,

'Federated Identity' in Office365"

and is the most seamless identity scenarios. Organization's IT support team, don't have track two separate, user identities, but can have the same user identity that is used at the corporate domain. It give the easy and more fine grained control, as adding/removing a user from Office365 cloud, simply means add/remove the user object from the corporate active directory.

Apart from this, 'two more identity scenarios' are possible with Office365 subscription, not as seamless as 'Federated Identity'. They are listed below;


Cloud Identity:
You have a separate, User identity for each user, other than the one used to login to your corporate active directory. Simply you've left with two separate user identities, one is for login into the Office365 cloud and once for login to your corporate domain.

Directory & Password Synchronization:
Similar to 'Federated Identity', but synchronized in one way only (i.e From corporate AD to Windows Azure AD). Also the passwords are stored as hash values in 'Windows Azure AD'. User can use the same corporate credentials for their Office365 cloud as well. (But they have to re-enter it, each time they access Office365)

Wednesday, October 22, 2014

The Paravirtualization Spectrum

As OS virtualization techniques evolved from Full Virtualization to Hardware assisted and then to Para Virtualized modes, all of these norms seems quite confusing and difficult to understand. As a beginner, I've searched a bit on this topic, and found a good graphical representation that describes all these terms, much more intuitively. Please note that, all terms are evolved based on Xen Hypervisor. We can also see, how KVM (Kernel Based Virtual Machine) fits in to this.


Full Virtualization (FV)

What is virtualized?

Everything from BIOS, motherboard, disk, network, CPU Privileged instructions etc.


As you can see, every hardware elements will be virtualized. This is the least performing mode. It requires frequent context switches.

Windows Guest Support?

As it does not requires Kernel recompilation.

Xen and KVM.

*Hardware Virtualization (HVM) - Hardware Extensions to Full Virtualization

What is virtualized?
Everything, except CPU privileged instructions.

Fully virtualized, but instead of virtualizing privileged instructions, they can be directly run with physical CPU, using Intel-VT/AMD-V extensions.

i.e CPU's with these extensions, provides 3 rings (instead of 2), where 'Ring -1' will be used by Hypervisor, So guest can directly execute privileged instructions in Ring0 (Supervisor Mode)

Better than Full Virtualization.

As running privileged instructions can be run directly under physical CPU, no emulation or context switch required and hence the performance improvement.

Windows Guest Support?

As it does not requires Kernel recompilation.

Xen and KVM.

This one is not included in the above figure, as it is most often used with para virtualized drivers (network/disk), for high performance.

Full Virtualization with Paravirtualized Drivers(PV on HVM)

What is virtualized?
Everything, except CPU privileged instructions + Disk + Network.

Para virtualized implementations have been provided for Disk and Network, and supporting drivers have made available to guests.

Better than HVM and Full Virtualization.

As additionally disk and network interfaces has been paravirtualized, this modes provides a huge performance improvement, as most of the time disk/network performances are the major bottle necks.

Windows Guest Support?

As it does not requires Kernel recompilation. Paravirtualized network and disk been supported by drivers specifically developed for windows guests.

Xen and KVM

Para virtualized Interrupt Controllers with HVM (PVHVM)

What is virtualized?
Everything, except CPU privileged instructions + Disk + Network + APIC/IOAPIC.

Interrupt controllers and CPU timers have been paravirtualized here.

Better than PV on HVM, HVM, Full Virtualization.

In addition to paravirtualized disk and network, it also paravirtualizes interrupts and timers.

Windows Guest Support?

To support paravirtualzed APIC and IOAPIC, kernel should be recompiled and only linux based guest's are supported in this mode with kernel recompilations.

Xen only.

Almost Paravirtualized (PVH)

What is virtualized?
Only memory/pagetables have been virtualized.

There are no emulated devices of any kind, like no Motherboard, PCI and BIOS or legacy boot.

Far better than PVHVM, PV on HVM, HVM, Full Virtualization.
As virtually everything except memory access have been paravirtualized.

Windows Guest Support?
This extensive paravirtualization support requires kernel recompilations.

Xen only.

Fully Paravirtualized (PV)

What is virtualized?
None. Everything paravirtualized.

Highest performance.

As virtually everything have been paravirtualized.

Windows Guest Support?

This extensive paravirtualization support requires kernel recompilations.

Xen only.


Now I know why even there is a comparison between Xen and KVM, though Xen have advanced high performing modes like PVM and PVHVM. Those modes can only run modified Linux Distros not windows guests.

Most often comparison will happen for windows guests, that will only supported in Full Virtualization, HVM and PV on HVM modes, that are common to both KVM and Xen.

So KVM may be performing equal or slightly better than, PV on HVM mode.

Otherwise Xen will simply outperforms KVM, as it having more advanced modes like PVM.

Original post


I request the readers to put any suggestions , corrections, if found as comments, so that I can update this article to be as accurate as possible.

Lambda Expressions - A Comparison between C# and Java

When we came to know about ‘Lambda Expression’ feature in C#, I really became obsessive with it. Using this feature, you can represent an anonymous function in a very crisp and compact way.

C#: "A lambda expression is an anonymous function that you can use to create delegates (Or simply Type safe function references)"

We've explored this feature in C# first, and lately in Java as well. From early on, C# (.NET 3.5) supports delegates and anonymous functions, So the 'Lambda Expression' feature addition, seems quite natural and intuitive in the C# language.

It seems like Java inspired by this enhancement in C#, and introduced their own version of 'Lambda Expression' in latest Java Language (From Java8 onwards). But the problem, is they have to incorporate this to an existing java specification, that revolves around only Class and interfaces. In java there is no concept of 'Delegates Or Function references', hence 'Lambda Expression' feature addition seems less intuitive in terms of implementation.

in C#, the lambda expression revolves around 'Delegates' with anonymous functions and in java it is implemented through 'interfaces - that contains a single function prototype' with anonymous classes

Lets see an example below to have a comparison between the two.

Build Your Lambda in Steps

(Delegate/Anonymous function based)
(Interface/Anonymous class based)

1. conceive your concrete function first

Lets take a simple function, that validates an even number. It returns 'true' for an even number and 'false' for an odd number. Let's say this is our concrete function, which we will convert to 'lambda expression' through subsequent steps.
bool IsEven(int number)
    return number % 2 == 0;

C# supports functions inside any class. No interface implementation required.

class EvenValidator implements Proto
    bool IsEven(int number)
        return number % 2 == 0;   

Java requires the function to be enclosed inside a class, that implements the interface prototype 

2. Define a Function Prototype to your concrete function 

delegate bool Proto(int number);

C# requires a delegate, A type safe function reference. No interface implementations!

interface Proto
    bool IsEven(int number);

Java requires the function prototype to be wrapped inside an interface. Extra overhead!

3. Assign concrete function to Function Prototype 

Proto objDlgte = new Proto(IsEven);

Clean and natural way of wrapping functions.

Proto objDlgte = new EvenValidator();

Java does this at the old way. 

4. Anonymous Declaration

As we're referring the function (IsEven) method body, only inside the Function Reference
and nowhere else, why don't we get rid of the function definition altogether, and inline the method body along with the Function reference.

This is exactly an anonymous function or class does. 
Proto objDlgte = (Proto) delegate(int number)
    return number % 2 == 0;

C# using Anonymous function.

Proto objDlgte = new Proto()
    bool IsEven(int number)   
        return number % 2 == 0;   

Java using anonymous class. 

5. Finally arrive at Lambda Expression 

Proto objDlgte = number => number % 2 == 0;  

Proto objDlgte = number -> number % 2 == 0;  

6. Usage/Inovke 

bool isEven = objDlgte(20);

Seems very natural way for invoking a predicate, as a function call.

bool isEven = objDlgte.IsEven(35);

Less intuitive, as you've to mention both the interface and the method to invoke it. 

Advanced Features - C# (Advantage)  Java (Limitations)

There are some feature limitations to Java, compared to C#. Explained below.

Capturing Variables 
C# allows us to capture variables inside lambda expressions, defined in the parent scope.
Very convenient.

Java: Java only allows to capture final or effectively final variables inside the lambda expressions, defined in the parent scope.Very Restricting.


void TestFunction()
    int outNumber = 5;
    int evenDivider = 2;
    Proto objDlgte = number =>
       return outNumber % evenDivider == 0;

C# support capturing almost all type of variables from outer scope.

void TestFunction()
    final int outNumber = 5;
    final int evenDivider = 2;
    Proto objDlgte = number ->
        return outNumber % evenDivider == 0;

You can only refer final/effectively final outer variables inside lambda.So you need to declare the variables as final. A major drawback. 


Lambda expression is a nice feature addition to high level languages like C# and java. C# lambda expression built with delegate and anonymous functions. Java lately added this feature, inspired from C#. Java's implementation seems not as clean as the way C# does it and having limitations while accessing variables from the declaring scope.

Sunday, October 5, 2014

Para virtualizing KVM - An Ultimate Virtualization Experience from your Desktop/Laptop

Recently, We’d gone for some research, on how to improve virtualization performance with KVM. As a result, we made KVM as our default choice for Virtualization due to it’s better performance and stability.

My friends talked about the performance issues with KVM, while working with windows guests. Especially the graphics performance. Mouse movement is painfully slow and irregular. Also the network and disk performances are not any better.

Upon analysis, its concluded that by default KVM (Or any other Hypervisor for that reason), uses ‘Full Virtualization’, to run the guests. In such a scenario, Guest OS’s run as unmodified (Full Virtualization), where they are not aware about their underneath virtual environment. They simply assume, as they are running under a normal hardware platform. Such a ‘Fully Virtualized’ guest, demands more hardware resources and memory from the Hypervisor as they are not fine tuned for the virtualized environment, simply because unmodified guest OSs have been built with native hardware in mind.

KMV also supports ‘Para Virtualization’. ‘Para Virtualized’ guests are very well aware of their virtualized environment and adjust themselves accordingly. They put a less demand on resources from their hosted hypervisor. In other words, guest OS know it is running under a virtualized environment and cooperate with Host, to relinquish resources and memory to  more demanding guests.

The two scenarios have been detailed in the below figure.


‘Para Virtualization’ techniques, make use of ‘device drivers’ (Para Virtualized Drivers – PVDs) at two levels. One set will be installed inside ‘Guests’ and other will be used inside the Host OS/Hypervisor. These two sets of ‘device drivers’ cooperate each other, and work cohesively as a single unit, to provide the optimizations.

To make a Windows guest, Para virtualized, drivers are available for respective Hypervisor. Once installed your windows guest are become virtualization aware. For KVM these are called ‘VirtIO’ drivers.

When applied in our environments, the performance gain is unbelievable. It is running with a ‘Near Native’ performance, something similar to Xen or VMWare and outperforms other Type2 hypervisors like Virtual Box. We are presenting that as a case study in the below sections.

A Case Study:

Physical Host Specs:

CPU: Intel Pentium Dual Core CPU (2.6 GHZ, 2.6  GHZ)

               Intel-VT/AMD-V has been enabled in BIOS, to get the Hardware assisted virtualization support.


HDD: (2 hard disks)

               SeaGate 250 GB (For Host OS installation, Swap space)

               Western Digital 1TB (To save guest disk images)

Host OS: Lubuntu 14.04

               Boot Time: 13 Seconds

               Startup Memory Usage: 175MB

               Shutdown Time: 4 Seconds

               Desktop: LXDE

One thing to note here that, your Host OS should be thinner as possible. We’ve chosen the best balance between ease of use and lightweightness, by selecting ‘Lubuntu14.04’ as the Host OS. Lubuntu is the lighter version of Ubuntu and one of the fastest Linux distribution.

Also it is better to have a secondary hard disk to keep your guest disk images, as Host OS does not have to compete with Guest OS’s while seeking data from hard disks. This gives the highest performance and long life to your drives, as it minimizes Read/Write head movements.

1. Install KVM and Management Tools

“sudo apt-get install qemu-kvm libvirt-bin python-spice-client-gtk virt-manager bridge-utils”

Add the current user to the ‘libvirtd’ group.

“sudo adduser `id -un` libvirtd”

2. Enable Nested Virtualization In KVM

echo 'options kvm_intel nested=1' >> /etc/modprobe.d/qemu-system-x86.conf

Download Para Virtualized Drivers for Windows:

Download SPICE Guest Tools here.

Purpose: Support Seamless integration with Host desktop, QXL GPU Para Virtualized drivers and guest tools.

Download Virtio Drivers (As ISO) here.

Purpose: Provides Para Virtualized Drivers for

Network Adapters (VirtIO Ethernet Adapter)

Block Storage         (VirtIO SCSI Driver)

PCI  Bus                    (VirtIO Serial)

RAM                          (VirtIO Memory Balloon Driver)

Create a Para Virtualized Windows Guest:

Open ‘Virt-Manager’ and start creating a windows guest.

Add ‘Nested Virtualization’ support.


Use ‘Virtio’ as Diskbus.


Use ‘Virtio’ as Network Adapter Device Model.


Use ‘Spice’ as Display Device and QXL as GPU.


Use PCI and Virtio Serial Bus.


To get the ‘VirtIO’ stuff work, you need to install Para Virtualized Drivers to the Windows Guest Machine. To accomplish that, first install the ‘SPICE Guest Tools’ EXE (You’ve downloaded earlier) inside the windows guest.

Once done, install other ‘VirtIO’ drivers, from the VirtIO ISO file (Downloaded earlier) for other devices (Network, Storage etc). Once completed you can open the ‘Device Manager’ to see the Para Virtualized Device Drivers (VirtIO) are in action. See the below figure.


When comes to performance, it truly runs with ‘Near Native’ speed. Opening applications like ‘Visual Studio' 2012’, Office 2007 are blazing fast. The folder/file browsing is so quick that, they will get opened as soon as you release your finger from the left mouse button. VirtualBox had took a noticeable amount of delay while performing the same operations. Also as number of guest increases, KVM simply outperform Virtualbox in terms of stability, speed and demand on hardware resources.

Especially you can run more number of guests with KVM, whose combined Virtual RAM requirement is more than the actual installed physical RAM, through a technique called ‘Memory Ballooning/Memory Overcommitting’. I’m planning to provide that idea in a separate article.

Tuesday, September 2, 2014

ThreadPool implementation using IO Completion Ports (Visual C++)

Looking for an efficient, fast and custom thread pool using Visual C++? Then implement a custom thread pool using IO-Completion ports.


There are two ways to manage a pool of threads.

One is, to depend on Microsoft’s straight forward way of thread pool implementation using Win32-API’s such as ‘QueueUserWorkItem’. Though simple, it lacks on the performance side. This is due to the fact that, frequent thread context switching can happen, while selecting worker threads from the pool to process requests. Also if we’ve a multi-core CPU, all cores may not be used in this implementation. Typically for inter-thread communications, we might be using ‘window messages’ that might also be slow.

The second way is to build your own thread pool using ‘I/O Completion ports (IOCP)’ API’s. It is known to be an extremely fast and efficient solution for custom thread pooling, that can far outweigh the rudimentary thread pooling described above. At the very heart, IOCP implementation is revolving around the usage of the below 3 Win32-API’s.


This API will create the IO queue, to which work requests will be posted and retrieved. In a thread pool scenario, work requests will be submitted and retrieved to/from this queue and worker thread will process the requests.


This API is used to post a work request to the queue, by any client to the thread pool.


Worker threads, will call and wait on this API, to retrieve work request from the queue. This API, will not add any overhead to the CPU, like the conventional way of waiting on handles. 


The following are the advantages of using ‘IO-Completion ports’ for thread pool implementations.

1. Maximum CPU core usage

If you haven’t specified otherwise, IOCP will use the maximum number of CPU cores.

2. No unwanted context switching

The threads are used in LIFO (Last in first out) manner, hence the last thread in memory will get the priority to process the next work item and hence, no context switches will happen to load another thread to memory.

3. No clogging up CPU, for polling or waiting

IOCP is more efficient than, the conventional way of waiting on events, polling or messages.

4. Make a configured number of pre-created worker threads to be available in thread pool

While setting up the thread pool, we can create a configured number of threads and make them readily available for servicing the requests as soon as, we starts the thread pool.

That being said, this is the most recommended way of doing multithreading on a server scenario, as the server can have a very high loads of requests to be processed by a pool of threads. Also IOCP is not only for a custom thread pool implementation, it can be used with any object, that can work with overlapped IO and Asynchronous IO events to get the maximum performance and to use the least amount of server resources. Typical examples are WSA sockets, file handles e.t.c.

You can get a good understanding on IOCP by following articles in msdn (Threadpool, IOCP in msdn).


Ok that’s about the theory. Let’s go ahead and create a custom thread pool using IOCP. For this exercise, we will be using Visual C++ with MFC (Microsoft Foundation Classes). Below depicts the high level view on the implementation.

a. Create the IO-Completion port for the thread pool (Using API CreateIoCompletionPort)

b. Create pre-defined number of worker threads, that is to be readily available to the pool

c. For each ‘work request’ from client:

Queue work request to IOCP using API ‘PostQueuedCompletionStatus

d. Each worker thread will be waiting for queued jobs

using API ‘GetQueuedCompletionStatus

If any jobs available (eg: Step#c got invoked by a client), one worker thread will awaken to process the request

e. Once we’ve done with the thread pool, stop it:

Make all worker threads exit (using ‘PostQueuedCompletionStatus’, with a different parameter)

A work request constitutes of two arguments.

Work Request:

Data to be processed

This is a void long pointer, that can point to any structure or object. Using LPVOID, we can support any objects as the Data and it ensures scalability.

Function to act up on the Data

Client has the option to attach a pointer to ‘C++ class member function’ or a ‘Global C function’, that needs to be invoked by the worker thread to process the supplied data.

Demo – UI walkthrough

Below shows the UI of the test application.


You will start, by providing the desired number ‘Number of Worker Threads’ in the very first text box and hitting on the ‘Start Thread Pool Manager’ button. This will initialize the thread pool and creates the worker threads on the startup.

Now you can push as many job as you wish, by providing the ‘Jobs To Push’ as a desired integer value (Number of jobs to push) and hitting either ‘Push Jobs to Job Queue Using Member Function’ or ‘Push Jobs to Job Queue using Global Function’. The former button is to use ‘C++ class member function’ to process the data or the latter is to use a ‘Global C Function’.

These functions will simply write some text (given to it as its Data) to a file to make them simple enough.

Once you experiment enough, don’t forget to hit ‘Stop Thread Pool Manager’ button to gracefully shutdown the thread pool. It will remove any pending jobs in the queue and wait for any pending processing to finish.

Demo – Code Walkthrough

Below are the type definitions for the ‘Global C’ and ‘C++ class member’ prototype functions. Please note that, if you need to attach a ‘C++ class member function’, the class containing the member function should be derived from the very base class of MFC, the ‘CObject’.

// Function prototype of the worker function to be supported at the client side
// [For Non Member functions, i.e. For static or global functions

// Function prototype of the worker function to be supported at the client side
// [For Class Member Functions that should be invoked using an object]


The below listing will give you a summary on the codebase. Please note that, in actual project source attached, they are scattered around various places like inside button clicks and MFC dialog classes. But for simplicity we’ve consolidated all such code into a single place here.

The header file contains the thread pool manager object. The implementation file contains the ‘Global C Function’ and ‘C++ member function’ to be invoked by the worker threads in the thread pool. ‘InitializeThreadPool’ should be called first to initialize the thread pool. Then jobs are pushed the thread pool using the ‘EnQueue’ method. Supporting macros are being used to attach the functions to the structure members. ‘pJobInput’ is the member, that will contain the ‘Data’ to be processed. In our case its simply the for loop index. Finally ‘StopThreadPool’ is being called to shutdown the threadpool gracefully.

Header File (.h)

CThreadPoolEx m_ThreadPoolManager;

void MemberFunctionWorkerProc( LPVOID pDataIn_i );

Implementation File (.cpp)

void GlobalWorkerProc( LPVOID pDataIn_i )

// Worker function [ A global function ]that will call backed by the thread pool manager
void CThreadPoolManagerDlg::MemberFunctionWorkerProc( LPVOID pDataIn_i )

// Initialize the thread pool first
if( !m_ThreadPoolManager.InitializeThreadPool( m_nWorkerThreadCount ))

void CThreadPoolManagerDlg::OnButtonStartJobs()

    // Create the job item and set the worker and call back functon
    // Which is same for all the worker procedure

    CREATE_MEMBER_FUNCTION_CALLBACK( CThreadPoolManagerDlg::MemberFunctionWorkerProc, JobItem );

    bool bNotInitialized = false;
    int nIndex;
    for( nIndex = 1; nIndex <= m_nJobsToPush; ++nIndex )
        // Add the job input to the job item
        JobItem.pJobInput = (LPVOID)nIndex;      
        // Add the job item to the job queue
        m_ThreadPoolManager.EnQueue( JobItem );

void CThreadPoolManagerDlg::OnButtonStartJobsNonmember()
    // Create the job item and set the worker and call back functon
    // Which is same for all the worker procedure

    CREATE_FUNCTION_CALLBACK( GlobalWorkerProc, JobItem );

    bool bNotInitialized = false;
    int nIndex;
    for( nIndex = 1; nIndex <= m_nJobsToPush; ++nIndex )
        // Add the job input to the job item
        JobItem.pJobInput = (LPVOID)nIndex;      
        // Add the job item to the job queue
        m_ThreadPoolManager.EnQueue( JobItem );


void CThreadPoolManagerDlg::OnButtonStopThreadPoolMgr()
    // Stop the thread pool gracefully

Demo – Code Download

Download the complete source code from here.

Note: We’ve provided two version of thread pool classes. ‘CThreadPool’ and ‘CThreadPoolEx’. In the former version, we are creating all the worker thread at once during the startup of the thread pool. So a pre defined number of threads will always be available during startup. In the latter version, a single worker thread will be created during the startup of the thread pool. Later, as requests get queued and there is no free worker thread available to process the request, additional worker threads will be created on the fly.

Tuesday, August 26, 2014

Type1 (Bare Metal) Hypervisor for Desktops/Laptops–XenClient

Note: This product has been discontinued by Citrix. The below package is no more available for Download!
For a long time, Type1 (Bare Metal) hypervisors are reserved for server environments. They are known for speed, consistency and stability during heavy loads. They are the only hypervisors that provides ‘near native’ performance. Typically Type-1 hypervisors are not available or used under traditional desktops/laptops due the the below reasons.
A. Issues with Type1 hypervisors (For Desktop usage)
1. No GUI for Type-1 Hypervisors
Type1 hypervisors are installed to a machine, without any GUI. Most often the physical host machine (In which hypervisor has installed) will be controlled by a terminal/client installed on a second machine. This is perfect for a server scenario, where most of the time, the machines are sealed inside Datacenters and managed remotely.
This setup is not at all viable for a desktop/laptop machine, as user will be directly interacting with the machine.
2. Hardware compatibility
Desktops/Laptops are offered by a diverse list of manufacturers, than any server counter parts. So there is a huge effort for supporting all these diverse hardware environment, to any hypervisors targeting desktops. This is not the case with servers, as there are a few vendors manufacturing servers, as compared to personnel computer industry.
So typically Desktops/Laptops lives with Type2 hypervisors like Virtualbox, VirutalPC, VMWare Workstation e.t.c. Though KVM can be considered as a Type1 hypervisor, that statement is not completely true, as it requires an host OS to be present.
B. A true Type1/Bare Metal Desktop Hypervisor – XenClient
Does this means, desktops has to live with only Type2 hypervisors?
The answer is ‘No’, as Citrix has now come up with a BareMetal/Type1 hypervisor for desktop/laptop environments. It is a “Type1 BareMetal Desktop Hypervisor”. The product is XenClient’. More specifically the hypervisor is called ‘XenClient Engine’.
The product can be used free, for managing up to 10 virtual machines. It has the following advantages that are typically required for any desktop/laptop environments and that are not available in server environments.
i. Hypervisor is integrated with GUI, for managing virtual machines.
The hypervisor comes with a GUI frontend, by which we can create and manage virtual machine. So a single machine can be used to host virtual machine as well as managing them using GUI.
Also remote management is possible with client products installed on other machines.
j. Additional utilities are available.
One good thing with this desktop edition is, it contains ‘Google Chrome’ browser. It’s a must utility that every desktop can’t live without.
C. Download and Configure – XenClient in your desktop
XenClient is free for use (To create/manage up to 10 virtual machines), though it requires registration.
Go to XenClient Home Page and click on ‘Download Now’ button. You many need to create an account next, by providing a valid email id and other details. Once done you will redirected to the ‘XenClient’ download page.
Download ‘XenClient Enterprise Engine’ ISO file (See above fig.). Now burn it to a DVD and use it for installation. The installation procedure can be found in the manual, Please read it carefully. Normally the setup will install into the first available free space in the first hard disk and format it with LVM partition format (logical volume manager).
For experimental usage, we’ve installed ‘XenClient’ as a KVM Guest, as KVM supports nested virtualization. To install XenClient engine as a KVM guest, you can find the procedure here. Below given the screenshots from our XenClient Engine installed as a KVM Guest.
For more management capabilities, we can use ‘XenClient Synchronizer’, that can be installed into any windows machine (.NET Framework should be installed). The free usage allows us to manage at most 10 virtual machines.