Evaluating Iron scripting languages for use as a test framework

Not so long ago, I was passing through the hallway and someone from my team was giving praises about the XML based scripting language they were using to automate some tests.  This was particularly desirable because the process of compiling test changes, copying binaries to the correct location on the virtual machine, and rerunning the test is a bit tedious.  With scripting the XML file could be edited on the VM and the working file could be copied back to the development machine and checked in.

My reaction to this was to wonder why we needed a new scripting language when there were already so many in existence! This is especially true considering each component probably would require its own XML processing engine so there are probably multiple out there.

One additional benefit to using scripting for your tests is that since scripts are often interpreted, your build times should be reduced over implementing tests in C#. But there are also some down sides:

  • Detecting API changes and incompatibilities won’t happen until runtime when using scripting languages.
  • You may have to install additional binaries to run your scripts
  • Many people are probably not proficient enough in a scripting language such that they would consider using it for testing. It takes time to learn a new language.
  • Hosting a scripting language requires additional learning and research if you require a mixture of compiled and scripted code.

In this post, I will show some examples of scripts to run a simple scenario using the Unified Communications Managed SDK (UCMA), which I’ve used once or twice before.  I will attempt this in Iron Python 2.7 and Iron Ruby 1.1.3.

Note: PowerShell is another possible choice, but there are some additional things to overcome so I hope to handle it in a future post.

Challenges

Each language has some strengths and some challenges when it comes to dealing with existing .Net code.  In particular, the APIs we are testing have to have equal or better usability for the following C# code:

 endpoint.RegisterForIncomingCall<InstantMessagingCall>(this.CallReceived);
  
 call.StateChanged += this.CallStateChanged;

Basically, the scripting languages need to deal with passing generic delegates, and registering event handlers that use an EventHandler<TArgs> signature.

Using Iron Python

I am more familiar with Python than Ruby, so out of the gate Python has an advantage and it may be visible in my writing in this post.  If you are fluent in Iron Ruby and would like to point out some corrections to put Ruby on a more even playing field, feel free to leave a comment at the bottom of the post.

Iron Python has built in support for generics, which I happened to know when writing the sample scripts. This helped a lot in making the RegisterForIncomingCall usage easy. This section of code is shown here:

 print 'Registering for incoming call'
 self.__endpoint.RegisterForIncomingCall[InstantMessagingCall](self.__IMCallReceived)        

Also in Iron Python, registering for event handlers is similar to C# and doesn’t require much of a learning curve:

 self.__imCall.StateChanged += self.__CallStateChanged

But to load a dll so you can use it is a bit unintuitive at first:

 import clr
 import sys
  
 sys.path.append('<path to dll>')
 clr.AddReference("Microsoft.Rtc.Collaboration.dll")

and then python requires importing the symbols you need. I did it the hard way:

 from Microsoft.Rtc.Collaboration import ServerPlatformSettings,CollaborationPlatform,ApplicationEndpointSettings,ApplicationEndpoint
 from Microsoft.Rtc.Collaboration import Conversation,InstantMessagingCall,CallEstablishOptions
 from Microsoft.Rtc.Signaling import FailureResponseException

it could probably be simplified a bit like this:

 from Microsoft.Rtc.Collaboration import *
 from Microsoft.Rtc.Signaling import *

but it is probably the case you would pull in too many type names as .Net probably is not as conservative on what is exposed as Python libraries are.

The rest of the issues with python are the usual preference items that come up when comparing languages. For example, you will see “self” repeated a lot in python classes, and the space significance of the blocks may not be to your liking.

Using Iron Ruby

Not being too familiar with Ruby, I had to do a little research to get started. After picking up the Ruby necessities my first challenge was getting the reference set up to the API I was interested in testing. After trying a few things, I was able to get the API to test loaded, but it probably not the best way:

 require 'C:\windows\microsoft.net\assembly\GAC_MSIL\Microsoft.Rtc.Collaboration\<version>\Microsoft.Rtc.Collaboration.dll'
  
 include Microsoft::Rtc::Collaboration

Note that it is being included directly from the GAC. I should be able to specify the assembly information and it would hopefully find it in the GAC, but that didn’t seem to work unfortunately.

The next thing was to figure out the RegisterForIncomingCall invocation. This is what I came up with:

 # The C# equivalent of these two lines is:
 #
 # endpoint.RegisterForIncomingCall<InstantMessagingCall>(this.IMCallReceived)       
 #
 p = Proc.new { |sender,e| IMCallReceived(sender,e) }
 @endpoint.method(:RegisterForIncomingCall).of(InstantMessagingCall).call(p)

As you can see, this is probably not the most optimal way to do it in Ruby.  A small interactive experiment shows that Ruby does support generic syntax:

 >>> require 'System'
 => true
 >>> include System::Collections::Generic
 => Object
 >>> x = List[System::Int32].new
 => []
 >>> x.add(4)
 => nil
 >>> x
 => [4]
 >>> x.add(7)
 => nil
 >>> x
 => [4, 7]
 >>>

but that is how I initially got it working.  I probably could go back and change the code now, but finding the better way to register the event handlers is really bugging me.  It seems that this should have worked:

 # The following should also be valid, but doesn't seem to work. Because of generics?
 #
 #  @imCall.InstantMessagingFlowConfigurationRequested.add do |sender, e| 
 #    FlowConfigurationRequested(sender,e)
 #  end

But it didn’t work for me as you can see from the comment.  I eventually got it to work as follows:

 # In C#: call.InstantMessagingFlowConfigurationRequested += this.FlowConfigurationRequested
 p = Proc.new { |sender,e| FlowConfigurationRequested(sender,e) }
 @imCall.InstantMessagingFlowConfigurationRequested.add p

Although it works, it isn’t really intuitive to someone coming from C#. Maybe I can do something with the += operator to make this easier, but out of the box it didn’t seem to allow that. If you have experience with this in Iron Ruby feel free to leave a pointer in the comments on what the best way to register the event handlers.

Other than that, everything is pretty straight forward and again, the only thing left are tidbits related to language syntax and preferences.  Ruby is not sensitive to whitespace to determine its block boundaries like Python is, but you need to be sure to match up your “ends” correctly.

Ruby has more support for creating DSLs (sub languages) based on Ruby syntax which I am interested in trying out at some point. With a little more time spent with Ruby I’ll hopefully be able to pick up the best practices and improve my example.

Choosing your language

Given that some language aspects are likely to bug people and affect their choice of languages, it is very likely that both languages will need to be available in the test environment. Unfortunately, Iron Python and Iron Ruby currently use different versions of the hosting API which makes having them both present somewhat tricky.

The recommended solution is to download the source and recompile so they use the same hosting APIs.  This may be a barrier to entry that is too high for some people and drive them back to inventing their own scripting languages again. I quickly attempted this, but it will take more time and investigation to get it working.

Summary

In this post I presented two existing scripting languages that people could try instead of rolling their own.  Below I’ll include the full text of the implementations so you can see more clearly. Hopefully you will be inspired to spend one or two days learning a scripting language and consider using an existing one rather than rolling your own. By looking through the code below you may find one language appealing.  Maybe it will reduce the anxiety about learning one of the languages.

test.py

 import clr
 import sys
  
 sys.path.append('<path to dll>')
 clr.AddReference("Microsoft.Rtc.Collaboration.dll")
  
 print "Loaded ... "
  
 from Microsoft.Rtc.Collaboration import *
 from Microsoft.Rtc.Signaling import *
  
 class Client:
     
     def __init__(self,ownerUri,port):
         self.__platform = None
         self.__endpoint = None
         self.__ownerUri = ownerUri
         self.__port = port;
         self.__conversation = None
         self.__imCall = None
         self.__imFlow = None
         
     def get_Port(self):
         return self.__port
     
     def get_OwnerUri(self):
         return self.__ownerUri
             
     def Initialize(self,proxyPort):
         settings = ServerPlatformSettings("UserAgent", "localhost", self.__port, self.__ownerUri + ";gruu")
         self.__platform = CollaborationPlatform(settings)
  
         self.__platform.EndStartup(self.__platform.BeginStartup(None, None))
  
         settings = ApplicationEndpointSettings(self.__ownerUri,"localhost", proxyPort)
         print "creating app endpoint"
         self.__endpoint = ApplicationEndpoint(self.__platform, settings)
         
         print 'Registering for incoming call'
         self.__endpoint.RegisterForIncomingCall[InstantMessagingCall](self.__IMCallReceived)        
         print 'Establishing app endpoint'
         self.__endpoint.EndEstablish(self.__endpoint.BeginEstablish(None,None))
         
          
     def SendMessage(self,message):
         self.__imFlow.EndSendInstantMessage(self.__imFlow.BeginSendInstantMessage(message,None,None))
  
     def MakeCall(self,ownerUri):
         self.__conversation = Conversation(self.__endpoint)        
         self.__imCall = InstantMessagingCall(self.__conversation)
         self.__imCall.InstantMessagingFlowConfigurationRequested += self.__FlowConfigurationRequested
         self.__imCall.StateChanged += self.__CallStateChanged
         
         try:
             self.__imCall.EndEstablish(self.__imCall.BeginEstablish(ownerUri, None, None, None, None))
         except FailureResponseException,e:
             print "Exception:", e.Message
                
     def Terminate(self):
         if self.__imCall != None:
             self.__imCall.EndTerminate(self.__imCall.BeginTerminate(None,None))
  
     def __CallStateChanged(self,sender,e):
         s ="[" + self.__ownerUri + "] Call state changed: " + e.PreviousState.ToString() + " => " + e.State.ToString()
         print s
         
     def __IMCallReceived(self,sender,e):
         print "Call received."
         self.__imCall = e.Call
         self.__imCall.InstantMessagingFlowConfigurationRequested += self.__FlowConfigurationRequested
         self.__imCall.StateChanged += self.__CallStateChanged
         e.Call.BeginAccept(self.__CallAcceptCompleted,None)
                          
     def __FlowConfigurationRequested(self,sender,e):
         print "[",self.__ownerUri,"] Flow configuration requested."
         self.__imFlow = e.Flow
         e.Flow.StateChanged += self.__FlowStateChanged
         e.Flow.MessageReceived += self.__MessageReceived
  
     def __FlowStateChanged(self,sender,e):
         s = "[" + self.__ownerUri + "] Flow state changed: " + e.PreviousState.ToString() + " => " + e.State.ToString()
         print s
         
     def __MessageReceived(self,sender,e):
         s = '[' + self.__ownerUri + '] Message received: '+ e.TextBody
         print s
  
     def __CallAcceptCompleted(self,result):
         try:        
             self.__imCall.EndAccept(result)
         except RealTimeException,e:
             print "Call accept failed:", e.Message
  
 c1 = Client("sip:one@myhost", 5061)
 c2 = Client("sip:two@myhost", 5062)
  
 print 'Initializing client 1'
 c1.Initialize(c2.get_Port())
  
 print 'Initializing client 2'
 c2.Initialize(c1.get_Port())
  
 print 'Making call'
 c1.MakeCall(c2.get_OwnerUri())
  
 c1.SendMessage("Hello World!")
  
 c1.Terminate()
 c2.Terminate()
  

test.rb

 require 'C:\windows\microsoft.net\assembly\GAC_MSIL\Microsoft.Rtc.Collaboration\<version>\Microsoft.Rtc.Collaboration.dll'
  
 include Microsoft::Rtc::Collaboration
  
 class Client
     def initialize(ownerUri,port)
        @platform = nil
        @endpoint = nil
        @ownerUri = ownerUri
        @port = port
        @conversation = nil
        @imCall = nil
        @imFlow = nil
     end
  
     def Port
         @port
     end
  
     def OwnerUri
         @ownerUri
     end
  
     def CallAcceptCompleted(result)
         puts "Call accept completed."
         @imCall.EndAccept(result)
  
         #rescue Microsoft::Rtc::Signaling::RealTimeException => e
         #    puts "Accept failed: " + e.Message
         #end
     end
  
     def IMCallReceived(sender,e)
         puts 'Call received'
         @imCall = e.Call
  
         # The following should also be valid, but doesn't seem to work. Because of generics?
         #
         #  @imCall.InstantMessagingFlowConfigurationRequested.add do |sender, e| 
         #    FlowConfigurationRequested(sender,e)
         #  end
  
         # In C#: call.InstantMessagingFlowConfigurationRequested += this.FlowConfigurationRequested
         p = Proc.new { |sender,e| FlowConfigurationRequested(sender,e) }
         @imCall.InstantMessagingFlowConfigurationRequested.add p
  
         # In C#: call.StateChanged += this.CallStateChanged
         p = Proc.new { |sender,e| CallStateChanged(sender,e) }
         @imCall.StateChanged.add p 
  
         p = Proc.new { |result| CallAcceptCompleted(result) }
         @imCall.BeginAccept(p,nil)
     end
  
     def MessageReceived(sender,e)
         s = '[' + @ownerUri + '] Message received: ' + e.TextBody
         puts s
     end
  
     def CallStateChanged(sender,e)
         s = '[' + @ownerUri + '] Call state changed: ' + e.PreviousState.ToString() + '=> ' + e.State.ToString()
         puts s
     end
  
     def FlowStateChanged(sender,e)
         s = '[' + @ownerUri + '] Flow state changed: ' + e.PreviousState.ToString() + ' => ' + e.State.ToString()
         puts s
     end
  
     def FlowConfigurationRequested(sender,e)
         s = '[' + @ownerUri + '] Flow configuration requested'
         puts s
         @imFlow = e.Flow
         
         p = Proc.new { |sender,e| FlowStateChanged(sender,e) }
         @imFlow.StateChanged.add p
  
         p = Proc.new { |sender,e| MessageReceived(sender,e) }
         @imFlow.MessageReceived.add p
  
     end
  
     def Start(proxyPort)
         settings = ServerPlatformSettings.new("UserAgent", "localhost", @port, @ownerUri + ";gruu")
         @platform = CollaborationPlatform.new(settings)
  
         puts 'Creating app endpoint'
         settings = ApplicationEndpointSettings.new(@ownerUri, "localhost", proxyPort)
         @endpoint = ApplicationEndpoint.new(@platform,settings)
  
         # The C# equivalent of these two lines is:
         #
         # endpoint.RegisterForIncomingCall<InstantMessagingCall>(this.IMCallReceived)       
         #
         p = Proc.new { |sender,e| IMCallReceived(sender,e) }
         @endpoint.method(:RegisterForIncomingCall).of(InstantMessagingCall).call(p)
  
         puts 'Creating platform'
         @platform.EndStartup(@platform.BeginStartup(nil, nil))
         @endpoint.EndEstablish(@endpoint.BeginEstablish(nil,nil))
     end
  
     def SendMessage(message)
         @imFlow.EndSendInstantMessage(@imFlow.BeginSendInstantMessage(message,nil,nil))
     end
  
     def Terminate()
         if @imCall then 
             puts '[' + @ownerUri + '] Terminating call.'
             @imCall.EndTerminate(@imCall.BeginTerminate(nil,nil))
         end
  
         if @endpoint then
            puts '[' + @ownerUri + '] Terminating endpoint.'
            @endpoint.EndTerminate(@endpoint.BeginTerminate(nil,nil))
         end
  
         if @platform then
             puts '[' + @ownerUri + '] Terminating platform.'
             @platform.EndShutdown(@platform.BeginShutdown(nil,nil))
         end       
     end
    
     def MakeCall(ownerUri)
         @conversation = Conversation.new(@endpoint)
         @imCall = InstantMessagingCall.new(@conversation)
  
         # In C#: call.InstantMessagingFlowConfigurationRequested += this.FlowConfigurationRequested
         p = Proc.new { |sender,e| FlowConfigurationRequested(sender,e) }
         @imCall.InstantMessagingFlowConfigurationRequested.add p
  
         # In C#: call.StateChanged += this.CallStateChanged
         p = Proc.new { |sender,e| CallStateChanged(sender,e) }
         @imCall.StateChanged.add p
  
         @imCall.EndEstablish(@imCall.BeginEstablish(ownerUri, nil, nil, nil, nil))
  
         #rescue Microsoft::Rtc::Signaling::FailureResponseException => e
         #    print 'Exception: ' + e.Message
         #end
     end 
 end
  
 c1 = Client.new("sip:one@myhost", 5061)
 c2 = Client.new("sip:two@myhost", 5062)
  
 puts 'Starting client 1'
 c1.Start c2.Port
  
 puts 'Starting client 2'
 c2.Start c1.Port
  
 puts 'Making call to ' + c2.OwnerUri
 c1.MakeCall c2.OwnerUri
  
 puts 'Sending message.'
 c1.SendMessage("Hello world!")
  
 puts 'Terminating client 1'
 c1.Terminate
  
 puts 'Terminating client 2'
 c2.Terminate

201101127_Scripting.zip