MSTest v2 - Run Settings for Selenium tests

July 13, 2017    UnitTesting DevOps VSTS TFS AutomatedTesting Selenium

MS Test v2 RunSettings for Selenium Tests on Multiple Browsers

Our team has been using the DataSource attribute to run multiple permutations of browser sizes since 2015. We’ve used the .testsettings file and the DeploymentItems to get the TestMatrix.csv to the Vm, which uses that as the DataSource.

We want to move to .runSettings. The TestRun Parameters and TFS/VSTS build steps to provide the browser type and size (ex. Chrome, Large) at runtime.

The DataSource attribute is great for MSTest running the same test multiple times, just based on how many lines in the .csv file. That feature was useful to get large and small showing next to eachother in the build results (they’d look like MyTest (Data Row 0) and MyTest (Data Row 1)). This was a quick indication that it wasn’t a timing issue. It was also good to add 10 lines in the .csv to try and replicate a difficult to replicate failure.

However, .runSettings seems the way forward and the VSTS build definitions have made it much easier than the old XAML builds to add in new build steps.

Note: I’m on an OnPremise TFS installation Version 15.112.26307.0. VSTS will be different. I need to spend time there with the Deployment Groups and Release Management and then come back and update this article. Thanks for your understanding.

My Approach to moving to runsettings

I created a new local.runsettings files for running Selenium test from Visual Studio (2017 at this moment) with TestRunParameters.

  <TestRunParameters>
    <Parameter name="browserType" value="Chrome" />
    <Parameter name="browserSize" value="Large" />
  </TestRunParameters>

We can read the value from the TestContext.Properties dictionary. This StackOverflow answer pointed the way.

    var browserType = TestContext.Properties["browserType"].ToString();

Now we can use the browser type and size to tell Selenium how to run the test.

Using the parameters in the Release

First we need to use Windows Machine File Copy to get all the dlls and needed Selenium drivers to the VM. I’m using the $(UITestFiles) variable to send them to “C:\UiTestFiles”.

Then we can use the Deploy Test Agent and Run Fuctional Tests steps in the release management.

Test Build steps - TFS onPremise

The is version 2, but on VSTS it looks different and there is only version 1. Make sure to check Run UI tests. Test Build steps - TFS onPremise

The settings will be similar to this. The $(TestMachines) variable is a comma list of VMs we have in our VMWare setup. UILocal.runsettings has the <TestRunParameters> I have above.

The Override Test Run Parameters allows us to override the TestRunParameter. Run Large Test Settings - TFS onPremise


Here’s a similar article on MSDN from 2015.

Using Docker with VSTS and Selenium we may be able to get more browsers running in parallel without more Vms. This isn’t directly related to the runsettings, but a link I came across and something I want to investigate.



comments powered by Disqus

Please consider using Brave and adding me to your payment ledger. Then you won't have to see ads!

Support me and download Brave!

Use Brave