My visit to Let’s Test 2015 – Day 1

Test Pappy

Welcome to the second part of my visit to Let’s Test 2015. You can find the arrival day here.

The bright Swedish sun woke me up around 0530 for the first time, and I panicked, because I thought I missed the alarm going off. AfterI checked the time I realized that I need to get used to Swedish “nights” quickly. With a sunrise before 5 am the early mornings are really bright and the curtains are not that useful at all.

At breakfast I finally met Dan Billing and had time for a chat with him. This guy has a lot of knowledge around security testing and more. And he is also one of the facilitators of Weekend Testing Europe. I also met Chris for the first time. Not my last encounter with this really nice Swiss fellow.
A fun moment at the table was meeting Nicola and…

View original post 781 more words


Cynefin, testing & auditing

James Christie's Blog

Over the last few weeks following my CAST 2014 talk in New York, while the Stop 29119 campaign has been raging, I have been thinking more about some of the underlying issues.

One of these has been the idea of “best practice”, which led me back to the Cynefin Framework. If you don’t know about Cynefin then I strongly recommend that you learn about it and reflect on its implications. The article is a good start, not least because Dave Snowden, Cynefin’s creator, keeps an eye on it. This short presented by Snowden is also helpful.

An overview of the Cynefin Framework

Cynefin_as_of_1st_June_2014

I have carelessly described the Cynefin Framework as being a quadrant in the past, but that was sloppy. It isn’t. It merely looks like one. It is a collection of five domains that are distinct and clearly defined in principle, but which blur into one another in practice.

View original post 1,847 more words


“Software testers balk at ISO 29119 standards proposal” – my Infoworld.com interview

James Christie's Blog

This is the full text of the email interview I gave to , which appeared on August 25th. They used only part of it, which was fine by me. I was delighted they approached me and was happy to get any part of my message out.

What don’t you like about ISO 29119? Will this “standard” have much impact anyway?

ISO 29119 puts too much emphasis on process and documentation, rather than the real testing. Of course that is not its purpose, or the intention of the people who have developed it. However, I have seen in practice how people react when they are dealing with a messy, complex problem, and there are detailed, prescriptive standards and processes on hand. They focus on complying with the standard, and lose sight of the real goal. This sort of goal displacement is a familiar problem in many situations. It is frustrating that…

View original post 766 more words


“ISO 29119: Why it is Dangerous to the Software Testing Community” (from uTest)

James Christie's Blog

This article orginally appeared on the appeared on September 2nd.

Two weeks ago, I gave a talk at CAST 2014 (the conference of the Association for Software Testing) in New York, titled “Standards: Promoting quality or restricting competition?”


It was mainly about the new ISO 29119 software testing standard (according to ISO, “an internationally agreed set of standards for software testing that can be used within any software development life cycle or organization”), though I also wove in arguments about ISTQB certification.

My argument was based on an economic analysis of how ISO (the International Organization for Standardization) has gone about developing and promoting the standard. ISO’s behavior is consistent with the economic concept of rent seeking. This is where factions use power and influence to acquire wealth by taking it from others — rigging the market — rather than by creating new wealth.

I argued that ISO has not…

View original post 622 more words


Valuable Information From TestResults

In the following example, I will show how to get meaningful information from tests. All examples are written in C# (.NET 4.0) and use NUnit 2.6.2 and TeamCity 8.0.

For all tests, the same system under test is used – to keep it simple, I just wrote one simple method. This method is used for all tests.

The basic idea is to create a CultureInfo from a string. As test input data, a list of strings is used.

The poor’s man approach – Execute all tests in a loop

This is the simplest way to do it: use a foreach-loop and fail if one or more tests are failing.

Image

Image

 

In this case, no information is available which test(s) were failing. A small improvement is to use NUnit’s overload methods of the Assert.Fail(), Assert.IsTrue() etc methods where a string can be provided:

Image

Image

One test per item – TestCaseSource

From NUnit 2.5, it is possible to use the TestCaseSource attribute. With this mechanism, a test is executed n times. 

In our example, you see that there are 24 items, and 3 of them fail:

Image

Image

 

If a test cannot be executed n times within the same session, the test has to be executed once in n sessions (loop of tests). A typical reason might be if a test relies on a value of a configuration file.

Execute single tests in a loop

Since the tests are executed in a loop outside of NUnit, only one test is visible in NUnit GUI:

Image

In TeamCity, the same MSBuild code is executed 24 times, each run with one test.

Image

As a summary, have a look at TeamCity’s output for all 4 variations:

Image

 

 


How To Provide Additional Data To MSBuild From TeamCity

Command Line

If you run MSBuild from the command line, you can use the “property” injector command line argument. This is basically a key value pair.

e.g. C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe <PathToMSBuildProjectFile> /t:Run /p:Configuration=Release /p:Property1=Value1

 

TeamCity

You can use the same construct also in TeamCity:

TeamCity.PropertyInjection

 

The problem arises if you have

  • more than one build step in a build configuration
  • more than one build configuration

which all use the same parameters.

Use Build Parameters

TeamCity provides the support of build parameters (http://confluence.jetbrains.com/display/TCD7/Configuring+Build+Parameters)

With this approach, the build parameters are automatically provided to the MSBuild command line. You have to use System Properties to achieve this goal:

TeamCity.BuildPara

Where to put Build Parameters

There are two possiblities where to put them:

  • build configuration
  • build project

Build Configuration

If you put the build parameters to Build Configuration, they are for this build configuration only.

TeamCity.BuildPara

Build Project

If you add the build parameters to the BuildProject, they are “inherited” by all build configurations of this project.

TeamCity.BuildPara.ProjTeamCity.BuildPara.Inherited

 

If one or more of the parameters are different for a build configuration, you can still overwrite it in the build configuration’s Build Parameters:

TeamCity.BuildPara.Overwrite


Welcome to my WordPress Blog site.

After I attended Let’s Test conference in 2013, I was really impressed by the openness of the attendes about sharing their experience and thoughts. With this site, I would also like to share mine and hope I can help someone else as well.