Field Testing
The art of testing is in large part simply making sure that the end user is not the first person to discover problems. In order to do this, you have to go through the same actions as the end user. Most situations are fairly straightforward and involve things like trying to fill a form in a large variety of ways, use a service with many users at the same time, accessing a website from different browsers, etc. However, when testing mobile applications, another aspect comes into play: the environment. The user can take a phone anywhere and it will behave differently in different environments. Recently, I had the pleasure of testing in two extreme situations: in -30°C and next to a racing track near the Dead Sea in Jordan.
Jordan
I'll start with the racing track. One of the key features of Nokia's WRC Live application is the ability for fans to follow the race live, either at home or while 'on stage'. The fan can watch a map that shows the cars in real-time. This feature is awesome only if it truly works; the experience isn't quite the same if the fan sees the car pass in front of him and 30 seconds later on his phone.
There is only one way to test the entire chain from the GPS unit in the car to your handset: to be on stage. So we went to Jordan to see the WRC rally while testing a very early version of the application. We were impressed with the results. There were only a couple of seconds lag between the actual passing of the car and the passing on the map, which was accurate enough for fans to know in advance who is passing and what their position is in the race.
We did find some problems, however, which we simply could not foresee in the comfort of our Helsinki office. One of those issues is the amount of dust next to the track. Fans would need to be able to clean their screen without activating the touch buttons or moving the map. We fixed the problem by adding a lock feature. This nicely confirmed how important it is to test in an environment that is as close as possible to the actual usage scenario.
Fridge
The second extreme testing environment was at minus 30 degrees Celsius. We developed an application which is going to be used throughout the year, outside, in Finland, and would come preinstalled on a specific handset. We needed to make sure the appliance - the handset and application - worked in the harsh Finnish winter conditions. This is slightly problematic in June. To start, it is about 55 degrees too warm. Time to go to a fridge, a big fridge.
We found a research facility that tests large trucks in extreme conditions to make sure they don't break down. In other words, a fridge that should have room for a couple of people and phones. Our main concern for the test was battery life. We know that extreme cold shortens battery life, but our questions was by how much? We had pruned the possible handsets down to two contenders, and also had to make a final decision about which one to use in production.
On a warm sunny day in June, we dug out our winter clothes and set out to do some testing. It turned out to be tougher than we thought. We ended up leaving the phones in the fridge and one of us would quickly run some test while the other two reheated themselves outside. In the end, we could safely conclude that the phones would be fine as they lasted longer in the fridge than the humans operating them.
All in all, a great set of experiences. For me, it reinforces my idea that the best way to test is real life testing.
