Skip to main content

On Automated vs. Human Testing

A lot of people are pointing to Joel Spolsky's Talk at Yale: Part 1 of 3

and it's a great read. If you read it further, I guess closer to the end of Part 1, he makes some very valuable comments on automated testing and the need for human intervention, using Vista as an example:

Speaking of the old approach to testing - "they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one."

And because of the reliance on automated scripts - "And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug” by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP. "

"nobody wrote the automated test to check if Vista provided users with a compelling reason to upgrade from XP."

Now that sounds a little harsh but it's also very true. No one will argue (ok, I'm sure some will) that UAC is a better way for managing security, or that putting all the but the reasons that Joel mentions above are precisely the reasons why Chris went off on his rants about the Vista UI last year. Now Ed did counter with some great points but many did agree and the result (6 months after the release of Vista) was still the same.

I'm all for automated testing and test suites, etc - but let's remember that software is only useful if people can use it. And the people who write it aren't always the ones who find the bugs. (great post by Goran Zidar on that)

I often think of this when I wonder why we haven't done more implementing new VFP 9 technology in various products (including Foxfire!). After all, many of the features are completely awesome to put in. But the user experience has to count for something. And while putting a variety of code into the Comments field is one way and while sure, you can build a dialog with a bunch of checkboxes on it - I don't know if you come away making an interface easier to use. Now, most of my readers here are developers - so we want the cool features and the UI on these dialogs certainly works the way most of us would expect it. But I'm taking the leap here from a development IDE to an end user product. Just as a user doesn't always get all the different format options that can be dropped into the Picture box, nor do I think they will get all of the other features. So the user experience has to be considered priority and that is not something that can happen properly with automated testing.

I'm doing some work with outsourced resources from Russian but I've also seen the same results with local contractors. Early on, I have learned that if I want a dialog to say something, I have to spell it out completely - as in word for word, spelling out each dialog. If I don't, I know others will come back and clobber me for it.

Likewise with good user interfaces, it's something everyone needs to struggle with. What is the easiest way for users to discover a feature? (some might argue that users shouldn't have to discover a feature but in this case, I equate discover to inductive user interface). Now some don't get IUI and use obviously bad examples to describe its shortcomings. But that's not the point.

It's all about consistency and while I'm sure there are some automated tests that ensure buttons are lined up perfectly and the right fonts are used, they still don't take the "human impact" into consideration. I don't think any automated test would have shown that Apple's UI decisions would be as impactful as they have been in the software world.

Those that are looking for automated tests to do everything should look into good usability tests. Morae's a useful tool here but sometimes, it's simply enough to set up a Snag It  or a GoToMeeting and just record what people are going through. 

Blogged with Flock



Update: Argh! And just as I posted this from Flock, I was presented with another UI inconsistency. WHO was it that decided that any open-source or non-MS tool should reverse the order of the buttons in a dialog so that the most commonly used option was on the furthest RIGHT instead of the Cancel?

Comments

Tamar E. Granor said…
Andrew - I'd argue that programmers/developers shouldn't be designing UI at all. We know too much, we like computers too much, and we don't see the world the way most of our users do. UI should be designed by designers.
Andrew MacNeill said…
But how realistic is that?

While it would be nice for a programming team to be made up of:

a) a designer
b) a programmer/developer
c) a documenter

That could add a fairly big overhead to a project.

Sounds like a good discussion for a FoxShow - you up for it?

Popular posts from this blog

Blogs and RSS come to Microsoft.com

MS has just introduced their portal and it's pretty comprehensive. Nothing quite like learning that some people use AIM instead of MSN messenger, or that there really may be a need for supporting 4 monitors ( Cyrus Complains ) However, it's really a great sign that MS is serious about supporting the blogging community which seems to have um, exploded in size in the past year. Blogs and RSS come to Microsoft.com

FoxInCloud Stats

FoxInCloud sent this link a while back about their statistics regarding visits to their site: http://foxincloud.com/blog/2017/12/27/VFP-community-lessons-from-foxincloud-site.html What's interesting here is the breakdown of people. Yes, I think it's understandable that the Fox community is getting older. Another factor is the growth of the mobile and web environments taking over development. These environments really do push people towards the newer non-SQL or free SQL/hosted environments but more towards hosted storage options like Amazon and Google. A tool like FoxInCloud that helps MOVE existing applications to the cloud inherently competes with those environments. But FoxInCloud also allows developers to extend their application further by giving them a starting point using Javascript and the basic CSS (such as Bootstrap). If you're not rebuilding your application from scratch, it's certainly a great step forward. FoxPro VFP

5 Great Reasons to attend Virtual FoxFest

What's coming up? Virtual FoxFest is coming up soon (sessions start October 14th). Like last year, the conference is entirely virtual yet includes great breakdown rooms and sessions to add that nice one-on-one feel that you get in person. It's also staggered so you can choose which days you want to attend - October 14th, 20th and 26th. This is great if you can't break away for a consecutive three days. But really, I've gone through the sessions and I see five great sessions that I'm eager to check out. 1. A Decade of Thor (Rick Schummer) Thor has been an extension for Visual FoxPro that many developers swear by, yet many don't know even exists. Visual FoxPro's built-in extensions are great but Jim Nelson's Thor supercharges your IDE. I can't believe it's been ten years - so Rick's session should be able to not just whet your appetite but give you all the reasons you should be using it. 2. VFP C++ compiler.  Last year, we saw DotNetX as well