29 January

Bug Tracking 2: Keep the Focus on the Product

by Maura van der Linden

The basic purpose of creating and tracking bugs is to improve the product you are testing. It seems really simple, almost too simple, but every other purpose bugs and bug tracking is made to serve in addition to this one can dilute this focus and create problems. You have to take each choice to use the bugs and your bug tracking system and consider what that choice can mean to the goal of product improvement.

Here are several real-world examples of situations where a secondary use for the bugs and bug tracking system affected the primary focus:

  • Zero Bug Mandates
    There’s a very common rule on teams I’ve worked on that the product cannot be shipped if there are any open bugs. Obvious and makes sense, no? It does make sense in theory, but what I’ve seen happen in practice is that the team focus shifts from making the right decision for the product to making sure that bug count is at zero and stays there. Good bugs are discarded with inadequate investigation or review. Bugs are closed instead of deferred. Bugs aren’t filed because the tester doesn’t believe they will receive the correct attention and instead they are forgotten or scrawled on a notepad so the tester might file them later-if they remember to do so.

    If your project has a Zero Bug Mandate, you have to make sure you have a process that supports bug retention, addressing the bugs that don’t make the cut for the current release but which do need to be addressed. You also need to encourage people to still file the bugs they find right away. Depending on how you track bugs, you may want to create a new database or release for the next iteration of the project for these bugs. If you are using an agile process, this are part of the product backlog.

  • Bug Metrics as Indicators of Individual Performance
    Sometimes a decision is made to use bug metrics as a way to measure the performance of team members. The typical logic is that bug activity correlates to “work” on the part of the various team members and there should be a way to use these concrete bug metrics to judge how much work any team member is doing relative to other team members. It sounds good, but what it does not take into account is that all bugs are not created equal in terms of severity, work to find, work to fix or effort to find or regress. The effort moves from a focus on finding and solving bugs as expeditiously as possible to finding and fixing a large quantity of bugs, regardless of their quality or the quality of the fix.

    If you are using bug metrics as a measure of individual performance, you need to insure it’s not the main way performance is judged. Bugs should be assessed for root cause and similarity and duplicates or related bugs identified as such to prevent cluttering the bug tracking system and unduly influencing your measurement process. Honestly, bugs are a very poor measurement of performance and I never advocate using them for that purpose for individual team members.

Stay tuned for Bug Report Basics!

25 January

Microsoft’s January 2009 Layoff – My Position Was Eliminated

by Maura van der Linden

As part of the January 22, 2009 layoff at Microsoft, my postition was one of those eliminated. A brief interruption and distraction is unavoidable as I hunt for a new job.

Will manage projects, test or write if you have a position for me!

15 January

Bug Tracking 1: Potential Red Flags

by Maura van der Linden

A discussion on a group I belong to on LinkedIn got me thinking about bug/defect tracking and the different systems I’ve used for it over the years along with the relative successes or failures of these systems.

Tracking bugs is a core need in software development. Bugs can be generated by the development team (all disciplines within the team), by customers, by product support, etc. This insures bugs that are found are followed, even if not fixed, and metrics from bug tracking help determine things like when a version of the product is ready to release, when a QFE or patch may be needed, etc.

For such a simple task, bug tracking can quickly become a complex and sometimes frustrating chore and this is often due to one or more of the following:

  • Tracking More Than Bugs
    This tends to lead to field creep – an explosion in the fields in the bug form – in order to accommodate things like specifications, work items, customer issues, etc.
  • Tracking Every Available Piece of Information
    Without a basic agreement on what information MUST be in a bug report, the tendency is to track every piece of available information so nothing is missed. This is another contributor to the issue of field creep.
  • Lots of Required Fields
    If fields are made “required” when they are not always applicable, people will put junk in them to get past the requirement. At the point this starts to happen, the usefulness of that field’s information disappears. The same thing happens when the list of required fields is large – people will put anything in just to be done with the bug report.
  • Field Overlap
    If there is more than one place to put the same exact same piece of information, users will begin to make mistakes, omit that information, or enter contradictory information.
  • Lack of User Training and Documentation
    Users of the bug system should be trained in the way its implemented and used and what is required of them along with a good idea of the process used to address bugs and the bug lifecycle in use in their company. When the first solution to lack of compliance is to set up the bug tracking system to enforce rules, it can cause worse behavior because the users still don’t know the rules, instead they work at getting around the automatic enforcement.
  • Shared Bug Tracking
    Sharing bug tracking among multiple projects, sometimes throughout a company, can lead to huge bug forms with a lot of redundancy or fields that are meaningless to all but one project.
  • No Central System Control
    Lack of a single person or core v-team that is responsible for managing the bug tracking system can lead to both having no one in charge of fixing issues that inevitably arise, but also to people just randomly adding fields or field contents without a master plan or vision.
  • Off the Shelf Configuration
    Using a bug tracking solution off the shelf, without reviewing it and possibly customizing it for the product, the development process and the needs of the team is akin to the square peg, round hole problem.
  • Bug Tracking not Periodically Reviewed
    Using the same bug tracking solution and setup over and over without reviewing it periodically to insure it’s still applicable and optimized for the needs of the company, team and product can eventually erode its usability and usefulness.

These are the potential red flags from my own list and years of managing bug databases and bug tracking systems. I’m sure there are more and you can leave those in comments.

3 January

Mentoring in Software Testing

by Maura van der Linden

I was reading a blog post on uTest.com by James Whittaker called Thoughts on the Future of Software Testing and one thing that really jumped out at me from it is the issue of mentoring in Software Testing.

I’ve had the benefit of several very good mentors and, in turn, have mentored other testers over the course of my thirteen years in software testing. I know James has put out a call to testers to take mentoring seriously and I’d like to second that but also offer a short list of does and don’ts from my own experience:

  • DON’T wait for someone else to break the ice. If you have another tester your respect and would like to learn from, go ahead and ask if they can spend a little time teaching and mentoring you. If you see a promising tester you think you can help, ask them if they’d like you to mentor them. The worst thing that can happen is that they turn you down.
  • DON’T treat a mentor/mentee relationship as if it were a marriage for life. Even informal or short term mentoring relationships can have considerable value without having to sign up for a long-term or potentially unsustainable relationship.
  • DO set expectations on both sides. Part of a good mentor/mentee relationship is having a clear and upfront agreement on expectations from both mentor and mentee. Lack of this typically leads to disappointment, resentment and a failure of the relationship.
  • DO keep an open mind. If you go into a mentoring relationship, you won’t get much out of it if you aren’t open to other ways of doing things or the learning opportunities you may encounter.
  • DON’T limit yourself to only other testers as mentors or mentees. Learning from other disciplines and teaching test theory and techniques to other disciples can be valuable as well.

Mentoring can be a great help to any professional, and testers in particular.

20 October

Stay away from Grouply

by Maura van der Linden

There’s a site/service called Grouply that is purported to help you manage all your Yahoo groups in a centralized place. Sounds good, right? Especially if you have a lot of group memberships.

But rumors abound about Grouply and how it operates. Some reports are that when you sign up for it, it changes all your membership emails to be username@grouply.com and then any mail in the groups you subscribe to are indexed and resourced by Grouply. This means that anyone else on those groups has, without their permission, had their assumed privacy removed.

Now, everyone should realize that there is no real privacy on the web. None. And the memory of the web is pretty darned infinite. But we’ll ignore the fact that illusions of privacy are just that.

But my most basic issue with Grouply is very simple – it requires you to disclose your Yahoo username and password to a third party service. At that point, you’ve lost all control. You have no idea what, how or when the third party will use that information. You certainly have no control over it.

So the basic security advice applies. Never give out your username and password.

2 February

Skeptical Computing

by Chuck

So Maura just shared with you a recent little adventure we had with a pushbot worm.  I’d like to give you my side, and a bit of insight into the thinking that should be going on in your head when something like this happens.

 It comes down to the practice of what is called Skeptical Computing, or as Maura might say, not misplacing your Trust.  This means questioning when something is sent to you out of the blue and making sure it’s genuine.  It also means trusting your instincts if something seems ‘not right’, and not just blindly clicking links and running files sent to you with relatively ‘reckless abandon’ as some folks seem to do.  It is also the single best way I know of to avoid falling prey to things like Phishing scams, and worms/trojans/viri that spread via a social engineering vector.  In this particular case it’s what saved me from being infected with a nasty little worm that would have tried to spread to everyone in my messenger list, and opened a back door on my computer to persons with almost certain bad intentions.

Lets set the scene.. I’m at work, working on a loadtest script when I get a messenger popup from an ex-coworker who I’ve not chatted with in over a year.   The message is not characteristic of her, but still somewhat believable..  it’s short, asks the question ‘is this you?’ and contains a link..  Okaaay.   Stop. Think.  Does this fit the profile of a ‘social engineering’ attack, ala the infamous ‘I love you’ worm?  Yes it does.  It’s a very short, generic message, designed to appeal to anyone and get them to click that link.   Ratchet suspicion level up one notch.. 

Lets just stop and consider the situation.   You are now faced with the virtual equivalent of a friendly looking cup of unknown stuff that says “drink me”,  set down on the table in front of you by what appears to be someone you know.   So what do you do?   What would you do if this happened in person?  Would you just pick up the cup and drink?

How about if we ask ‘em what’s in the cup?  “What’s in the link“  I send back.  Now, since their IM was JUST sent, it’s reasonable that IF it was sent by my friend, I’ll get a response.  You techies will recognize this as a simple ‘touring test.’  While I wait for a response, let’s have a closer look at the link.  it LOOKS like it’s going to serve me up some kind of image, since part of the server path is \images\  but it’s ALSO got my email address in there as part of a ‘query-string’  presumably used to find the right image.  STOP. THINK.   This isn’t some ‘imageid’ number, it’s my email address.  how would this site have my email address?  Now, my friend has my email address, because it’s part of the messenger contact info, but why would some site hosting images have the image indexed by my email?

My friend has also not responded back. Ratchet the suspicion level up another few notches because not only is the link suspicious, but it’s looking less and less likely that the IM I got was sent by a human.  In our analogy, the person who set the ‘drink me’ cup on the table has not moved or said a word, and not responded to a direct query made to them.   The cup is very enticing, and really almost engineered to pique someone’s curiosity and get them to drink it.  But on closer inspection, the stuff in the cup is looking kinda weird if you ask me.

So, ok,  at this point if I was not a tester by nature, I’d leave well enough alone, and wait for my friend to respond, and if I could not ascertain what’s in the link and that they REALLY sent it to me, I’d not touch it.  But hey that’s not me.  Still, at this point I’m fairly well up on full alert.

Should we click the link and see what happens?  Actually that’s what I did, but in hindsite it’s not what I SHOULD have done. Did I get owned by clicking?  No, but what I did do by clicking the link was send my email address to an unknown server.  So I may have just signed myself up for some spam on an address that was up to this point almost entirely spam free (joy).  So yeah  what I SHOULD have done was copy the link, change the email address, and then fire it off (I actually did this a bit later, but more on that in a moment).

So I click the link.  Now I should mention at this point that my browser security settings are pretty high, popups are blocked, scripts need permission to execute etc..  am I protected 100%?  No, not likely, but chances of getting owned by merely clicking the link are low.  Do I get rewarded with an image? Nope, I get a popup asking me if I want to run or save a file.  What?  Oh my, this is interesting. Let’s look at the details: the filename is long, and of a form like image7.jpg.something, so it’s trying hard to LOOK like an image but the system isn’t treating it like one, and in fact for file type it says ‘ms-dos application’ which means this is an executable file.

An executable file that is trying to look like it’s a harmless image. Riiiight.  Yeah, this can’t be good.

The Cancel Button is your Friend at times like this.  Nope, I don’t want to execute this, I don’t want to save it, I don’t want it anywhere near my machine.

I sent mail to my friend  that stated “I think your box is owned with some kind of messenger worm”  and a few details of the message she supposedly sent me, etc. 

Now the tester in me comes out in full force. I do one of the things I should have done in the first place, I first look at the main root of the site  (which was www.mainmsn.com). Oh ho,  what’s this? It’s a generic default page for a webserver, similar to the kind you see when you first setup a webserver, but have not provided any content.  Interesting. So the main site is pretending that nobody is home, yet there is this \images\ directory that is serving what looks like a worm.  Uh uh,  more and more suspicious all the time

So next I do another thing I should have done in the first place, I take the link, change the email address to ‘nobody@nowhere.com’  and submit it.  Same response, SAME file. So the email address part either has nothing to do with this, or the server is harvesting them and will pass them along later.  (I curse myself for not editing link before I first submitted it. Hopefully I’ve not just made that account into a spam magnet.)

At this point I contact a friend I have who knows folks on the MS anti-virus/anti-malware team and tell ‘em to have their folks check this thing out.  Sure enough, a little later I get confirmation it’s a worm. 

Am I done?  No because there’s more I can do to try and shut this thing down now that I know what it is.  I know that, despite their clout, MS is unlikely to try and get that site shut down. They get enough flak from folks for being big brother, the 800pound bully etc. that they are unlikely to do anything that might come across as that sort of thing. So they won’t contact the site operators or anyone else to get that thing shut down.

But I’m not MS and I don’t work for them, so I don’t have to worry about that.  In fact, I CAN take steps to notify ISP’s etc and get the thing shut down. So first I ping the site and get the IP, then use a who-is tool to find the owner of the IP address range (it turns out to have been the business services division of a major ISP). I contact their security folks and tell them they have a system on an IP address leased from them that is serving up a messenger worm.  I give them details of the site, the IP being used, etc.  Odds are someone had some brand new site setup and it wasn’t secure and it got owned by someone who’s using it to serve this worm.  But I’ll let them worry about contacting their customer, and hopefully cleaning off their compromised server.

Next I track down the registration on the ‘mainmsn’ domain itself. Why?  Because the worm used the site name, not the IP, and unless the name gets invalidated, it can simply be shifted to another IP to keep the worm alive and spreading.  So I track down the folks who sold that domain name, and and let them know it’s being used as part of a messenger worm and that I suspect the registration data is bogus (it’s listed as some guy in cleveland or somesuch, but the contact email name given belongs to a domain in bulgeria or thereabouts). In any event, they might want to revoke the registration and see if they can put some kind of tombstone on the DNS data so the name can’t continue to be used in this way. 

Did my efforts make a difference? Hard to know in the long run, but for right now I can tell you that four days later that domain name no longer resolves to an IP address which means, for now, the worm is no longer able to spread using the link I saw.  There’s still a default page at the IP address that mainmsn was using, but it will no longer try to serve me the worm from the path that was used, so the server might have been cleaned up as well.  On the other hand, the worm itself included some back-door access to affected machines, so the authors could have changed to a different domain name, different path on the server etc.  So yeah, minor victory in shutting down one site/domain name  but I’m sure there will be another varient soon, using a different site and different domain name.   And as long as some folks click the link, and then run the exe it downloads, the worm will continue to spread.

However, because I was skeptical, and didn’t just click the link and run the program it sent me, I won’t be part of spreading the worm.  I won’t have my system owned by some punk, and instead I can actually be a small part of making it harder for their junk to spread around the web.

Did I need to be some computer wiz to spot what was wrong?  No, not really. All that was really needed was a healthy dose of skepticism and the attempt to check back with the sender before opening a link in a very generic and non personal message. Barring that, asking the question “Why does an image file want to RUN on my computer?” can also save you a lot of grief. That sort of thing should be a red flag that has you checking back with the sender to find out what this thing is, and what it’s going to install on your system if you run it.

1 February

Recent MSN Messenger Worm Variant – Worm:Win32/Pushbot.BE

by Maura

Here’s a mantra everyone, especially every tester, should learn and repeat regularly….


Users are warned constantly to never open files from someone they don’t know but they really need to be even more pessimistic than that. A lot more pessimistic. Just because something comes from the account of someone you know or recognize does not mean it’s actually from that person or that they know it’s being sent.

Chuck and I became aware of this variant when he received an instant message from a former co-worker he’d not talked to in many months. The text of the message said “Is this YOU?” and it contained a link that, at first glance, appeared like it might be an image of some sort due to a ViewImage word in the link. When he clicked on it, it wanted to either run or save an MS-DOS application. Needless to say, he didn’t allow it to do either.

Instead we passed the information on to an antivirus group.

Sure enough, it was a new variant of Win32/Pushbot called Win32/Pushbot.BE – at least that’s what one vendor calls it. Naming of viruses and variants, especially if they are not a single type only tends to be more than a little inconsistent across vendors.

Now if Chuck had been a little less suspicious, he’d have likely fallen for that bit of social engineering. It played on people’s desire to see whether someone who knows them found a picture of them and what that picture might be. Because it’s someone they at least know well enough to add to their messenger list. It goes back to a mix of trust and human curiousity.

If he’d allowed the file to run, it would have spread by sending itself to all the members of his friends list as well.

In case you DID fall victim to this worm, you can see how to remove it on Microsoft’s Malware Protection Center entry.

31 January


by Maura

My name is Maura van der Linden and I’ve been a software test engineer for about a decade. I tend to specialize in test theory and security testing and I talked my husband, Chuck, into this blog as a way to tell more than just each other about testing, the trials and tribulations of testing and even some real test content!

We’ll see how it goes!