Posts Tagged ‘testing’

23 May

How to write an effective bug report

One of the most basic skills anyone dealing with software needs, especially testers, is the ability to write an effective bug report. I’m sure when you first see “bug report” you think it only matters for testers but, trust me, this skill is a very necessary skill for just about everyone. Unfortunately, it’s also a skill that is either undeveloped or underdeveloped in probably 90% of the people I deal with, even technical professionals. Sad but true. I place a lot of the blame for this lack of knowledge and skill on the fact that no one ever seems to teach what a good bug report should contain.

After encountering yet more “bug reports” in my bug database at work today that were completely unactionable because they didn’t include even the most basic information I needed to have in order to start investigations, I decided to publish my own little guide to effective bug reports.

Note that there are multiple pseudonyms for “bug” in use around the industry but I’m just keeping it simple here. Use whatever pseudonym makes you happy. Also note that there are many, many ways of tracking bug reports and a lot of them require that the person filing the bug report fill out information in addition to what I have talked about below. This varies so much by system, team, etc. that I can’t cover them all, but the basics listed below will get you 90% of the way to a useful, actionable, clear bug report. I’m not going to address severity, priority, etc. as those are constructs put on top of bug reports in order to further classify them.

The Purpose of a Bug Report

The purpose of a bug report is to give as much information on a bug as necessary to allow the bug report recipient to investigate the issue and help you to resolve it. That’s it, really, but what I consistently see people doing is forgetting that phrase “as much information on a bug as necessary” and just saying something like “I tried this and it didn’t work.” or “This broke.” These are completely useless bug reports as they contain no details that would even allow someone to begin an investigation. In fact, they aren’t even bug reports – they are just complaints.

Note that I don’t say “all information” because there will be information that does not need to be in the bug report. Typically bugs are not caused by things like what material your desk is made of (as an extreme example) so you do have to exercise some filtering.

Basic Bug Information

All bug reports should include the following information, no matter how they are filed (bug database, email, contact form, etc).

Descriptive Title
The title is the first thing the investigator will see, it’s the most visible and usually one of the easiest to search for. This is your prime area for key information and to clearly state what the bug is. The title should be concise but contain any unique information that may get quick action or allow the investigator to relate your bug to other known issues or reports. Consider these two examples of bug titles:

  1. “MyApplication crashes when USB is plugged in.”
  2. “MyApplication fails with error 1234567 on MyOSv1 when MyCamera is plugged in via USB.”

Notice that the second title gives a lot more information right up front? The error code, the camera make, the OS, etc are all immediately present and can be searched for. The title concisely explains what happens, in what environment and when. You do have to work within any character limits imposed but a great title is a huge help.

There is always some argument when I tell people that I include a statement of impact in my basic bug information list but it makes a big difference in getting a bug fixed if the impact it has is clearly understood up front. Impact can be as minor as a bug that causes the screen to flicker briefly or as major as a system crash and data destruction. Think about what impact the failure itself has and then what impact the steps to recover have and clearly state them for the investigator. You may even know impacts that the investigator doesn’t but should know in order to appropriately classify and prioritize the bug.

I write impact statements like:

Impact: Unable to use MyCamera’s USB download with its OEM supplied software with MyApplication running. I have to manually quit MyApplication before plugging in MyCamera. Recovering from the MyApplication crash requires a system reboot.

Clear Repro Steps
Repro steps are what the investigator will use to try to reproduce your bug for investigation and that means they need to be correct, complete and sequential. I strongly favor these are a numbered list of instructions similar to those shown below:

Repro Steps:

  1. Start MyApplication
  2. Plug USB cable into system and MyCamera usb ports
  3. Turn camera power on.

The repro steps should be free from anything that doesn’t really pertain to the bug so, for example, unless the type of user mattered, I wouldn’t bother saying which user and what security level that user has. In software testing, a goal is always to reduce the repro steps down to the minimum needed to cause the bug to occur. If there is more than one way to get a bug to repro, then I typically use the steps for the shortest or most basic repro scenario. If you have to have a particular state before the bug will occur, that goes at the top of the Repro Steps. So you may start with something like “System needs to be freshly rebooted.” or similar.

Expected Result
Including an expected result in your bug report will help the investigator to determine whether your expectations are accurate. You can be expecting something to happen that won’t because it’s not included in the design or you could be correct in your expectations. Either way, a clear statement of what you expect should always be included. To expand on the sample I started with, I’d include an expected result of:

Expected Result: MyCamera software will autolaunch and I will be asked if I want to download the pictures off MyCamera.

Actual Result
This is where you state what actually happened in as much detail as you can. Just like in the title, this is not the place for “it crashed” but rather a place to tell the investigator whatever you can about what happened when the bug occurred.

Actual Result: MyApplication crashed with a Windows error 1234567. MyCamera software never launched and MyCamera was listed as an unrecognized device.

Each part of this result can give the investigator a clue as to what happened, so they are all worth including. Even if what happens is as seemingly simple as “no device was recognized” or “software did not autolaunch”, you should always have this Actual Result section.

Environment Information
This is an area for you to list the specifics of the environment in which the bug occurred. How much detail you go into depends on what you are reporting a bug on, but some of the common ones are:


  • Operating System and Version
  • Processor Architecture
  • Affected Product Versions and Build #s
  • Browser and Version

If your bug has nothing to do with a browser, you probably don’t need to list the browser. You may need to add in specific information that matter to your particular bug like hardware versions, drivers, etc.

The last section I always have as part of my Basic Bug Information is a section for notes. This is the place to list whatever solutions you’ve tried and what happened, a place to list if the bug occurs when a different repro step set is carried out. You can list any guesses as to what might be causing it to help out the investigator. If you have any log files, screenshots, etc, you should indicate so here and attach them if you have a way to do so. If there is no way to attach them, indicate you have them available if needed. If you have discovered your bug only occurs if you do things one way but not a very similar but different way, indicate that here. Try to make it as easy to read and understand as possible and break it into coherent paragraphs.


Writing an effective bug report is a skill everyone should develop to some extent but it’s critical for software professionals.

Below is a quick snapshot of my Basic Bug Information list that I’ve used in software testing for more than a decade and still use as a technical writer. I’m formatting it as code so you should be able to copy and paste it easily.



Repro Steps:

Expected Result:

Actual Result:



25 June


(Cross-posted from my personal blog as it’s germaine to testing.)

Last week, in the course of commenting on James Bach’s blog, I explained that I liked to learn by laying my beliefs and understandings in front of others for them to question things I may not have questioned and, in turn, to question them to see if knowledge or views they have fill in something for me. James responded that this was called “transpection.”

It has a name? Who knew?

Transpection basically means to learn by putting yourself in place of another (to quote James’ blog post on transpection). This is different from introspection (contemplation of one’s own thoughts and feelings) and extrospection (examination of things outside oneself). It’s also different than the often-referred to “put yourself in someone else’s shoes” because you need to put yourself in someone else’s brain, really. It’s less role-playing where you imagine what that person might be thinking (which I think would be still be highly subject to your own views) and more based on really asking questions of someone else about a subject you are interested in.

Honestly, this truly fascinated me. I was not only interested in the fact there was a word for this process but the fact there was spurred me into reading and thinking about the process of transpection. As with many things that catch my interest, I mentioned it a few times and James kindly pointed me at a few other definitions of it.

Then James invited me to join him on skype to have him run me through a transpection session with him. This was a really interesting experience – not terribly comfortable in some ways and yet quite eye-opening in others. Now, mind you, this is coming from a person who has experience with James’ personality and does understand what he’s trying to do in advance. I got frustrated once or twice and felt obtuse several times but it was really fascinating to experience it and, at the end, hear what James’ response to the questions he asked me were.

I’ve read James’ list of problems with transpection on his blog and thought I’d give you my views:

  • Feeling interrogated or tortured – I did feel a bit interrogated but wasn’t that part of the point? On the other hand, because I can see what the goal was, I knew repetition of a question/set of questions meant either that I had an interesting view or I really needed to look at my answers. Tortured, no – other than when I could NOT, even to myself, validate a stance I had taken. Time for that stance to go.
  • Being judged – We all judge each other all the time. I expected it but, in return, I was judging my reactions and James’ approach too. It was definitely a two-way street.
  • Being treated like a lab rat – Yep. But I had a psych minor and undergrads are the lab rats of the psych world. So it was familiar ground and I don’t feel demeaned by it at all. I learned and James learned and that’s the point.

What did I learn?

  • I have fallen a bit into a trap of reliance on metrics and my own fixed idea of what a test/test case consist of. This is more limiting than I would have guessed, actually. This is probably rooted, at least in part, in having done test automation recently.
  • My particular order of examination seems to put visual appearance lower on my list than many people would have it. I need to consciously push it further up my list to compensate. This is almost certainly rooted in the fact I have no real visual memory so it can be quite difficult for me to deal with visual appearance and it’s never my default behavior.

I saved our skype session so I can examine it a few more times to see what else I learn from it. I will note that I did experience a bit of James’ excitement when he hears something interesting. In this case it was my mentioning the fact I had no visual memory. The questions were rapid and many but I didn’t interpret it as sharp or angry – just very curious and wanting to gather information.

I will point out that this type of questioning does require that the need to “be right” or “win” be analyzed as well. I suggest that if you go into a transpection session to prove you are right, you will limit the benefits you can get from it. I also think this desire could be at the root of people feeling attacked or offended. Being right is not the point. Learning is the point and, to learn, you need to be willing to examine what you think you know and your views and reassess them in light of new information. If I went into this session with something to prove, I would not have gotten out of it what I did.

After the skype session with James, I ended up thinking more about my lack of visual memory and ended up trying a somewhat clumsy version of it on my husband, Chuck. I asked him questions about how he finds things around the house and remembers where they are. It sounds simple but it can be cause for friction. I thought it might be interesting because he, unlike myself, is highly visual.

  • We both start by looking for an object in its designated “home” if it has one.
  • If it is not in its home or has no home, Chuck pictures where he last saw it and looks for it there.
  • If it is not in its home or has no home, I remember where I last put it and look for it there.
  • If that fails, we both try nearby areas to the one we expected to find the object in.
  • If that fails, we both resort to trying to figure out where the other person would have decided to put in instead and then check there.

It’s a very simple example but it pointed out something interesting to us. It answers the question of why if I cannot find something, Chuck usually can and vice versa. It also points out a symptom of my lack of visual memory. I cannot picture where I saw it. Instead I rely on where it SHOULD be or where *I* put it.

Since then I’ve done a lot of thinking about how I cope with my weird brain wiring and compensate for it, when necessary. I’ll have to write a separate post on that.

22 March

CHM Text Searches – Lesson Learned

The current project I am working on involves working on the CHM (Microsoft Help Files) for Windows Embedded Standard 7. This was a project underway before I joined the team and it had even changed managers during that time, which always creates at least a bit of chaos.

We just dropped our CHM files to the product team for integration last week and then discovered we had not completely purged the contents of the project’s internal name and prior release same in favor of the final release name. This happened despite having myself and four other people all search the Help for the string. It even happened despite a tool that would automatically check for the strings you tell it to at build time.

After dealing with the aftermath and rebuild/redrop (my boss was out of town), I looked into why this had happened and discovered I had let my tester mindset go to sleep and had not really thought in depth about how the searches were being done and the potential holes in these approaches.

The stage is set by:

  • The tool was checking for the term in the files at build time.
  • The individual searches done ad-hoc were using the search functionality provided in the Help by Windows when reading the CHM file.

The following discoveries were made as we looked into where this failed:

  • The tool does not check the index. Several instances of terms entered into the tool were found in the index.
  • Not all “bad terms” were entered into the tool’s database.
  • The Search functionality in Help does not search the index and no one thought to look there because the index was generated from files thought to be clean.
  • The Search functionality in Help is a regex-base search. All ad hoc searches were done with the same string, but no one thought to look for substrings with use of a wildcard before, after, or both in relation to the “bad string.” This meant no instances where the “bad string” was a substring could be found, though quite a few existed.

It seems obvious now, of course, but it really does help to look at these occurrences and see what the root cause was so that it can be caught in the future. In this case, we took away the following actions for the next release:

  • All terms must be entered in the tool’s database.
  • The CHM must be decompiled and the HTML contents searched for the “bad terms”. This will catch substrings and index occurrences.
  • Ad hoc searching must be done with wildcards and regex to also look for string and substring occurences.

Lesson learned and I paid the price. Both in terms of this instance’s particular issues and in terms of forgetting to look for holes in the process/testing – even if it’s not my job.

30 January

Bug Tracking 3: Minimal Bug Report Information

Bug reports vary greatly in length and complexity depending on your product, corporate culture and needs but there are some fields that I believe are a must for any bug report. A bare minimum, if you will. This happens to be my own list and other testers will have different fields they consider mandatory but I tend toward a bit of minimalism.

A clear and concise title helps everyone. I make it a practice of putting unique information in the title like error codes or exe names. This makes it easier to later search the bug tracking system for duplicates or regressions rather than file a new bug.

A summary of the environment the bug occurs in as it relates to what you are testing. It could be as simple as the operating system or as complex as the os, browser, version of flash, etc.

How important do you think this bug is to fix? This should be important to the overall product and it may be higher priority if it’s blocking other testing or progress.

Product or Feature Area
What product (in a shared tracking system) or what feature area this bug occurred in. This serves to help when deciding who to assign bugs to or what areas of the product need concentrated attention.

Assigned To
This is the current owner of this bug and who is responsible for the next action with it.

Repro Steps
These are the steps, in order, required to reproduce this bug. They should be complete enough that someone unfamiliar with that bug can accurately reproduce it. I recommend a numbered list. If an additional file or script is needed to reproduce the bug, the location of that should be noted here or the file or script attached if the bug tracking system supports it.

This is a clear statement of what the issue is and why this is a bug. This is the place to make a case for this bug to be fixed so customer impact as well as expected behavior should be listed. If the bug is not high priority, that should be listed here as well. This is not a place to try to game the system and artificially cry wolf on bugs that really don’t deserve it.

Opened By
Who opened the bug. This allows questioning of the person, if necessary.

Opened Date
Date the bug was opened. This is useful for determining bug age.

Opened Version
This is the version of the product the bug is being reported for. It tracks what version was tested, no matter the date the bug was filed.

Fixed By
Who fixed the bug.

Fixed Date
Date the bug was fixed.

Fixed Version
Version of the product the fix will appear in. This allows the person verifying the fix to know when to expect it to be fixed instead of guessing.

Closed By
Who verified the bug was fixed.

Closed Date
Date the bug fix was verified.

29 January

Bug Tracking 2: Keep the Focus on the Product

The basic purpose of creating and tracking bugs is to improve the product you are testing. It seems really simple, almost too simple, but every other purpose bugs and bug tracking is made to serve in addition to this one can dilute this focus and create problems. You have to take each choice to use the bugs and your bug tracking system and consider what that choice can mean to the goal of product improvement.

Here are several real-world examples of situations where a secondary use for the bugs and bug tracking system affected the primary focus:

  • Zero Bug Mandates
    There’s a very common rule on teams I’ve worked on that the product cannot be shipped if there are any open bugs. Obvious and makes sense, no? It does make sense in theory, but what I’ve seen happen in practice is that the team focus shifts from making the right decision for the product to making sure that bug count is at zero and stays there. Good bugs are discarded with inadequate investigation or review. Bugs are closed instead of deferred. Bugs aren’t filed because the tester doesn’t believe they will receive the correct attention and instead they are forgotten or scrawled on a notepad so the tester might file them later-if they remember to do so.

    If your project has a Zero Bug Mandate, you have to make sure you have a process that supports bug retention, addressing the bugs that don’t make the cut for the current release but which do need to be addressed. You also need to encourage people to still file the bugs they find right away. Depending on how you track bugs, you may want to create a new database or release for the next iteration of the project for these bugs. If you are using an agile process, this are part of the product backlog.

  • Bug Metrics as Indicators of Individual Performance
    Sometimes a decision is made to use bug metrics as a way to measure the performance of team members. The typical logic is that bug activity correlates to “work” on the part of the various team members and there should be a way to use these concrete bug metrics to judge how much work any team member is doing relative to other team members. It sounds good, but what it does not take into account is that all bugs are not created equal in terms of severity, work to find, work to fix or effort to find or regress. The effort moves from a focus on finding and solving bugs as expeditiously as possible to finding and fixing a large quantity of bugs, regardless of their quality or the quality of the fix.

    If you are using bug metrics as a measure of individual performance, you need to insure it’s not the main way performance is judged. Bugs should be assessed for root cause and similarity and duplicates or related bugs identified as such to prevent cluttering the bug tracking system and unduly influencing your measurement process. Honestly, bugs are a very poor measurement of performance and I never advocate using them for that purpose for individual team members.

Stay tuned for Bug Report Basics!

3 January

Mentoring in Software Testing

I was reading a blog post on by James Whittaker called Thoughts on the Future of Software Testing and one thing that really jumped out at me from it is the issue of mentoring in Software Testing.

I’ve had the benefit of several very good mentors and, in turn, have mentored other testers over the course of my thirteen years in software testing. I know James has put out a call to testers to take mentoring seriously and I’d like to second that but also offer a short list of does and don’ts from my own experience:

  • DON’T wait for someone else to break the ice. If you have another tester your respect and would like to learn from, go ahead and ask if they can spend a little time teaching and mentoring you. If you see a promising tester you think you can help, ask them if they’d like you to mentor them. The worst thing that can happen is that they turn you down.
  • DON’T treat a mentor/mentee relationship as if it were a marriage for life. Even informal or short term mentoring relationships can have considerable value without having to sign up for a long-term or potentially unsustainable relationship.
  • DO set expectations on both sides. Part of a good mentor/mentee relationship is having a clear and upfront agreement on expectations from both mentor and mentee. Lack of this typically leads to disappointment, resentment and a failure of the relationship.
  • DO keep an open mind. If you go into a mentoring relationship, you won’t get much out of it if you aren’t open to other ways of doing things or the learning opportunities you may encounter.
  • DON’T limit yourself to only other testers as mentors or mentees. Learning from other disciplines and teaching test theory and techniques to other disciples can be valuable as well.

Mentoring can be a great help to any professional, and testers in particular.