Posts Tagged ‘Maura’

23 May

How to write an effective bug report

One of the most basic skills anyone dealing with software needs, especially testers, is the ability to write an effective bug report. I’m sure when you first see “bug report” you think it only matters for testers but, trust me, this skill is a very necessary skill for just about everyone. Unfortunately, it’s also a skill that is either undeveloped or underdeveloped in probably 90% of the people I deal with, even technical professionals. Sad but true. I place a lot of the blame for this lack of knowledge and skill on the fact that no one ever seems to teach what a good bug report should contain.

After encountering yet more “bug reports” in my bug database at work today that were completely unactionable because they didn’t include even the most basic information I needed to have in order to start investigations, I decided to publish my own little guide to effective bug reports.

Note that there are multiple pseudonyms for “bug” in use around the industry but I’m just keeping it simple here. Use whatever pseudonym makes you happy. Also note that there are many, many ways of tracking bug reports and a lot of them require that the person filing the bug report fill out information in addition to what I have talked about below. This varies so much by system, team, etc. that I can’t cover them all, but the basics listed below will get you 90% of the way to a useful, actionable, clear bug report. I’m not going to address severity, priority, etc. as those are constructs put on top of bug reports in order to further classify them.

The Purpose of a Bug Report

The purpose of a bug report is to give as much information on a bug as necessary to allow the bug report recipient to investigate the issue and help you to resolve it. That’s it, really, but what I consistently see people doing is forgetting that phrase “as much information on a bug as necessary” and just saying something like “I tried this and it didn’t work.” or “This broke.” These are completely useless bug reports as they contain no details that would even allow someone to begin an investigation. In fact, they aren’t even bug reports – they are just complaints.

Note that I don’t say “all information” because there will be information that does not need to be in the bug report. Typically bugs are not caused by things like what material your desk is made of (as an extreme example) so you do have to exercise some filtering.

Basic Bug Information

All bug reports should include the following information, no matter how they are filed (bug database, email, contact form, etc).

Descriptive Title
The title is the first thing the investigator will see, it’s the most visible and usually one of the easiest to search for. This is your prime area for key information and to clearly state what the bug is. The title should be concise but contain any unique information that may get quick action or allow the investigator to relate your bug to other known issues or reports. Consider these two examples of bug titles:

  1. “MyApplication crashes when USB is plugged in.”
  2. “MyApplication fails with error 1234567 on MyOSv1 when MyCamera is plugged in via USB.”

Notice that the second title gives a lot more information right up front? The error code, the camera make, the OS, etc are all immediately present and can be searched for. The title concisely explains what happens, in what environment and when. You do have to work within any character limits imposed but a great title is a huge help.

Impact
There is always some argument when I tell people that I include a statement of impact in my basic bug information list but it makes a big difference in getting a bug fixed if the impact it has is clearly understood up front. Impact can be as minor as a bug that causes the screen to flicker briefly or as major as a system crash and data destruction. Think about what impact the failure itself has and then what impact the steps to recover have and clearly state them for the investigator. You may even know impacts that the investigator doesn’t but should know in order to appropriately classify and prioritize the bug.

I write impact statements like:

Impact: Unable to use MyCamera’s USB download with its OEM supplied software with MyApplication running. I have to manually quit MyApplication before plugging in MyCamera. Recovering from the MyApplication crash requires a system reboot.

Clear Repro Steps
Repro steps are what the investigator will use to try to reproduce your bug for investigation and that means they need to be correct, complete and sequential. I strongly favor these are a numbered list of instructions similar to those shown below:

Repro Steps:

  1. Start MyApplication
  2. Plug USB cable into system and MyCamera usb ports
  3. Turn camera power on.

The repro steps should be free from anything that doesn’t really pertain to the bug so, for example, unless the type of user mattered, I wouldn’t bother saying which user and what security level that user has. In software testing, a goal is always to reduce the repro steps down to the minimum needed to cause the bug to occur. If there is more than one way to get a bug to repro, then I typically use the steps for the shortest or most basic repro scenario. If you have to have a particular state before the bug will occur, that goes at the top of the Repro Steps. So you may start with something like “System needs to be freshly rebooted.” or similar.

Expected Result
Including an expected result in your bug report will help the investigator to determine whether your expectations are accurate. You can be expecting something to happen that won’t because it’s not included in the design or you could be correct in your expectations. Either way, a clear statement of what you expect should always be included. To expand on the sample I started with, I’d include an expected result of:

Expected Result: MyCamera software will autolaunch and I will be asked if I want to download the pictures off MyCamera.

Actual Result
This is where you state what actually happened in as much detail as you can. Just like in the title, this is not the place for “it crashed” but rather a place to tell the investigator whatever you can about what happened when the bug occurred.

Actual Result: MyApplication crashed with a Windows error 1234567. MyCamera software never launched and MyCamera was listed as an unrecognized device.

Each part of this result can give the investigator a clue as to what happened, so they are all worth including. Even if what happens is as seemingly simple as “no device was recognized” or “software did not autolaunch”, you should always have this Actual Result section.

Environment Information
This is an area for you to list the specifics of the environment in which the bug occurred. How much detail you go into depends on what you are reporting a bug on, but some of the common ones are:

Environment:

  • Operating System and Version
  • Processor Architecture
  • Affected Product Versions and Build #s
  • Browser and Version

If your bug has nothing to do with a browser, you probably don’t need to list the browser. You may need to add in specific information that matter to your particular bug like hardware versions, drivers, etc.

Notes
The last section I always have as part of my Basic Bug Information is a section for notes. This is the place to list whatever solutions you’ve tried and what happened, a place to list if the bug occurs when a different repro step set is carried out. You can list any guesses as to what might be causing it to help out the investigator. If you have any log files, screenshots, etc, you should indicate so here and attach them if you have a way to do so. If there is no way to attach them, indicate you have them available if needed. If you have discovered your bug only occurs if you do things one way but not a very similar but different way, indicate that here. Try to make it as easy to read and understand as possible and break it into coherent paragraphs.

Summary

Writing an effective bug report is a skill everyone should develop to some extent but it’s critical for software professionals.

Below is a quick snapshot of my Basic Bug Information list that I’ve used in software testing for more than a decade and still use as a technical writer. I’m formatting it as code so you should be able to copy and paste it easily.


Title:

Impact:

Repro Steps:
1.
2.
3.

Expected Result:

Actual Result:

Environment:

Notes:

22 March

CHM Text Searches – Lesson Learned

The current project I am working on involves working on the CHM (Microsoft Help Files) for Windows Embedded Standard 7. This was a project underway before I joined the team and it had even changed managers during that time, which always creates at least a bit of chaos.

We just dropped our CHM files to the product team for integration last week and then discovered we had not completely purged the contents of the project’s internal name and prior release same in favor of the final release name. This happened despite having myself and four other people all search the Help for the string. It even happened despite a tool that would automatically check for the strings you tell it to at build time.

After dealing with the aftermath and rebuild/redrop (my boss was out of town), I looked into why this had happened and discovered I had let my tester mindset go to sleep and had not really thought in depth about how the searches were being done and the potential holes in these approaches.

The stage is set by:

  • The tool was checking for the term in the files at build time.
  • The individual searches done ad-hoc were using the search functionality provided in the Help by Windows when reading the CHM file.

The following discoveries were made as we looked into where this failed:

  • The tool does not check the index. Several instances of terms entered into the tool were found in the index.
  • Not all “bad terms” were entered into the tool’s database.
  • The Search functionality in Help does not search the index and no one thought to look there because the index was generated from files thought to be clean.
  • The Search functionality in Help is a regex-base search. All ad hoc searches were done with the same string, but no one thought to look for substrings with use of a wildcard before, after, or both in relation to the “bad string.” This meant no instances where the “bad string” was a substring could be found, though quite a few existed.

It seems obvious now, of course, but it really does help to look at these occurrences and see what the root cause was so that it can be caught in the future. In this case, we took away the following actions for the next release:

  • All terms must be entered in the tool’s database.
  • The CHM must be decompiled and the HTML contents searched for the “bad terms”. This will catch substrings and index occurrences.
  • Ad hoc searching must be done with wildcards and regex to also look for string and substring occurences.

Lesson learned and I paid the price. Both in terms of this instance’s particular issues and in terms of forgetting to look for holes in the process/testing – even if it’s not my job.

15 January

Bug Tracking 1: Potential Red Flags

A discussion on a group I belong to on LinkedIn got me thinking about bug/defect tracking and the different systems I’ve used for it over the years along with the relative successes or failures of these systems.

Tracking bugs is a core need in software development. Bugs can be generated by the development team (all disciplines within the team), by customers, by product support, etc. This insures bugs that are found are followed, even if not fixed, and metrics from bug tracking help determine things like when a version of the product is ready to release, when a QFE or patch may be needed, etc.

For such a simple task, bug tracking can quickly become a complex and sometimes frustrating chore and this is often due to one or more of the following:

  • Tracking More Than Bugs
    This tends to lead to field creep – an explosion in the fields in the bug form – in order to accommodate things like specifications, work items, customer issues, etc.
  • Tracking Every Available Piece of Information
    Without a basic agreement on what information MUST be in a bug report, the tendency is to track every piece of available information so nothing is missed. This is another contributor to the issue of field creep.
  • Lots of Required Fields
    If fields are made “required” when they are not always applicable, people will put junk in them to get past the requirement. At the point this starts to happen, the usefulness of that field’s information disappears. The same thing happens when the list of required fields is large – people will put anything in just to be done with the bug report.
  • Field Overlap
    If there is more than one place to put the same exact same piece of information, users will begin to make mistakes, omit that information, or enter contradictory information.
  • Lack of User Training and Documentation
    Users of the bug system should be trained in the way its implemented and used and what is required of them along with a good idea of the process used to address bugs and the bug lifecycle in use in their company. When the first solution to lack of compliance is to set up the bug tracking system to enforce rules, it can cause worse behavior because the users still don’t know the rules, instead they work at getting around the automatic enforcement.
  • Shared Bug Tracking
    Sharing bug tracking among multiple projects, sometimes throughout a company, can lead to huge bug forms with a lot of redundancy or fields that are meaningless to all but one project.
  • No Central System Control
    Lack of a single person or core v-team that is responsible for managing the bug tracking system can lead to both having no one in charge of fixing issues that inevitably arise, but also to people just randomly adding fields or field contents without a master plan or vision.
  • Off the Shelf Configuration
    Using a bug tracking solution off the shelf, without reviewing it and possibly customizing it for the product, the development process and the needs of the team is akin to the square peg, round hole problem.
  • Bug Tracking not Periodically Reviewed
    Using the same bug tracking solution and setup over and over without reviewing it periodically to insure it’s still applicable and optimized for the needs of the company, team and product can eventually erode its usability and usefulness.

These are the potential red flags from my own list and years of managing bug databases and bug tracking systems. I’m sure there are more and you can leave those in comments.

31 January

Maura

My name is Maura van der Linden and I’ve been a software test engineer for about a decade. I tend to specialize in test theory and security testing and I talked my husband, Chuck, into this blog as a way to tell more than just each other about testing, the trials and tribulations of testing and even some real test content!

We’ll see how it goes!