Resolved To Test

As both an author and a consumer of technical documentation as well as having some background in instructional design, I have some strong opinions about the role of documentation for a product. Over a decade in testing has also had an effect on how I view that documentation.

The most basic documentation is the help file that ships with the product. In Microsoft products, this is called a .chm (”chum”) file. Yes, I’ve probably thought of my share of bad jokes about that slang, too. I find myself currently involved in cleaning up the previously written .chm file for a new product as well as authoring new content for it and there have been some challenges.

Out of these challenges have come the start of my own personal list of DOs and DON’Ts for technical documentation.

• DO know your consumer and know HOW they use the product to do their jobs.
• DON’T sacrifice your reference sections in favor of only user story or scenario-based content.
• DO use a more conversational voice to make the documentation easier to consume.
• DO review customer feedback for opportunities to update your documentation to cover problem areas when appropriate.
• DO produce other documentation besides just the .chm help files.
• DON’T use the excuse of having other documentation to short-change the help files.
• DO edit for professional voice and errors.
• DON’T over-edit and lose technical accuracy in favor of prettiness of language.
• DO offer tips and make sure gotchas are included.
• DO make content easily findable and clearly identifiable by the consumers.
• DO have your content thoroughly reviewed by the product team.
• DON’T let the process of creating documentation get in the way of doing the right thing for the customer.

My background in shipping products has given me some valuable insight and gives me a solid base on which to work with the product team. I understand what they are talking about and don’t need a lot of hand-holding. They know I take deadlines seriously and am honest with them about where I am on my deliverables.

But I think documentation teams also really need to be able to turn on a dime. Things can change rapidly and you have to be able to roll with those changes.

Processes are required, both within the documentation team and where the documentation team has to interact with the product development team, but you really have to watch for signs that the process may be causing more overhead and pain that it’s solving. When a five minute task takes three days, an email thread of ten or more emails plus several requests being filed…you may have a problem.

You have to keep your eyes on the prize. The prize is excellent documentation to assist your consumers – not how well processes were followed or how many you have. No one will judge your documentation by how many people worked on it, how many processes were followed, who did what tasks or what tools were used to produce it. All consumers care about is how well it meets their needs.

It can be useful to view the RDoc information for the various gems you may have installed on your local system. RubyGems provides a way to do this by starting up a local RDoc server.

Local Gem Server
To start the server, open a command window on the machine that has RubyGems installed and type:

gem server

Once the you get a message that the server is starting, you can minimize this command window (don’t close it or the server will stop).

Using your browser of choice, navigate to http://localhost:8808.

When you open this page, you will see a list of all the gems you have installed on the local system and each one will have an WWW link that will open the project’s homepage as well as an RDoc link that will open the RDoc information for that particular gem. At the top of the page are three columns: Files, Classes and Methods. I tend to find Classes and Methods the most useful when trying to discover usage and syntax for a gem.

If you click on methods displayed in the RDocs, an small pop-up will open to show you the code relating to the method. This can be even more useful if you are trying to figure out how to use a method.

Missing RDoc Information
There might be a situation where you have a gem installed that does not have RDoc information but you can force RubyGems to generate RDoc information for already installed gems.

gem rdoc --all

This will generate the RDoc information for all installed gems. You can also generate RDoc for select gems only by using:

gem rdoc GEMNAME

Ruby Gems can be a little interesting to manage, especially to someone as new to it as I am. In the course of my own self-education, I’ve compiled a little list of what I consider the basic commands. Hopefully this will be of some help to others as well.

Gem Sources

To see which gem servers or repositories your installation of RubyGems is using, open a command prompt and type:

gem sources

The system will return with a list of the repositories and caches Ruby is using to look for gems. In the case of my local system, I got back:

C:\>gem sources
*** CURRENT SOURCES ***


http://gems.rubyforge.org/

Which shows that my system is only looking at RubyForge for its gems right now. This can be useful to know if you get errors that gems cannot be found.

Installing Gems

To install a new gem, you can open a command prompt and type:

gem install gemname

This will install the latest version of the gem gemname from the repository(ies) your local system knows about.

If you have a local copy of the gem and you need to install that particular one, you can open a command prompt, navigate to the directory that contains the local copy and type:

gem install gemname --local

If you need only a particular version of a gem that might not be the latest version you’d get by default, you can open a command prompt and type:

gem install gemname --version #.#.#

This will install version #.#.# of gem gemname.

Uninstalling Gems

Uninstalling a gem can be done by opening a command prompt and typing:

gem uninstall gemname

This will uninstall the gem gemname. If you have more than one version of that gem installed, RubyGems will display a numbered list of the versions and you can enter the number of the version you want to uninstall or choose to uninstall all versions of gem gemname.

Updating Gems

You can update installed gems by opening a command prompt and typing:

gem update gemname

This will update the gem gemname to the latest version in the gem repository RubyGems is pointing to.

Updating the RubyGems System

Sometimes you will need to update the actual RubyGems management system. To do this, open a command prompt and type:

gem update --system

If you are having problems updating your gems or using them, it usually won’t hurt to try to do this update.

Check Gem Dependencies

If you need to know what the dependencies of an installed gem are, you can open a command prompt and type:

gem dependency gemname

This will list the dependencies of the gem gemname.

Gem Help

There is a good amount of help embedded in the RubyGems system. Often you can answer your own questions on how to manipulate your gems using this help. To access the list of commands, open a command prompt and type:

gem help commands

To get help on an individual command, open a command prompt and type:

gem help command

This has been an interesting week for me. I am immersing myself in Watir (Web Applications Testing in Ruby), an open-source library for automating web browsers that works with Ruby, an open-source, object-oriented programming language I’m starting to really like it!

This is a great boon to those of us who have to write automation quickly that need to also be maintainable and not cost an arm and a leg to buy tools for. While some tools do exist, they tend to be expensive or brittle – or both.

The one thing I really am discovering is that there is a mixed bag of documentation and a lot of room for better documentation and training materials. Maybe it’s a place I can turn my writing skills.

Bug reports vary greatly in length and complexity depending on your product, corporate culture and needs but there are some fields that I believe are a must for any bug report. A bare minimum, if you will. This happens to be my own list and other testers will have different fields they consider mandatory but I tend toward a bit of minimalism.

Title
A clear and concise title helps everyone. I make it a practice of putting unique information in the title like error codes or exe names. This makes it easier to later search the bug tracking system for duplicates or regressions rather than file a new bug.

Environment
A summary of the environment the bug occurs in as it relates to what you are testing. It could be as simple as the operating system or as complex as the os, browser, version of flash, etc.

Priority
How important do you think this bug is to fix? This should be important to the overall product and it may be higher priority if it’s blocking other testing or progress.

Product or Feature Area
What product (in a shared tracking system) or what feature area this bug occurred in. This serves to help when deciding who to assign bugs to or what areas of the product need concentrated attention.

Assigned To
This is the current owner of this bug and who is responsible for the next action with it.

Repro Steps
These are the steps, in order, required to reproduce this bug. They should be complete enough that someone unfamiliar with that bug can accurately reproduce it. I recommend a numbered list. If an additional file or script is needed to reproduce the bug, the location of that should be noted here or the file or script attached if the bug tracking system supports it.

Summary
This is a clear statement of what the issue is and why this is a bug. This is the place to make a case for this bug to be fixed so customer impact as well as expected behavior should be listed. If the bug is not high priority, that should be listed here as well. This is not a place to try to game the system and artificially cry wolf on bugs that really don’t deserve it.

Opened By
Who opened the bug. This allows questioning of the person, if necessary.

Opened Date
Date the bug was opened. This is useful for determining bug age.

Opened Version
This is the version of the product the bug is being reported for. It tracks what version was tested, no matter the date the bug was filed.

Fixed By
Who fixed the bug.

Fixed Date
Date the bug was fixed.

Fixed Version
Version of the product the fix will appear in. This allows the person verifying the fix to know when to expect it to be fixed instead of guessing.

Closed By
Who verified the bug was fixed.

Closed Date
Date the bug fix was verified.

The basic purpose of creating and tracking bugs is to improve the product you are testing. It seems really simple, almost too simple, but every other purpose bugs and bug tracking is made to serve in addition to this one can dilute this focus and create problems. You have to take each choice to use the bugs and your bug tracking system and consider what that choice can mean to the goal of product improvement.

Here are several real-world examples of situations where a secondary use for the bugs and bug tracking system affected the primary focus:

• Zero Bug Mandates
  There’s a very common rule on teams I’ve worked on that the product cannot be shipped if there are any open bugs. Obvious and makes sense, no? It does make sense in theory, but what I’ve seen happen in practice is that the team focus shifts from making the right decision for the product to making sure that bug count is at zero and stays there. Good bugs are discarded with inadequate investigation or review. Bugs are closed instead of deferred. Bugs aren’t filed because the tester doesn’t believe they will receive the correct attention and instead they are forgotten or scrawled on a notepad so the tester might file them later-if they remember to do so.

  If your project has a Zero Bug Mandate, you have to make sure you have a process that supports bug retention, addressing the bugs that don’t make the cut for the current release but which do need to be addressed. You also need to encourage people to still file the bugs they find right away. Depending on how you track bugs, you may want to create a new database or release for the next iteration of the project for these bugs. If you are using an agile process, this are part of the product backlog.

• Bug Metrics as Indicators of Individual Performance
  Sometimes a decision is made to use bug metrics as a way to measure the performance of team members. The typical logic is that bug activity correlates to “work” on the part of the various team members and there should be a way to use these concrete bug metrics to judge how much work any team member is doing relative to other team members. It sounds good, but what it does not take into account is that all bugs are not created equal in terms of severity, work to find, work to fix or effort to find or regress. The effort moves from a focus on finding and solving bugs as expeditiously as possible to finding and fixing a large quantity of bugs, regardless of their quality or the quality of the fix.

  If you are using bug metrics as a measure of individual performance, you need to insure it’s not the main way performance is judged. Bugs should be assessed for root cause and similarity and duplicates or related bugs identified as such to prevent cluttering the bug tracking system and unduly influencing your measurement process. Honestly, bugs are a very poor measurement of performance and I never advocate using them for that purpose for individual team members.

Stay tuned for Bug Report Basics!

As part of the January 22, 2009 layoff at Microsoft, my postition was one of those eliminated. A brief interruption and distraction is unavoidable as I hunt for a new job.

Will manage projects, test or write if you have a position for me!

A discussion on a group I belong to on LinkedIn got me thinking about bug/defect tracking and the different systems I’ve used for it over the years along with the relative successes or failures of these systems.

Tracking bugs is a core need in software development. Bugs can be generated by the development team (all disciplines within the team), by customers, by product support, etc. This insures bugs that are found are followed, even if not fixed, and metrics from bug tracking help determine things like when a version of the product is ready to release, when a QFE or patch may be needed, etc.

For such a simple task, bug tracking can quickly become a complex and sometimes frustrating chore and this is often due to one or more of the following:

• Tracking More Than Bugs
  This tends to lead to field creep – an explosion in the fields in the bug form – in order to accommodate things like specifications, work items, customer issues, etc.
• Tracking Every Available Piece of Information
  Without a basic agreement on what information MUST be in a bug report, the tendency is to track every piece of available information so nothing is missed. This is another contributor to the issue of field creep.
• Lots of Required Fields
  If fields are made “required” when they are not always applicable, people will put junk in them to get past the requirement. At the point this starts to happen, the usefulness of that field’s information disappears. The same thing happens when the list of required fields is large – people will put anything in just to be done with the bug report.
• Field Overlap
  If there is more than one place to put the same exact same piece of information, users will begin to make mistakes, omit that information, or enter contradictory information.
• Lack of User Training and Documentation
  Users of the bug system should be trained in the way its implemented and used and what is required of them along with a good idea of the process used to address bugs and the bug lifecycle in use in their company. When the first solution to lack of compliance is to set up the bug tracking system to enforce rules, it can cause worse behavior because the users still don’t know the rules, instead they work at getting around the automatic enforcement.
• Shared Bug Tracking
  Sharing bug tracking among multiple projects, sometimes throughout a company, can lead to huge bug forms with a lot of redundancy or fields that are meaningless to all but one project.
• No Central System Control
  Lack of a single person or core v-team that is responsible for managing the bug tracking system can lead to both having no one in charge of fixing issues that inevitably arise, but also to people just randomly adding fields or field contents without a master plan or vision.
• Off the Shelf Configuration
  Using a bug tracking solution off the shelf, without reviewing it and possibly customizing it for the product, the development process and the needs of the team is akin to the square peg, round hole problem.
• Bug Tracking not Periodically Reviewed
  Using the same bug tracking solution and setup over and over without reviewing it periodically to insure it’s still applicable and optimized for the needs of the company, team and product can eventually erode its usability and usefulness.

These are the potential red flags from my own list and years of managing bug databases and bug tracking systems. I’m sure there are more and you can leave those in comments.

I was reading a blog post on uTest.com by James Whittaker called Thoughts on the Future of Software Testing and one thing that really jumped out at me from it is the issue of mentoring in Software Testing.

I’ve had the benefit of several very good mentors and, in turn, have mentored other testers over the course of my thirteen years in software testing. I know James has put out a call to testers to take mentoring seriously and I’d like to second that but also offer a short list of does and don’ts from my own experience:

• DON’T wait for someone else to break the ice. If you have another tester your respect and would like to learn from, go ahead and ask if they can spend a little time teaching and mentoring you. If you see a promising tester you think you can help, ask them if they’d like you to mentor them. The worst thing that can happen is that they turn you down.
• DON’T treat a mentor/mentee relationship as if it were a marriage for life. Even informal or short term mentoring relationships can have considerable value without having to sign up for a long-term or potentially unsustainable relationship.
• DO set expectations on both sides. Part of a good mentor/mentee relationship is having a clear and upfront agreement on expectations from both mentor and mentee. Lack of this typically leads to disappointment, resentment and a failure of the relationship.
• DO keep an open mind. If you go into a mentoring relationship, you won’t get much out of it if you aren’t open to other ways of doing things or the learning opportunities you may encounter.
• DON’T limit yourself to only other testers as mentors or mentees. Learning from other disciplines and teaching test theory and techniques to other disciples can be valuable as well.

Mentoring can be a great help to any professional, and testers in particular.

There’s a site/service called Grouply that is purported to help you manage all your Yahoo groups in a centralized place. Sounds good, right? Especially if you have a lot of group memberships.

But rumors abound about Grouply and how it operates. Some reports are that when you sign up for it, it changes all your membership emails to be username@grouply.com and then any mail in the groups you subscribe to are indexed and resourced by Grouply. This means that anyone else on those groups has, without their permission, had their assumed privacy removed.

Now, everyone should realize that there is no real privacy on the web. None. And the memory of the web is pretty darned infinite. But we’ll ignore the fact that illusions of privacy are just that.

But my most basic issue with Grouply is very simple – it requires you to disclose your Yahoo username and password to a third party service. At that point, you’ve lost all control. You have no idea what, how or when the third party will use that information. You certainly have no control over it.

So the basic security advice applies. Never give out your username and password.