Posts Tagged ‘Software Testing Discipline’

22 March

CHM Text Searches – Lesson Learned

The current project I am working on involves working on the CHM (Microsoft Help Files) for Windows Embedded Standard 7. This was a project underway before I joined the team and it had even changed managers during that time, which always creates at least a bit of chaos.

We just dropped our CHM files to the product team for integration last week and then discovered we had not completely purged the contents of the project’s internal name and prior release same in favor of the final release name. This happened despite having myself and four other people all search the Help for the string. It even happened despite a tool that would automatically check for the strings you tell it to at build time.

After dealing with the aftermath and rebuild/redrop (my boss was out of town), I looked into why this had happened and discovered I had let my tester mindset go to sleep and had not really thought in depth about how the searches were being done and the potential holes in these approaches.

The stage is set by:

  • The tool was checking for the term in the files at build time.
  • The individual searches done ad-hoc were using the search functionality provided in the Help by Windows when reading the CHM file.

The following discoveries were made as we looked into where this failed:

  • The tool does not check the index. Several instances of terms entered into the tool were found in the index.
  • Not all “bad terms” were entered into the tool’s database.
  • The Search functionality in Help does not search the index and no one thought to look there because the index was generated from files thought to be clean.
  • The Search functionality in Help is a regex-base search. All ad hoc searches were done with the same string, but no one thought to look for substrings with use of a wildcard before, after, or both in relation to the “bad string.” This meant no instances where the “bad string” was a substring could be found, though quite a few existed.

It seems obvious now, of course, but it really does help to look at these occurrences and see what the root cause was so that it can be caught in the future. In this case, we took away the following actions for the next release:

  • All terms must be entered in the tool’s database.
  • The CHM must be decompiled and the HTML contents searched for the “bad terms”. This will catch substrings and index occurrences.
  • Ad hoc searching must be done with wildcards and regex to also look for string and substring occurences.

Lesson learned and I paid the price. Both in terms of this instance’s particular issues and in terms of forgetting to look for holes in the process/testing – even if it’s not my job.

15 January

Bug Tracking 1: Potential Red Flags

A discussion on a group I belong to on LinkedIn got me thinking about bug/defect tracking and the different systems I’ve used for it over the years along with the relative successes or failures of these systems.

Tracking bugs is a core need in software development. Bugs can be generated by the development team (all disciplines within the team), by customers, by product support, etc. This insures bugs that are found are followed, even if not fixed, and metrics from bug tracking help determine things like when a version of the product is ready to release, when a QFE or patch may be needed, etc.

For such a simple task, bug tracking can quickly become a complex and sometimes frustrating chore and this is often due to one or more of the following:

  • Tracking More Than Bugs
    This tends to lead to field creep – an explosion in the fields in the bug form – in order to accommodate things like specifications, work items, customer issues, etc.
  • Tracking Every Available Piece of Information
    Without a basic agreement on what information MUST be in a bug report, the tendency is to track every piece of available information so nothing is missed. This is another contributor to the issue of field creep.
  • Lots of Required Fields
    If fields are made “required” when they are not always applicable, people will put junk in them to get past the requirement. At the point this starts to happen, the usefulness of that field’s information disappears. The same thing happens when the list of required fields is large – people will put anything in just to be done with the bug report.
  • Field Overlap
    If there is more than one place to put the same exact same piece of information, users will begin to make mistakes, omit that information, or enter contradictory information.
  • Lack of User Training and Documentation
    Users of the bug system should be trained in the way its implemented and used and what is required of them along with a good idea of the process used to address bugs and the bug lifecycle in use in their company. When the first solution to lack of compliance is to set up the bug tracking system to enforce rules, it can cause worse behavior because the users still don’t know the rules, instead they work at getting around the automatic enforcement.
  • Shared Bug Tracking
    Sharing bug tracking among multiple projects, sometimes throughout a company, can lead to huge bug forms with a lot of redundancy or fields that are meaningless to all but one project.
  • No Central System Control
    Lack of a single person or core v-team that is responsible for managing the bug tracking system can lead to both having no one in charge of fixing issues that inevitably arise, but also to people just randomly adding fields or field contents without a master plan or vision.
  • Off the Shelf Configuration
    Using a bug tracking solution off the shelf, without reviewing it and possibly customizing it for the product, the development process and the needs of the team is akin to the square peg, round hole problem.
  • Bug Tracking not Periodically Reviewed
    Using the same bug tracking solution and setup over and over without reviewing it periodically to insure it’s still applicable and optimized for the needs of the company, team and product can eventually erode its usability and usefulness.

These are the potential red flags from my own list and years of managing bug databases and bug tracking systems. I’m sure there are more and you can leave those in comments.