Who should validate computer software used in blood banks/transfusion services?

A colleague reports that in her opinion the validation of blood bank/transfusion service computer software performance is best established by objective evidence that all software specifications conform to user needs and intended uses, and that the particular requirements implemented through software can be consistently fulfilled. She is also of the opinion that an independent validation of software by a knowledgeable "3rd Party" is better than a "self validation" by the user of the software or a validation by the developer of the software. In her opinion, self-validation is extremely difficult, and validation by a software system developer or vendor might be subject to conflict of interest and/or commercial bias. Thus, in her opinion, an independent 3rd party validation is the best approach. In fact, she adds that IEEE standards state that in the classical approach to validation, the responsibility is vested in an organization that is separate from the development organization. The inquiring colleague is curious to know what approach others have taken for their computer validation, and what experiences others have had with each approach. In other words, have colleagues employed self validations, employed the services of the software developer/vendor to perform the validation, or employed the services of an independent 3rd Party to perform the validation?

The following responses have been submitted.

1. According to J. Wade Atkins, QA Specialist in the Department of Transfusion Medicine at the Warren G. Magnuson Clinical Center of the National Institutes of Health (attribution used with permission), the posted topic sparked some very interesting and timely conversation in the Department of Transfusion Medicine in the Clinical Center at the National Institutes of Health. Here are the comments from the responding colleague at the NIH (verbatim) "These are very interesting comments posted by the colleague that is a proponent of contracted validation. I agree that validation of blood bank/transfusion service computer software performance is best established by objective evidence that all software specifications conform to user needs and intended uses, and that the particular requirements implemented through software can be consistently fulfilled and in the classical approach to validation, the responsibility is vested in an organization that is separate from the development organization. I disagree though that a third party is best suited to conduct the validation. Although the vendor/ third party approach does alleviate the burden of extra workload to the line staff, it does not account for the lost productivity from the staff member(s) required to get the consultant familiarized with the internal business practices and policies in order to simulate institutional practices, it prevents end users in becoming comfortable (i.e. really learning the system functionality) in using the software application prior to "go live", it prevents the end user from developing process improvement ideas prior to implementation, it eliminates the likelihood of modification to current processes impacted by the new software prior to implementation, it fails to validate the entire process from training, SOPs and compliance, and I think most importantly it would not allow me the personal sound feeling of confidence that the system was functioning as I intended. From a quality systems perspective these are key elements that ensure you are in control and often go beyond just the validation test cases. YES, validation is sometimes painful, but when approached in an open-minded manner can be more beneficial than just compliance with a seldom understood and often misrepresented requirement."
   
   
2. A colleague from the Department of Veteran's Affairs (VA) reports that they have their own computer system. Since the requirement for FDA licensure, their developers send them a comprehensive validation procedure for every patch that affects the Blood Bank. The responding colleague says that it is still cumbersome to do, but at least they have a guide.

ADDENDA April 3, 2003

3. A colleague who has done 3rd party validations of blood bank/transfusion service software wants to go on the record as saying that she too believes the USER is in the best position to do the validation IF they are capable of doing so. However, if the USER is not qualified to do the computer validation themselves and chooses to use outside competent help (rather than depend entirely on the software developer/vendor), then the USER should be allowed to use a qualified 3rd party. The responding colleague is concerned about reports that some software developers/vendors now impose a contractual restriction that limits access to outside validation support, which in effect prohibits a 3rd party independent review.

ADDENDA April 6, 2003

4. A colleague in the US, who is a strong proponent of using third parties for computer validation, wanted to share the following case example that came to her attention when a company modified its computer system. In the opinion of the responding colleague, the Transfusion Service module of the computer system received short shrift and as time went on fixes were added to it. The in-house IT department did its own validation.
   
   YEARS LATER it was discovered that some printers used in the interfaces between hospital and lab computers printed in upper case only. The printing of reports ONLY IN UPPER CASE impacted on patient results when antibody identifications were also discovered to be all in upper case.
   
   Thus, anti-c was reported as 'ANTI-C', and so on. Although an attempt was made to review and correct the erroneous patient results, no one really seems to know how many patients were affected by this system error.
   
   EDITOR's note: This is a perfect example of why one needs to carefully validate all aspects of a computer system, including the output that is seen by the ordering physician and other healthcare providers. Test results can look quite different in a hard copy printout versus a computer terminal display. It would seem that a deficiency in reporting, such as that described above, should have been prevented by a comprehensive validation, regardless of who performed the validation - 'in-house' users, an installing vendor/software developer, or a third party.
   
   Postscript: In the opinion of the colleague in reply #4, third party auditors are more neutral and less invested in the internal politics, personalities, and budget issues than in-house staff. That is not to say that all issues will be captured in a third party audit. However the auditors are less likely to be vested in the area of interest and are better able to "say the things which must be said" without fear of reprisal. As a follow up of the anti-c versus ANTI-C reporting debacle, the colleague adds that they later found out that the problem HAD BEEN REPORTED to their information technology (IT) department when the system was first installed. For whatever reason, IT had not followed up on it. After installation, the IT department decentralized and the problem fell through the cracks. In retrospect the reporting colleague thinks a third party would have followed up on this incident to ensure corrective action had been taken and was effective.

ADDENDA April 7, 2003

5. A Compliance Officer, who is also certified by the ASQ as a Certified Quality Auditor and a Certified Software Quality Engineer believes that the FDA takes the position that the user is responsible for making sure that the validation is performed at their own site. Further, she believes that the vendor should provide information about how the system works and how upgrades impact and affect the system's overall performance, but that is the only role that the vendor should play is in the site validation. Validation is becoming more and more complicated and demanding as the FDA moves transfusion services and blood banks toward computer industry standards. She adds that in this age of specialization, the user cannot be expected to become a software engineer who is capable of performing the tasks necessary to validate a system thoroughly and correctly. That is the role of the consultant when the user is unable to actually perform the validation and is busy with the everyday tasks associated with providing safe products to the public.

ADDENDA April 9, 2003

6. The following comment is verbatim from a third party consulting company that does validation on computer systems:  'I agree that validation is a great way for the staff to learn the system. However, today most sites already have budget constraints and cannot afford to divert staff members from their current duties and give them more. Consultants should be more than validators. Implementing a new computer system is more than just validation. Consultants should be partners in the implementation and well versed in many different business processes, recommend different approaches, assist with SOP's, and perform training. Consultants should supply the site with documentation on how the validation will be performed before execution takes place. This ensures the end users will have input and comfort with the final product. Readers should consult CFR 21 Part 11 Draft Guidance section 5.7 that states the following on this issue.'

   5.7 Independence of Review. It is a quality assurance tenet that objective self-evaluation is difficult. Therefore, where possible, and especially for higher risk applications, computer system validation should be performed by persons other than those responsible for building the system. Two approaches to ensuring an objective review are (1) Engaging a third party; and, (2) dividing the work within an organization such that people who review the system (or a portion of the system) are not the same people who built it.

7. A colleague who is extremely knowledgeable about blood bank and transfusion service computer systems is of the opinion that essentially the question is, who should validate a computer system? Here is his verbatim answer to that question.
   
   "The ultimate responsible party is, of course, the "end user". But there may be options available to assist the end user that can enhance or expedite the process. Each option comes complete with price and trade-offs.
   
   Consider some of the more common practices.
   • The end user designs and executes their own validation plan with little or no outside assistance.
   • In another approach, the end user hires an independent consultant to advise on or even perform the validation. The user may choose to only purchase validation materials such as documents and plans from the consultant.
   • Still another possibility is to hire the developer, owner, or vendor of the system to perform the validation for the end user. I am amazed that anyone would consider this option. The value of an independent perspective is ignored. The risk of liability is sizeable. Liability for the user's organization because they are paying a vendor to collect proof that the system will live up to their sales claims. The risk here is that evidence of fraud or misrepresentation would never come to light.
   Here are some arguments in favor of an independent company performing validation:
   • Both the CEO and the board will have a greater level of confidence in their investment decision when they receive a positive, unbiased, third party testimonial of the system validation.
   • Independent validation consultants can save time in the planning and training stages because they know many of the intricacies and limitations of the system they are validating, whereas the vendor of the system may be reluctant to reveal unfavorable details because to do so could place the vendor in a compromising position.
   While validation is at times difficult, it is the process that must be undertaken by the user to document that the system is in control by the user and that the end user is competent. If done properly, validation will help to minimize the likelihood of failures or errors."

Page 2

Submit comments to the e-Network Forum at enetworkforum@cbbsweb.org

Ira A. Shulman, MD
CBBS e-Network Forum Senior Editor & Moderator

W. Tait Stevens, MD
CBBS e-Network Forum Editor & Moderator

Elizabeth M. St. Lezin, MD
CBBS e-Network Forum Associate Editor & Moderator