Checklist to Evaluate IT Project Vendors

For many organizations the success or failure of IT initiatives is predicated on the selection of the appropriate technology vendor. Despite the critical nature of this process, many organizations underestimate the time and effort it takes to make a well-informed decision. This article is my personal experience & learning while doing complete IT projects in Pay Point India is meant to serve as a guide to help you understand and think through the critical steps in the vendor selection process.

As you read this, please keep in mind that as an organization goes through the vendor selection process it is not uncommon for other business processes or organizational needs to be revealed. It is important to remember that technology projects are often not just about the technology, but rather the health and effectiveness of the entire organization. This learning experience focuses on the process of selecting a vendor, and assumes that other important organizational change management issues are being addressed in concert to support this process.

( Read more: Security Technology Implementation Report- Annual CISO Survey )

 

Seven Step Model

  • ASSESS FEASIBILITY - Is this viable for my organization?
  • GATHER REQUIREMENTS - What does my organization need?
  • RESEARCH & REFINE OPTIONS - What solutions/vendors might fit my needs?
  • EVALUATE VENDORS - What is the best fit for my organization’s needs?
  • SELECT & ENGAGE VENDOR - Is this a reasonable price and contract?
  • MANAGE IMPLEMENTATION - Has the vendor delivered on its promises?
  • SUPPORT & MAINTENANCE - How will we maintain the solution and support it?

 

STEP 1:   ASSESSING FEASIBILITY

Organizational Readiness - Consider important elements to project success such as getting buy-in from staff and overcoming technology fears and resistance to change.

Budgeting - Ensure that you have the appropriate budget level to successfully execute on the project. Make sure that your budget can withstand reasonable variances from original estimates. Technology projects have varying degrees of  financial risk based on the complexity of the project. At a minimum, your project budget should be able to withstand a 15% variance.

Staff Availability - Most technology projects require a significant investment of time by your organization’s staff. Your staff will be involved in many stages of the process, such as requirements gathering, training, testing, and disruptions during deployment. You will also need to designate a project advocate from your staff to manage the vendor relationship and internal resources associated with the project. Before embarking on any large technology project, ensure that your organization can free up time from the appropriate staff members to make this project successful.

Sustainability - Ensure that you have the proper resources in place to sustain the technology at the conclusion of the project. This could include budgeting for ongoing support, hiring a technology manager, or giving ownership of maintenance to a staff member.

Return on Investment (ROI) - Is the project worth the investment? Will it allow you to serve your constituents better or serve more of them? Will it improve your operations and/or lower costs?

Arriving at a Decision - After careful review of the aforementioned factors, you are now ready to make a decision. Most organizations will have a clear “go” or “no-go” decision. If the limiting factor is budget or staff availability you may decide to opt for a “go-later” decision.

OUTCOME: “GO”, “NO GO”, “GO LATER” DECISION

STEP 2:   GATHER REQUIREMENTS

Review Business Strategy - Identify the business goals you hope to accomplish with this technology project.

Ensure Alignment - Make sure that the application of technology will be an enabling factor and will not create a disruptive influence on the organization.

Process Mapping - Document critical business processes that your organization performs. This understanding will be critical for a vendor to understand how its solution should be implemented at your organization.

Process Re-engineering - Technology implementation often provides an opportunity to change the way certain business tasks are managed at your organization. Consider this element and make a determination if it would be valu-able to include.

Requirements Analysis - Identify critical requirements (such as number of users, current technologies in use, need for remote access, training, etc.) that you will need as a part of your technology solution.

Prioritization of requirements - Prioritize your list of requirements and determine which ones are essential and which ones are “nice to have” but not required for success.

Environmental assessment - If your project involves environmental or physical location factors, make sure a thorough assessment is conducted and that all findings are well documented. 

Technical assessment - Document your current technology and catalog all areas that may interface with your new solution.

OUTCOME: REQUIREMENTS DOCUMENT/REQUEST FOR PROPOSAL

( Read more:  Top 5 Application Security Technology Trends )

STEP 3:   RESEARCH & REFINE OPTIONS

Buy/Blend/Build - Most technology solutions can be categorized into one of three areas: Buy an off-the-shelf solution, Build a custom solution, or Blend a solution by combining an off-the-shelf product with some customization.

Establish Evaluation Criteria - Develop a set of criteria on which you would like to evaluate your prospective vendors. Appendix A has an example of some common criteria used in evaluations.

Conduct Research - Use the resources at your disposal to learn more about existing products or solutions that could meet your needs. Discuss your project objectives with related organizations, trusted advisors, and technology consultants.

Define Targeted List - Based on your requirements and your research into solutions, create a short list of vendors who may be able to meet your requirements. The size of your short list of vendors should correlate to variability in proposed solutions and project complexity. For instance, for a small defined project a short list of 3 vendors may be appropriate. For large complex projects with many different approaches, you may consider a list as large as 8 vendors. Make sure that you keep your short list of vendors to a manageable scale.

Send RFP - Send the vendors your requirements information and ask them to submit a proposal. Typically requirements are sent in the form of a Request for Proposal (RFP) document.

OUTCOME: TARGETED LIST OF VENDORS/SOLUTIONS TO PURSUE

 

STEP 4:   EVALUATE VENDORS

Evaluation Matrix - Develop an evaluation matrix (see Appendix B) to help you objectively evaluate each vendor’s proposal and product demonstration.

Proposals - Each invited vendor should respond to your RFP with a written proposal. Carefully evaluate each proposal and encode the proposal information into your evaluation matrix.

Product Demonstrations - Many vendors will request an in-person or web-based opportunity (a “demo”) to show-case the capabilities of your solution. Demos are a valuable way to get more information and also evaluate intangible aspects of a vendor.

Reference Checks - Don’t forget to check the vendor’s references as a part of your evaluation process. Consider site visits if you are making a large investment.

OUTCOMES: VENDOR PROPOSALS, VENDOR DEMOS, WEIGHTED VENDOR MATRIX

STEP 5:   SELECT & ENGAGE VENDOR

Primary and Secondary Options - At the conclusion of your evaluation process, you will need to identify a primary option (your winner) and some secondary alternatives.

Negotiations - Do not burn the bridges with secondary option vendors as they will serve as a valuable resource in the negotiation process. While you are in the negotiation process, keep in mind your secondary options as they serve as your best alternative if your negotiation falls through. Make sure that the final deal you strike with your preferred vendor is at least as favorable as your secondary options. 

Contracting - Identify a clear set of objectives, deliverables, timeframes, and budgets for your project with the vendor. Make sure these are clearly written in the terms of the contract.

OUTCOME: FINAL VENDOR SELECTED & CONTRACTED

( Watch more : Attacks on Smart TV and Connected Smart Devices )

STEP 6:   MANAGE IMPLEMENTATION

Dedicate Project Manager - Your organization should dedicate one or more staff to oversee the solution implementation .These staff should have regular checkpoints with the vendor to ensure that delivery matches expectations.

Ensure Timely Delivery - Vendors often juggle many clients at once and as such it is important for your organization to keep track of deliverable dates and ensure that the vendor is meeting them. Be conscious of your deadlines and deliverables to your vendor so they can make their target delivery dates. Keep an eye out for contract terms that apply additional fees for late delivery of necessary project materials from you to the vendor.

Ensure On-Budget Delivery - If your organization negotiates a Time & Materials (T&M) contract with vendor, then it will become imperative to track hours spent and budgeted hours remaining on a project. Without careful consideration of these elements, project costs could spiral out of control.

Manage Scope - The greatest area of risk for most technology projects is in controlling project scope. Once an organization begins to see the possibility of technology, they often attempt to do too much in the initial development and launch of the solution. If this is the case, consider your project with the vendor a “Phase 1 deployment” and try to push back on new additions until a future phase. If a new addition is essential to a project, then you should clearly define it in an addendum to the scope of work and negotiate the price with the vendor.

Manage Expectations - Manage the expectations of all parties involved in the implementation support. Be sure to provide realistic timeframes and advance warning of any variances in budgets and timeframes.

OUTCOME: ON TIME & ON BUDGET DELIVERY OF EXPECTED SOLUTION

 

STEP 7:   SUPPORT & MAINTENANCE

Resources: Ensure that the appropriate resources are dedicated to support the technology on an ongoing basis. Your support and maintenance plan could include some or all of the following:

  • Support Hours/Contract
  • Hiring of tech resources to manage it
  • Assignment of staff member to take ownership
  • Patches & Maintenance
  • Ongoing Training


Upgrades: If the technology solution becomes mission critical, plan an upgrade path for it. Technology tends to change dramatically every 3 years and should never be considered a one-time investment.

OUTCOME: STABLE & EFFICIENT TECHNOLOGY SOLUTION THAT EMPOWERS THE ORGANIZATION

 

CONSIDER EXTERNAL FACTORS

The framework proposed in this paper assumes that your organization is operating in a completely neutral framework and has great latitude in making a decision. Our experience of working through this process with many clients indicates that this is often not the case. Most vendor selection efforts are often influenced by external factors such as foundation recommendations, group purchasing decisions, or donations/discounts discovered through board contacts. Consider these external factors in your assessment phase. The presence of these external factors does not mean that you should forgo the vendor selection process; however, it can mean considering your options in a different light.

These external factors can sometimes lead to significant benefits such as discounts with vendors, financial support, leveraging existing research on vendors, implementation experience, and technical support. The equation you should take into consideration is whether the cumulative benefits outweigh the costs of potentially selecting a less optimal vendor.

Is your organization being asked to use a vendor that really doesn't match your needs? If such a case does
arise, the vendor evaluation matrix can become a huge asset for your organization. Conduct the evaluation
using the externally recommended vendor as a baseline and see where your options fall. You can then present the evaluation matrix to your funders or board members to make an argument for or against a specific
course of action.

( Read more:  5 easy ways to build your personal brand ! )

APPENDIX A: DIMENSIONS OF EVALUATION FOR VENDORS

The following list contains typical dimensions along which vendors can be evaluated. While comprehensive, the list is not exhaustive and you should consider adding your own dimensions to the evaluation criteria.

FEATURES

■   Essential Features

■   Cool to Have Features

■ (Add Requirements Criteria)

 

VENDOR STABILITY

■   Vendor Size

■   Vendor Financials

■   Years in Business

■   Number of Clients

■   Size of Tech Team

■   References

■   Future Direction - Roadmap

 

TECHNOLOGY ELEMENTS

■   Usability/Ease of Use

■   User Interface/Visuals

■   Flexibility

■   Extensible? Customizable?

■   Compatibility

■   Security

■   Backups

■   Virus Protection

 

GENERAL IMPRESSIONS

■   Positives

■   Risks

■   Friendliness

■   Responsiveness

■   Experience/Skill Level

■   Actual Project Team

 

PRODUCT STABILITY

■   Performance Levels

■   Uptime Percentage

■   Last Downtime

■   Duration of Downtime

■   Load/Capacity


TIMEFRAME FOR DEPLOYMENT

■   Phase 1

■   Phase 2

■   Additional phases (if any)

■   Project Completion

■   Training

 

COSTS

■   One-Time (Setup, Configuration, Development)

■   Ongoing (Maintenance, Licensing)

■   Add-Ons

■   Hardware/Software

■   Training

■   Support

■   Data Migration

■   Fixed or Variable

■   TCO = Total Cost of Ownership

 

TRAINING & SUPPORT

■   Support Availability

■   Support Coverage Hours

■   Support Response Time

■   Training Plan

■   Online Help Resources

■   Availability of Support Talent

■   Documentation

 

OTHER CONSIDERATIONS

■   Hosted Externally/ASP

■   Additional Equipment

■   Platform Considerations

■   Locked In to Vendor Solution?

■   Implementation Plan

■   Data Migration

 

SECURITY & BACKUPS

■   Backup Policies

■   Recovery Procedures

■   Virus Protection

■   Data Security

■   Application Security

■   Hardware Security

( Watch more : South Asia's Cyber Security Landscape after the Snowden Revelations )

APPENDIX B: CREATING A WEIGHTED VENDOR EVALUATION MATRIX

It is important to keep yourself objective when going through the vendor evaluation process. It is easy to get swayed by an impressive product demonstration or an eloquent sales representative. In order to avoid falling into this trap, we often use a weighted matrix to rank vendors. Below is an example of how to structure your own vendor evaluation matrix.

 

SAMPLE WEIGHTED MATRIX : (for 3 Vendor evaluation )

 

A spreadsheet program is a great tool for plotting your evaluation matrix. When developing the matrix, you will need to make decisions regarding the following:

 

  • How important is each of the dimensions to your organization? For instance, if support hours are critical, you may
    assign it 10 points instead of 4.

 

  • How do the scores relate to each other? For instance, if you are evaluating three vendors it is usually good to score
    using a 3 point scale or a multiple of a 3 point scale. The vendor who performs best in this category would get a 3 and the worst performer would get a 1. If two vendors are equal on a given dimension, then give them the same score. If the dimension is a very important one, you may make it worth 12 points with the top vendor getting 12, the second getting 8, and the last one getting 4.

 

  • What is a substantive difference in scores? If you are evaluating on a 100 point scale and you get a final list of three
    vendors all within a score range of 51 to 59, then there may not be a substantive difference between them. Take a deeper look at the relative strengths and weaknesses of each vendor before making a final decision.

 

Do not add any elements to your weighted scores that are worth more than 25% of the total points on the matrix. These dimensions should be looked at side by side with the weighted scores. The two most common elements we normally do not include in our weighting are PRICE and TIMEFRAME. Including elements such as these in the matrix would really skew the results, so it works better to consider them independently.

 

YOUR END RESULT should be something like the following:

 

- With Sachin Lokhande, Pay Point India Network Ltd on How To Evaluate A Vendor in IT Projects ClickToTweet

Which above steps will be the most helpful for your organizations ? Share your thoughts with us below in the comments or Write your article here

Views: 4772

Comment

You need to be a member of CISO Platform to add comments!

Join CISO Platform

Follow Us

Contact Us

Email: contact@cisoplatform.com

Mobile: +91 99002 62585

InfoSec Media Private Limited,First Floor,# 48,Dr DV Gundappa Road, Basavanagudi,Bangalore,Karnataka - 560004

© 2018   Created by CISO Platform   |   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service

Related Posts