As you may already know, we have automated our selection process. In this blog, we are just going to focus on the demonstration evaluation. Our process includes a script for every demonstration. A script is essential as it compels the vendors to focus on what our clients deem most important rather than giving their usual demonstration of what they do best. It also enables comparison between systems.
We usually have two demonstrations 1) Requirements Demonstration based on a subset of the most important requirements in the RFP. 2) Business Process Demonstration based on business processes linked to requirements. In both demonstrations, the script identifies the requirements to be demonstrated and the requirements that don’t need to be demonstrated. On average, the first demonstration is half a day and the second one is a full day, but these times will vary based on scope and complexity. The evaluation takes about 1.5 times as many hours as those spent attending the demonstration. Assuming a 4-hour demonstration, that will take 10 hours of our time in total.
We now give AI (Claude) the script, a transcript of the demonstration and some screenshots. Claude will update our evaluation with:
- What was demonstrated for the requirements that were to be demonstrated
- What was demonstrated for the requirements that did not need to be demonstrated
- Which requirements were not demonstrated
- Which requirements were added, as sometimes our client or the vendor discuss requirements not in the script.
I
ncredibly, this takes Claude only 15 minutes. Claude is not so good, at least not now, at evaluating the strengths and weaknesses of the demonstration, so we still need to be there, but the time may come soon when Claude will attend for us and we will just review Claude’s evaluation to make sure the strengths and weaknesses of the demonstration have been properly identified.
One caveat relates to security and the risk of exposing confidential information to the public. There are two considerations:
- The information that is shared with Claude must not be considered confidential by the client.
- The AI plan and settings. Claude has multiple plans, and there may be a risk of exposure if the “Help improve our AI models” setting is turned on.
We encourage you to unleash the power of AI and we are available to help you if you have any questions.
