What are the key criteria you consider when evaluating AI vendors, beyond just technical capabilities? (for example, company culture, data security practices, long-term viability). What would be deal breakers?

When evaluating AI vendors, understanding the different types of agents they provide is key:

Note that a strong vendor or vendor partner, should show how these agents work together to address both immediate tasks and long-term enterprise goals, ensuring scalability, transparency, and measurable value.

 • Chatbots: Basic, task-specific systems handling simple interactions like FAQs. Look for adaptability and contextual understanding.

 • Assistants: Task-driven helpers that work alongside users. Evaluate their ability to support workflows and provide meaningful insights.

 • Co-Pilots: Collaborative tools offering suggestions while humans stay in control. Transparency and relevance in recommendations are critical.

 • Agentic Systems: Task-oriented and autonomous, focused on executing specific actions efficiently. Ensure scalability and integration.

 • AI Agents: Goal-oriented systems driving broader objectives autonomously. They must align with enterprise goals and offer governance.

 • Expert Agents: Persona-oriented specialists providing deep domain knowledge. Verify their expertise and ability to deliver reliable insights.

In the university space, our vendor evaluation process is quite exhaustive. Regardless of the type of purchase—be it security, AI, or applications—everything goes through the same rigorous process. Specific to AI, third-party risk is a key consideration, including the vendor's willingness to work with us on contractual agreements. A deal breaker for us is if a vendor cannot guarantee that our data will be stored in the US, which is crucial for our compliance requirements. We are also concerned about long-term viability. I often engage with venture capital firms to identify promising startups, but procurement always questions the stability and scalability of these smaller companies. The main considerations are whether we can exit the agreement if needed and whether it's wise to invest in a small company. These are the larger issues we face when evaluating AI vendors.

For us, budget constraints are a significant factor. Our budget is set years in advance, so we have limited flexibility. For instance, we had a frustrating experience with a vendor whose pricing model was untenable, increasing costs by 50% for a new tool. This kind of pricing is simply not feasible for us. Beyond budget, company culture is less of a concern. Our team, which includes many scientists and engineers, is generally open to change and new tools, despite some resistance to updates. Data security practices are crucial, but we strive to minimize internal PII data to reduce management burdens. Long-term viability is another challenge, particularly with pay-as-you-go models. While they offer flexibility, they can also become costly if not carefully monitored. We prefer to scale usage as needed rather than committing to large upfront purchases.