Questions to Ask When Selecting an AI Tool for Data Synthesis

As the head of product at Voxpopme , I’ve spent a lot of time evaluating and working with teams that are searching for AI tools to help our customers synthesize data. Asking the right questions when evaluating an AI tool is the difference between getting something that will actually help your team vs getting roped into the hype around these two letters that are redefining a lot about our work life. Asking these questions when looking at any AI tool is going to save you headache, heartache, time and money.

TL;DR - Questions to Consider

• How do you price it? Is it usage based?
• What's your base LLM?
• Are you training on my data? What benefits do we get from that training?
• How do you verify the results? Do you show your work?
• How can you help us get through security and AI governance?

1. Pricing: Understand the Cost Dynamics

Pricing can be a complex puzzle when it comes to AI tools. Is it token-based? What’s the usage going to be like? As product teams, we’re just starting to get used to the technological costs associated with AI. These costs aren’t going away. I'm expecting that they’re likely to become more standardized as the market matures. Understanding these dynamics early will help you make informed decisions as these technologies evolve.

• How do you price your AI toolset?
• Do I have unlimited usage or am I token-limited?

2. Base LLM: Choose a Market Leader

There are numerous base LLMs out there, and preferences can vary across teams, companies, and privacy professionals. It’s crucial to align with a market leader in this space. Make sure you know which LLM the AI tool is leveraging, as this can have significant implications for performance, security, and future scalability.

• Which LLM are you using?
• Do you have a partnership or relationship with that provider?
• How is the infrastructure set up around my account?

3. Training: Ask the Right Questions

The word “training” often raises red flags for anyone purchasing an AI tool, let alone one that is utilizing proprietary or customer data. Is the model being trained on my data? At Voxpopme, we don’t train models on customer data, but it’s surprisingly a double-edged sword. While some teams prefer not to have their data used for training, others can see value in it. Make sure you understand the implications and communicate your preferences early and clearly.

• Is the model being trained on my data? If so, how?

• Is the training happening for my instance, all your customers, or the LLM itself?

4. Evidence Pulling and Citations: Verify the AI’s Outputs

If the AI tool doesn’t point to evidence for its conclusions, run away. In the realm of data synthesis, transparency is key. Tools that provide clear citations and evidence for their outputs ensure that you’re not just getting data but getting reliable data.

• How does your product 'show it's work'?
• Does it link to citations or evidence?
• How do you combat hallucinations?

5. Security: Engage Early & Provide the Basics First

Security is non-negotiable, but honestly, many legal teams are still catching up with the ins and outs of AI toolsets’ nuances. They’re working hard to keep things safe and mitigate risk, but these teams need more in-depth understanding than before. AI has changed things for these teams quickly, so start the security conversations early and provide comprehensive overviews of how the AI tool works. This not only speeds up the process but also ensures that all stakeholders are on the same page.

• What security resources do you have?
• Do you have an overall 'what our company does' one pager for my legal team?
• Who is your point of contact for security questions?

Navigating the AI landscape can be challenging, but with the right approach, you can find a tool that meets your needs and aligns with your company's values.

AI was used to write the subtitles, spell check, and create a wrap up of this article.