The Dichotomy: Choosing Between Open-Source and Externally Hosted LLMs

When it comes to enterprise AI, the choice between open-source large language models (LLMs) and externally hosted (paid) LLMs isn’t just about technical specs; it’s also aligning costs with company values and priorities. This article is focused on the later part, and here is a breakdown of what to consider when deciding which model suits your organization best.

1. Where Does Your Data Go?

In a highly regulated industry, where moving data outside the company network is a concern, externally hosted solutions may be off the table, despite their convenience and low upfront costs. In such cases, an open-source model like LLaMA 3 could be the best fit, keeping sensitive data in-house while providing flexibility in how the model is used and modified.

The debate about open source vs. hosted opens when there are no company policy restrictions on data movement outside of companies’ network. 

2. A Quick Cost Comparison

 Let’s dive into the numbers with a simple comparison of ongoing costs for each option:

 Externally Hosted (Paid LLMs) – Variable Cost:

For illustration, I have used ChatGPT-4 Turbo to estimate the cost.

Let’s review an example, and you could apply this to your company based on your need.  The variables are the token consumption, and the cost per token.  For E.g., 

• Average Input Token Consumption: 200,000 tokens/day = 6 million/month 
• Average Output Token Consumption: 300,000 tokens/day = 9 million/month 
• Total Token Consumption: 500,000 tokens/day = 15 million/month 

Open-Source LLM (Fixed Cost):

With open-source LLMs, initial setup costs can be high, including:

• Hardware (e.g., GPUs): Servers with powerful GPUs are essential, and costs vary with model complexity (e.g., 8 billion vs. 70 billion parameters). 
• Infrastructure Maintenance: Headcount and power costs for cooling and maintenance add up over time.

 However, once set up, there’s a key advantage: predictable scaling costs. Unlike paid models, the cost remains the same regardless of the number of tokens consumed.

3. Conclusion: How to Decide:

In short:

• If data regulations prevent off-site storage, go open source.
• Know your token usage rate, for low-scale needs, paid LLMs keep costs predictable and require little setup.
• For high-scale needs, if the costs of paid models exceed open-source solutions, consider open-source to control expenses.