From the course: Large Language Models on AWS: Building and Deploying Open-Source LLMs

Unlock this course with a free trial

Join today to access over 24,600 courses taught by industry experts.

Key concepts in llama.cpp walkthrough

Key concepts in llama.cpp walkthrough

From the course: Large Language Models on AWS: Building and Deploying Open-Source LLMs

Start my 1-month free trial Buy for my team

Key concepts in llama.cpp walkthrough

“

- [Instructor] Let's talk through this Qwen2 coder deployment pipeline, A comprehensive guide from a high level view. And this guide is really interesting because it shows probably the most cutting edge AI coding assistant workflow that's local that you can use because we use Llama.cpp and we have full control of every single step. And this process involves several stages. Each one is a crucial role in making it run efficiently on my specific hardware. So first up here we have the Hugging Face model download stage, and this place we have a central repository for AI models. You can think of it as a GitHub for AI. It has thousands of models, including this Qwen2.5 coder from Ali. And this provides a access to a state-of-the-art coding assistant that's on par with mini commercial models. First up, we have the heavy lifting, which is we have to download this model and it's 32 gigabytes. So it's a huge, huge model. But then we go through and we get the Hugging Face CLI to download it, and…

Contents