llama.cpp-python setup
How to run LLM Entirely on Your Machine
Hosted on our YouTube channel Watch on YouTube ↗
messages = [
{
"role": "system",
"content": "You are a helpful assistant.",
},
{
"role": "user",
"content": "Hello! Please introduce yourself briefly.",
},
]
Run it: python test_llamapython.py
from llama_cpp import llama_cpp
def main():
print("llama-cpp-python imported successfully.")
print(f"llama.cpp version: {llama_cpp.llama_print_system_info().decode()}")
if __name__ == "__main__": main()
Hello! I am a large language model, trained by Google. Think of me as your versatile AI assistant. I can help you with a wide range of tasks, such as: * **Answering questions** on almost any topic. * **Writing and editing** content (emails, essays, stories, code, etc.). * **Summarizing** long articles or documents. * **Translating** languages. * **Brainstorming** creative ideas or solving complex problems. How can I help you today?
flowchart LR
A[Local Machine] --> B[llama.cpp]
B --> C{Build}
C -->|CPU| D[CPU Backend]
C -->|CUDA| E[NVIDIA GPU]
D --> F[llama-cpp-python]
E --> F
F --> G[Load GGUF Model]
G --> H[Configure Llama]
H --> I[Chat Completion]
I --> J[Local Inference]
J --> K[Generated Response]
K -.-> L[Nothing leaves the machine]