If you are using Python 3.11 you can use this instead to get a working pytorch: pip install -pre torch -extra-index-url īefore running the conversions scripts, models/7B/consolidated.00.pth should be a 13GB file. Next, install the dependencies needed by the Python conversion script. Your folder structure should look like this: % ls. You need to create a models/ folder in your llama.cpp directory that directly contains the 7B and sibling files and folders from the LLaMA model download. I use pipenv and Python 3.10 so I created an environment like this: pipenv shell -python 3.10 Next you need a Python environment you can install some packages into, in order to run the Python script that converts the model to the smaller format used by llama.cpp. Next, checkout the llama.cpp repository: git clone I've only tried running the smaller 7B and 13B models so far.
The model is a 240GB download, which includes the 7B, 13B, 30B and 65B models. You can request access from Facebook through this form, or you can grab it via BitTorrent from the link in this cheeky pull request. You also need Python 3 - I used Python 3.10, after finding that 3.11 didn't work because there was no torch wheel for it yet, but there's a workaround for 3.11 listed below. To run llama.cpp you need an Apple Silicon MacBook M1/M2 with xcode installed. LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla70B and PaLM-540B Setup Georgi previously released whisper.cpp which does the same thing for OpenAI's Whisper automatic speech recognition model. I'm using llama.cpp by Georgi Gerganov, a "port of Facebook's LLaMA model in C/C++".
I just ran the 7B and 13B models on my 64GB M2 MacBook Pro! It claims to be small enough to run on consumer hardware. See also: Large language models are having their Stable Diffusion moment right now.įacebook's LLaMA is a "collection of foundation language models ranging from 7B to 65B parameters", released on February 24th 2023. Simon Willison’s TILs Running LLaMA 7B and 13B on a 64GB M2 MacBook Pro with llama.cpp Running LLaMA 7B and 13B on a 64GB M2 MacBook Pro with llama.cpp | Simon Willison’s TILs
0 kommentar(er)
