llama-node

LLaMA Node

Introduction

LLaMA Node is a library for Node.js designed to work with large language models. Created to help developers easily use models such as LLaMA, RWKV, and others derived from LLaMA, this project is currently in its early stages and not yet ready for production. Therefore, future changes to the Node.js API might occur. It doesn't follow the traditional semantic versioning convention, so users should exercise caution when integrating it into applications.

The library leverages several underlying technologies, including llm (originally named llama-rs), llama.cpp, and rwkv.cpp, utilizing napi-rs to facilitate communication between Node.js and the LLaMA execution thread.

Supported Models

LLaMA Node supports various models through different backends, primarily using the GGML format:

• llama.cpp Backend:

  • LLaMA 🦙
  • Alpaca
  • GPT4All
  • Chinese LLaMA / Alpaca
  • Vigogne (French)
  • Vicuna
  • Koala
  • OpenBuddy 🐶 (Multilingual)
  • Pygmalion 7B / Metharme 7B

• llm (llama-rs) Backend:

  • GPT-2
  • GPT-J
  • LLaMA family (Alpaca, Vicuna, Koala, GPT4All v1, GPT4-X, Wizard)
  • GPT-NeoX (including StableLM, RedPajama, Dolly v2)
  • BLOOM (including BLOOMZ)

• rwkv.cpp Backend:

  • RWKV

Supported Platforms

LLaMA Node supports several operating systems and architectures, including:

• darwin-x64
• darwin-arm64
• linux-x64-gnu (requires glibc >= 2.31)
• linux-x64-musl
• win32-x64-msvc

Note that the Node.js version required to use this library is 16 or above.

Installation

Adding LLaMA Node to a project is straightforward. You can install it using npm with the following command:

npm install llama-node

After installing the primary package, choose and install at least one inference backend:

• For llama.cpp:

  npm install @llama-node/llama-cpp
  

• For llm:

  npm install @llama-node/core
  

• For rwkv.cpp:

  npm install @llama-node/rwkv-cpp
  

Manual Compilation

Details on manual compilation can be found in the project's contribution guide, which provides instructions for those who wish to get involved in the project's development or manage more specific build requirements.

CUDA Support

For users interested in leveraging CUDA support, detailed instructions are available on the LLaMA Node documentation site concerning manual compilation related to CUDA.

Acknowledgments

The LLaMA Node library is released under the MIT/Apache-2.0 license. The developers recommend giving credit if you use the code in other projects. Key dependencies and inspiration include:

• LLaMA models: facebookresearch/llama
• RWKV models: BlinkDL/RWKV-LM
• llama.cpp: ggerganov/llama.cpp
• llm: rustformers/llm
• rwkv.cpp: saharNooby/rwkv.cpp

Some portions of the code are adapted from:

• llama-cpp bindings: sobelio/llm-chain
• rwkv logits sampling: KerfuffleV2/smolrsrwkv

Community

For those interested in joining discussions, sharing insights, or seeking help, LLaMA Node maintains an active Discord community. You can join by clicking here.