Skip to content
This repository was archived by the owner on Jun 3, 2025. It is now read-only.

Commit 3dfc256

Browse files
authored
Update integration-langchain.md (#1320)
* Update integration-langchain.md * Update integration-langchain.md * Update integration-langchain.md
1 parent b476ac8 commit 3dfc256

File tree

1 file changed

+46
-6
lines changed

1 file changed

+46
-6
lines changed

docs/llms/integration-langchain.md

Lines changed: 46 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,9 @@ It is broken into two parts: installation and then examples of DeepSparse usage.
2525
- Choose a [SparseZoo model](https://sparsezoo.neuralmagic.com/?useCase=text_generation) or export a support model to ONNX [using Optimum](https://github.com/neuralmagic/notebooks/blob/main/notebooks/opt-text-generation-deepsparse-quickstart/OPT_Text_Generation_DeepSparse_Quickstart.ipynb)
2626
- Models hosted on HuggingFace are also supported by prepending `"hf:"` to the model id, such as [`"hf:mgoin/TinyStories-33M-quant-deepsparse"`](https://huggingface.co/mgoin/TinyStories-33M-quant-deepsparse)
2727

28-
## Wrappers
28+
## Using DeepSparse With LangChain
2929

30-
There exists a DeepSparse LLM wrapper, which you can access with:
30+
There is a DeepSparse LLM wrapper, which you can access with:
3131

3232
```python
3333
from langchain.llms import DeepSparse
@@ -40,22 +40,62 @@ from langchain.llms import DeepSparse
4040
llm = DeepSparse(model='zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none')
4141
print(llm('def fib():'))
4242
```
43-
44-
And provides support for per token output streaming:
43+
## Streaming
44+
The DeepSparse LangChain wrapper also supports per token output streaming:
4545

4646
```python
4747
from langchain.llms import DeepSparse
4848
llm = DeepSparse(
49-
model="zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base_quant-none",
49+
model="hf:neuralmagic/mpt-7b-chat-pruned50-quant",
5050
streaming=True
5151
)
5252
for chunk in llm.stream("Tell me a joke", stop=["'","\n"]):
5353
print(chunk, end='', flush=True)
5454
```
55+
## Using Instruction Fine-tune Models With DeepSparse
56+
Here's an example of how to prompt an instruction fine-tuned model using DeepSparse and the MPT-Instruct model:
57+
```python
58+
prompt="""
59+
Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: what is quantization? ### Response:
60+
"""
61+
llm = DeepSparse(model='zoo:mpt-7b-dolly_mpt_pretrain-pruned50_quantized')
62+
print(llm(prompt))
63+
"""
64+
In physics, the term "quantization" refers to the process of transforming a continuous variable into a set of discrete values. In the context of quantum mechanics, this process is used to describe the restriction of the degrees of freedom of a system to a set of discrete values. In other words, it is the process of transforming the continuous spectrum of a physical quantity into a set of discrete, or "quantized", values.
65+
"""
66+
```
67+
You can also do all the other things you are used to in LangChain such as using `PromptTemplate`s and parsing outputs:
68+
```python
69+
from langchain import PromptTemplate
70+
from langchain.output_parsers import CommaSeparatedListOutputParser
71+
72+
llm_parser = CommaSeparatedListOutputParser()
73+
llm = DeepSparse(model='hf:neuralmagic/mpt-7b-chat-pruned50-quant')
74+
75+
prompt = PromptTemplate(
76+
template="List how to {do}",
77+
input_variables=["do"])
78+
79+
output = llm.predict(text=prompt.format(do="Become a great software engineer"))
80+
81+
print(output)
82+
"""
83+
List how to Become a great software engineer
84+
By TechRadar Staff
85+
Here are some tips on how to become a great software engineer:
86+
1. Develop good programming skills: To become a great software engineer, you need to have a strong understanding of programming concepts and techniques. You should be able to write clean, efficient code that meets the requirements of the project.
87+
2. Learn new technologies: To stay up-to in the field, you should be familiar with new technologies and programming languages. You should also be able to adapt to new environments and work with different tools and platforms.
88+
3. Build a portfolio: To showcase your skills, you should build a portfolio of your work. This will help you showcase your skills and abilities to potential employers.
89+
4. Network: Networking is an important aspect of your career. You should attend industry events and conferences to meet other professionals in the field.
90+
5. Stay up-to-date with industry trends: Stay up-to-date with industry trends and developments. This will help you stay relevant in your field and help you stay ahead of your competition.
91+
6. Take courses and certifications: Taking courses and certifications can help you gain new skills and knowledge. This will help you stay ahead of your competition and help you grow in your career.
92+
7. Practice and refine your skills: Practice and refine your skills by working on projects and solving problems. This will help you develop your skills and help you grow in your career.
93+
"""
5594

95+
```
5696
## Configuration
5797

58-
It has arguments to control the model loaded, any configs for how the model should be loaded, configs to control how tokens are generated, and then whether to return all tokens at once or to stream them one-by-one.
98+
The DeepSparse LangChain integration has arguments to control the model loaded, any configs for how the model should be loaded, configs to control how tokens are generated, and then whether to return all tokens at once or to stream them one-by-one.
5999

60100
```python
61101
model: str

0 commit comments

Comments
 (0)