You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jun 3, 2025. It is now read-only.
Copy file name to clipboardExpand all lines: docs/llms/integration-langchain.md
+46-6Lines changed: 46 additions & 6 deletions
Original file line number
Diff line number
Diff line change
@@ -25,9 +25,9 @@ It is broken into two parts: installation and then examples of DeepSparse usage.
25
25
- Choose a [SparseZoo model](https://sparsezoo.neuralmagic.com/?useCase=text_generation) or export a support model to ONNX [using Optimum](https://github.com/neuralmagic/notebooks/blob/main/notebooks/opt-text-generation-deepsparse-quickstart/OPT_Text_Generation_DeepSparse_Quickstart.ipynb)
26
26
- Models hosted on HuggingFace are also supported by prepending `"hf:"` to the model id, such as [`"hf:mgoin/TinyStories-33M-quant-deepsparse"`](https://huggingface.co/mgoin/TinyStories-33M-quant-deepsparse)
27
27
28
-
## Wrappers
28
+
## Using DeepSparse With LangChain
29
29
30
-
There exists a DeepSparse LLM wrapper, which you can access with:
30
+
There is a DeepSparse LLM wrapper, which you can access with:
31
31
32
32
```python
33
33
from langchain.llms import DeepSparse
@@ -40,22 +40,62 @@ from langchain.llms import DeepSparse
for chunk in llm.stream("Tell me a joke", stop=["'","\n"]):
53
53
print(chunk, end='', flush=True)
54
54
```
55
+
## Using Instruction Fine-tune Models With DeepSparse
56
+
Here's an example of how to prompt an instruction fine-tuned model using DeepSparse and the MPT-Instruct model:
57
+
```python
58
+
prompt="""
59
+
Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: what is quantization? ### Response:
In physics, the term "quantization" refers to the process of transforming a continuous variable into a set of discrete values. In the context of quantum mechanics, this process is used to describe the restriction of the degrees of freedom of a system to a set of discrete values. In other words, it is the process of transforming the continuous spectrum of a physical quantity into a set of discrete, or "quantized", values.
65
+
"""
66
+
```
67
+
You can also do all the other things you are used to in LangChain such as using `PromptTemplate`s and parsing outputs:
68
+
```python
69
+
from langchain import PromptTemplate
70
+
from langchain.output_parsers import CommaSeparatedListOutputParser
output = llm.predict(text=prompt.format(do="Become a great software engineer"))
80
+
81
+
print(output)
82
+
"""
83
+
List how to Become a great software engineer
84
+
By TechRadar Staff
85
+
Here are some tips on how to become a great software engineer:
86
+
1. Develop good programming skills: To become a great software engineer, you need to have a strong understanding of programming concepts and techniques. You should be able to write clean, efficient code that meets the requirements of the project.
87
+
2. Learn new technologies: To stay up-to in the field, you should be familiar with new technologies and programming languages. You should also be able to adapt to new environments and work with different tools and platforms.
88
+
3. Build a portfolio: To showcase your skills, you should build a portfolio of your work. This will help you showcase your skills and abilities to potential employers.
89
+
4. Network: Networking is an important aspect of your career. You should attend industry events and conferences to meet other professionals in the field.
90
+
5. Stay up-to-date with industry trends: Stay up-to-date with industry trends and developments. This will help you stay relevant in your field and help you stay ahead of your competition.
91
+
6. Take courses and certifications: Taking courses and certifications can help you gain new skills and knowledge. This will help you stay ahead of your competition and help you grow in your career.
92
+
7. Practice and refine your skills: Practice and refine your skills by working on projects and solving problems. This will help you develop your skills and help you grow in your career.
93
+
"""
55
94
95
+
```
56
96
## Configuration
57
97
58
-
It has arguments to control the model loaded, any configs for how the model should be loaded, configs to control how tokens are generated, and then whether to return all tokens at once or to stream them one-by-one.
98
+
The DeepSparse LangChain integration has arguments to control the model loaded, any configs for how the model should be loaded, configs to control how tokens are generated, and then whether to return all tokens at once or to stream them one-by-one.
0 commit comments