By: Samuel Chan · October 1, 2024
Table of Content
ReAct-style agent) and the older implementations featuring
AgentExecutor` (in chapter 1 and chapter 2).typing
annotations in Python. For example, the get_top_companies_by_tx_volume
tool might have the following signature:
Stock
, that represents a stock and its attributes. We are inheriting from BaseModel
of the pydantic
library, a popular data validation library in Python.
Stock
class, we can count on pydantic
to validate the output of our tools and call the .with_structured_output()
on the LLM to obtain structured outputs.
The .with_structured_output()
method always takes a schema as input, respecting the names, types and descriptions of the fields, or attributes, of the schema. It then returns objects that conform to the schema (instead of strings), which can be used in downstream tools when orchestrated in a chain-like fashion.
When using Pydantic, the model-generated output will be validated against the schema, with Pydantic raising an error if the output does not match the schema (e.g. required fields missing, or fields of the wrong type). Below is an example of creating a structured LLM with Pydantic. I’m choosing to use the llama3-groq-70b-8192-tool-use-preview
model from Groq, which is fine-tuned for tool use.
with_structured_output
is most commonly used in tool-use scenarios (i.e. function-calling, or tool-calling) and these
information essentially act as model prompts to guide the language model in generating structured outputs.Stock
class.
Stock
class. One can imagine a common use case being a chatbot that can extract information about companies from news articles, press releases, PDF documents or other unstructured text sources into a structured format adhering to the schema of the Stock
class ready for database insertion.
We can similarly use the LLM to generate structured outputs from natural language prompts. We specify our instructions, and the structured LLM will respect the schema of the Stock
class when generating the output.
Chaining can mean making multiple LLM calls in a sequence. Language models are often non deterministic and can make errors, so making multiple calls to check previous outputs or to break down larger tasks into bite-sized steps can improve results. Constructing the Input to LLMs Chaining can mean combining data transformation with a call to an LLM. For example, formatting a prompt template with user input or using retrieval to look up additional information to insert into the prompt template Using the Output of LLMs Another form of chaining refers to passing the output of an LLM call to a downstream application. For example, using the LLM to generate Python code and then running that code; using the LLM to generate SQL and then executing that against a SQL database.The essential functionalities are:
|
(pipe) operator to chain multiple functions together.invoke()
to call the LLM.batch()
to run the chain against a list of inputs.stream()
to return an iterable.batch()
takes a list of inputs and will perform optimizations such as batching calls to LLM providers when possible.
.stream()
returns an iterable:
Stocks
that represents a list of Stock
objectsConversationalResponse
that represents a response to a conversational query (e.g “how are you?”, “what’s your name?”)FinalResponse
that represents the final response of our AI agentprompt
object and use the chaining syntax to construct our RunnableSequence
FinalResponse
schema that can either contain a list of Stock
objects or a ConversationalResponse
object. This is a common pattern in AI systems where the output can be of different types depending on the context of the query.
Pay attention to the use of |
operator as well. This is the chaining syntax that allows us to compose multiple functions together, composing the final chain (“runnable sequence”) that can be executed by calling .invoke()
.
In fact, let’s now call .invoke()
to run the chain and see how the AI agent generates structured outputs with our new schemas:
PydanticOutputParser
Stock
model that we have created earlier. We will add two model_validator
as class methods to the Stock
class, used to validate the output of the model. We will then create a parser
object which we can include in the downstream of our chain.
These are the changes we need to make to the Stock
class:
parser
object:
get_format_instructions
to see what the output generated for us by PydanticOutputParser
looks like:
PromptTemplate
to create a prompt and chain it with our LLM and the parser
object:
prompt | llm | parser
uses LCEL syntax and runnable.invoke()
is equivalent to:
Stock
class.
PydanticOutputParser
validate the output of the model, it also provides instructions on how the output should be formatted and uses model_validator
to ensure that the output meets our expectations and conforms to the schema of the Stock
class.
In the optional exercises below, you will be tasked with writing a test for the PydanticOutputParser
to ensure that it raises a ValueError
when the market capitalization is negative.
SimpleJsonOutputParser
SimpleJsonOutputParser
which is a simpler output parser that does not use Pydantic. It is useful when you do not need to validate the output of the model against a schema, but still want to generate structured outputs in JSON format.
Stock
and Data
classes are the same as before, but we have added a followup_question
field to the Data
class. We will use this field to ask the user a follow-up question after extracting the data about the stocks.
PromptTemplate
to include the followup_question
field with some additional instructions, then chain the PromptTemplate
, the LLM, and the SimpleJsonOutputParser
together.
Data
class. The SimpleJsonOutputParser
does not validate the output of the model against a schema, but it does generate structured outputs in JSON format.
out.keys()
call returns a dictionary with the keys stocks
and followup_question
, with the extration to structured data matching the schema of the Data
class.
A caveat with this approach is that the SimpleJsonOutputParser
, as mentioned above, does not validate the output of the model against a schema, so this is a task that you will have to perform manually (better prompts, few-shot prompting, or any useful techniques) perhaps implemented as a downstream tool. This is especially important if you are generating structured outputs with a strict schema that you want to enforce.
company_name
, industry
, location
, and product
, which, while structured, are not validated against our schema.
test_parser.py
file that tests the extraction of structured data from unstructured text using the PydanticOutputParser
.
text
above contains a negative market capitalization value. The test should validate that the PydanticOutputParser
raises a ValueError
when the market capitalization is negative, instead of continuing with the parsing and feed this output into a downstream tool (i.e. report creation tool, database insertion tool etc).
unittest
or pytest
to write the test. pytest
needs to be installed with pip install pytest
if you choose the latter (unittest
is part of the Python standard library).test_parser.py
and run it using pytest test_parser.py
or python -m unittest test_parser.py
.