Generative AI series

Ongoing (Q4 '24):

5-part Generative AI Series
Create RAG systems and AI agents with Sectors Financial API, LangChain and state-of-the-art LLM models -- capable of producing fact-based financial analysis and financial-specific reasoning. **Continually updated** to keep up with the latest major versions of the tools and libraries used in the series.

Generative AI Series: Table of Contents

1

Generative AI for Finance

An overview of designing Generative AI systems for the finance industry and the motivation for retrieval-augmented generation (RAG) systems.
2

Tool-Use Retrieval Augmented Generation (RAG)

3

Structured Output from AIs

From using Generative AI to extract from unstructured data or perform actions like database queries, API calls, JSON parsing and more, we need schema and structure in the AI's output.

Tool-use ReAct Agents w/ Streaming

Updated for LangChain v0.3.2, we explore streaming, LCEL expressions and ReAct agents following the most up-to-date practices for creating conversational AI agents.
5

Conversational Memory AI Agents

Updated for LangChain v0.2.3, we dive into Creating AI Agents with Conversational Memory

This article is part 4 of the Generative AI for Finance series, and is written using LangChain 0.3.0 (released on 14th September 2024).

For best results, it is recommended to consume the series in order, starting from chapter 1.

For continuity purposes, I will point out the key differences between the current version (featuring ReAct-style agent) and the older implementations featuring AgentExecutor (in chapter 1 and chapter 2).

​
Tool Use LangGraph Agents

Part 4 of this series will feel familiar to readers who have gone through the materials in chapter 2, with the addition of the LangGraph library and a prebuilt ReAct agent that comes with LangGraph.

To make use of the LangGraph library, you will need to install it:

pip install langgraph

LangGraph is a library by LangChain, intended for more complex agentic systems and greater control than LangChain’s agents. The orchestration isn’t much different from what you have seen up to this point, so let us walk this process through:

  1. Setting up our secrets and retriever utility
  2. Creating our tools with the @tool decorator
  3. Use create_react_agent to create our agent equipped with tools we created from (2)
  4. Using the LCEL syntax to set up our runnables
  5. Invoke the runnables

​
Setting up a retriever utility

The following code should feel familiar to you if you have gone through chapter 2: tool use LLMs.

With the retriever utility in place, we can proceed to create the various tools that our agent will use. Our tools are thin wrappers around the retriever utility, designed to fetch data from Sectors Financial API based on the user’s query.

​
Creating information retrieval tools

from langchain_core.tools import tool

@tool
def get_company_overview(stock: str) -> dict:
    """
    Get company overview
    
    @param stock: The stock symbol of the company
    @return: The company overview
    """

    url = f"https://api.sectors.app/v1/company/report/{stock}/?sections=overview"

    return retrieve_from_endpoint(url)


@tool
def get_top_companies_ranked(dimension: str) -> List[dict]:
    """
    Return a list of top companies (symbol) based on certain dimension (dividend yield, total dividend, revenue, earnings, market cap,...)

    @param dimension: The dimension to rank the companies by, one of: dividend_yield, total_dividend, revenue, earnings, market_cap, ...
    @return: A list of top tickers in a given year based on certain classification
    """

    url = f"https://api.sectors.app/v1/companies/top/?classifications={dimension}"

    return retrieve_from_endpoint(url)

With the @tool decorator, we turn our functions into LangChain’s structured tools, of the type <class 'langchain_core.tools.structured.StructuredTool'>, and these tools can be used, either directly or indirectly by our agent.

Just to see how the tools work, let us invoke them directly:

out = get_company_overview.invoke({"stock": "BBRI"})
out2 = get_top_companies_ranked.invoke({"dimension": "dividend_yield"})
print(out)
print(out2)

Notice that at this point we are not using the ReAct agent yet. In fact, we don’t even have a Language Model (neither Llama3.1 nor GPT-4) to work with. Here is the result of the above code:

{"symbol": "BBRI.JK", "company_name": "PT Bank Rakyat Indonesia (Persero) Tbk", 
"overview": {"listing_board": "Main", 
        "industry": "Banks", 
        "sub_industry": "Banks", 
        "sector": "Financials", 
        "sub_sector": "Banks", 
        "market_cap": 732761686016000, 
        "market_cap_rank": 4, 
        "address": "Gedung BRI I Lantai 20\r\nJl. Jenderal Sudirman Kav.44-46, Jakarta Pusat 10210", 
        "employee_num": 80257, 
        "listing_date": "2003-11-10", 
        "website": "www.bri.co.id", 
        "phone": "021 - 575 1966", 
        "email": "humas@bri.co.id; corsec@bri.co.id", 
        "last_close_price": 4860, 
        "latest_close_date": "2024-10-04", 
        "daily_close_change": -0.0121951219512195}}

and:

{"dividend_yield": 
    [{"symbol": "ITMG.JK", "company_name": "Indo Tambangraya Megah Tbk", "dividend_yield": 0.114533205004812}, 
    {"symbol": "BJBR.JK", "company_name": "Bank Pembangunan Daerah Jawa Barat dan Banten Tbk", "dividend_yield": 0.0984974124888682}, 
    {"symbol": "BJTM.JK", "company_name": "Bank Pembangunan Daerah Jawa Timur Tbk", "dividend_yield": 0.0962654856453955}, 
    {"symbol": "ADMF.JK", "company_name": "Adira Dinamika Multi Finance Tbk", "dividend_yield": 0.0865924276169265}, 
    {"symbol": "MCOL.JK", "company_name": "PT Prima Andalan Mandiri Tbk", "dividend_yield": 0.0771300448430493}]}

The output gives us some assurance that the tool functions are working as expected. We are calling each of the tool with the appropriate parameters, allowing each tool to then fetch the data from Sectors Financial API through our utility function.

At this time of writing, Indo Tambangraya Megah Tbk (ITMG.JK) has the highest dividend yield among the companies listed on IDX (yield of 11.45%), followed by 3 companies in the finance sector each with a yield of around 8.5% to 9.8%. Rounding off the list is PT Prima Andalan Mandari — owner of the Mandiri Coal brand — with a dividend yield of 7.71%.

​
Bringing in our LLM

Now that we have our tools in place, let us also create a prompt object and bind our tools to a LLM model of our choice. Instead of invoking each tool directly like we did earlier, we will create a runnable that chains the prompt with the tool-use LLM model.

Our expectation is to be able to prompt the runnable with something like “overview of BBRI” and have the agent invoke the correct tool — in this case, get_company_overview — along with the correct parameters for us.

from langchain_core.prompts import ChatPromptTemplate
from langchain_groq import ChatGroq

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            f"""Answer the following queries factually. Whenever you return a 
            list of names, return also the corresponding values for each name. 
            Always answer in markdown table if you can.
            """,
        ),
        ("human", "{text}"),
    ]
)

llm = ChatGroq(
    temperature=0,
    model_name="llama3-groq-70b-8192-tool-use-preview",
    groq_api_key=GROQ_API_KEY,
)
tools = [
    get_company_overview,
    get_top_companies_ranked,
]

llm_with_tools = llm.bind_tools(tools)

runnable = prompt | llm_with_tools
out_with_runnable = runnable.invoke("overview of BBRI")
print(out_with_runnable.tool_calls)

# output:
# [{'name': 'get_company_overview', 'args': {'stock': 'BBRI'}, 'id': 'call_9dp1', 'type': 'tool_call'}]

What we observe, is that the runnable has correctly identified the tool to call, and the parameters to pass to the tool. If the query require more than one tool, we will see this being reflected in the list of .tool_calls.

At this point, you might be tempted to chain the output of tool_calls to further runnables, thus actually calling the API for information retrieval and then using structuring the output into the desired format. However, LangChain provides some utility functions to make this process easier. Recall from chapter 2 we have the AgentExecutor class that helps us orchestrate the tools and the LLM model:

from langchain.agents import create_tool_calling_agent, AgentExecutor

agent = create_tool_calling_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools)

query_1 = "What are the top 3 companies by transaction volume over the last 7 days?"
query_2 = "Based on the closing prices of BBCA between 1st and 30th of June 2024, are we seeing an uptrend or downtrend? Try to explain why."
query_3 = "What is the company with the largest market cap between BBCA and BREN? For said company, retrieve the email, phone number, listing date and website for further research."
query_4 = "What is the performance of GOTO (symbol: GOTO) since its IPO listing?"
query_5 = "If i had invested into GOTO vs BREN on their respective IPO listing date, which one would have given me a better return over a 90 day horizon?"

queries = [query_1, query_2, query_3, query_4, query_5]

for query in queries:
    print("Question:", query)
    result = agent_executor.invoke({"input": query})
    print("Answer:", "\n", result["output"], "\n\n======\n\n")

For the following sections, I will be using of LangGraph’s ReAct agent instead, as it is also now the recommended way to create agents in LangChain. The AgentExecutor class is still available, but its official documentation now recommends the use of ReAct agents instead.

​
LangGraph pre-built ReAct agent

Source: ReAct: Synergizing Reasoning and Acting in Language Models

An in-depth discussion of the ReAct paper is beyond the scope of this article, I have linked to the papers in the references section1,2 for interested readers. The authors posit that large language models (LLMs) research has been focused either on reasoning (e.g. chain-of-thought prompting) or acting (e.g. action plan generation), but not together.

Its own abstract reads:

We explore the use of LLMs to generate both reasoning traces and task-specific actions in an interleaved manner, allowing for greater synergy between the two: reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it to interface with external sources, such as knowledge bases or environments, to gather additional information. ReAct outperforms imitation and reinforcement learning methods by an absolute success rate of 34% and 10% respectively, while being prompted with only one or two in-context examples.

​
Introduction to pre-built ReAct agents

We’ll start off by importing the create_react_agent function from langgraph.prebuilt. This function requires two arguments:

  • model: The LLM model to use
  • tools: A list of tools to bind to the agent, essentially replacing the llm.bind_tools(tools) set-up we did earlier
  • (optional) state_modifier: A function that modifies the state of the agent. This is useful for adding system messages or other state changes to the agent.

The object returned by create_react_agent is of type <class 'langgraph.graph.state.CompiledStateGraph'> which conveniently also implements the invoke method. The most basic usage of the agent is as follows:

from langgraph.prebuilt import create_react_agent

tools = [
    get_company_overview,
    get_top_companies_ranked,
]

app = create_react_agent(llm, tools)
app.invoke({"messages": "..."})

The .invoke() method takes a dictionary with a key messages and a value that is a string. The output is a dictionary that would contain, among other things, a messages key with a list of HumanMessage and AIMessage objects.

x.invoke({"messages": "..."})

# output:
{'messages': [
    HumanMessage(content='...', additional_kwargs={}, response_metadata={}, id='928afdfd-a566-4890-8352-36d836e73e21'), 
    AIMessage(content="I'm sorry but I do not have the capability to perform this task for you, I am happy to help you with any other queries you may have.", 
    ...)]}

If desired, you can also visualize this agent:

from IPython.display import Image, display
display(Image(app.get_graph().draw_mermaid_png()))
# print(app.get_graph().draw_mermaid()) produces underlying code

​
A Financial Data Agent with ReAct

With the newly equipped knowledge, we can now create a financial data agent that can retrieve companies ranked by a certain metric, as well as providing overviews of companies based on their stock symbols.

Our code is slightly modified from the one in the previous section, with the addition of a state_modifier and another utility function to simplify the invocation of the agent.

from langgraph.prebuilt import create_react_agent

tools = [
    get_company_overview,
    get_top_companies_ranked,
]

system_message = "You are an expert tool calling agent meant for financial data retriever and summarization. Use tools to get the information you need. If you do not know the answer to a question, say so. Whenever possible, answer with markdown-formatted code."

app = create_react_agent(llm, tools, state_modifier=system_message)

# instead of: 
# app.invoke({"messages": "..."})
def query_app(text: str) -> str:
    out = app.invoke(
        {
            "messages": [
                HumanMessage(text),
            ]
        }
    )
    # return out["messages"][-1].content
    return out["messages"]

With the utility function in place, let’s query the agent with a few questions and see how it responds:

out_agent = query_app("Give me an overview of BBRI")

# content of last message from the agent
print(out_agent[-1].content)

# output:
# PT Bank Rakyat Indonesia (Persero) Tbk, listed as BBRI.JK, is a bank operating in the financial sector, 
# specifically in the banks sub-industry. It is headquartered at Gedung BRI I Lantai 20, Jl. Jenderal Sudirman Kav.44-46, Jakarta Pusat 10210. 
# The company has a market capitalization of 741,808,095,100,928, ranking it 4th in the market. 
# It has 80,257 employees and is listed on the Main board of the stock exchange since November 10, 2003. 
# The latest closing price was 4,920 on October 3, 2024, with a daily close change of -0.00404858299595142. 
# You can contact them via email at humas@bri.co.id or corsec@bri.co.id, or by phone at 021 - 575 1966. Their website is www.bri.co.id.

When invoking the tool directly (without the agent, and without any language models), we would have to manually pass the stock symbol to the get_company_overview tool. Even then, the response would be in the original format returned by the API, i.e. a JSON object — making it less readable for the end-user.

With the ReAct agent, we’re now querying in natural language, and getting a far more human-friendly response, generated by the agent.

We can also query the agent for the top 5 companies ranked by a certain metric:

query = "Top 5 companies with the highest market cap, along with their full company name and market cap"

out_agent2 = query_app(query)

print(out_agent2[-1].content)

# output:
# The top 5 companies with the highest market cap are as follows:
# 1. **PT Bank Central Asia Tbk.** with a market cap of 1285141859139584
# 2. **PT Barito Renewables Energy Tbk.** with a market cap of 916434083381248
# 3. **PT Chandra Asri Pacific Tbk.** with a market cap of 763463991296000
# 4. **PT Bank Rakyat Indonesia (Persero) Tbk** with a market cap of 741808095100928
# 5. **PT Amman Mineral Internasional Tbk.** with a market cap of 654476713132032

We observe that the agent is able to understand the query, and the instructions in state_modifier were also being followed as the agent responded with a markdown-formatted answer (ordered lists, along with bolded text syntax).

​
Streaming the agent’s response

The last section of this chapter will demonstrate the idea of streaming. Streaming is a concept where we want to make our AI feels a bit more human-like and responsive, by having it to stream its response in chunks, with our program yielding each chunk as soon as they are available. This is especially true when the agent can take a long time to process the query, or when the underlying large language models are slow to respond.

For the purpose of this demonstration, we will be using the same query as the one above. Notice that we are printing the the response as they come in, rather than waiting for the entire response to be generated before displaying it.

for chunk in app.stream(
    {
        "messages": [
            HumanMessage(
                content="Top 5 companies with the highest market cap, along with their full company name and market cap"
            )
        ]
    }
):
    print(chunk)
    print("# ----")

On a given runnable,

  • Calling .invoke() gets us a final response;
  • Calling .stream() let us stream messages as they occur (useful if agent takes a while, i.e. multiple steps)

The output of the streaming code will be similar to the one below:

{'agent': {'messages': [
    AIMessage(content='', 
        ...,
        tool_calls=[{'name': 'get_top_companies_ranked', 'args': {'dimension': 'market_cap'} ... }],
        ),
]}}

# ----
{'tools': {'messages': [
    ToolMessage(content='{"market_cap": [{"symbol": "BBCA.JK", "company_name": "PT Bank Central Asia Tbk.", "market_cap": 1291305674080256}, {"symbol": "BREN.JK", "company_name": "PT Barito Renewables Energy Tbk.", "market_cap": 929812705181696}, {"symbol": "TPIA.JK", "company_name": "PT Chandra Asri Pacific Tbk", "market_cap": 748324500013056}, {"symbol": "BBRI.JK", "company_name": "PT Bank Rakyat Indonesia (Persero) Tbk", "market_cap": 732761686016000}, {"symbol": "BMRI.JK", "company_name": "PT Bank Mandiri (Persero) Tbk", "market_cap": 648666427686912}]}', 
        name='get_top_companies_ranked', ... 
    )]}}

# ----

{'agent': {'messages': [
    AIMessage(content='Here are the top 5 companies with the highest market cap:\n1. **PT Bank Central Asia Tbk.** - Market Cap: 1,291,305,674,080,256\n2. **PT Barito Renewables Energy Tbk.** - Market Cap: 929,812,705,181,696\n3. **PT Chandra Asri Pacific Tbk** - Market Cap: 748,324,500,013,056\n4. **PT Bank Rakyat Indonesia (Persero) Tbk** - Market Cap: 732,761,686,016,000\n5. **PT Bank Mandiri (Persero) Tbk** - Market Cap: 648,666,427,686,912', 
        ...
    )]}}

flush the buffer!

Disk write and read operations are slow in comparison to RAM-based operations. One way programs can speed up their operations is by batching characters in a RAM buffer before writing them at once to disk, dramatically reducing the number of disk write operations.

The flush=True argument in print forces the buffer to be written to disk immediately, thus flushing the buffer. In the case of streaming responses from our AI agent, this has the benefit of ensuring our users see the output as soon as it is generated, rather than waiting for the buffer to fill up.

​
Streaming JSON and Structured Output

Streaming sounds like a really great idea, but also one that would fail if the output is JSON (or any structured syntax, like xml). If we were to stream a JSON object, and use json.loads (or json.dumps) on the partial JSON, the parsing would fail due to the incomplete syntax.

The solution is to apply the parser on the input stream so that it could attempt to auto-complete the partial json into a valid and complete JSON object.

Here is a simple example of one such implementation:

from langchain_core.output_parsers import JsonOutputParser

model = ChatGroq(model="llama3-8b-8192")
app2 = model | JsonOutputParser()


async def fetch_companies():
    max_companies = 10

    if max_companies > 0:
        async for chunk in app2.astream(
            "generate a list of companies in indonesia and their corresponding ticker in json format. Use a dict with an outer key of 'companies' which contain a list of companies. Each company should have the key 'name' and 'ticker'. Stop at 10 companies."
        ):
            print(chunk, flush=True)
            max_companies -= 1


# Call the async function
asyncio.run(fetch_companies())

And when we observe the output, we will see that the JSON object is being streamed in chunks, and the parser is able to correctly infer a valid JSON object from the partial JSON syntax:

{}
{'companies': []}
{'companies': [{}]}
{'companies': [{'name': ''}]}
{'companies': [{'name': 'PT'}]}
{'companies': [{'name': 'PT Bank'}]}
{'companies': [{'name': 'PT Bank Central'}]}
{'companies': [{'name': 'PT Bank Central Asia'}]}
{'companies': [{'name': 'PT Bank Central Asia T'}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk'}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': ''}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': 'B'}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': 'BBC'}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': 'BBCA'}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': 'BBCA'}, {}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': 'BBCA'}, {'name': ''}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': 'BBCA'}, {'name': 'PT'}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': 'BBCA'}, {'name': 'PT Bank'}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': 'BBCA'}, {'name': 'PT Bank Mand'}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': 'BBCA'}, {'name': 'PT Bank Mandiri'}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': 'BBCA'}, {'name': 'PT Bank Mandiri T'}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': 'BBCA'}, {'name': 'PT Bank Mandiri Tbk'}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': 'BBCA'}, {'name': 'PT Bank Mandiri Tbk', 'ticker': ''}]}
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': 'BBCA'}, {'name': 'PT Bank Mandiri Tbk', 'ticker': 'BB'}]}
...
{'companies': [{'name': 'PT Bank Central Asia Tbk', 'ticker': 'BCA'}, {'name': 'PT Bank Mandiri Tbk', 'ticker': 'BMRI'}, {'name': 'PT Bank Rakyat Indonesia Tbk', 'ticker': 'BBRI'}, {'name': 'PT Astra International Tbk', 'ticker': 'ASII'}, {'name': 'PT Unilever Indonesia Tbk', 'ticker': 'UNVR'}, {'name': 'PT Indofood Sukses Makmur Tbk', 'ticker': 'INDF'}, {'name': 'PT Indocement Tunggal Prakarsa Tbk', 'ticker': 'INTP'}, {'name': 'PT United Tractors Tbk', 'ticker': 'UNTR'}, {'name': 'PT Aneka Tambang Tbk', 'ticker': 'ANTM'}, {'name': 'PT Telekomunikasi Indonesia Tbk', 'ticker': 'TLKM'}]}

​
Challenge

Earn a Certificate

There is an associated challenge with this chapter. Successful completion of this challenge will earn you a certificate of completion and possibly extra rewards if you’re among the top performers.

Implement a ReAct-style agent using the code you’ve written so far. Use this agent to answer the following questions:

  • True to the spirit of value investing, find the top 7 companies based on their P/E values (lower is better).
  • Issue a second query to get an overview of the fourth company in the list.

You should submit the code that implements the agent, and outputs of both queries in your notebook.

For reference, at the time of this writing, the top 7 companies by P/E values returned by the query are as follows. Depending on the time you attempt this challenge, the companies and their P/E values may differ:

query = "Top 7 companies based on P/E values, along with their full company name and PE values"

# response:
"""
Here are the top 7 companies based on their P/E values:
1. **ABM Investama Tbk** (ABMM.JK) - P/E: 3.09
2. **PT Jasa Marga Tbk** (JSMR.JK) - P/E: 4.43
3. **Pabrik Kertas Tjiwi Kimia Tbk** (TKIM.JK) - P/E: 4.50
4. **Adaro Energy Indonesia Tbk** (ADRO.JK) - P/E: 4.79
5. **Energi Mega Persada Tbk** (ENRG.JK) - P/E: 5.01
6. **PT Jasa Marga Tbk** (JSMR.JK) - P/E: 4.43
7. **Pabrik Kertas Tjiwi Kimia Tbk** (TKIM.JK) - P/E: 4.50
"""

Learn: What is P/E value and how is it used?

If you are unfamiliar with the concept of P/E values and value-investing in general, I have an article that explains the concept in more detail. Giving it a read will help you understand the context of the challenge better.

​
References

  1. ReAct: Synergizing Reasoning and Acting in Language Models (Introduction)
  2. ReAct: Synergizing Reasoning and Acting in Language Models (Paper)
3. Structured Output AIs5. Conversational Memory AI Agents