Continue Analysis on Generated Charts¶

While TableGPT2 excels in data analysis tasks, it currently lacks built-in support for visual modalities. Many data analysis tasks involve visualization, so to address this limitation, we provide an interface for integrating your own Visual Language Model (VLM) plugin.

When the agent performs a visualization task—typically using matplotlib.pyplot.show—the VLM will take over from the LLM, offering a more nuanced summarization of the visualization. This approach avoids the common pitfalls of LLMs in visualization tasks, which often either state, "I have plotted the data," or hallucinating the content of the plot.

We continue using the agent from the previous section to perform a data visualization task and observe its final output.

NOTE Before you start, you can install Chinese fonts using the following command:

apt-get update && apt-get install -y --no-install-recommends fonts-noto-cjk
mplfonts init

In [9]:

from datetime import date

# Define the human message that asks the model to draw a pie chart based on gender data
human_message = HumanMessage(content="Draw a pie chart based on gender and the number of people of each gender.")

async for event in agent.astream_events(
    input={
        "messages": [human_message],
        "parent_id": "some-parent-id2",
        "date": date.today(),
    },
    version="v2",
    # We configure the same thread_id to use checkpoints to retrieve the memory of the last run.
    config={"configurable": {"thread_id": "some-thread-id"}},
):
    evt = event["event"]
    if evt == "on_chat_model_end":
        print(event["data"]["output"])
    elif event["name"] == "tools" and evt == "on_chain_stream":
        for lc_msg in event["data"]["chunk"]["messages"]:
            print(lc_msg)
    else:
        # Handle other events here
        pass

from datetime import date # Define the human message that asks the model to draw a pie chart based on gender data human_message = HumanMessage(content="Draw a pie chart based on gender and the number of people of each gender.") async for event in agent.astream_events( input={ "messages": [human_message], "parent_id": "some-parent-id2", "date": date.today(), }, version="v2", # We configure the same thread_id to use checkpoints to retrieve the memory of the last run. config={"configurable": {"thread_id": "some-thread-id"}}, ): evt = event["event"] if evt == "on_chat_model_end": print(event["data"]["output"]) elif event["name"] == "tools" and evt == "on_chain_stream": for lc_msg in event["data"]["chunk"]["messages"]: print(lc_msg) else: # Handle other events here pass

content="好的，我将基于性别绘制一个饼图，以展示每个性别的人数。首先，我们需要统计每个性别的人数，然后使用 `seaborn` 和 `matplotlib` 来绘制饼图。\n\n```python\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# Count the number of people for each gender\ngender_counts = df['Sex'].value_counts()\n\n# Create a pie chart\nplt.figure(figsize=(8, 6))\nplt.pie(gender_counts, labels=gender_counts.index, autopct='%1.1f%%', startangle=140, colors=sns.color_palette('pastel'))\nplt.title('Gender Distribution')\nplt.show()\n```" additional_kwargs={} response_metadata={'finish_reason': 'stop', 'model_name': 'TableGPT2-7B'} id='run-6115fe22-3b55-4d85-be09-6c31a59736f6'
content=[{'type': 'text', 'text': '```pycon\n<Figure size 800x600 with 1 Axes>\n```'}, {'type': 'image_url', 'image_url': {'url': 'data:image/png;base64,iVBORw0KG...'}}] name='python' id='226ba8f2-29a7-4706-9178-8cb5b4062488' tool_call_id='03eb1113-6aed-4e0a-a3c0-4cc0043a55ee' artifact=[]
content='饼图已经成功生成。' additional_kwargs={} response_metadata={'finish_reason': 'stop', 'model_name': 'TableGPT2-7B'} id='run-83468bd1-9451-4c78-91a3-b0f96ffa169a'

Now let's set up the Visual Language Model (VLM) and create a new agent with VLM support:

In [10]:

# Initialize the VLM instance
vlm = ChatOpenAI(openai_api_base="YOUR_VLM_URL", openai_api_key="whatever", model_name="YOUR_MODEL_NAME")

# Assume llm, pybox_manager, and memory_saver are defined elsewhere
agent_with_vlm = create_tablegpt_graph(
    llm=llm,
    pybox_manager=pybox_manager,
    vlm=vlm,
    checkpointer=checkpointer,
    session_id="some-session-id",
)

# Initialize the VLM instance vlm = ChatOpenAI(openai_api_base="YOUR_VLM_URL", openai_api_key="whatever", model_name="YOUR_MODEL_NAME") # Assume llm, pybox_manager, and memory_saver are defined elsewhere agent_with_vlm = create_tablegpt_graph( llm=llm, pybox_manager=pybox_manager, vlm=vlm, checkpointer=checkpointer, session_id="some-session-id", )

We use a time travel feature to go back to before the last time the agent gave an answer, to avoid past memories hallucinating the model:

In [11]:

state_history = agent.get_state_history(config={"configurable": {"thread_id": "some-thread-id"}})

to_replay = None
for state in list(state_history)[::-1]:
    if state.next and state.next[0] == "__start__":
        to_replay = state

state_history = agent.get_state_history(config={"configurable": {"thread_id": "some-thread-id"}}) to_replay = None for state in list(state_history)[::-1]: if state.next and state.next[0] == "__start__": to_replay = state

Send the same question to the model via the new agent with VLM support

In [12]:

async for event in agent_with_vlm.astream_events(
    None,
    to_replay.config,
    version="v2",
):
    evt = event["event"]
    if evt == "on_chat_model_end":
        print(event["data"]["output"])
    elif event["name"] == "tools" and evt == "on_chain_stream":
        for lc_msg in event["data"]["chunk"]["messages"]:
            print(lc_msg)
    else:
        # Handle other events here
        pass

async for event in agent_with_vlm.astream_events( None, to_replay.config, version="v2", ): evt = event["event"] if evt == "on_chat_model_end": print(event["data"]["output"]) elif event["name"] == "tools" and evt == "on_chain_stream": for lc_msg in event["data"]["chunk"]["messages"]: print(lc_msg) else: # Handle other events here pass

content="好的，我将绘制一个饼图来展示数据集中男性和女性乘客的数量。\n```python\n# Count the number of passengers by gender\ngender_counts = df['Sex'].value_counts()\n\n# Plot a pie chart\nplt.figure(figsize=(8, 6))\nplt.pie(gender_counts, labels=gender_counts.index, autopct='%1.1f%%', startangle=140)\nplt.title('Gender Distribution')\nplt.show()\n```\n" additional_kwargs={} response_metadata={'finish_reason': 'stop', 'model_name': 'TableGPT2-7B'} id='run-2d05b2ab-32f4-481f-8fa5-43c78515d9c3'
content=[{'type': 'text', 'text': '```pycon\n<Figure size 800x600 with 1 Axes>\n```'}, {'type': 'image_url', 'image_url': {'url': 'data:image/png;base64,iVBORw0K...'}}] name='python' id='51a99935-b0b1-496d-9a45-c1f318104773' tool_call_id='918d57ee-7362-4e0d-8d66-64b7e57ecaf6' artifact=[]
content='饼图显示数据集中性别分布为 50% 女性和 50% 男性，这表明男性和女性乘客数量相等。' additional_kwargs={} response_metadata={'finish_reason': 'stop', 'model_name': 'qwen2-vl-7b-instruct'} id='run-d9b0e891-f03c-40c8-8474-9fef7511c40b'

We observe that the answer provided by the agent with VLM support is significantly more detailed, including a comprehensive description of the generated images.