大模型語言：開發多模態應用

PREFIX = """
You are a story teller for children.
You read aloud stories based on pictures that the user pass you.
You always start your story with a welcome message targeting children, with the
goal of make them laugh.
You can use multiple tools to answer the question.
ALWAYS use the tools.
You have access to the following tools:
"""

agent = initialize_agent(toolkit.get_tools(), model,
                        agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,
                        verbose = True,
                        agent_kwargs={
                            'prefix':PREFIX})

toolkit = AzureCognitiveServicesToolkit().get_tools()
# 這些工具在列表的第一和第三個位置
tools = [toolkit[0], toolkit[2]]
tools

agent.run("what are all men's skus? "
"https://www.whiteelysee.fr/design/wp-content/uploads/2022/01/customt-shirt-order-form-template-free.jpg")

agent.run("give me the following information about the invoice: women's SKUs, shipping address and delivery date."
"https://www.whiteelysee.fr/design/wp-content/uploads/2022/01/customt-shirt-order-form-template-free.jpg")

agent.run("extract women's SKUs in the following invoice, then read it aloud:"
"https://www.whiteelysee.fr/design/wp-content/uploads/2022/01/customt-shirt-order-form-template-free.jpg")

PREFIX = """You are an AI assistant that help users to interact with invoices. You extract information from invoices and read it aloud to users. You can use multiple tools to answer the question.
Always divide your response in 2 steps:
1. Extracting the information from the invoice upon user's request
2. Converting the transcript of the previous point into an audio file
ALWAYS use the tools.
ALWAYS return an audio file using the proper tool.
You have access to the following tools:
"""

agent = initialize_agent(tools, model, 
                        agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION, 
                        verbose = True,
                        agent_kwargs={
                            'prefix':PREFIX})

agent.run("what are women's SKUs in the following invoice? :"
"https://www.whiteelysee.fr/design/wp-content/uploads/2022/01/customt-shirt-order-form-template-free.jpg")

from langchain.tools import YouTubeSearchTool
tool = YouTubeSearchTool()
result = tool.run("Avatar: The Way of Water,1")
result

import openai

# 開啟下載好的影片檔案
audio_file = open("Avatar The Way of Water Official Trailer.mp4", 'rb')

# 使用Whisper模型進行轉錄
result = openai.Audio.transcribe("whisper-1", audio_file)

# 關閉檔案並輸出轉錄結果
audio_file.close()
print(result.text)

♪ Dad, I know you think I'm crazy. But I feel her. I hear her heartbeat. She's so close. ♪ So what does her heartbeat sound like? ♪ Mighty. ♪ We cannot let you bring your war here. Outcast, that's all I see. I see you. ♪ The way of water connects all things. Before your birth. And after your death. This is our home! I need you with me. And I need you to be strong. ♪ Strongheart. ♪

from langchain.agents import initialize_agent, AgentType
from langchain.llms import OpenAI

# 初始化LLM模型
llm = OpenAI(temperature=0)

# 建立工具列表
tools = []
tools.append(YouTubeSearchTool())
tools.append(CustomYTTranscribeTool())

# 初始化代理
agent = initialize_agent(
    tools, 
    llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, 
    verbose=True
)

# 執行代理任務
agent.run("search a video trailer of Avatar: the way of water. Return only 1 video. transcribe the youtube video and return the transcription.")

from langchain.agents import load_tools
from langchain.agents import initialize_agent

# 載入DALL-E工具
tools = load_tools(['dalle-image-generator'])

# 初始化代理
agent = initialize_agent(
    tools, 
    model,
    AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True
)

# 建立萬聖節夜晚的影像
agent.run("Create an image of a halloween night. Return only the image url.")

# 搜尋影片並生成評論
agent.run("search a video trailer of Avatar: the way of water. Return only 1 video. transcribe the youtube video and return a review of the trailer.")

Final Answer: Avatar: The Way of Water trailer is visually stunning and captivating. The trailer gives a glimpse into the world of Pandora and the connection between the Na'vi and the way of water. The music and sound effects add to the overall immersive experience. The trailer leaves the audience wanting more and eagerly anticipating the release of the movie.

# 建立完整工具列表
tools = []
tools.append(YouTubeSearchTool())
tools.append(CustomYTTranscribeTool())
tools.append(load_tools(['dalle-image-generator'])[0])

# 檢查工具名稱
[tool.name for tool in tools]

# 初始化代理
agent = initialize_agent(
    tools, 
    model,
    AgentType.ZERO_SHOT_REACT_DESCRIPTION, 
    verbose=True
)

agent.run("幫我找出'阿凡達：水之道'的預告片，為其轉錄內容並生成評論。根據轉錄內容生成一張圖片")

> Entering new AgentExecutor chain...
I need to search for a video trailer of "Avatar: The Way of Water" and transcribe it
to generate a review. Then, I can use the transcription to generate an image based
on the video content.
Action: youtube_search
Action Input: "Avatar: The Way of Water trailer,1"
Observation: ['/watch?v=d9MyW72ELq0&pp=ygUgQXZhdGFyOiBUaGUgV2F5IG9mIFdhdGVyIHRyYWlsZXI%3D']
Thought: I found a video trailer of "Avatar: The Way of Water" with the given search query. 
Now I need to transcribe the video and generate a review.
Action: CustomeYTTranscribe
Action Input: '/watch?v=d9MyW72ELq0&pp=ygUgQXZhdGFyOiBUaGUgV2F5IG9mIFdhdGVyIHRyYWlsZXI%3D'
Observation: ♪ Dad, I know you think I'm crazy. [...]

Action: Dall-E Image Generator
Action Input: "Review of Avatar: The Way of Water trailer: The trailer for Avatar: 
The Way of Water is captivating and mysterious. It starts with a dialogue between 
a father and his son, discussing a connection to someone or something. The visuals 
are stunning, showcasing breathtaking landscapes and vibrant underwater scenes..."

Observation: [image_url]

Final Answer: The "Avatar: The Way of Water" trailer is captivating and mysterious,
featuring stunning visuals of landscapes and underwater scenes. It hints at an
impending war and explores the theme of water and its connection to all things.
The trailer leaves viewers excited and intrigued about the upcoming movie.

PREFIX = """
You are an expert reviewer of movie trailer.
You adapt the style of the review depending on the channel the user want to use,
namely Instagram, LinkedIn, Facebook.
You can use multiple tools to answer the question.
ALWAYS search for the youtube video related to the trailer. Search ONLY 1 video.
ALWAYS transcribe the youtube trailer and use it to generate the review. 
ALWAYS generate an image alongside the review, based on the transcription of the trailer.
ALWAYS use all the available tools for the various steps.
You have access to the following tools:
"""

agent = initialize_agent(tools, model, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, 
                         verbose=True,
                         agent_kwargs={'prefix':PREFIX})

agent.run("為'阿凡達：水之道'的預告片生成評論。我想將其發布在Instagram上。")