Top

Published in:

2025 | OriginalPaper | Chapter

ControlLLM: Augment Language Models with Tools by Searching on Graphs

Authors : Zhaoyang Liu, Zeqiang Lai, Zhangwei Gao, Erfei Cui, Ziheng Li, Xizhou Zhu, Lewei Lu, Qifeng Chen, Yu Qiao, Jifeng Dai, Wenhai Wang

Published in: Computer Vision – ECCV 2024

Publisher: Springer Nature Switzerland

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

We present ControlLLM, a novel framework that enables large language models (LLMs) to utilize multi-modal tools for solving complex real-world tasks. Despite the remarkable performance of LLMs, they still struggle with tool invocation due to ambiguous user prompts, inaccurate tool selection and mismatched input arguments. To overcome these challenges, our framework comprises three key components: (1) a task decomposer that breaks down a complex task into clear subtasks with well-defined inputs and outputs; (2) a Thoughts-on-Graph (ToG) paradigm that searches the optimal solution path on a pre-built tool graph, which specifies the parameter and dependency relations among different tools; and (3) an execution engine with a rich toolbox that interprets the solution path and runs the tools efficiently on different computational devices. We evaluate our framework on diverse tasks involving image, audio, and video processing, demonstrating its superior accuracy, efficiency, and versatility compared to existing methods. The code is available at https://github.com/OpenGVLab/ControlLLM.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Bridging the Gap: Studio-Like Avatar Creation from a Monocular Phone Capture

next chapter UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction

Available only for authorised users

https://github.com/Significant-Gravitas/Auto-GPT.

https://huggingface.co/docs/transformers/transformers_agents.

Ahn, M., et al.: Do as I can, not as I say: grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 (2022)

Besta, M., et al.: Graph of Thoughts: Solving Elaborate Problems with Large Language Models (2023)

Brown, T., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020)

Chiang, W.L., et al.: Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality (2023). https://lmsys.org/blog/2023-03-30-vicuna/

Chowdhery, A., et al.: Palm: scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022)

Chung, H.W., et al.: Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022)

Dai, W., et al.: Instructblip: towards general-purpose vision-language models with instruction tuning (2023)

Dong, Q., et al.: A survey for in-context learning. arXiv preprint arXiv:2301.00234 (2022)

Dong, R., et al.: DreamLLM: synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499 (2023)

10.

Gao, Z., et al.: Clova: a closed-loop visual assistant with tool usage and update. arXiv preprint arXiv:2312.10908 (2023)

11.

Gupta, T., Kembhavi, A.: Visual programming: compositional visual reasoning without training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953–14962 (2023)

12.

Hao, S., Liu, T., Wang, Z., Hu, Z.: ToolkenGPT: augmenting frozen language models with massive tools via tool embeddings. arXiv preprint arXiv:2305.11554 (2023)

13.

He, Y., et al.: LLMs meet multimodal generation and editing: a survey. arXiv preprint arXiv:2405.19334 (2024)

14.

Ho, N., Schmid, L., Yun, S.Y.: Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071 (2022)

15.

Jiang, Y., et al.: Effectiveness assessment of recent large vision-language models. arXiv preprint arXiv:2403.04306 (2024)

16.

Koh, J.Y., Fried, D., Salakhutdinov, R.: Generating images with multimodal language models. arXiv preprint arXiv:2305.17216 (2023)

17.

Lai, X., et al..: Lisa: reasoning segmentation via large language model. arXiv preprint arXiv:2308.00692 (2023)

18.

Latombe, J.C.: Robot Motion Planning, vol. 124. Springer, Cham (2012)

19.

Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)

20.

Li, J., Li, D., Xiong, C., Hoi, S.: Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning, pp. 12888–12900. PMLR (2022)

21.

Liu, C., Jiang, X., Ding, H.: Primitivenet: decomposing the global constraints for referring segmentation. Vis. Intell. 2(1), 16 (2024)CrossRef

22.

Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. arXiv preprint arXiv:2304.08485 (2023)

23.

Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: NeurIPS (2023)

24.

Liu, Z., et al.: InternGPT: solving vision-centric tasks by interacting with chatbots beyond language. arXiv preprint arXiv:2305.05662 (2023)

25.

Ma, L., Han, J., Wang, Z., Zhang, D.: CephGPT-4: an interactive multimodal cephalometric measurement and diagnostic system with visual large language model. arXiv preprint arXiv:2307.07518 (2023)

26.

Moon, S., et al.: Anymal: an efficient and scalable any-modality augmented language model. arXiv preprint arXiv:2309.16058 (2023)

27.

Mu, Y., et al.: EmbodiedGPT: vision-language pre-training via embodied chain of thought. arXiv preprint arXiv:2305.15021 (2023)

28.

OpenAI: Chatgpt (Mar 14 version) [large language model]. 6 (2023)

29.

Ouyang, L., et al.: Training language models to follow instructions with human feedback. In: Advances in Neural Information Processing Systems, vol. 35, pp. 27730–27744 (2022)

30.

Parisi, A., Zhao, Y., Fiedel, N.: Talm: tool augmented language models. arXiv preprint arXiv:2205.12255 (2022)

31.

Patil, S.G., Zhang, T., Wang, X., Gonzalez, J.E.: Gorilla: large language model connected with massive APIs. arXiv preprint arXiv:2305.15334 (2023)

32.

Peng, Z., et al.: Kosmos-2: grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 (2023)

33.

Qin, Y., et al.: ToolLLM: facilitating large language models to master 16000+ real-world APIs. arXiv preprint arXiv:2307.16789 (2023)

34.

Rae, J.W., et al.: Scaling language models: methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 (2021)

35.

Schick, T., et al.: Toolformer: language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 (2023)

36.

Shen, Y., Song, K., Tan, X., Li, D., Lu, W., Zhuang, Y.: HuggingGPT: solving AI tasks with ChatGPT and its friends in huggingface. arXiv preprint arXiv:2303.17580 (2023)

37.

Silver, D., et al.: Mastering the game of go without human knowledge. Nature 550(7676), 354–359 (2017)CrossRef

38.

Surís, D., Menon, S., Vondrick, C.: ViperGPT: visual inference via python execution for reasoning. In: Proceedings of IEEE International Conference on Computer Vision (ICCV) (2023)

39.

Tang, Q., Deng, Z., Lin, H., Han, X., Liang, Q., Sun, L.: Toolalpaca: generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301 (2023)

40.

Touvron, H., et al.: Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

41.

Touvron, H., et al.: Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

42.

Vemprala, S., Bonatti, R., Bucker, A., Kapoor, A.: ChatGPT for robotics: design principles and model abilities. Microsoft Auton. Syst. Robot. Res. 2, 20 (2023)

43.

Wang, W., et al.: VisionLLM: large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305.11175 (2023)

44.

Wang, X., et al.: Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)

45.

Wei, J., et al.: Chain-of-thought prompting elicits reasoning in large language models. In: Advances in Neural Information Processing Systems, vol. 35, pp. 24824–24837 (2022)

46.

Weiss, S.M., Kulikowski, C.A., Amarel, S., Safir, A.: A model-based method for computer-aided medical decision-making. Artif. Intell. 11(1–2), 145–172 (1978)CrossRef

47.

Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., Duan, N.: Visual ChatGPT: talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671 (2023)

48.

Wu, S., Fei, H., Qu, L., Ji, W., Chua, T.S.: Next-GPT: any-to-any multimodal LLM. arXiv preprint arXiv:2309.05519 (2023)

49.

Yang, R., et al.: GPT4tools: teaching large language model to use tools via self-instruction (2023)

50.

Yao, S., et al.: Tree of thoughts: deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601 (2023)

51.

Yao, S., et al.: React: synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022)

52.

Yao, Y., Li, Z., Zhao, H.: Beyond chain-of-thought, effective graph-of-thought reasoning in large language models. arXiv preprint arXiv:2305.16582 (2023)

53.

You, H., et al.: Ferret: refer and ground anything anywhere at any granularity (2023)

54.

Zeng, A., et al.: GLM-130b: an open bilingual pre-trained model. arXiv preprint arXiv:2210.02414 (2022)

55.

Zhang, R., et al.: Llama-adapter: efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 (2023)

56.

Zhang, S., et al.: GPT4RoI: instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601 (2023)

57.

Zheng, K., He, X., Wang, X.E.: MiniGPT-5: interleaved vision-and-language generation via generative vokens. arXiv preprint arXiv:2310.02239 (2023)

58.

Zhou, D., et al.: Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625 (2022)

Title: ControlLLM: Augment Language Models with Tools by Searching on Graphs
Authors: Zhaoyang Liu
Zeqiang Lai
Zhangwei Gao
Erfei Cui
Ziheng Li
Xizhou Zhu
Lewei Lu
Qifeng Chen
Yu Qiao
Jifeng Dai
Wenhai Wang
Publisher: Springer Nature Switzerland
Book: Computer Vision – ECCV 2024
Print ISBN: 978-3-031-73253-9

Electronic ISBN: 978-3-031-73254-6

Copyright Year: 2025
DOI: https://doi.org/10.1007/978-3-031-73254-6_6

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"