[2206.08853] MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Computer Science > Machine Learning
arXiv:2206.08853
(cs)
[Submitted on 17 Jun 2022 (
v1
), last revised 22 Nov 2022 (this version, v2)]
Title:
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Authors:
Linxi Fan
Guanzhi Wang
Yunfan Jiang
Ajay Mandlekar
Yuncong Yang
Haoyi Zhu
Andrew Tang
De-An Huang
Yuke Zhu
Anima Anandkumar
View a PDF of the paper titled MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge, by Linxi Fan and 9 other authors
View PDF
Abstract:
Autonomous agents have made great strides in specialist domains like Atari games and Go. However, they typically learn tabula rasa in isolated environments with limited and manually conceived objectives, thus failing to generalize across a wide spectrum of tasks and capabilities. Inspired by how humans continually learn and adapt in the open world, we advocate a trinity of ingredients for building generalist agents: 1) an environment that supports a multitude of tasks and goals, 2) a large-scale database of multimodal knowledge, and 3) a flexible and scalable agent architecture. We introduce MineDojo, a new framework built on the popular Minecraft game that features a simulation suite with thousands of diverse open-ended tasks and an internet-scale knowledge base with Minecraft videos, tutorials, wiki pages, and forum discussions. Using MineDojo's data, we propose a novel agent learning algorithm that leverages large pre-trained video-language models as a learned reward function. Our agent is able to solve a variety of open-ended tasks specified in free-form language without any manually designed dense shaping reward. We open-source the simulation suite, knowledge bases, algorithm implementation, and pretrained models (
this https URL
) to promote research towards the goal of generally capable embodied agents.
Comments:
Outstanding Paper Award at NeurIPS 2022. Project website:
this https URL
Subjects:
Machine Learning (cs.LG)
; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
Cite as:
arXiv:2206.08853
[cs.LG]
(or
arXiv:2206.08853v2
[cs.LG]
for this version)
arXiv-issued DOI via DataCite
Submission history
From: Linxi Fan [
view email
[v1]
Fri, 17 Jun 2022 15:53:05 UTC (7,675 KB)
[v2]
Tue, 22 Nov 2022 07:59:47 UTC (9,526 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge, by Linxi Fan and 9 other authors
View PDF
TeX Source
view license
Current browse context:
cs.LG
< prev
next >
new
recent
2022-06
Change to browse by:
cs
cs.AI
cs.CL
cs.CV
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
1 blog link
what is this?
export BibTeX citation
Loading...
BibTeX formatted citation
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer
What is the Explorer?
Connected Papers Toggle
Connected Papers
What is Connected Papers?
Litmaps Toggle
Litmaps
What is Litmaps?
scite.ai Toggle
scite Smart Citations
What are Smart Citations?
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv
What is alphaXiv?
Links to Code Toggle
CatalyzeX Code Finder for Papers
What is CatalyzeX?
DagsHub Toggle
DagsHub
What is DagsHub?
GotitPub Toggle
Gotit.pub
What is GotitPub?
Huggingface Toggle
Hugging Face
What is Huggingface?
Links to Code Toggle
Papers with Code
What is Papers with Code?
ScienceCast Toggle
ScienceCast
What is ScienceCast?
Demos
Demos
Replicate Toggle
Replicate
What is Replicate?
Spaces Toggle
Hugging Face Spaces
What is Spaces?
Spaces Toggle
TXYZ.AI
What is TXYZ.AI?
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower
What are Influence Flowers?
Core recommender toggle
CORE Recommender
What is CORE?
IArxiv recommender toggle
IArxiv Recommender
What is IArxiv?
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs
Which authors of this paper are endorsers?
Disable MathJax
What is MathJax?