FRESH

Hacker News

A General Goal-Conditioned Minecraft Model

42 points by agajews

by agajews

2 subcomments

Hey everyone! I'm Alex, one of the founders of Pantograph. We've spent the last six months building a pretty smart Minecraft model, coming soon to a server near you!
We trained it on about 500k hours of Minecraft videos, and it learned how to fight creepers, build walls and other structures, and explore to find visual goals.
We're considering putting up a public API for larger models like this one, let us know if you'd like to be able to put Pan in your own server :)
What's most interesting about the model isn't the performance that it gets in Minecraft, but how general the method is. When we scale it up, it should be able to act in any kind of video game, as well as robots in the real world (which are really just another video game).

by Onavo

1 subcomments

I disagree with the approach. It's a good approach for limited domain problems, but not for general purpose. Take something like this where you will need to be able to refer to wikis and research and ask questions on Reddit and Discord to optimize playthrough, none of the goal conditioning will be useful.
https://www.feed-the-beast.com/modpacks/125-ftb-evolution
I think a properly fine-tuned VLA with access to tool calls can scale way better.

by xnx

1 subcomments

Reminds me of Google's AI Dreamer that could mine diamonds in Minecraft (https://www.scientificamerican.com/article/google-deepmind-t...) or OpenAI training on video to play Minecraft (https://openai.com/index/vpt/)

by codeulike

1 subcomments

Does it have language skills? I always thought it would be interesting to train a model within minecraft as a sortof proxy for 'embodiment'. You could then try asking it about its experiences. "whats your favourite food?", "How does it feel when you hear a spider?", "how low does your food bar need to go before it feels really urgent?" etc

by nee1r

1 subcomments

how do you pick good goal conditioning images/do you have to hand pick a dataset of good goal images? seems hard if you don't have full context. really cool though!