Our community never fails to impress with clever, creative, and downright fun builds—and this one might just take the cake (block?). Recently, we’ve been talking about how Stream Deck is showing up in more places than ever: with Stream Deck Modules and Virtual Stream Deck on your computer. But now? It’s in Minecraft.
Yep, that’s right. A fully functional Stream Deck—inside Minecraft.
This project was built by community member TheAppleFreak, who turned a clever concept into an actual working setup. Here’s how they made it happen, in their own words:
Inspiration
About a month ago, an Elgato community specialist posted a Minecraft build of a 15-key Stream Deck to the Elgato Twitter. Possessed by the spirit of cordial one-upsmanship, I began trying to figure out how to turn this concept into an actual functional reality. While what I made isn't perfect by any stretch of the imagination, and does have a tendency to break if you stare at it the wrong way, I'm more than pleased with the results!
This build is powered by a six key Virtual Stream Deck hidden surreptitiously off on my other monitor, with a very hacky Node.js script handling the hard parts of pressing SD buttons based on what's activated in game, as well as converting the VSD display into something that can be represented in Minecraft.
What’s happening in Minecraft
On the Minecraft side, the redstone involved is extraordinarily simple. Behind the screen, every button uses a pair of calibrated sculk sensors (in-game sound sensors) to monitor for input events, which in this case are either snowballs landing on the button (signal strength 2) or the sound of a wind charge exploding (signal strength 15). Each pair leads into a single command block that, when triggered, puts between 1 to 8 spaces in the in-game chat. The page back/forward buttons off to the sides also have a tiny bit of extra circuitry that controls the indicator lights, but otherwise function identically.
What’s happening in my custom script
The bulk of the magic happens in the custom Node.js script I wrote for this project. In the initial program startup, I get some special information that lets me select either a Virtual Stream Deck (VSD) window or the Minecraft game window, then also get the coordinates of the VSD window. Once all the setup is done, there’s two loops that run in tandem with each other: input and display.
The input half of the program is the simpler of the two to reason about. Whenever Minecraft runs, it keeps a log file of most important things that happen in a text file aptly named latest.log. Importantly, this log file also includes any messages received in chat, which means with a little bit of clever code we can look at only the log entries that contain our spaces-only messages. When such a message arrives, we count the number of spaces and use that to determine which button to press. I ordered the buttons top-to-bottom, left-to-right (there’s also two additional buttons for page back/forward). Once we get the button selected, I use a specialized desktop automation library to very quickly move the mouse out of the Minecraft window, click the corresponding button on the Virtual Stream Deck powering everything, and return it back to Minecraft’s control (you have to press F3+P to make this work; otherwise, it’ll pause when the game loses focus). If you’re not moving the mouse when these moves happen, it’s pretty seamless! if you are, it tends to break horribly lol
That handles pressing buttons fine, but the real magic of the Stream Deck is that you have the programmable display under each key. How do you show that within Minecraft? The solution that I chose is perhaps not the most elegant possible, but it works, which is what I care about! Four times a second, the script takes a screenshot of the VSD window. Assuming the buttons are at a specific size (the default size of 52x52 px works quite well), it crops the screenshot to only the size of a button, then resizes it to 8x8 px large. Once that’s done, it goes over each pixel in the cropped image and runs them through some clever code to figure out what block matches its color the closest. (Many thanks to kult0922 on GitHub, who made a website that converts images to Minecraft pixel art and published the source code for it! Without that code, this project likely would have taken a good bit longer to figure out). Just make sure to rinse and repeat for the other five buttons
Getting the list of blocks you’d need is a good step, but getting them into the game is still a problem. If you have a way to run chat commands, you could use the /setblock command a bunch to update each pixel one at a time, but that’s very slow and very inefficient. A better tool to use would be the Structure Block, a tool in the vanilla game that among other things lets you place premade “schematics” of structures into the game world. I figured out how to have my script create schematics in the way that Minecraft was expecting them, and once they get saved in the schematics folder I can issue a chat command that tells the Structure Block in game to look at the correct schematic for the current state of the screen (each individual frame needs to have a unique filename, which complicates things a little bit). Once that’s done, you just have a tiny redstone circuit repeatedly paste the active schematic into the world, and you’ve got a working screen!
Conclusion
At the end of the day, you’ve got a mostly usable Stream Deck, but in Minecraft! While my script is designed primarily to run on my machine only, I don’t see any reason why it can’t be improved to maybe run on multiplayer servers! The original prototype of this project actually ran on an instance of Stream Deck Mobile running in Windows Subsystem for Android, which technically means you could run everything on a completely separate device or in a virtual machine. Whether that’s a good idea is an entirely different question, of course… but don’t let concerns like “practicality” stop you from playing around :)
Fun fact: the Stream Deck logo at the top was made using 24 maps. The logo’s structure is located in the End void, but it’s so big that it stretches nearly from one end of the void to the other.
Check out more of TheAppleFreak’s work:
PRODUCTS IN ARTICLE