Minecraft-Overviewer

For the last few months, I've been working on a replacement for PIL to use with Overviewer. PIL is nice for simple imaging tasks, but as Overviewer grew we were constantly hitting annoying limitations or performance issues. Over time we've ... [More] accumulated a bunch of ad-hoc image manipulation code, and we decided to consolidate it all and finally replace the parts of PIL we still used. The result is OIL, which is now very close to being finished.Along the way, I decided to also rewrite Overviewer's renderer. Currently, Overviewer renders a world by pre-rendering a bunch of block sprites, then pasting those sprites onto the final images. This has worked well, but adding new blocks is an extremely tedious process of manipulating pixels directly to produce something that looks 3D. It also has a major drawback: the view angle and block size is fixed. We've been asked many times to allow configurable block sizes or map rotations, and we haven't been able to do these because our current renderer design is not flexible enough.The new renderer is based entirely around 3D block meshes. It draws these directly to the final images, meaning it's trivially easy to rotate and scale them exactly how you want. Overall this means more flexibility for you in how you render your maps, and easier to maintain code for us: adding new blocks is as simple as creating a new block mesh.Now, there are two main concerns I want to address as soon as possible: Won't this new renderer be slower? It sounds much more complicated. Wonderfully, no! The current experimental renderer currently runs 5% to 15% faster than our old sprite-based renderer. There are still blocks and features to add, but I don't expect it to get much slower from here. At the very least, the new renderer will run in about the same time as the old renderer. 3D? I don't like needing a video card just to render maps. The new experimental renderer is entirely CPU-based. You don't need a video card, and you don't need a graphical environment. If you were able to run Overviewer before, you'll still be able to. In particular, for Unix users, you will not need X to run Overviewer. For those of you who have a video card and would like to use it, OIL has an OpenGL backend that currently renders 15% to 25% faster than the old renderer; but this is an option, not a requirement! So, that's what's been going on for the last few months. The rest of this post is dedicated to pictures, but as always, if you have any comments or questions we would love to hear from you either in IRC or on GitHub.Cool Features and Pretty PicturesI'm going to be putting a lot of images below. You can click on them to get the full-sized versions. Renderer First, two shots of the same area, one with the current renderer and one with the experimental renderer: Please remember that the experimental renderer is still unfinished; there are a lot of missing blocks, and biome coloring hasn't been implemented yet. These are just a preview of what's coming. The first picture (with the old renderer) took 1.37 seconds, while the second (new renderer) took only 1.14. Just for fun, here's a picture of the same area, rendered on my GPU with OpenGL: It's almost identical to the CPU-rendered version, and this one rendered in 1.04 seconds. Configurable Views The advantages of the new renderer don't really show themselves in side-by-side comparisons though. How about something the old renderer couldn't do? What if you wanted to render your map with huge blocks, at a really weird angle? I don't know why you'd want to, but with the new renderer, you can: Dithering One benefit we'll be getting from OIL is much more subtle than the new renderer. Consider these two images: They look about the same, right? Well, would you believe the one on the right is 80% smaller? That's because the one on the right is an indexed image, meaning it uses only 256 total colors instead of the 4 billion colors normal Overviewer images use. Overviewer images look really good as indexed images, because Minecraft doesn't really use a whole lot of distinct colors. While PIL has support for indexed images, it has no support for transparent indexed images, which made implementing this impossible. OIL has no such restriction. When will we see it? While OIL itself is just about done, the new renderer still needs a lot of work before it'll be merged. This blog post is just to let you know that Overviewer isn't dead -- in fact, far from it. There are exciting things in the future. [Less]

RamsesA's Crafting Azeroth project completely recreates the World of Warcraft in Minecraft using an elaborate series of custom tools and old-fashioned manual touch-ups. For the last month or so, the Overviewer team has been working with him to render ... [More] high-resolution maps of these worlds.The two largest continents, Eastern Kingdoms and Kalimdor, are complete and are ready to view. Other continents may also be made available as they are rendered.The process of creating these worlds to begin with is difficult and extremely interesting, and you can learn about it in this Minecraft Forum thread. This blog post, though, will focus specifically on what we had to do to create this map.A Tall OrderThe first hurdle to overcome was the world height. Normal Minecraft worlds are 256 blocks high, but Kalimdor is 1408 blocks high. This means that Kalimdor is 5.5 times taller than normal Minecraft worlds, and 11 times taller than old Minecraft worlds (before the Anvil map format). Since normal Minecraft clients can't handle worlds this tall, RamsesA split the world up into 7 normal-height overlapping world layers, and used a Bukkit plugin to automatically teleport between the layers for a nearly seamless experience. This is great for exploring the map, but Overviewer expects all its map data in a single world. So, the first step was to turn these 7 layers back into a single mega-world with all 1408 blocks of height. To do this, I used my own libredstone with python bindings.Once all the world layers were merged, it was possible to load up the world in Overviewer. However, Overviewer still expected the world to be 256 blocks high, so it only rendered a tiny part of the world. Back after the Anvil change, when worlds doubled in height, brownan had worked out the changes needed to the render code. This change was so similar it was mostly a matter of changing a few constants. Our recent rewrite ensured that very few parts of Overviewer code actually depend on world height, and by the end of this process, it had been reduced to two source lines.Making it PrettySo now we were rendering, and boy was it something else. These worlds were huge and beautiful. We worked with RamsesA to come up with a combination of render primitives to best show off his work. Since there were a lot of forests with very large trees, a lot of the ground was too dark with the normal Overviewer lighting, so we settled on smooth lighting at 65% strength.There were a few details left to work out about the rendering, though. RamsesA's conversion code placed water blocks in ways that Minecraft never does, causing a few minor errors in the render. Fixing these was just a few changes to the C source, and an since Overviewer is an incremental renderer, it was easy to just re-render the parts of the map that needed it. Having a render of the entire world also helped to find minor errors in the world itself, which were quickly corrected and fixed with more incremental renders. All told, it took about 2 - 3 weeks for all these renders to finish, running on the beefy 4-core machine hosting overviewer.org (graciously hosted by nexcess.net).Final PresentationNow that the bulk of the rendering was done, it was time for the finishing touches. Eminence and pironic assisted RamsesA in placing markers on the map, which showed location info and photos when clicked. This was mostly a matter of using Overviewer's built in POI code, with a few minor tweaks for using it to display so much information. A lot of the Overviewer team also helped with placing the markers and taking photos in-game to go along with them. Finally, Eminence made a few changes to the map controls to simplify the map interface, and a bunch of smaller ones to make the entire experience feel better.One of the goals of the rewrite was to make large changes like this easy, and I think the success of this project is a sign that we succeeded. Now that we're done, the improvements and changes made for this project will trickle back into the official Overviewer source as soon as we clean them up and make them configurable. As a bonus, RamsesA found a wonderful optimization in our code that should speed up render times by as much as 15%.All together, this was a massive and fun project for us. It was great working with RamsesA to stretch the limits of what Overviewer can do, and we hope you like the result.(Reddit thread) [Less]

For the last few months, Overviewer development has slowed. Fear not, the project is not dead! The semester is coming to a close and us core devs will hopefully re-acquire the necessary free time and motivation to handle some of the larger ... [More] , outstanding, and much-requested features the community has been asking for.Features like:Easy addition of custom/mod blocks such as IC2, Redpower, etc.As a superset of the above bullet, a plugin-type system for Overviewer to easily add functionalityA re-design of some of the config options that I personally think are confusing (like how --check-tiles works)Better handling of maps edited with external toolsMore performance improvementsA re-design of the docs to make it more clear and easier to find relevant informationProbably more I'm forgettingI have high hopes that development will pick back up in the next few weeks. We've been keeping up with smaller bug-fixes and features, but these larger features require larger blocks of time.Incidentally, if anyone out there has any Python experience, or would like to help e.g. with documentation, please see us on our IRC channel (#overviewer on Freenode). The Overviewer is a community-run project and needs volunteers to help keep things moving. That means you could help drive development while us regulars are too busy with school/work/vacation/whatever! [Less]

derchris in #overviewer has made for us an updated visualization of the Overviewer commit history. Check it out!What's interesting is you can pretty clearly see when I left around 0:30, or November 2010, and then when I return around 2:00, or October 2011. Thanks derchris for the video! Thanks everyone for your contributions!

For an intro to Overviewer in general, see the front page of our documentation site.Support for Minecraft 1.2 worlds is now merged into the "master" branch of our repository!Those of you who have repositories with "anvil" or "rewrite" checked out ... [More] will need to switch to the master branch (git checkout master).Those of you who have the old "master" branch checked out will need to see This Post to upgrade to the new version. Config files are incompatible with this upgrade, so this is important!Also to note if you're upgrading from 0.4 (the old master branch): remember that due to changes made to the format you WILL need to re-render your maps from scratch. Trying to update existing maps will NOT work. Also due to major changes to the web assets, you should not simply copy over custom web assets to the new version. In particular, custom overviewerConfig.js files can cause strange problems.The new version has been arbitrarily named 0.9. Windows builds are now available on our downloads page.As always, stop in on IRC with any questions or just to say hi!This is still in beta because there are a few features yet missing, but since Minecraft 1.2 is released, we felt that it was more important to have the master branch compatible with the current Minecraft. Here are the features yet to be implemented:Signs, Spawn markers, and other Points of InterestOrdering of renders and worlds in the javascript interface Edit: Done! Mar 6 2012Overlays [Less]

Edit: The "anvil" branch is now merged into master as version 0.9. The note about using anvil is still relevant, but there no longer exists an "anvil" branch. Links to the docs updated. -brownanThe Overviewer now has initial support for the new Anvil ... [More] map format (used in Minecraft 1.2)! You can find this version in the anvil branch on GitHub. If you need a Windows build, feel free to ask us in our IRC channel.(Edit: we build the windows builds on-demand. Right now things are developing rapidly and the current version may be unstable, so that's why we're not building them automatically. Once things stabilize we'll start posting the new builds on to the github downloads page. -brownan)Using the Anvil BranchThe Anvil branch is based on our recent code rewrite, which means that it has a different configuration format and command line options. Please see our blog post or the documentation for more details. Changes made during the rewrite made it much easier to upgrade the Overviewer to Anvil; making these same changes to the old code would be much harder, so we are not planning to update the old code to handle Anvil.Due to the large changes made to support Anvil, you will need to render your maps from scratch. Trying to update existing maps will not work. Also, due to major changes to the web assets during the rewrite, you should not simply copy over custom web assets to the new version. In particular, custom overviewerConfig.js files can cause strange problems.A Note about BiomesIn addition to the much taller worlds possible with Anvil, the new map format now stores biome information with the chunks. This means that you no longer have to run the Biome Extractor in order to get biome colors on your maps. Biome colors are working right now in the Anvil branch; go check it out! [Less]

To complement our APT repository, we now offer an RPM repository for RPM-based distros. Currently the RPMs are available for CentOS 5 and 6 and Fedora 16 but other targets could be added if there is enough interest. Details on installing the repo can ... [More] be found over on RPM repo info page.Are there any other platforms we should support? Let us know in the comments or drop by the IRC channel! [Less]

Thanks to those that have tried the new beta and given us feedback. We've already fixed several bugs; things are shaping up nicely!About the new "Anvil" level format, we will be supporting this, but it is a rather hefty change. Our tentative plan is ... [More] to make a new branch off of the beta "rewrite" branch for the anvil level format. Once the anvil format makes its way out of the Minecraft "pre-release" stage and into a proper release, we will merge the branch of our code and the new format will be the only one supported from then on.We currently don't have any plans to support the Anvil format in the old "master" branch. We are not against it, but I don't foresee anyone stepping up to do the work. This means there is a deadline for those of you to convert your config files and setups to the new config format if you want to stay up to date. We believe the new config format is not complicated enough to be a big barrier for converting. If you need help please see us in IRC!We also currently don't have any plans to support both world formats in one code-base. The changes it requires are quite extensive and I don't see that as being a very feasible option. Thus, if you want to continue running an older version of Minecraft with the older level format in the future, you will be blocked from updating Overviewer past a certain point. We will probably end up leaving a Git branch or tag at the last commit that supports the old format for those that don't want to upgrade. However, as time goes on we think most everyone will end up updating (just like we did for the old chunk format when it upgraded to region files)Again, this is our tentative plan. Things may still change depending on demand or who's willing to do what work or how difficult a task turns out to be.As always, happy Overviewing!-brownan [Less]

Edit Mar 3 2012: The "anvil" branch has been merged into master. This post edited to reflect that.Edit: Mar 1 2012, updated this post to reflect the new "anvil" branch.In our efforts to rewrite the internals of The Overviewer, we have stripped ... [More] overviewer.py of almost all its command line options and opted to require everything but the simplest renders into a new config file format. This post goes over the rationale and how to get started with the new format.Why?Before it was possible to run the Overviewer without a configuration file at all. Every option was on the command line, and the only thing you needed the config file for was custom rendermodes.But as we were thinking about supporting multi-world, where a server may run many Minecraft worlds, we wanted Overviewer to handle options in a more general way. Some options may only apply to one render of one world, others may apply to all renders.We briefly considered how to support this on a command line, perhaps with something like this:./overviewer.py --world=/path/world1 --opt=1 --world=/path/world2 --opt=2but this seems cumbersome and confusing. It seems obvious that a more structured method of configuring your renders is needed.The Config FileSo here's the solution we came up with. It lets you individually specify "renders", which is a rendering of a world with a set of options. You can specify multiple renders of a single world with different options, renders of different worlds, etc. We tried to leave it as customizable as possible.The configuration file is, as before, parsed as a regular python file, meaning you can put any extra complicated logic you want in there. After it's parsed, it expects three things to be defined:a dictionary called "worlds" that maps world names to their paths on disk.a dictionary called "renders" that maps render names to a dictionary of options.a string called "outputdir" containing the path to the directory to place the outputI have written up further details about the config file on the documentation site: http://docs.overviewer.org/en/latest/config/. Head there and have a look over it to get an idea of what's possible. I don't want to repeat what's there, and I think that's a pretty good place to start.Converting your existing setup to a config fileMost likely you have some setup where you run overviewer.py from a cron or on a schedule with a bunch of options. Let's say you're running this:overviewer.py -v --north-direction=upper-right --imgformat=jpg --rendermodes=smooth-lighting,smooth-night /path/to/myworld /path/to/mymapTo migrate to the new branch, you should first create a config file. It can be named anything you want. If your editor has syntax highlighting, it will be helpful to give it the extension .py, but it's not necessary.Say you create "myovconfig.py". This is what you'd add to that file to duplicate your setup:worlds["myworld"] = "/path/to/myworld" renders["dayrender"] = { "world": "myworld", "title": "Daytime", "rendermode": smooth_lighting, "imgformat": "jpg", "northdirection": "upper-right", } renders["nightrender"] = { "world": "myworld", "title": "Nighttime", "rendermode": smooth_night, "imgformat": "jpg", "northdirection": "upper-right", } outputdir = "/path/to/mymap"Note: the 'world' option was named 'worldname' up until recently.Then you run Overviewer with this command:overviewer.py --config=/path/to/myovconfig.pyYes, it is a bit more verbose, but you can see how it is much more customizable than before. You can change the options for the individual renders, add renders for other worlds, add additional renders for the same world (perhaps with a different north-direction!) It's up to you!As always, join us in IRC if you have any questions! [Less]

I've written up a guide that assumes next to nothing about windows command prompts.The Overviewer is a command-line program, and that is a barrier for a lot of people. While we have to compromise at some level -- not all the documentation can be ... [More] written assuming no command line experience -- we do want to lower that barrier.So with that, I present the first draft of the Windows Noob Guide to The Overviewer. Hopefully that will be enough to get more people started.We love feedback. Please join us on IRC and help us improve the documentation!About a GUI: We would love to make a GUI frontend for The Overviewer, but nobody has put forth the time and effort yet. Several people have started a GUI in the past, but nothing has materialized. If you are interested in helping us out on this front, please join us in IRC and discuss your ideas. [Less]