New Code

A dozen classes later and I've injected a mock into my rendering thread so I can interrupt draw calls and simulate almost the entire process. This basically makes everything up to the draw call unit testable and turns my code into the most complicated of bagatelle machines.


Network packet arrives in one end, bounces from pillar to post, draw call comes out the other end!


This brings me a little close to Alpha, but more in a holiday-spirit-playing-with-code sense than the typical slaving over a hot keyboard.  As I don't get paid for developing Pioneer I've been putting it aside for my birthday and then Xmas and have only snatched a couple of half-days this month.


In general the classes are easier to work with, and places the last stepping stone before adding variable level of detail rendering modes and dynamic shadows.  Its also a little faster/smoother when playing and has fixed a really obscure visibility bug that would sometimes leave you with an invisible brick.


The large benefits have been the removal of a cyclic dependency which has meant I can straighten the code out into a directed acyclic graph. All elementary stuff but removing this particular spanner from the works has been kind of a personal quest. The new object graph (and a couple of mocks) has meant I can easily increase the test coverage, which is the next thing I'm going to fiddle with because I was a bit lazy on that front in the last couple of sprints. I'm probably going to see what I can do about cutting those mock objects out of the test harness too, since they add a potential maintenance cost.

Code Mines

This is another no update blogpost, but a real update is getting closer.  A previous code review left me pondering... What if I just did it right?
So I thought I'd practice what I preach and do all of those fancy things that smart programmers dao.

The Pioneer codebase has always been dominated by the Brick class which describes the size, shape, position and colour of a building brick, and the Chunk class which measures 16x16 studs, contains zero to many Bricks and is arranged spatially with other Chunks to form the World.  Almost everything was either in one of those classes, or operated on them.

So the code was largely a bit of a lump. The Brick and Chunk classes became tightly coupled, with the World becoming a resource locator for chunks. A simple shared pointer made it far too easy to pass ChunkPtrs around and hold on to them even though - theoretically - they were short lifespan objects.

While I stuck to reasonable guidelines and practices, it became easy to add lazy features and it wasn't until I wanted to add mesh LoD and runtime LoD switching and discovered that it was a bit tricky. The possible solutions
- expose more members as public, either through access specifiers or get accessors.
- bundle more responsibility onto the already bloated Chunk class
- refactor so that the object graph makes the new feature easy to add.

Both of the least-effort solutions felt pretty bad, but it looked like it should be fairly easy to extract a responsibility FROM the chunk and isolate the rendering in a new object.  Even if it was a RenderableChunk class, it could still steal all of the occlusion, rendering and become a container for that stuff where it should be easy to add that mesh LOD functionality too.

And so the project went through whatever the opposite of suffering is as I pruned back the almighty Chunk. It lost a lot of fat, it also became a Directed Acyclic Graph and I could strip out a now redundant cycle.

Compile times were a little slow so I retypedeffed my shared pointers to incomplete types and made all of their clients rely on the abstract interface instead of the implementation.  This reduced the amount of include files getting chained together and allowed me to hack away in small, fast iterations.

The new acyclic object graph made the test suite simpler and much easier to expand to be more comprehensive. I was never satisfied with my test coverage and having that easy to improve is a load off. Its almost as if the "better" code was also easier to unit test. Astounding!

Programming can be a bit like a new LEGO build. You know what its supposed to be when its finished but you haven't got any instructions so you add and remove bricks, and each iteration will get you incrementally closer to your goal. One brick removed, two bricks added.

There will need to be more time at the keyboard before anything appears, but the last of the big changes has happened. From here it looks like I'll just be putting bricks back together. Maybe the odd one or two will be shifter, but I've solved the problem I wanted to solve and a few more to boot.

Code Quality

I generally believe that code should be easier to read than to write. There aren't many exceptions to this and hard-to-read code is often a signifier that a brief code review might be in order.

Tonight I discovered a class that doesn't do what it says on the tin, so didn't do what I expected it to do and had two responsibilities so boot. So its been split in two, and each half aptly named. In retrospect its a lot of typing and when you finally run the game its *exactly the same* but that's kind of the point of this kind of refactoring. Its made adding new stuff easier and what I've got is easier to read for future revisions.

The clean-up has exposed a juxtaposition of bad habits, itself a more interesting point than the code or class layout in question. The contents of the class was place holder and had been overlooked for quality because it was "only temporary". Another symptom of being left to rot was the suspicious 'todo' comment that had never been followed up.

The experience has highlighted a serious difference between a naive implementation and bad code. Knowing that I was writing an implementation that was going to be thrown away meant I'd turned a blind-eye to a poor choice of interface, and that's where the rot started.

The old lesson relearned... there is NO SUCH THING as throwaway code. As you write, you just don't know how many weeks, months, or years a temporary class is going to last. I could just have easily have looked at it, saw it was ugly and left it alone. Or compounded the problem by just hacking my new code in. But the clean-up has left me with more functionality, less code and more readable, intuitive classes with shallower inheritance.

Sprint 17 Complete

Sprint 17 has landed.

It has the new pantones, a few new bricks and a couple of tweaks. The vertex and fragment shaders have been disabled for now but will be back in a future sprint soon. A lot of graphical tweaks are pencilled in for release 19, while I'm expecting to focus on core building and useability for 18.

Vertex Shader. Yeaj!

And as if by magic, the answer game to me!

Just the act of making a tasty tasty coffee was enough to stimulate the little grey cells. I had a play with vertex and fragment shaders recently but I was only really having a play in the 'play for fun' rather than nerdcore critical-path programming.

One of the shaders I wanted to write was to make my blocks look a little more like physical LEGO bricks. The LEGO company make the plastic construction toy that has been my inspiration and physical building materials have different properties to intangible polygons.

Simulating the look of a real world item with computer graphics is always a fungineering task, but I hit upon the technique I'm going to use to get that extra va va voom out of my graphics pipeline using a vertex shader in the base render pass instead of during post-process or using a second render pass and additional VBO set.

More news as it breaks!

p.s. You can support the LEGO company by buying sets, bricks, toys or apparel from their online or storefront retail outlets. I highly recommend basically everything they sell.

Shader Frenzy

New Cols

I've tweaked the colours tonight. They are brighter, and more complementing. They just go together better really. There are other lighting and shading effects I want to play with, but getting the base colours right was fairly high on the list.

Sprint 16 Complete... 17 is underway

I can't really tell if 16 is complete or not. Its got an (NPE) crash, but that wasn't found until I said 'finished' so fixing the thing is technically the first part of Sprint 17. It feels horribly wrong to say 16 is complete now I know there is a bug though.

Either way, its heads-down on 16, which is checking/asserting some pointers but I don't have a repro so just can't be sure.

The edge of the world...

This is what the edge of the world looks like. The green area was created first, with a Hills world generator, and the grexbox was filled in later after I had explored an area.

I was testing out the new Jetpack, which has an added 'hover' option - this allows you to fly around your constructions with ease. Its almost cheating though, there was something very real about building scaffolding to climb as you place a new row of bricks. Time will tell if its a good thing to have of if it'll fall by the wayside.

New rules

Here we can see the old baseplate edge issue has been squashed. Bricks can span between two baseplates now, which means your constructions can be grander and more detailed than ever before.

I've no idea how well it works, but the initial "this is how to do it" line has been crossed.

There are still CPU spikes when you walk around as baseplates are streamed in and out of memory - and actually they are worse now as overlapping bricks are resolved at the same time.

Building between baseplates was the main task in this sprint, so I'm going to hit the small items and have the next release ready to go as soon as I can.

A couple of other pics from Google+

One of these memory allocations is not like the other.

ChunkRenderer * Alpha = new ChunkRenderer_Debug( NULL );

ChunkRendererPtr Beta = new ChunkRenderer_Debug( NULL );

ChunkRendererPtr Charlie ( new ChunkRenderer_Debug( NULL ) );

Only one of those three is wrong, but only one of those three is right.

A couple of builders have been busy

Sprint 16 on its way

Both on its way and well underway.

I've 99% fixed the crash introduced in 15. It was memory leak, schoolboy error I know... Actually it was two memory leaks. One caused by me relying on the garbage collection sweep working, and the other I'm not so sure about. A boost::shared_ptr<> looks like the only reference to my object and that reference is cleared... but the dtor isn't called. In this instance I could 'just simply' not use a shared_ptr and the delete operator would do fine - but I want to get to the bottom of it.

As for the GC... Well, I don't have any GC which kind of explains why it failed. That was an easy fix.

Second to this, building between baseplates is in progress. I can safely add/remove bricks that span two or more baseplates. I think. They don't stream in and out of memory very cleanly, and phantom invisible or intangible bricks are sometimes the result as you move through the world.

However, the fix is to cleanly stitch together the two baseplates - accounting for the bricks that span - and all should be well.

Lastly, aside from a couple of minor points, the Jetpack is getting a little tweak based on some use and feedback. Its getting a cruise-control so you can hover at a constant height. I went through all sorts of physics models for it, and whether or not it should have some sort of limited fuel, propellant etc.. But right now its just going to be pretty much free, unrestricted flight.

Sprint 15 Complete

Sprint 15 is working, folks.

Here is the outline:

Fixed Framebuffer startup issue

Alt-Tab is working

Pressing Enter Twice on the main menu will no longer break the world load

A rare startup (NPE) crash has been fixed

Improved WalkRun movement controller

New Jump/Jetpack

Added config option for spherifynormals and clientradius

Fixed the 'invisible brick' startup bug.

*Very* pleased its over. Time to get cracking on 16.

The wise man knows he is a fool.

So I was trying to be clever, including supplying twizzle vector operators like Vector.XZY() that returns a transformed vector or Vector.XZX(). These give you some clever(ish) results for noise generation, geometry transforms and lighting. I was playing with them - a while ago now - and they've always just 'been there'... Until a typo, and now they have been banished. Causes a REALLY weird physics bug, getting the wrong transform from a matrix and I'm glad to be shot of it.

Secondly, that means I'm close to finishing Sprint 15. Really close.

Lastly - something I can't work on now even though I'm excited about it. After spending breakfast staring at a couple of minifigs from Collection #4, I worked out how to write the brick shader I was thinking of. Sometime in the past twenty years, the plastic used for (at least) minifig legs and probably all bricks has changed. Anybody notice the difference?

On reflection, I really like the new plastic. Its not fully opaque, which means the edges of bricks let a *tiny* amount of light through. This gives a sharp, crisp well defined edge that is ever so slightly lighter than the rest of the brick as if the edge is highlighted.

Its a very nice visual effect, and something that I couldn't put my finger on for ages. I really wanted that crisp edge definition but didn't want the geometry overhead of adding a bevel. In a bright room, its almost emissive as the edge between two faces not hit by the dominant directional light is still brighter than either face in a way that couldn't be achieved with a bevel.

I'd already decided to try adding a bevel using a normal map instead of geometry, but if I can add this effect too then I'll be very happy. Take a close look at an old and a new brick, you'll see what I mean.

Sprint 14 Complete

Sprint 14, which contained mainly render optimisations, is complete. Its difficult to hand-on-heart say its totally done but I'm content with the results for now. Vertex Transforms are down, the draw speed it way up, and I've added the first significant command line switches and config options - more will appear now the code to read them is in place.

There are a couple of bugs to wrap up the Pre-Prototype so sprint 15 will be a series of consolidation tasks before mainline features-first development continues. Signing off sprint 14 with a couple of known bugs feels pretty low, but they are the highest priority item in sprint 15 and I'll hit those first.

Other than ensuring stability, I haven't got much planned for sprint 15. There are some physics improvements to make so it'll be a good time to roll those in.

Quantum Bricks

New brick occlusion code is in - well, actually old block-based occlusion has been updated to correctly work with bricks. Its sub-optimal at the moment - and could be sped up by caching and reusing previous results to significantly reduce the work it does.

Also, the brick template doesn't have an occlusion cache, which I'm going to add in future brick data so it doesn't have to be derived from the brick collision. This will also make supporting transparent blocks easy too.

Being GPU bound, the occluded bricks have taken a lot of load off of the poor GPU at a very small CPU overhead so really makes a difference. They also reduced the VBO construction CPU spike as it iterates over fewer bricks to build the buffer, this has a huge impact

In general development, each baseplate object represents a 16x16 studded area with an AddBrick()/RemoveBrick() interface, as well as a smart pointer to the surrounding four baseplates. It has containers for AllBricks and VisibleBricks, as well as block data used for occlusion and collision. While it seemed like a good idea at the time, this one class is now running most of the game (VBO+Rendering, Collision, Occlusion, Streaming...) and I'm uncomfortable about it having so many responsibilities even if a lot of the work is done by external classes. In general, its a non-issue as the baseplate is constructed and configured by a factory, and managed as needed by the correct system(s) but the part of me that knows its wrong wants to fix it before it breaks rather than after.

Factory

Earlier I was contemplating protected constructors with friend class factories for a couple of classes that really shouldn't be instantiated without configuring.

Clearly this is feedback from The Code. Its trying to tell me that the class in question is too bloated and complicated, with too many responsibilities, methods or members.

I guess the first step is components? I could refactor into components that would have more granular unit tests I guess. But then I'd have to atomically construct several components for it to work - which I'd do in a factory. The system under test would be made of a graph of interconnecting objects. Am I unfairly prejudiced against irreducible complexity, or is this the way of the future?

The question has arisen because I have a bug in an object that requires several steps to initialise. While I may have been aware of this, it wasn't until adding a unit test did I really notice that I didn't know how to construct and configure the object. My factory knows the magic to make it happen, but its not obvious from the interface which is a problem to address.

Now I've spotted the problem I'm going to fix it so that Future Roger will like my code more than I currently like it.

Hidden secrets

The first optimisation path I tried when I had a world made of blocks was a block-based occlusion scheme. Since I know whether each block is occluding or not, its easy to check the six neighbours of a block and mark it as occluded.

Occluded blocks are removed from the visible set, and their vertex data is ignored for VBO construction so they lighten the load on the GPU at the cost of a small CPU and memory overhead.

In early versions occlusion data could be frozen - allowing you to manually check for problems and the scene could be rendered in wireframe, which again shows up if too many blocks are rendered. Both of these visual tests were removed when I changed from regular uniform cubes to irregular cuboid blocks, and somewhere along the line the occlusion check broke too.

Secondly, I'm rendering quite a lot of large Vertex Buffer objects and have just added a vertex dot product check to only render the ones that are a bit onscreen - this is all simple stuff but does serve as a reminder that I've built a lot of code from scratch and the rendering - until now - has all been placeholder.

As I'm approaching prototype, the graphical look and feel and specifically the framerate are turning up as items that need to be addressed as the project shifts gear from a whimsical code experiment to a functional tech demo. Thrashing the GPU with millions of polys isn't quite acceptable, and 100ms CPU spikes for vertex transforms or perlin noise gerneration are being smoothed out too.

The usual suspects are being wheeled out in the form of LOD models, fog, environment skybox et. al. in preparation for a prototype release. Like the previous development, each of these will only have to exist in its crudest state so I can cross them off as bug-free and feature-complete, two thirds of the development trinity. Agile Homebrew has the advantage of hiding the third "on-schedule" bar of the triangle and turning it into a non-issue. As with everything, it'll be done when its done.

The entire of Sprint 14 looks like render optimisations with no end in sight, but I'll probably get it running a bit faster and then cast my eye at the next set of tasks. I'm already drafting a list of what I want to see in Sprint 15.

Slightly More Technical

Axis-aligned building blocks provide a number of advantages and cheap optimisations for collision, occlusion and rendering. This adds to the gameplay and development reasons for using building blocks.

The pioneer world is divided into chunks that are sixteen blocks square. Each chunk is processed atomically and owns a vertex buffer object that it maintains when a brick is added or removed from it.

Whole chunks are constructed/deconstructed by a Horizon object that keeps track of which chunk the player is standing in and ensures a few chunks in each direction are visible. This allows very large landscapes to be represented and to stream in and out to allow seamless travel over long distances. It also provides a base for distance culling to scale the renderer to match the performance of different computers.
Each brick is an arrangement of blocks that can be atomically added to or removed from a chunk.

Each chunk contains a list of the bricks inside it, and each time the brick list changes the brick list is iterated over to construct a Vertex Buffer Object for the chunk.

Because of the chunk architecture, its not yet possible to place a brick so that it spans two or more chunks. Each brick must lay entirely within one chunk. This restriction has proved pretty annoying when building, and does need to be resolved.

Overview

Pioneer is a game development experiment written in C++ using OpenGL and SDL. The unusual development approach taken is to code before design, instead of the other way around. To set this up I’m starting with a development sandbox to try ideas in.

Since the entire project is a construction toy, It seemed appropriate to make the environment a construction toy, similar to popular plastic building blocks. This provides a direction for development and a surprisingly specific set of goals, functionality and user-stories that can be summarised as “a world made of bricks”

The second advantage of making a world out of building blocks is it provides inspiration for gameplay mechanics. Just looking at a brick-based landscape calls to mind what you could do with buildings, bridges and castles or with roads, trains and planes. Each idea can be tried out as a gameplay experiment, and either forge a direction for the project or form a fork that can be worked on in isolation.

The third advantage of using building blocks is one of nostalgia. If you have an affinity for the construction medium or just fond memories, the environment becomes easy to pick up and to visualise constructions.

The prototype is nearly finished and aims to provide a sandbox environment for construction with a reasonable UI.