Pseudo Random Journal Generator Predictable permutations on programming, technology, and more.

adam_12oct2007.plan

So while I had created a graphics test that had a textured polygon rendering to the screen, it was still relying on some sample code that is provided with the SDK to set up the console's graphics system.

About a week ago we made the decision not to depend on any of this sample code, which is why we had to write our own memory management code. Since then I also wrote our own code to set up the input from the controllers (Once I got it working I implemented: A button turns the motor (vibrator) on, and B button turns it off... we are planning to market it to the small but growing market of lesbian gamers 😉 ), and yesterday I finally wrote all the code to setup the graphics system.

So finally free of the constraints of the SDK demo code, we started looking into a few remaining issues before we start the actual game. Geometry loading for one.

With our new memInit.h file, we can pull in any C or C++ file we want and have it work out of the box now. So after searching around a bit we have decided to go with the MD3 model format. Used in ID Tech 3 ( aka Quake 3 to the rest of us ) it is a great format that has been evolving for some time and is battle tested. And since it is such a widely used format, we can just grab an open source loader and drop it in with our memInit.h doing all the porting 🙂

So I guess the next thing to do is going to be, create an extremely simple menu from which u can exit or enter the game, just to show the system works, and then I will begin integrating ODE (Open Dynamics Engine, the open source physics engine we plan on using) and start working towards a playable version!

adam_03oct2007.plan

I think when we are kids and we hear some one does such and such "professionally" we are taught that it means they are very good at it. And that because product X comes from big huge company Y it must be very well designed. I mean it sells well and they have been making products like X for a long time so it must be good.

But as I've gotten older it's become starkly apparent that these assumptions simply do not hold true. Many times Mr. Big Professional actually sucks at his job and makes bad decisions despite his 10 years of experience. And product X probably has many design flaws that simply should not exist considering the fact that big huge company Y made it and has been making products like it for a long time.

Case and Point. The Source engine powering, among others, the hugely successfully Half-Life 2. The full source is available for mod developers and I personally have looked at much of the source and even coded a very simple mod with it. And I can say with out pause, it is a disgusting mess.

Let me digress for a second. There is a big difference between academia and the commercial world. And while, for example, in academia global variables are the root of all evil, in real world commercial software, that are sometimes necessary. My point here is, products, and software in particular, in the commercial world may have to be a bit "dirty" by academic standards. But if it works, and it works well, then it's all good.

With that in mind lets return to my main point here about the Source engine. It is a total, complete, disgusting, nauseating, F$%*ing MESS. Commercially acceptable *dirt* was left way way WAY behind when making this engine. Their poor design decisions alone would make baby Jesus cry, not to mention their archaic coding practices that make debugging all but impossible. THE PREPROCESSOR IS NOT A COMPILER PEOPLE! STOP TRYING TO USE IT AS SUCH!

And I'm not the only one that feels this way, go on any game development forum (Where game engine design is usually discussed, these conversations are usually more academic, but still) or even a software engineering forum, and when you see people talking about the source engine it's usually something along the lines of:
"I used the Source engine once... *stares off into the distance*"

But big supposedly well designed software that sucks doesn't stop there. Talk to anyone who has worked on open source projects such as GAIM and they will tell you horror stories of how dirty the code is. (To be fair, GAIM is now called Pidgin, uner new management now, and from what I hear the code base is hugely cleaned up)

Now if you can even remember my original point, it was that big commercial products suck a lot of the time. I could go on, but I think I've provided enough examples.

This console how ever... is. awesome. I've been digging through the technical docs for weeks now. The hardware design decisions, the API design decisions, all of it is genius. Yes there is a huge learning curve to the APIs, but the reason is what I described in my last .plan file, they give you an incredible level of control very close to the hardware to be able to squeeze every last ounce of speed out of that hardware.

I've heard horror stories about the PS3 hardware and SDK. And while the console's SDK leaves something to be desired in terms of the IDE and some of the documentation. It is still fairly good. And once you do find the info you need, you realize even more how well it actually has been designed.

What is clearly evident when developing for this console is how they have learned lessons from every console they have made. I wouldn't be surprised if it was mostly the same team sticking together to make each new console.

Anyway, this is dragging on. It may seem like all I do here is sing Nintendo's praises but let there be no mistake about it. I'm no Nintendo fan boy. The last Nintendo system I bought was the SNES. And while I did love that system, I scowled at their latter decisions with N64 and GameCube. I didn't like the DS (the 2nd screen seemed gimmicky, no real use). In short, I hadn't owned, or for that matter, played, a Nintendo product in years. When I saw the Wii and it's new controller, I figured it was just another gimmick like the 2nd screen on the DS and I didn't get very excited.

So when I got this job developing a game for this console I entered there a unbiased scientifically minded observer. And I can honestly say, Nintendo has impressed me with their technically decisions, their marketing decisions, and their ingenuity.

So heres to Nintendo!

- Adam

adam_26sep2007.plan

Long time no update, so let me fill you in. I finished the Decrypt beta and it was fairly well received by the gamedev.net community. It is of course a very niche game, but I have had some very enthusiastic responses. I believe once I get some time to do some bug fixing and improvements and add another mission things will take off a bit more for it.

But more to the meat of what is going on now! I'm now working for local company developing a game for a major game console. It's awesome seeing the console side of game development.

Here you can see the first graphics I have got rendering on the our console with code of my own:

What you are seeing here is I have set the "clear color" to green, which is the color the graphics system resets the image buffer with. And I have drawn a polygon (quad) with a texture I stole from some demo in the SDK. I would have used my own texture but the tools to encode it in the proprietary format kept crashing :/

It's been very interesting seeing how the console was designed. I think it's quite genius.

See the Playstation3 and XBox360 went with the approach the computers have gone with. They say, computers are fast enough as a whole that we can sacrifice some processor cycles to gain some developer cycles. Essentially, make it easier and less time consuming for the developer at the cost of run time performance. It's what computers have been doing ever since the first compiler was written. And every time we go another step towards RAD (Rapid Application Development) you always have some one who cries about the lost performance, and in the end the higher level approach proves to win out.

But in this case, the console's designers has a good point. See, for old consoles like the SNES and Sega, you had to code entirely in ASM. Think back on the complexity of some of the games you played on those systems! And they had that all running on hardware that is laughable at best even for the day. Thats why consoles used to be cheap. But then newer consoles like Playstation began to use higher level languages. Now the XBox 360 uses an interpreted languages even (XNA is based on the .NET framework).

SO! (Sorry for being so long winded) MS and Sony have to pack a ton of hardware muscle into their consoles to coupe with the lose of performance and still get the jaw dropping visuals you see.

Mean while, the console's designers thinks the price of these consoles are ridiculous, and the developer should absorb the cost (in terms of development time) and develop closer to the hardware.

Thus cheaper, less powerful hardware can produce at least equal visuals.

Sure the learning curve is steep, that image I posted took 2 days of head smashing to get rendering, but it's more efficient at run time (if only slightly) then say, OpenGL, and DirectX. (That time also has to do with the fact that the documentation is in many cases not even correct. I'd go into more detail but I feel that could breach the NDA.

Any way, if you happen to be interested in console development or game development in general stay tuned, I plan on chronically this process.

- Adam

adam_31jul2007.plan

Quite a bit happening lately. I finished the mission system which utilizes the event handlers I finished last time. And I implemented a one sequence mission.

Then I went on to implement the in-game chat I had been talking about. I ended up using PHPFreeChat which is an AWESOME little script. I plan on either donating or finding some code to contribute so I can remove the badge. Not that I don't want to recognize their efforts, I just want a very cohesive feel to it all.

With that finally implemented, tweaked, and bug tested, I started working on the Web Browser which is going to be a huge part of the game going forward.

It shouldn't take much more code to get the browser functioning. Then I will implement a more complex mission involving the browser and email together.

I also plan on implementing a console. Mainly for easily outputting debug info but maybe later for use as a command line interface.

So thats it for now, stay tuned, very interesting things to come!

- Adam

adam_17jul2007.plan

So I haven't been spending much time programing lately. But I have been thinking quite a bit about all aspects of Decrypt from technical, to game play, to story.

I have finally come up with how the mission system will work. It should be very flexible. And Over the last 2 or 3 days I implemented the way in which applications and thus the game world, will be affected by missions. There are any number of ways to go about doing this, I opted for an event system.

Each application, when it registers it's self with the operating system has the option of registering a default event handler. Then an active mission can then register event handlers for any application it wishes to affect, and these 'over riding' event handlers get pushed onto the handler stack. The stack has fall through, so if a certain event is not handled in a certain event handler, then that event falls through to the next handler and so on.

The missions will be quite simple. They will be described in an XML file by sequence points. Each stage of a mission will describe the game world by specifying a list of handlers for different applications. When a player completes a task that requires anything in the game world to change, they will change sequence points which will remove the old sequence point handlers by popping them off the handler stack, then pushing the new sequence point's handlers on.

I think you could provide additional flexibility by allowing for sub-sequence points, so can change small parts of the world while retaining the majority of handlers. Then when u change the main sequence point you would change all the handlers.

Anyway, with the event system in place, and the Email client now working off of the event system, and the email client the only component absolutely required for the most basic mission. I'm going to implement the mission system now and create an ultra basic mission!

adam_05jul2007.plan

So, long time no post.

But I have been very, very, busy with work, and contract work on the side as well as summer activities.

But more to the point I polished a few code features in Black Engine and have a new version I'm going to release v0.6a which is much closer to the functionality I had originally wanted in v0.6.

That being said, after much research into refactoring and additions I want to make to the engine, I have decided to take a break from Black Engine development for a while.

With what little spare time I do have, I have started another project. It's called Decrypt (working title), you can check it out in the projects section.

The initial idea has expanded a bit but the core thing is this. I want to challenge the players mind. When a player over comes an obstacle in the game I want them to feel a real sense of accomplishment. Not just the fact that they progressed, but that what they did was actually an accomplishment for them selves, in real world terms. I want to teach the player things that I think are generally fascinating. The player will actually learn real world things that will help them to complete parts of the game.

You will play the role of a sort of modern day private eye. The game takes place on a "virtual computer". The entire thing resides inside your web browser.

You will receive an email from a friend asking for your help in decrypting this message he found. You will use the in-game web browser to go to this forum site that he runs and find the text he is talking about. Then you will have to use real world cryptography skills to decrypt the message. Once decrypted the message will unveil the seeds of the plot and set in motion a chain of events that will sweep you up in an international event.

There will be certain hacking-esk elements much in the spirit of Uplink ( http://www.uplink.co.uk/ ) that will lead you to the next message to decrypt. The story will be dynamic in certain ways. Depending on your actions different people in the game world will react differently and change the story.

I really think I can produce a rich, entertaining, and intellectual experience. I've also taken some influence and ideas from sites like: www.hackthissite.org

Another thing in the back of my mind is a game that came out just before 9/11. I forget the name of it. But when you signed up you agreed to let them do tons of different things to you in the real world. Such as call your cell, email you, send you text messages, etc. It reminded me a lot of the move "The Game" with Michael Douglas. Sadly, because of the paranoia following 9/11 the game silently closed it's doors. While Decrypt won't be doing any of these things to your real world devices, I am trying to re-produce atleast some of this in the "virtual computer".

The greatest thing about this project is that, since I am doing it as a web based app, I can take advantage of so many different tools and libraries that aim at aiding in RAD. It took me 1 hour to set up the basic client side (Javascript) code for launching programs! After 2 more days of coding I had the basic operating system and process management code written on the server side and had it hooked up to the client side display.

I've already written 2 "Applications", one is an OS settings application that allows you to change the theme of your virtual computer and such. And the other is the Email Client which will be a hugely important part of the game.

Right now I am working on the last major technical hurdle which will manage the story and how it affects the game world.

I want to make a system that is flexible enough to support more then 1 story. So once you finish one, you can start another. I envision the main character as almost like a Jack Ryan (Tom Clancy's main character in many of this books) where it's the same character in many different stories that are loosely tied to each other.

Two ideas I am toying with are:
1) Allowing users to create missions
2) Allowing users to write programs for the virtual operating system. The application system is flexible enough it would be very very easy to support user made programs. Users could pass "Install Links" around to other users to allow them to install a program they made. Because the email system is very generalized, you can email any other user of the game, and they will get it in their virtual In box just like any another other email in the game. I'm also planning on making a chat so you can chat to anyone else who is currently logged onto the system.

I'm still trying to flesh out some of the major details of the plot, and I'm still working on the mission/story system which will have to control scripted emails, intelligent responses from NPCs to player emails. And modify contents of fake web sites, and news wires.

So I've got a ways to go. But things are coming along astonishingly fast.

- Adam

adam_31may2007.plan

So no time really spent lately on coding for Black Engine, what with moving, unpacking, cleaning, and starting a new job. But I have been doing a lot of research.

I printed out the entire ODE manual and read through it cover to cover, then went back and read some more. Then I got on some forums and talked and read some more. I have a pretty good grip on how I will implement ODE now.

But something that seemed like a small caveat that could be easily solved has blossomed into a scarily large issue.

The problem is simulation time vs. real time. This was a problem I faced down years ago. The problem is simple. My computer runs faster then yours (probably not really, but you get the idea). So if you hold down W to move forward what do I do? Well in the simplest form, I just add 10 to your forward velocity.

So we read input every frame, I run at 30 Frames Per Second (FPS) which means if I hold the W key for 1 second I have a velocity of 300 ( 30 * 10 ) (assuming my velocity accumulates and there is no friction). Now you run it on your Commador64 (Jesus man, upgrade already) and only get 10 FPS. This means you only have an accumulated velocity of 100 ( 10 * 10 ) by the end of 1 second. As you can see this is a rather large problem.

So when I first encountered this problem like 2 years ago, I wrote a timer class which times the length of time it takes to render each frame and stores it in as deltaTime. Thus everything that is time sensitive uses the value of deltaTime as a scaling factor. So the amount of velocity to add each frame is computed as follows:
currentVelocity = velocityConstant * deltaTime;
totalVelocity += currentVelocity;

This yielded fairly stable results except my custom timing class was *too* accurate, and it produced different results on different computers. Finally I moved to a cross-platform timer with a more coarse resolution and things were perfectly stable. Thus the simulation time is determined by the deltaTime and the real time is determined by the frame rate.

So thats the history of it. Since things were handled all internally, synchronously, and most importantly, deterministically with the coarse grain timer, there was no problem. The problem comes up when non-deterministic elements are introduced. The first of these elements is the physics engine. It is a non-deterministic physics engine which uses calculus (integration) to interpolate between "simulation steps". Now I *do* have simulation steps currently, each frame is a simulation step, and since we use deltaTime, we can say that our simulation steps are of variable width or size. This does not work well with the calculus that the physics engine uses. The physics engine wants fixed size steps.

After looking into that I started wandering in my research merely out of boredom and started looking into networking again (which I hadn't planned on looking at for a very long time) and realized a lot of my questions of how the server would remain authoritative in a real time environment were solved by fixed size simulation steps.

Now this doesn't sound like a huge deal, converting from variable size to fixed size, but the implications are huge, absolutely massive. First it only makes sense to implement them in a client server model if a move to that model is ever planned, other wise you'd essentially be rewriting everything later when you wanted to switch.

Second, you now introduce a mind boggling number of issues intrinsically related to both the move to fixed size steps as well as the change of logic in how client/server model should communicate, and how the client should work as a whole.

See one issue is this, like I said ODE (and many others) is non-deterministic, so it suffers from floating point rounding errors which vary from one computer to another (much like my original fine grain timer). The server must be authoritative, but to reduce bandwidth you might want to only send updates, like player X wants to move with force Y. Thats a small update. Then the client side physics engine takes that data and computes the new position based on that data as well as all the other world data that that affects player X (gravity, friction, collision detection, the list goes on). Problem is, my uber awesome computer will computer that to a high accuracy, but your Commodore 64 will make several rounding errors due to, among other things, hardware limitations. Now we both get the next update player X wants to move with force Z. My player X is at a slightly different location then your player x due to the rounding errors, and now they both move again in another direction, and again your's loses some precision and moves on a slightly different vector then mine. As you can see these errors compound and over time the 2 clients become totally out of sync with each other and even the server world (which is the "true" world).

This could be fixed with sync (synchronization) packets which contain the full data on an object, location, orientation, velocity, friction, and every other world force. But thats a huge amount of data to send to every player, every step!

Another problem is this, as you all know, the internet is unreliable (damn you Al Gore!). The speed of the connection will vary, packets will be lost, etc. So what does a client do between update and sync packets? Just sit there? You would experience very studdered, non-physical behavior if this were the case, even if the connection was decent.

The solution to this is to do client side extrapolation or interpolation between step updates. Interpolation is the more favored method, BUT! You run into the same problem with the physics engine and update packets! Client side floating point inaccuracies! Either resolution method however still requires a very difference structure for even the game loop!

In short (wwwaaaayyyyyy too late for that) no matter what I choose, I want to eventually make this a networked engine, the advantages provided by an authoritative server for physics (as well as many other aspects) are undeniable, thus I have to go back and refactor to a client server model now rather then later 🙁

This is going to be a huge task... Not as big as the change from v.5 to v.6, but still huge. And it's going to take a lot more research and planning to get it to work.

adam_16may2007.plan

WOW. So I Conquered two HUGE and extremely long standing problems in the past 2 days.

The first is, I FINALLY got the ATI drivers working for my Radeon 9800 Pro under Linux ( Ubuntu ) which all in all is pretty damn important considering I primarily use Linux for developing a 3D graphics engine. So now I'm not running about 30 FPS in Linux 😉

Let me tell you, it was a pretty damn good feeling seeing:
"Direct rendering: Yes"
after typeing:
glxinfo | grep rendering
So SO SSSOOOO many times and seeing No.

Any way, with that finally set, I went back to debugging my memory corruption bug and low and behold! I KILLED IT! You can now load a level, then load another one and it will clean up the memory and get it going again.

There is still quite a bit of memory leaks to track down, but with Valgrind, my new best friend, that won't take very long!

So looking forward. The big news is, after very long deliberation I have decided to integrate 3 new 3rd party libraries.

The first is the Configurable Math Library (CML). I'm going to retire my custom Math Lib to get a more standardized and unit tested base math library. It should be faster as well.

The second is the Open Dynamics Engine. It is an open source physics engine that looks really sweet. It will handle movement, collision, all that fun stuff. Should allow for some REAL cool stuff in the future and doing all that physics math isn't the point of this project. I'm making a Game Engine, not a Physics Engine, they are two totally separate projects these days.

The third is the Irrklang 3D Sound Engine. This is absolutely the sweetest library I've worked with in a long time. It is just freaking perfect and has influenced me on certain design ideologies I would like to implement as I go along.

At some point I will be integrating a networking library as well, I have my eye on the networking layer that garage games puts out. I forget the name at the moment. But it looks sweet.

So probably the next thing will be to clean up some of the more egregious memory leaks ( My poor poor texture manager not deleting properly 🙁 ). Then research the ODE and see how I will be integrating it into Black Engine. Once I have a good handle on that I will integrate the CML, then ODE.

Hopefully during this time Naxos will be working on the model loading and rendering!

Well I've said it before and I'll say it again but I mean it more then ever this time, stay tuned because theres going to be some very neat things to come!!

- Adam