Pseudo Random Journal Generator Predictable permutations on programming, technology, and more.

adam_17jul2007.plan

So I haven't been spending much time programing lately. But I have been thinking quite a bit about all aspects of Decrypt from technical, to game play, to story.

I have finally come up with how the mission system will work. It should be very flexible. And Over the last 2 or 3 days I implemented the way in which applications and thus the game world, will be affected by missions. There are any number of ways to go about doing this, I opted for an event system.

Each application, when it registers it's self with the operating system has the option of registering a default event handler. Then an active mission can then register event handlers for any application it wishes to affect, and these 'over riding' event handlers get pushed onto the handler stack. The stack has fall through, so if a certain event is not handled in a certain event handler, then that event falls through to the next handler and so on.

The missions will be quite simple. They will be described in an XML file by sequence points. Each stage of a mission will describe the game world by specifying a list of handlers for different applications. When a player completes a task that requires anything in the game world to change, they will change sequence points which will remove the old sequence point handlers by popping them off the handler stack, then pushing the new sequence point's handlers on.

I think you could provide additional flexibility by allowing for sub-sequence points, so can change small parts of the world while retaining the majority of handlers. Then when u change the main sequence point you would change all the handlers.

Anyway, with the event system in place, and the Email client now working off of the event system, and the email client the only component absolutely required for the most basic mission. I'm going to implement the mission system now and create an ultra basic mission!

adam_05jul2007.plan

So, long time no post.

But I have been very, very, busy with work, and contract work on the side as well as summer activities.

But more to the point I polished a few code features in Black Engine and have a new version I'm going to release v0.6a which is much closer to the functionality I had originally wanted in v0.6.

That being said, after much research into refactoring and additions I want to make to the engine, I have decided to take a break from Black Engine development for a while.

With what little spare time I do have, I have started another project. It's called Decrypt (working title), you can check it out in the projects section.

The initial idea has expanded a bit but the core thing is this. I want to challenge the players mind. When a player over comes an obstacle in the game I want them to feel a real sense of accomplishment. Not just the fact that they progressed, but that what they did was actually an accomplishment for them selves, in real world terms. I want to teach the player things that I think are generally fascinating. The player will actually learn real world things that will help them to complete parts of the game.

You will play the role of a sort of modern day private eye. The game takes place on a "virtual computer". The entire thing resides inside your web browser.

You will receive an email from a friend asking for your help in decrypting this message he found. You will use the in-game web browser to go to this forum site that he runs and find the text he is talking about. Then you will have to use real world cryptography skills to decrypt the message. Once decrypted the message will unveil the seeds of the plot and set in motion a chain of events that will sweep you up in an international event.

There will be certain hacking-esk elements much in the spirit of Uplink ( http://www.uplink.co.uk/ ) that will lead you to the next message to decrypt. The story will be dynamic in certain ways. Depending on your actions different people in the game world will react differently and change the story.

I really think I can produce a rich, entertaining, and intellectual experience. I've also taken some influence and ideas from sites like: www.hackthissite.org

Another thing in the back of my mind is a game that came out just before 9/11. I forget the name of it. But when you signed up you agreed to let them do tons of different things to you in the real world. Such as call your cell, email you, send you text messages, etc. It reminded me a lot of the move "The Game" with Michael Douglas. Sadly, because of the paranoia following 9/11 the game silently closed it's doors. While Decrypt won't be doing any of these things to your real world devices, I am trying to re-produce atleast some of this in the "virtual computer".

The greatest thing about this project is that, since I am doing it as a web based app, I can take advantage of so many different tools and libraries that aim at aiding in RAD. It took me 1 hour to set up the basic client side (Javascript) code for launching programs! After 2 more days of coding I had the basic operating system and process management code written on the server side and had it hooked up to the client side display.

I've already written 2 "Applications", one is an OS settings application that allows you to change the theme of your virtual computer and such. And the other is the Email Client which will be a hugely important part of the game.

Right now I am working on the last major technical hurdle which will manage the story and how it affects the game world.

I want to make a system that is flexible enough to support more then 1 story. So once you finish one, you can start another. I envision the main character as almost like a Jack Ryan (Tom Clancy's main character in many of this books) where it's the same character in many different stories that are loosely tied to each other.

Two ideas I am toying with are:
1) Allowing users to create missions
2) Allowing users to write programs for the virtual operating system. The application system is flexible enough it would be very very easy to support user made programs. Users could pass "Install Links" around to other users to allow them to install a program they made. Because the email system is very generalized, you can email any other user of the game, and they will get it in their virtual In box just like any another other email in the game. I'm also planning on making a chat so you can chat to anyone else who is currently logged onto the system.

I'm still trying to flesh out some of the major details of the plot, and I'm still working on the mission/story system which will have to control scripted emails, intelligent responses from NPCs to player emails. And modify contents of fake web sites, and news wires.

So I've got a ways to go. But things are coming along astonishingly fast.

- Adam

adam_31may2007.plan

So no time really spent lately on coding for Black Engine, what with moving, unpacking, cleaning, and starting a new job. But I have been doing a lot of research.

I printed out the entire ODE manual and read through it cover to cover, then went back and read some more. Then I got on some forums and talked and read some more. I have a pretty good grip on how I will implement ODE now.

But something that seemed like a small caveat that could be easily solved has blossomed into a scarily large issue.

The problem is simulation time vs. real time. This was a problem I faced down years ago. The problem is simple. My computer runs faster then yours (probably not really, but you get the idea). So if you hold down W to move forward what do I do? Well in the simplest form, I just add 10 to your forward velocity.

So we read input every frame, I run at 30 Frames Per Second (FPS) which means if I hold the W key for 1 second I have a velocity of 300 ( 30 * 10 ) (assuming my velocity accumulates and there is no friction). Now you run it on your Commador64 (Jesus man, upgrade already) and only get 10 FPS. This means you only have an accumulated velocity of 100 ( 10 * 10 ) by the end of 1 second. As you can see this is a rather large problem.

So when I first encountered this problem like 2 years ago, I wrote a timer class which times the length of time it takes to render each frame and stores it in as deltaTime. Thus everything that is time sensitive uses the value of deltaTime as a scaling factor. So the amount of velocity to add each frame is computed as follows:
currentVelocity = velocityConstant * deltaTime;
totalVelocity += currentVelocity;

This yielded fairly stable results except my custom timing class was *too* accurate, and it produced different results on different computers. Finally I moved to a cross-platform timer with a more coarse resolution and things were perfectly stable. Thus the simulation time is determined by the deltaTime and the real time is determined by the frame rate.

So thats the history of it. Since things were handled all internally, synchronously, and most importantly, deterministically with the coarse grain timer, there was no problem. The problem comes up when non-deterministic elements are introduced. The first of these elements is the physics engine. It is a non-deterministic physics engine which uses calculus (integration) to interpolate between "simulation steps". Now I *do* have simulation steps currently, each frame is a simulation step, and since we use deltaTime, we can say that our simulation steps are of variable width or size. This does not work well with the calculus that the physics engine uses. The physics engine wants fixed size steps.

After looking into that I started wandering in my research merely out of boredom and started looking into networking again (which I hadn't planned on looking at for a very long time) and realized a lot of my questions of how the server would remain authoritative in a real time environment were solved by fixed size simulation steps.

Now this doesn't sound like a huge deal, converting from variable size to fixed size, but the implications are huge, absolutely massive. First it only makes sense to implement them in a client server model if a move to that model is ever planned, other wise you'd essentially be rewriting everything later when you wanted to switch.

Second, you now introduce a mind boggling number of issues intrinsically related to both the move to fixed size steps as well as the change of logic in how client/server model should communicate, and how the client should work as a whole.

See one issue is this, like I said ODE (and many others) is non-deterministic, so it suffers from floating point rounding errors which vary from one computer to another (much like my original fine grain timer). The server must be authoritative, but to reduce bandwidth you might want to only send updates, like player X wants to move with force Y. Thats a small update. Then the client side physics engine takes that data and computes the new position based on that data as well as all the other world data that that affects player X (gravity, friction, collision detection, the list goes on). Problem is, my uber awesome computer will computer that to a high accuracy, but your Commodore 64 will make several rounding errors due to, among other things, hardware limitations. Now we both get the next update player X wants to move with force Z. My player X is at a slightly different location then your player x due to the rounding errors, and now they both move again in another direction, and again your's loses some precision and moves on a slightly different vector then mine. As you can see these errors compound and over time the 2 clients become totally out of sync with each other and even the server world (which is the "true" world).

This could be fixed with sync (synchronization) packets which contain the full data on an object, location, orientation, velocity, friction, and every other world force. But thats a huge amount of data to send to every player, every step!

Another problem is this, as you all know, the internet is unreliable (damn you Al Gore!). The speed of the connection will vary, packets will be lost, etc. So what does a client do between update and sync packets? Just sit there? You would experience very studdered, non-physical behavior if this were the case, even if the connection was decent.

The solution to this is to do client side extrapolation or interpolation between step updates. Interpolation is the more favored method, BUT! You run into the same problem with the physics engine and update packets! Client side floating point inaccuracies! Either resolution method however still requires a very difference structure for even the game loop!

In short (wwwaaaayyyyyy too late for that) no matter what I choose, I want to eventually make this a networked engine, the advantages provided by an authoritative server for physics (as well as many other aspects) are undeniable, thus I have to go back and refactor to a client server model now rather then later 🙁

This is going to be a huge task... Not as big as the change from v.5 to v.6, but still huge. And it's going to take a lot more research and planning to get it to work.

adam_16may2007.plan

WOW. So I Conquered two HUGE and extremely long standing problems in the past 2 days.

The first is, I FINALLY got the ATI drivers working for my Radeon 9800 Pro under Linux ( Ubuntu ) which all in all is pretty damn important considering I primarily use Linux for developing a 3D graphics engine. So now I'm not running about 30 FPS in Linux 😉

Let me tell you, it was a pretty damn good feeling seeing:
"Direct rendering: Yes"
after typeing:
glxinfo | grep rendering
So SO SSSOOOO many times and seeing No.

Any way, with that finally set, I went back to debugging my memory corruption bug and low and behold! I KILLED IT! You can now load a level, then load another one and it will clean up the memory and get it going again.

There is still quite a bit of memory leaks to track down, but with Valgrind, my new best friend, that won't take very long!

So looking forward. The big news is, after very long deliberation I have decided to integrate 3 new 3rd party libraries.

The first is the Configurable Math Library (CML). I'm going to retire my custom Math Lib to get a more standardized and unit tested base math library. It should be faster as well.

The second is the Open Dynamics Engine. It is an open source physics engine that looks really sweet. It will handle movement, collision, all that fun stuff. Should allow for some REAL cool stuff in the future and doing all that physics math isn't the point of this project. I'm making a Game Engine, not a Physics Engine, they are two totally separate projects these days.

The third is the Irrklang 3D Sound Engine. This is absolutely the sweetest library I've worked with in a long time. It is just freaking perfect and has influenced me on certain design ideologies I would like to implement as I go along.

At some point I will be integrating a networking library as well, I have my eye on the networking layer that garage games puts out. I forget the name at the moment. But it looks sweet.

So probably the next thing will be to clean up some of the more egregious memory leaks ( My poor poor texture manager not deleting properly 🙁 ). Then research the ODE and see how I will be integrating it into Black Engine. Once I have a good handle on that I will integrate the CML, then ODE.

Hopefully during this time Naxos will be working on the model loading and rendering!

Well I've said it before and I'll say it again but I mean it more then ever this time, stay tuned because theres going to be some very neat things to come!!

- Adam

adam_01may2007.plan

It may be that I'm running on only 2 hours of sleep but debugging is especially frustrating to me today.

Any way, a few days ago I solved several more big bugs which allowed me to load, unload, and load another level several times before it crashed.

Then I ran it through valgrind again to identify more trouble spots. More spots found, and fixed today. Pretty much the last memory corruption bug now has to do with the Camera class deleting it's Scene Graph Nodes. It is a bit of a special case so it makes sense it is the one causing problems. But I have been over it and over it, fixing little logic errors associated with very obscure code paths along the way, and I can't find a single thing wrong with it now. It's very frustrating.

Once that is solved ( some day ) there is quite a bit of memory leakage happening. Mostly obvious stuff. Part of my texture memory, a temp buffer during creation, isn't being deleted. It should be pretty quick to get most of that under control.

A few steps forward. But so much has been revealed to me during this debugging that is necessary to make the code base clean. I'm going to be spending a lot of time cleaning and refactoring...

- Adam

adam_28apr2007.plan

Well. what follows is the meanderings of thoughts I have had during the past 2 weeks.

If ur not terribly interested in Black Engine / Software Engineering you’ll probably want to pass this one by. But I think for those of us that are new to trying to learn as we create, this should be of some interest.

So I began by tracking down a very intricate heap corruption bug involving a string of virtual destructors which after learning many many new techniques for debugging memory corruption I tracked it down to several places in my more basic code. Code that I had tested to my satisfaction. This was a big slap in the face. Some of the mistakes were minor and forgivable in my mind because of the huge time crunch it was all developed under. But some were more integral.

I found several rather severe faults. Some where in design, some were in general ideology.

Let me go into them.

The first, the Element class in my mind should handle and subsequently hide all of the interactions with the Element Manager Service. Essentially wrap all the functionality and provide higher level, task oriented access to the Service’s functionality. This all seemed logical to me. So in the constructor (all constructors are virtual) the Element class handles registration with the EMS. Now, if the EMS rejects the registration, which can happen if the Element Id is not unique for example, the instance of that element is not valid and should be deleted. Well, this constructor is being called from a much higher level virtual constructor. So neither the EMS register() function now the Element constructor it’s self can handle deleting the memory b/c at that point during the construction it only knows that it is an Element, there for the Element memory would be deleted, nothing else, and the rest of the constructors would proceed as normal. Even if you don’t delete it right then, you then have the issue of communicating back to the calling code the fact that it failed. Also the calling code must then have to take care of cleaning it up conditionally which I didn’t like.

Now I thought up several solutions but when pitching them to other programmers got the sense they were very uneasy about the whole thing as was I. It just indicated there was a deeper design flaw. Their general response was, it should be a 2 stage initialization. Construction, then initialization. This is probably something close to what I will be going with.

Another thing. I didn’t use exception any where in the design. I have used exception and understand them just fine. But I had never used them in C++, with the time constraints I had at the time and the amount of things I had to learn any way I decided to not tack on another unknown. That was a mistake. Exceptions would provide a very clean solution to the previous issue as well as many others. So some point soon I plan on developing the exception scheme and implementing it.

The last big thing is. I tracked the majority of my memory corruption bugs back to my hash table. Which. Christ. I should be able to code a freaking hash table. That was the big slap in the face. For many reasons though besides just the memory bugs, I decided to switch to STL::map which is a simple unique hash map. While doing so I learned a lot about 2 things which I had known existed but hadn’t spent time to get to know well. Iterator and References. The sad fact is, I was taught C and C++ together so until recently had a confused idea of what belonged to which language. If you don’t know already, while they can be used together, they shouldn’t be, and there is a clear separation between the two.

In C++, looping through arrays should when EVER possible be done with Iterators. And heap allocated objects should almost ALWAYS be passed by reference (which does NOT mean passing a pointer!), for speed, safety, everything. Use References!

So I plan at some point removing as many as possible (the VAST majority) of raw pointers and replacing them with references.

That being said, this is the view of my code through the most critical spot light possible. My own. I believe that when these things mentioned are refactored, the code base will be rock solid and modern well coded C++.

So takes a breath…
Any way, I started writing the 2nd tutorial which goes in depth about the Extension project which is the actually game and graphics engine portion of the whole framework. I’ll probably finish that tomorrow as well as hammer away at some more of the bugs and maybe a few of these refactoring tasks.

Well, if your still reading then you’re a freaking trooper lol.

That’s it for me.

- Adam

adam_11apr2007.plan

So, I've spent almost a week on this memory corruption bug. Talked to a lot of people and read a lot of things and finally come up with a plan to crush it after it evaded all my previous methods.

I downloaded valgrind which is "the" memory manager / tracker for Linux. This thing has been used by NASA for the Mars rover and EA for games like Battle Field, I mean this thing is the heavy weight champ of it's class of programs.

So I booted into Linux, wrestled with my video drivers a bit more and updated everything since I haven't booted up into Linux in a few months. Then I got valgrind running straight from the Ubunutu repository.

And let me tell you. This program is a dream. I will have this memory corruption bug plus many other potential bugs killed in short time.

I'm leaving for the weekend to get my bike ready for the season and bring it up to school, but you can bet there will be a flurry of activity next week, killing this bug plus starting work on collision detection!

Stay tuned!

- Adam

adam_08apr2007.plan

So lots of news with Black Engine. Some good some bad.

The good is, I've brought on one new developer Naxos, who is going to be helping code, and working on brining on at least one more, Crazedbrit.

The release of v.6 went over really well in my opinion. It got a ton more attention then any other release so far. 76 downloads in the first 4 days. Not bad for an engine in such early stages as Black Engine.

Now the bad. The BSP unloading bug I've been hunting is proving to be an exceedingly elusive memory corruption issue I believe. I've eliminated a few very stupid stupid bugs already ( deleting stack allocated memory, derived classes not having virtual destructors ) and caught quite a few potential memory leaks. I even get a level to unload and reload now and then, but it's just luck, there is still severe memory corruption some where so after loading the new level it dies quickly.

Also, in the process of hunting this bug I have found a rather... large. design flaw having to do with an Element failing to register with the Element Manager Service and thus having to delete it's self but being unable to do so. I've been discussing a few solutions on the gamedev boards but none of them are very elegant. So we'll see where that goes.

All in all progress is being made though. And hopefully I'll be able to crush this memory corruption bug soon.

- Adam