Please view the full post to see the Unity content.

I believe this could just as easily been produced in processing but I find the lack of assistance from the IDE to be too frustrating to manage.  On the other hand, though, processing / Java is apparently very similar to the kind of instruction set that can be compiled on to arduino micro-controllers.  Hopefully more on that later! In the meantime, here is a video of my anemone:

Note: the jitter is from the capture process – I am using CamStudio to grab the footage, which requires a little more of the system than Java wants to give up.  If you want to install Java3D, you can view the Applet at a much higher resolution.

One thing I hadn’t thought of was the fact that, on a laptop with a monitor-fixed webcam, in a room where the monitor is the only light,  The relief map does a reasonable job of being a depth map!  The luminance might map to the square of the height or something but you can see from the image above how well it works.

You can try it for yourself if you like – Note the usual applies with the webcam… If you can see what it’s like in a darkened room, do try it! I’d love to see what other people come up with!

(mouse controls rotation, buttons toggle params – try the relief map with the original bitmap laid on top for freaky 3d-ness!)

Update 2009-12-26

It is possible to make a 3d depth map in realtime, although there are a couple of hurdles to jump through. Even on a reasonable computer, the highest poly count you can expect out of Flash at present is about 5,000 triangles, so the resolution of the mesh used in this Flash experiment is 64 x 48.

If you were to do the same thing in a language that gave you access to DirectX or openGL you can expect to have poly counts in the millions, so one vertex per pixel is an easily achievable goal.

The next part is more complicated: Getting a color channel to look like a depth channel. The experiment above is using the red channel as the depth map. This will yield a real depth map, but only under these two conditions:

1. The light must originate from (close to) the camera. That way, the brighter a point is, the closer to the camera it is. For objects that are reasonably distant – say, 50 cm away, it is sufficient for the light to be within 5cm, but the closer the lights are, the better. You can buy some webcams with a built-in ring of LEDs – these are ideal.

2. All the objects must exhibit Diffuse / Lambertian luminance. Not as complicated as it sounds: Basically, when an object has a mirrored or a shiny surface, its brightness depends on both how it is lit and the camera position (where you are looking at it.) When you change your point of view, you change the brightness of the surface. There are commercially-available Lambert materials like Spectralon, but an unfired white clay will do just as well.  In fact, you could probably dust an object in any white powder – something like flour or baking soda,  light it under the right conditions and see what it looks like.

If anyone tries this I would love to see the results!  Under the two conditions above,  you should be able to use my mesh experiment above to make  meshes that show depth.  If you’re interested in some of the lower-level stuff in this subject, it’s part of Computer Vision, specifically Shape from Shading. As with any vision project, the best port-of-call is the openCV project.

Click in the panel to get focus, then use left and right to steer your accumulation buffer-derived snake to victory.  Good luck!

(you need a webcam to make it work)
I partied to this all night.

Long time no post, I have been (and still am) on-site on a contract job, and haven’t had a lot of time outside of work.  Nevertheless, I have been doing some interesting things!  Here is one of them – a test of the practicality for particle fields to be drawn from cached BitmapDatas, rather than generated per-frame.  It’s not fast,  but it’s much faster than the alternative:

In case you’re wondering what is going wrong, this is what happens when the pre-drawn circle bitmap isn’t offset appropriately.  To see it working properly, click the flash file and just hold down any key for the shapes to make proper circles.  Releasing again will revert to the crazy, checkerboard-like state.

The project I’m working on is very exciting, and should be ready for public unveiling in 3 weeks or so.  In the meantime, I can happily say that video uploads and downloads with Red5 and a basic apache HTTP install are far, far easier than I had ever hoped. Thank you to both project teams for being so awesome!

Yes, I am aware that there are occlusion issues.  I’m not completely certain how the 3d is going to integrate into the 2d environment, so it may prove to be unnecessary to do anything at all about it.  I’m thinking that only dynamic characters need be rendered in 3d, though it depends on whether the camera view will have any aspect of dynamic-ness,  and how well 2d / pre-rendered assets can integrate.

Re: the core game engine:  I’m running up against a bit of an irritating problem with keeping the optical, mechanical and visual representations of polygons in sync with one another.  There are a good number of advances since the last display, but unfortunately they’ll have to wait until this niggly issue is fixed!

I’ve also started making serious progress with Flash-PHP-SQL interactions, and vice versa, as well as beginning to make hooks into Facebook.  Interesting times ahead!

Another thing I noticed when pushing PV3d to the performance limit (and beyond, according to some people), is something that I have seen a lot of other complaints about: Papervision is very slow.  This is a fair accusation in comparison with, say, the Unreal engine (or even the Quake 1 engine at this point, but I’m convinced it’ll improve with time)  – but the thing about real-time 3d applications, games included, is that they have always been about the cheapest hacks available.

With the Chills animation I did things the hard way – hard for me, and unfortunately, hard for Papervision.  It looks pretty – there’s an aurora, reflections, changing colours for the clouds and the sky, dynamically generated igloos and a shaded, textured mountainscape.  And a motion blur.    All good things, save for the fact that it runs at about 5FPS without hardware acceleration. When I added more features in – a better motion blur, more detailed shading and mesh detailing, it brought even the most powerful desktops to their knees.  Screenshots looked fantastic, though.

This is a great movie, save for the framerate.

That’s not a movie, that’s a slideshow

Let’s think of the possibilities for a second.  Instead of being required to produce 25+ FPS, if you could render one frame every second or two, and give the user a fancy zoom/fade/wipe/etc. transition between them,  you’d get away with much more.  It might even give them a more enjoyable experience than struggling to cram everything you can into the constraints of realtime.  If you wanted to go further, you could even pre-render an entire sequence to play back at realtime later – providing you had something to show now, and you could properly bridge the two sequences.

The slideshow idea is still something I’m interested in but we didn’t have time on that particular project.  The second option, using Papervision as a pre-renderer – was impractical because of the first-person, full-scene animation of the thing.  It was an idea I considered, discounted as inappropriate, and forgot about.

Until recently.   Earlier last month, Tim posted about Playfish, who are doing some amazing things with Flash and social media.  After seeing this mention, and generally seeing Pet Society being spammed about for a while, I was curious about their offerings, so I signed up for a few of their games.

I wasn’t impressed with the first experience, but it turns out that Pet Society is probably the weakest title in their stable.  Since then they have made a series of Isometric titles with customizable characters.  Since I’m always fascinated by new technical developments in flash, and the fact that these titlesseem to run surprisingly well given the amount of stuff happening on-screen, I tried to dig up what the underlying technology is.

It’s Papervision, but not as we know it

Minigolf party cleverly sidesteps Pv3d's tendency to crawl

They’ve done what I was pondering over earlier this year – According to some discussion on old papvervison3d mailing lists, they’re using pv3d to pre-render elements like characters to a series of bitmaps in a preload phase, and just playing them as simple movies as required.  This can be seen in titles like Minigolf Party, where the load time includes the pre-render stuff.  In more recent titles like Restaurant City, where there is a constant flow of characters in and out of the player’s view, I’m guessing they’re being generated just prior to display, and (hopefully) being dumped afterward.

RC displays a rolling cast of new, dynamically generated characters

Aside from sounding like a somewhat perverse approach to art, this is genius.  While the artists lose the fine-grained control of painting pixels or drawing out vector points, they can create much more content by rendering all necessary angles automatically, as well as change a hairstyle or a shirt without having to redraw the whole thing.

It cuts down on bandwidth, too – instead of sending out all the character animation that a user might need to see, you send them the model and the animations, and they handle it.

That’s all advantage which is shared between real-time and pre-rendered 3d.  Where they differ, though, is performance.   In most 3d engines, and in most Papervision-based systems, the whole scene is drawn every frame.  Papervision is fast, but it’s still running inside Flash, which means it is (at least) thousands of times slower than a modern game engine.   In order to have any performance at all, the detail has to be thousands of times simpler.

Unless you’re pre-rendering!  For animations that can be repeated (walk cycles, attacks etc), you can take as long as you want to render in a ‘load’ phase, and then just play back the results as necessary.  This only works when you can guarantee you’ll use the same animation at least twice, and what you gain in performance comes at the cost of more storage  – the inevitable space / time tradeoff we always run up against in computing.  In most applications – consoles, mobile devices and such – the storage (in RAM) and processing capabilities are more or less equal.  In Flash, though, the storage is vastly greater than the processing power, so it makes sense to cache heavily.

Since I plan on using using this in my Opticks game,  I thought it would be worth trying out how hard it is to make this stuff happen.  Toward that end, I have been experimenting.  Below is a Playfish-style representation of what I have been doing. To do the things it says to, click in the panel and press keys ( Press “J”! It’s amazing!)

Maybe not everyone.  Gamers seem to be in violent opposition to the idea of changing input methods.  On Slashdot, Gametrailers and other forums I still see people insist that the two-thumb controller is the best input method for platformers, a mouse the best method for Realtime Stragety (RTS), and a mouse and keyboard for First Person Shooter (FPS) games.

Gaming in 1990 (click to enlarge)

I find that reaction disappointing.  For a start, It’s probably not true.  Prior to Quake, most ‘gamers’ (though they did not exist in the same numbers or have the same recognition as a demographic) insisted that a mouse was an unnecessary and fiddly complication to the task of moving a character around, and that looking up or down could be done easily enough entirely with the keyboard. As strange as it seems, gamers and technophiles in general appear to be a very conservative lot.*

Second, even if it were true, and it is a possibility – it is only because the RTS, FPS and platformer genres have been forced to work as effectively as they can the available systems.  They’ve been designed with these input methods in mind, so of course they’re going to seem like the only option available.

Gaming in 2000 (click to enlarge)

When you change the range of input devices, though, the options broaden.  It’s not hard to imagine a shooter where the player can lean around corners, duck projectiles or operate machinery with a little more nuance than just a ‘use’ button.  With RTS games, we could issue two orders simultaneously with each hand, draw out formation plans with gestures or guide the attitude a subordinate AI officer commands with through a facial expression.

While I’m glad that “Motion Control” appears to be the bandwagon to jump on at present, I urge developers, gamers, and the public at large to see motion gaming as a ‘gateway drug’ to better and weirder things. We can turn this into a watershed moment in history not just for gaming but for computing in general, where we decide that the main challenges in computing today are with Human-Computer Interaction (HCI), rather than computation or processing.

Gaming in 2010 (click to enlarge)

One relatively recent breakthrough in Artificial Intelligence is the realization that agents are far, far more effective if they are properly embedded in the environment in which they operate.  That means having both consistency of – and plenty of – sensory information. Problems involving vision become (comparatively) simple when a number of camera views are added and combined as input, the problem of balance is made much easier if acceleration and force data is collected directly from multiple points rather than modelling ‘expected’ values internally.

It follows, then, that the task of making sense of what a user wants from a computer would be made easier by giving it more information.  What may be less obvious is that it works the other way around, too.  The more sensory input from a system we have, the more sense the system is going to make, and the more immersive that system becomes.  The “next big thing” in mobile devices after touch-screen interfaces is tactile feedback from interfaces.  The Nokia Haptikos display will try to simulate the press of a button by shaking on confirmation.

I welcome the shift in emphasis of the console vendors.  We need to ignore the grumbling from current gamers and game developers, who are very stuck in their ways.  These detractors are ignoring the trends in gaming that have led us to this point, and where these trends indicate that gaming will go.  I hope that by the time I have kids who are old enough to lecture, I can tell them to go outside and play a videogame.

Gaming in 2020 (click to enlarge)

* As an aside on the conservatism of technophiles, I remember that, at the ripe old age of 10, I was incensed at the suggestion that people were installing “sound cards” and other add-on hardware into IBM-PC compatible computers.  PCs were supposed to be generalized computing devices! If you wanted sound that badly, you should wait until the CPU could process it all in software! If I had heard that people were going to then go on and make cards that dealt exclusively with 3d graphics, I probably would have passed out from sheer indignation.  I was an unusual child.

Over the course of this Opticks project, I’ve been wondering a lot about that – after adding in lensing and additive/subtractive light systems it started to get important.  The next step is chromatic aberration – making red split from green and green from blue – but what’s the point of this? How far is it worth going?

Fiat Lux!

I think the answer is this:  When you create an artificial system that mimics the laws of nature, you’re in control of when the laws are the same and when the laws are different.  In reality, mirrors are never perfect reflectors – they might reflect 99.99% of light (many do) – but if you bounce light between them indefinitely then it’s eventually going to dissipate.

In Opticks, I can make a perfect reflector – even a mirror that reflects 110% of the light that falls on it.  I don’t know why I’d want to right now, but the point is that I can.

And the errors in simulated reality can look so funny!

Arbitrarily altering the simulated reality doesn’t even need to be useful, either – it can be done just for the enjoyment of experiencing an altered reality.  If you want to change your perception of a thing, you can either change your perception, or change the thing.  Drugs do the former,  simulations allow me to do the latter.  This way, I can take screenshots of what I see!