Fucking [NaN](https://en.wikipedia.org/wiki/NaN). A *not a number* value, invented in 1985 by computer people who had enough of error handling divisions by zero. A stupid concept. A thing so useless, and yet so dangerous, that I don't know how NaNs aren't considered bugs. What the actual fuck?! # Rambling about NaNs Okay let me explain from the beginning. I feel shit. You probably can tell that I feel shit. So NaNs... What the hell are they? And why the hell I feel so strongly about them? ![Dani's Race Screenshot](/pictures/danisrace/DRscreens/26.jpg) I develop a game called [Dani's Race](/games/Dani's_Race). It is a kind of silly GTA clone ( or something ) where you play as a little boy who steals cars and gets into all sorts of shenanigans. At one point, already long time ago, I started trying to make [a multiplayer mode](https://forg.madiator.cloud/BlenderDumbass/DanisRace_Multiplayer) for this game. While developing the multiplayer mode I stumbled upon a bug that kept re-occurring that I couldn't comprehend. Here is a version of the same bug from much earlier. Probably one of the first recorded instances of it happening. ( I posted about it in [the engine's bug-report page](https://github.com/UPBGE/upbge/issues/1872) ). As you can see it is very early *Dani's Race*. Like from the first half of the first year of development. I drive a red car ( called "Redkiss" in the game ) and while trying desperately to control it I run into two strange issues: - First the shadows disappear. - Then the whole frame becomes white. When I was in the mids of the height of the effort on the multiplayer for the game, I ran into this ( or very similar ) bugs constantly. And I couldn't explain or understand what the fuck was going on. Shit just kept breaking all the time. More then a year ago I published this video on Bugs in Dani's Race. It is an excellent starting point to comprehend what is going on with the NaNs. ![peertube-video](https://peer.madiator.cloud/w/9zqtajv8Jxh9Qo44TufQpZ) In said video, which I hope you actually took time to watch, I explain the core mechanics of the NaN problem. The way I found out about it. And the way I ultimately gave up on trying to fix it. To recoup: While testing I found out that just standing in one spot for a long time, might make the shadows disappear. And with that ( which was not yet implemented in the version I ended up bug-reporting to the devs of the engine ) there were some sound issues. Mainly a noise of an engine sounded as if it was right next to the camera. I decided to look into this engine noise, trying to figure out what could get teleported to where the camera is, but what I found was more strange. Instead of a random car teleporting to camera, the random car teleported to coordinates of NaN x, NaN y and NaN z. Just to clarify. In 3D locations of objects are represented with 3 values: XY and Z. Those represent an arbitrary distance away from the center of the world. In Dani's Race those are measured in meters. So if a car in the game is at coordinates 100 x, -200 y and 55 z, it means that it is 100 meters to the right from the center, 200 meters backwards from the center, and 55 meters up from the center. The car, who's engine I heard loudly all of a sudden, was at coordinates NaN, NaN and NaN, which makes no bloody sense. NaN in computer science stands for "Not a Number". How the hell is this car, not a number meters away from the center? What? Well, the game-engine also was like "What?" and couldn't figure out where the sound of the car's motor should come from. So it gave up. And made the sound of the motor non-3D. Which I heard as if I was right next to it. Okay so, something happened. And a car ended up getting those Not numbers as coordinates. How does that effect the rest of the bug? Well here is the thing. Shadows are calculated using a voxel based shadowbox, that is stretched over the entire scene. All objects included. If one of those objects is at coordinates that are not a number, the stretching of the shadow-box collapses, and you ultimately get no shadows ( while technically they are still pretty much calculated ). And the white screen happens when the camera itself gets NaNed. At the time of that video I didn't care much about the NaN issue. So I patched it in a very round about way. I just made a function that runs every once in a while and searches for those NaNed objects. If it finds one. It deletes it. Problem solved. Shadows never go away for longer than a few seconds. And the annoying car engine is a temporary issue. Until the NaN plagued the game again! Not feeling much about the game, I focused on other projects for some time. Which faded interest in its multiplayer and other stuff in the eyes of people. And which ultimately made me forget about the whole NaN issue. Until I had time to burn all of a sudden. I'm pitching a movie script right now. I probably wrote about it here and there. In any case, dealing with producers involves a lot of waiting around. They like to take their time. So I started doing more Dani's Race, this time entering the so-called "Road Rage" phase of development. This time I'm not taking any of it seriously. I'm not "vibe-coding" it. God forbid I ever touch anything even remotely related to that shit. So don't worry about it. I even didn't update the version of the game-engine because [they decided to use AI](/articles/upbge___the_blender_game_engine___is_now_ai_slop). I use the older, human-made version. And now ship a copy of it with the game. The beauty of "Libre Software" I suppose... anyway... what was I talking about? Yes... So I don't take the game seriously, which makes it ultimately a lot more fun in the end of the day. Both in terms of gameplay and in terms of development. I never thought, for example, that I would ever add weapons into the game. But fuck it. We have bazookas and mountable rocket launchers now. And you can shoot other racers. And the police can shoot you. Fuck yeah... Anyway... With the new life in the game itself, there came new interest from the public. And some people ( particularly Miner34 ) expressed an issue that I knew existed in the game. Sometimes when going to the garage ( where the game store your cars ) one or more of those cars will violently despawn from there. There will be a flash of light, a loud sound of the engine, and then a few seconds later, a huge freeze. Well this is the good old NaN bug. But this time not just being annoying. It literally steals cars you worked hard to get and save. When you exit the game it saves the savefile. If you don't know where it is, to back it up, this NaN bug can potentially ruin your whole progress, just like that. Just by being this nasty thing that keeps happening randomly. The flash of light, is the shadows. The engine is the collapsing 3D space. The freeze is the deletion of the car. The patch that I made earlier. This is a serious bug. The problem is, the bug seems very random. Sometimes it happens, sometimes it doesn't. I cannot predict when it will. I don't know what causes it. I was sure it was the multiplayer code. Hell, it started happening much more frequently after I worked on the multiplayer code. Finally today magic happened. I got a savefile ( that I backed up ) that consistently produces the bug. It was a sign. I had to fix it now. # Fixing the NaN bug So the setup was clear. I back up the file. I exit the game by killing it. I start the game and type `GARAGE` which is the cheat-code to teleport to the garage. As soon as I do that, the game spawns in the cars it has recorded into the save-file. And the NaN bug occurs. GOOD! Perfect. Now let's dig. To save and load the car I use a `JSON` formatted version of it, that I can dump into a file, generated by 2 functions in [Scripts.Vehicle.py](/danisrace/download/Scripts/Vehicle.py). Specifically the functions `Encode` and `Decode`. The `Encode` function is straight forward. It just takes whatever relevant info the car has and formats it such that it all could be dumped into `JSON`. The `Decode` part is interesting. It is both used to load in the saved cars from the garage, and both used for the multiplayer, to decode changes to cars, as received from the server. Maybe if this whole thing started with the Multiplayer, the bug is in the code dealing with the Multiplayer. ( Technically it is, but let's not go ahead of ourselves ). So I look into it and see 2 contenders: ``` if "position" in changed: newposition = client.extrapolate(car.position.copy(), data.get("position", (0,0,0)), obj = car) Reuse.Move(car, newposition) ``` We have these 2 functions that I clearly remember writing for the Multiplayer. The `client.extrapolate` function that tries to smooth out the positions of the cars received from the network. And the `Reuse.Move` function that tries to lower the impact of moving cars around on [Depsgraph](/articles/upbge_-_what_is_depsgraph__and_how_to_optimize_for_depsgraph_). Looking at [Scripts.Reuse.py](/danisrace/download/Scripts/Reuse.py) you can see that `Reuse.Move` function looks like this: ``` def Move(object, position, times=1): # Moving object using applyMovement object.applyMovement( ( mathutils.Vector(position) - object.position ) * times ) ``` Yes it looks a bit weird. Why not just write the location directly into `car.position`? And frankly I don't have enough science to see if it works the way it should. If anything, on paper it looks to be slower than writing the position directly. But... Depsgraph is a bitch. And it feels like using this actually improves performance a little bit. Why do I think that I might improve performance? ( I know, we will get to NaNs ). Well... it seems like whoever coded the engine ( UPBGE ) coded a more strict sort of recalculation thingie ( maybe not even Depsgraph, but I think it is Depsgraph ) when you assign things directly to variables on game objects. I talked about something like that in [my optimization video](https://peer.madiator.cloud/w/sfkKtRYpt4ccKsVxZ8fpYd). While running their functions seem to not do that. So think about it this way. If I assign a location to an object, this object needs to recalculate things internally to accommodate my assigning. If I run `object.applyMovement` instead, it doesn't need to recalculate anything. So I could give it only the difference of where it is, to where I want it to be ( which is trivial to calculate, as it is just 3 subtractions (xyz) ), and hopefully it will give me a way to move objects a little bit faster. The problem here is that I might be asking a function ( implemented in C++ internally in the game-engine ) to move an object zero meters away from where it is. Which internally might confuse it and result in a NaN. I know for a fact that the only way to make a NaN in `python` ( the language of the code of the game ) is to do `float("nan")`. As in, to literally tell python that I want specifically a NaN. I tried diving by zero and it just gave me errors. Like it should. I tried messing with precisions. It wouldn't budge. But I know at least one program in C++ susceptible to NaNs. Specifically [Bullet Physics](https://github.com/bulletphysics/bullet3/issues/976). Which coincidentally is the physics engine in the game engine I'm using for my game. And coincidentally the game engine is as well, written in C++. So maybe, by giving an object a request to move 0 meters I cause some calculation internally to collapse? Maybe I cause something on the C++ side of things to give me a NaN? No! I tried everything. The `Reuse.Move` function just won't budge. It is not it. It doesn't do the NaNs. Okay... what else is there? Right... `client.extrapolate` Well if you look into [Scripts.Multiplayer.client.py](/danisrace/download/Scripts/Multiplayer/client.py) you will see this function: ``` def extrapolate(original, value, obj=None, objKey="position"): # This function will extrapolate motion # of objects. # Make sure both are Vectors if obj: try: original = obj["net-finals"][objKey].copy() except: pass original = mathutils.Vector(original) value = mathutils.Vector( value ) # Calculate time offset offset = time.time() - queue.get("time", time.time()) frame = queue.get("time", time.time()) - queue.get("tick", time.time()) # caluclating the values diff = value - original withext = diff * ( frame + offset ) final = original + withext + diff # Smoothing data for the object. if obj: if "net-exp" not in obj: obj["net-exp"] = {} obj["net-finals"] = {} obj["net-originals"] = {} obj["net-exp"][objKey] = ( diff + withext ) * ( ( 1 / ( frame + offset ) ) / bge.logic.getAverageFrameRate() ) obj["net-exp"]["time"] = int(round(( frame + offset ) * bge.logic.getAverageFrameRate())) obj["net-exp"]["time-full"] = obj["net-exp"]["time"] obj["net-finals"][objKey] = final obj["net-originals"][objKey] = original return final ``` Look at this monstrosity. It is definitely it. I have FPS average stuff that I take from engine. I shove numbers back into engine. I do calculations with those numbers that could be unpredictable. It must be this function. No! I looked at the code a bit closer and found out that it not just not this function. This function doesn't even execute when loading cars from the garage. It is something else. I was desperate. I Decided to just dump locations of all cars per each frame of the game. And see what would happen. I see they spawn in. The locations are good. No NaNs. This is not even `Decode`. The cars spawns good. What the actual fuck? Next frame, the car is still there, still good. Next frame. Still there. Next frame? NaN! WHAT? WHAT? Sorry, WHAT? So the car spawns in normally. Doesn't break what so ever. 3 frames later suddenly decides to fucking NaN? I decided to comb through everything. What if I don't run it's Update function? Huh... the NaN problem disappears. Okay... what does the Update function run? `PhysicsUpdate` ? Not running just that... also no NaN. Hm... wait a second. I look inside. I see this: ``` def PhysicsUpdate(car): # Updating the car physics to # work with a more realistic # effect, while cheating at some # things. scene, car = GetSceneAndCar(car) toFPS = Opt.ToFPS() carD = Opt.FastDict(car) # If car has no physics yet, give it physics initfirst = False if not getCarPhysics(car): InitCarPhysics(car) initfirst = True # Forces ( Drag Force, Down Force ) x = 0 z = -DownForce(car) df = DragForce(car) car.applyForce(df, False) car.applyForce([0, 0, z], not car.get("underwater", False)) dirt = False ground = OnGround(car) if "ground" in str(ground).lower(): dirt = True s = 50 * abs(car.localLinearVelocity.y) vec = (random.uniform(-s, s), random.uniform(-s, s), 0) car.applyTorque(vec, True) if car.get("active") and ( dirt or ( abs(car.localLinearVelocity.x) > 2 and ground ) ) : speed = math.dist((0,0,0), car.localLinearVelocity) if speed > 10: Input.VibrateJoystick(speed/100, -car.localLinearVelocity.x) if car.get("active") or Opt.GoodFPS(str(id(car))+"_physics") or initfirst: GetPhysicsForces(car) SwingDoors(car) for n, wheel in enumerate(car["specs"]["wheels"]): wheelObject = car["wheels"][n] health = wheelObject["health"] ** 2 # Damping suspension = wheelObject["suspension"] * health hbf = 1.0 if car.get("handbraking"): if not wheel.get("front"): hbf = 0.5 else: hbf = 0.6 # Increasing suspension based on how much # the wheel is pressed against the bottom of the car. #if car["specs"].get("suspentionCorrect", True): #hight = car.getVectTo(wheelObject.position)[2][2] #if hight > 0: # suspension = suspension * max(1, ( 1 + ( ( hight ) * 20 ) )) #suspension += abs(z)/10 k = car["specs"]["suspentionDamping"] suspensionDamping = 2 * k * math.sqrt(suspension) getCarPhysics(car).setSuspensionStiffness(suspension, n) getCarPhysics(car).setSuspensionDamping(suspensionDamping, n) DFS = car["specs"].get("driftSpeed", 50) DRIFTINESS = car["specs"].get("driftiness", 1) DF = max(0, min(1, abs(car.worldLinearVelocity.y) / DFS ) ) if car.get("breaking"): DF = 1.0 try: DFT = max(0, min(1, (math.dist((0,0,0),car.localAngularVelocity) / car["specs"].get("maxAngularVelocity", 1.5)) / DF ) ) except: DFT = 0.0 DF *= DFT grip = car["specs"]["grip"] if (car.localAngularVelocity.z > 0) != (car.localLinearVelocity.x > 0): dgrip = wheel.get("drift_grip", car["specs"]["grip"]) grip = (grip*(1-DF)) + (dgrip * DF) car["DriftFactor"] = DF # if car.get("breaking"): # DFS /= 10 # DRIFTINESS *= 10 # DF = 1 / max(1, abs( car.localLinearVelocity.y ) / DFS ) # if abs(car.localAngularVelocity.z) >= car["specs"].get("maxAngularVelocity", 1.5) * DF\ # and Turning(car) >= (DF/2) : # grip = wheel.get("drift_grip", car["specs"]["grip"]) # car.applyTorque(car.localAngularVelocity*10*DRIFTINESS, True) # else: # grip = car["specs"]["grip"] # Simulating sand and dirt and stuff. roll = car["specs"]["roll"] if dirt: grip *= 0.4 roll += 0.8 grip = grip*math.sqrt(car.mass)*hbf getCarPhysics(car).setTyreFriction(grip, n) getCarPhysics(car).setRollInfluence(roll, n) # Applying the engine acceleration # to the wheels. ApplyEngine(car) # Making sure to clear the acceleration # for the next frame. # if not car.get("braking"): # car["driftturn"] = 0.0 car["accelerating"] = False car["braking"] = False ``` Surely somewhere here, somewhere in the `getCarPhysics(car).setTyreFriction(grip, n)` or something there is NaN. I disable all this code. No... Still NaNs. What? Where is the fucking NaN? `ApplyEngine(car)` `Apply` fucking `Engine(car)` !!! What? What could possibly be NaN worthy in Engine function? I look inside: ``` def ApplyEngine(car): carD = Opt.FastDict(car) toFPS = Opt.ToFPS() hp = HorsePower(car) force = HorsePowerToWatts(hp) / 100 grip = car["specs"]["grip"] # Here is a little cheat to help the car start notresistance = EngineResistance(car) if car.get("gear",0) == 0: notresistance = 1 FO = force * notresistance / GetGearRatio(car) / grip T = time.time() if force > 0: D = max(0, FO - carD["prevF"]) else: D = min(0, FO - carD["prevF"]) # TODO, this is a temporary fix. For some reason small cars such # as scooter and kart overaccelerate when this is turned on. # The math is designed to give players the same acceleration across # different performances. # TODO, investigate this for causing strange NaN bugs. # So fart the car.get("accelerating") stops most of it # from happening. But maybe there is more to it. if car.get("accelerating") and car["specs"].get("type") not in ("skating vehicle", "kart"): try: calc = ( 1 / ( D * ( ( T - carD["prevT"] ) ** 2 ) ) ) F = FO + calc except: F = FO else: F = FO carD["prevT"] = time.time() carD["prevF"] = FO for n, wheel in enumerate(car["specs"]["wheels"]): wheelObject = car["wheels"][n] health = (wheelObject["health"] + carD["health"]) / 2 rightwheel = ( wheel["front"] and force > 0 ) or ( not wheel["front"] and force < 0 ) if rightwheel and ( not car.get("braking") or car.get("drifting") ): getCarPhysics(car).applyEngineForce(F * health, n) else: getCarPhysics(car).applyEngineForce(0, n) # Decrease RPMS if not car.get("accelerating") and not NetworkControlled(car): carD["rpm"] *= 0.98 * EngineResistance(car) ``` What do I see? Oh! I see this: ``` ( 1 / ( D * ( ( T - carD["prevT"] ) ** 2 ) ) ) ``` This was another thing developed for the Multiplayer. It was co-developed with RowdyJoe. We were testing the multiplayer and found out that on the same car we had slightly different acceleration. The problem was with the granularity of the acceleration. See the physics updates were not happening in the same exact rate because we didn't have the same exact frame-rate. And therefor there was slight de-sync in the speed of the acceleration. Joe had an idea to use time-deltas to combat this problem. We would basically add a missing part of the acceleration to the engine, based on how much time was between this and the last frame. It worked. The cars felts much closer in speed. I tried removing this math operation for the test. And believe it or not, the NaN bug disappeared. I added a small check before it does this math, to see if the car is in fact trying to accelerate. And so far, there is no problems with this. But here is the problem I have. How? Why? What is it about this specific formula? Why this? When I ran the tests, the NaNed car, just before NaNing, had this formula output 1.63 ( roughly speaking ). This is not 0. This is nowhere near what should be NaN worthy. What the fuck? 1.63? And it fucked up? What? This NaN thing is a fucking rabbit-hole. What the fuck? WHAT THE FUCK?!!! Does anybody have a clue? A theory? Something? Please!!! I have to know!!! **Happy Hacking!!!**