let it leave me like a long breath

i ended up writing a todo list when i started and took some degree of day-by-day notes for this project, so i figured i might as well put together a writeup.

august 1 - 14: PICKING

original todo list:

• drawing an outlined hex prism
• drawing an outlined tri prism
  • ... at the correct grid locations
• picking hit from camera pos along front vector, hitting map plane surface
  • ... unlatticed into a grid coordinate
• full picking
• render tweak to reuse gpu buffers instead of reallocating
  • ... try to extend tweak to map surface
    • ... might need to add the (Int, a) -> (Int, a) -> Ord code in the spandrel math to make sure the surface remains coherent

picking-adjacent things:

• adding height & getting picking working for tiles of different height
• calculating what's under the hit
  • displaying a ui label or w/e for the thing that's hit (loose sand, cracked sand, well, scrubgrass, etc)
  • different picking shapes for hitting a tile vs. a plant (a la minecraft)

(stuff that's struck out is stuff i finished)

day log:

1. the rendering for having successfully picked a grid tile is just a prism, and i have a function to make a prism polyhedra alread: prism :: Int -> Float -> Polyhedra () (V3 Float) (). but, the only rendering functions i have for polyhedra 1. draw them as [TRecord a], which is something i'd like to move away from, and 2. draw them solid.

   i had vaguely recalled that gpipe didn't support drawing line or point primitives (so, triangle only), but that was wrong: it fully supports them, it's just not a matter of switching a single render call from GL_TRIANGLES to GL_LINES; you need a separate buffer type. so you need to generate different data to render lines vs. tris. this also required me to change the way my render chunks worked: they used to have TriangleList hardcoded, and i changed them to track their rendering topology so that it could be anything.

   then i wrote some functions like

   polyToFaceChunk :: (MonadIO m, ContextHandler ctx, BufferFormat f, BufferFormat v)
   	=> b -> Polyhedra e (HostFormat v) (HostFormat f)
   	-> ContextT ctx os m (RenderChunk os Triangles (f, v) b)
   
   polyToEdgeChunk :: (MonadIO m, ContextHandler ctx, BufferFormat e, BufferFormat v)
   	=> b -> Polyhedra (HostFormat e) (HostFormat v) f
   	-> ContextT ctx os m (RenderChunk os Lines (e, v) b)
   
   polyToEdgeChunk_ :: (MonadIO m, ContextHandler ctx, BufferFormat e, BufferFormat v)
   	=> b -> Polyhedra e (HostFormat v) f
   	-> ContextT ctx os m (RenderChunk os Lines v b)
   
   polyToVertexChunk :: (MonadIO m, ContextHandler ctx, BufferFormat v)
   	=> b -> Polyhedra e (HostFormat v) f
   	-> ContextT ctx os m (RenderChunk os Points v b)

   that take a polyhedra and turn it directly into face tris, edge lines, or vertex points, based on its structure. that was neat b/c that has a nice symmetry between Polyhedra e v f and the three different output topologies. (it's worth noting that as far as gpipe types go, these are simple)

   this also meant i needed to write a new shader, since shaders... okay i don't precisely know how gpipe shaders work, and which type constraints are ones that i'm putting there myself vs. ones that are inherently part of its system, but generally speaking i'm probably gonna want to write a new shader for lines vs. tris. so i did that, by stripping down my current shader a lot and trying to get it to still function.

   all that done, and i make a prism, turn it into a render chunk with those functions, and then use that render chunk with the outline shader to make a render object that i could then render. rendering them at a coordinate is just calling render on that coordinate (which is unrelated to actual rendering and is just render :: Hex -> V2 Float) to get its position in space. so that's effectively all the actual rendering math done, which is the easy part.

   the picking hit requires getting camera position and (i thought at the time) the camera's front vector, and i mucked about a little with that and didn't really get anywhere. i was vaguely aware this would involve a raycast from the camera, somehow affected by the mouse cursor position, but the details escaped me at the time.

2. it came to me in the night: opengl clip space is a cube, and it's actually very simple to construct the clip space position of the mouse. this is when i looked up gluUnProject. i had the vague idea that if you could just calculate a matrix inverse for the perspective/camera matrix, something i have no clue how to do, you might be able to run the computation in reverse and turn a clip space point into a world space point. from this, i wrote screenRay (which returns the clip space mouse points for the near and far plane) and started work on cameraRay (which would transform the camera plus the mouse cursor into a point and vector in world space).

   at the time i was thinking 0 was the camera near plane (not -1) and my cameraRay math kept being confusing in a hard-to-articulate way that maybe had something to do with scale. i did a bunch of ad-hoc scaling math that seemed to get a more-or-less correct result, but things didn't seem totally correct

3. i just could not figure out the camera scaling math. in the mean time, i restructured the way the ground mesh was generated -- previously there was a weird blur that happened on the spandrel tris between hexes, because they were calculated based on height and totally ignored the a value (which in this case was later used to calculate color). since i wanted the tris to all be one color, at first i had separated them out into separate 'connective layers', which was a gross hack that sometimes lead to missing tris entirely. at this point i added an a -> a -> Ordering function to the spandrel function, and used that to sort them by height and then a value. this meant that it would actually average out the values correctly, so that the generated mesh was actually continuous. it also caused further ugly glitches later on b/c just merging two ordering functions together isn't really what i wanted, but for now it was fine. i also added some ui to render text saying what a tile was when you mouse over it.

   this is when i started thinking of colliding the camera ray with the grid. previously, i was doing a simple ray/plane intersection with the XZ plane that was where the (completely flat) map was rendered, but i wanted to make steps towards adding height at some point, which meant that i wanted to have a list of possible hex collisions, in order, so i could check each one in turn and stop at the first hit. so, an actual collision raycast. that was all scary geometry math, so i didn't get around to it, but i roughed up some vague concepts of the data i'd want to store: each hex hit, plus the entry and exit points.

4. here's when i started taking more prose-style notes instead of just a list of functions written.

   wrote code to try picking through the hex grid. there are some precision issues so it fails sometimes still. also it's making it more clear that the camera picking ray isn't at the right positon or pointed in the right direction, which is presumably part of the whole ad-hoc scaling issue

   as you can see i was still confused about the camera thing. in retrospect i can say that the camera ray was in the right position and it was pointed in the right direction; the issue was that my visualization was flawed. but obviously, if you don't know that, seeing some weird outputs in relation to the camera code that you're already suspect of makes you kinda think "maybe this camera code is more busted than i thought".

5. i had thought maybe part of the scaling was something to... scale the vector in accordance with the scaling of the hex grid polys, since that's what i was testing it with to make sure it was positioned correctly, but the hex grid isn't actually scaled.

   i went through a series of checks. at first my assumption was that the scaling was somehow influenced by the camera distance value. that didn't really make sense since that was part of the camera matrix and would presumably be inverted along with the rest of the matrix math, but it had a vague sense of 'maybe that's right'. eventually i did the math and realized that the ratio between the near and far plane would always be the same, because whatever the camera distance was set to they'd just be a multiple of that, so it couldn't possibly be a varying quantity based on the camera distance.

   i also thought, as documented here, that it might be that i was scaling the hex grid, and since "does it hit the hex grid in the right place" was how i was measuring the correctness of the camera ray, if i was scaling the grid then maybe the ray math had picked up the same transform as i was doing to the hex grid. but i wasn't actually scaling the hex grid! i was very confused about where the scaling could possibly come from.

6. the scaling problem was due to not normalizing the vectors by dividing by w. so doing that removed all the weird scaling math, but didn't change the output -- it was still working in a way that wasn't what i wanted

   turns out i was dumb and was totally forgetting about w. i went over this in the other post so i'll leave it out here.

7. turns out actually doing the 3d math was what i wanted, and once i stopped rendering the output by projecting it into 2d and started rendering it by showing which hex prisms were being hit by the ray, everything started working exactly as expected.

8. added a height value to tiles, using a simple waveform generator

   got picking working basically correctly for tiles with heights

9. re-added slopes. didn't change the picking so it's slightly inaccurate but not in a major way

   there's some kind of infinite loop or endless picking or some other error that happens. maybe related to bigger maps? or just moving further away from the origin.

10. the hex raycast function has a lot of unnecessary strictness in it, due to being wrapped in Either. since the type was Either String [HexHit], that meant if there was an error anywhere during the hit it would return Left error; if it didn't it would return Right hits. so even if we were just reading the top five hit volumes before finding a hit, it still had to evaluate the entire raycast, to the range limit, in order to verify that there weren't any errors during that process and that it should return Right hits in the first place. i changed it to [Either String HexHit], which is a little more annoying but means we can just uncons each hit attempt at a time and we don't have to care about (read: evaluate) anything that happens after we find a hit.

    this doesn't solve the picking failures, but it does make picking in general a lot less costly, and seems to have resolved the incredibly slow-to-the-point-of-freezing picking on larger maps

11. solved most of the picking failures. they were caused by an error in the unrender code; the equation was:

    unrender :: V2 Float -> Hex
    unrender (V2 rx ry) = Hex (round x) (round y)
    	where
    		x = rx / 26
    		y = (ry - (x * 15)) / (-30)

    when the line for y should've been

    y = (ry - (fromIntegral $ round x * 15)) / (-30)

    since if we're inverting the render math, which is just

    render :: Hex -> V2 Float
    render (Hex x y) = V2 rx ry
    	where
    		rx = fromIntegral x * 26
    		ry = fromIntegral x * 15 + fromIntegral y * (-30)

    then x must always be an integer, so any fractional component involved will skew the result. the picking failures happened when the skew in combination with the camera position and orientation was enough to get the wrong hex for starting the raycast.

    there's still a picking failure that crops up rarely, but it's for like a single pixel and is presumably actually precison-based, and at that scale it's probably not worth it to fix right now. maybe later if it turns out to actually cause problems.

    ...that's picking all done, i guess. huh. when i wrote that todo list at the top, i was really expecting to not get everything checked off. i mean, i didn't check off the stuff that was actually rendering stuff, but that's fine since the next thing i'm gonna do is rendering stuff and that was more there to give me a lead-in rather than being stuff i seriously expected to get to. but it's been years since i last tried picking stuff, and my picking code has always been kind of a buggy mess, so it's pretty wild to have all of this coded up and working in under two weeks. this is kind of what i was talking about in that last post about these two week projects actually focusing me on a single thing, since if i had tried doing ten things at once i would've slowly picked away at all of this and made no progress and given up. so. it's nice to actually finish this, i guess.