Figured id make a post id love to see a little 5 minute video once a week for the guys at uber, just discussing on future ideas and how the builds are coming along, allow the community to talk about it and give some input Just a thought, let me know what you guys think
Actual diaries (ala .plan files) is far more interesting than a video camera in the office. THat said, I barely can remember to update my blog. Not sure most folks here will remember to update a .plan type update with any regularity. But it's certainly a viable idea.
Thansk warner yeah just figured could be a nice idea to give another level of input from the community before the next update
So when I'm working on a major coding project that's researchy I tend to write down a lot of my ideas in the form of tasks lists. This is basically what was left when I handed off planet generation to others. It's basically in the style of the task lists that Carmack would write back in the quake days. This is literally how I organize stuff for my brain when working on this kind of stuff. I'm leaving this completely raw so don't try to read too much into it. The status isn't always correct either and some things have diverged from this. However, since you guys are awesome backers and I've been away for the last few days I thought it would be a good time to dump it on ya'll. This is directly from planetbuilder.cpp as I tend to write the notes in the source files. /* [X] draw a color per face so we can see wtf is going on (switch to vert colors) - csg [X] - move into it's own file [X] - modify such that we keep track of the "leftovers" when doing csg [X] - make sure the ordering is correct by writing more tests - currently working on getting a test case working, fixed a few things already [X] - decide on rules for all cases and write example code - think I have them all? regardless moving on to using it more [X] - support plumbing through materials or be able to get original brush id somehow [X] - support plumbing through texture coordinates [X] - re-run tests to make sure bounding box code didn't **** them up. - have bsp do [X] - our hueristic code is N2 fix that [X] clean up bsptree by moving it into manifoldgeom.cpp [X] don't cut the polys when building the bsp as you don't need to do that work (I don't think) [X] make sure coplanar stuff is handled correctly (maybe just flip the current test) [X] balance the bsp [X] potentially use fake planes (kinda kd tree like) [X]- when filtering polys do a better job of not splitting - use recursive re-merging from the output list [X] - try merging out piece of loop [X] - when building tree coplanar stuff should be removed or normalized do as not to introduce errors [X] - re-write merge loop to not be N^2 - sorts the list then iterates it once doing the combining [X] - write a post filter merge that merges as much as possible when outputting final triangles - when re-merging check if we can merge an edge if it's colinear - so I had this working but it didn't seem to be performing as expecting so I took some time to figure out where I was going wrong (writing diagrams etc). Unfortunately it was simply me misinterpreting the data. [X] - allow building bsp with subset of faces that overlap a particular bounding box [X] - continue to tweak episolons and try to improve robustness as issues crop up related to testing csg [X] - when filtering through make sure we aren't filtering gons that don't intersect for gods sake, looked like this was happening. [ ] - we need to look at how slow it is sorting the polys through the tree as I think there is a lot of room for optimization there. [ ] - also bsp building can take a significant amoutn of time in some cases, we could build split archetypes for each brush which means it would go way faster.. like way faster so I should try this out - have manifold do [X] - add AABB [ ] - fix t-junctions? [ ] - triangle LOD decimation algoritm? or do we even need that? I would argue yess [ ] - connectivity [ ] - snapping to a grid [ ] - non-manifold cleanup [ ] - when we are looking at whether an edge can be combined we should also consider if flipping the edge would make for a better tesselation Planet builder: [ ] - generate a nav mesh from it [X] - create the concept of sampling cube so we can iterate over the surface in chunks that become plates, or something [X] - "cubemap" that can be used to store surface data on a grid with some sort of iterator (talk to william) - would allocate an array for each face to hold info at a certain density including the "height" and other data that can be used before the actual cube is generated. - this would then store an element at each spot for height or other attributes we want for eventually generation [ ] - different functions, chords, spirals?? [X] - basic terrain type gen and coloring for algorithm development [X] - generate **** water mesh [ ] - generate water mesh (properly this time but may be able to just intersect) [X] - it's sorting now [X] - fix the wrong lighting on the planet [X] - make subdivision of the planet be proportional to surface area [ ] - have things like bumpiness get driven from terrain type so we can have different types of terrain on same planet.. [X] - feed the simplex an offset or something that uses rand() so we don't always get the same planet [X] - feed size of planet into simplex somehow maybe in auto-scaling or something based on feature size. [X] - get brushes wrapping onto the surface [X] - ha, apparently my projection code needs a bit of work [X] - write a function that wraps a brush to the surface with 0 in z being the surface [X] - figure out wtf is up with the normals on the cube . They are wonk [X] - managed to **** up subtraction when I fixed the bounding box thing. It's probably an inside/outside thing [X] - move cubemapsurface into it's own file [X] - pass the cubemap surface into the brush [X] - refactor the cubemapsurface code and move into zu [ ] - get it sampling correctly at the center and along borders [X] - get bilinear samples working (with gradients) - still need to make it wrap on the faces correctly [X] - create spec lib [X] - make python copy them [X] - allow manifold geometry to be init'd from a papa file. [X] - perhaps keep around the link to the papa file so we can use it for materials since we need that anyway? [X] - nope, just decided to add materials to them [X] - modify code to assume manifold now has materials associated with it [X] - get a sample brush in one place on the surface [X] - get a bunch of them spread out - did so in a hacky way. [X] - rewrite manifold geometry to use materials per face. [X] - define brush blueprint and get brush loading [X] - have them convert to a manifold in the bp immediately on load or keep the papa file?? [X] - have the planet actually have a list of brush specs that we sample from for each terrain type. [X] - and the planet type has a list of terrain types, which have lists of brushes [X] - start drivint the terrain types from properties and mess around more with random generation [X] - update planetbp and actually use it [X] - planetsurface should have an addbrush function and then it should keep track of them and be able to generate the meshes later. changed my mind, instead just made csg be able to add a brush early [X] - create a function that can render weights onto our grid for where we have placed brushes. This then let's us evaluate how many other brushes are there for placement purposes. [X] - implement a planet surface iterator that can do sub surfel iteration for placing things like trees and rocks [X] - add features as just meshes that don't do csg [X] - define terrain type blueprint and get loading [X] - water mesh needs to be rendered separately so it sorts correctly. [ ] - define sliders and how they work with different terrain types [ ] - implement more types of data like distance from water, precipitation, wind?? [X] - create a feature spec [ ] - define basic terrain textures [ ] - generate terrain UV lauout [X] - check the bsp rebuilding every time to see if we do it whether we need it or not [ ] - create a a rule spec and use it to drive creation - including simplex type - after we have it figured out locally. [X] - create simpex noise sample spec so we have a way to control simple noise. [X] - try to figure out a consistent way to generate height data across planet sizes. - turns out using the position and then have a reasonable scale makes this work [X] - implement more params for biomes including using simplex as selector for multiple biomes that make sense - implement desert using distance from water - use simplex to choose between variants [X] - when referring to heigh in bp's I think 0..1 makes the most sense [X] - make planets that mirror in 2 or more dimensions for equal play - for placing props use simplex as a density function [ ] - look at normal sampler and see if we can get it correct [X] - need to figure out how to sample height at position on arbritrary mesh for adding in features [ ] - swirl function [X] - rewrite papa2manifold to not use addtriangle [X] - make an append geometry without the matrix that doesn't mul - did it, not worth the maintenance headache [X] - everything seem to be working-ish but we need to turn merge splits back on - not sure what's going on - the bottom level stuff is fuxored need to fix it still [X] - add centroids into faces and use those cause they are fast - currently turned off, doesn't help [X] - make water mesh entire sphere and use z-buffer for now - need to make it render second somehow [X] - check if the new appendGeoemtry is faster or slower in practice. - it's faster [X] - could potentially build the bsps only once for each type of brush - won't work because we project the brushes onto the terrain. [X] - have the builder take params about how much of a build it's doing. [X] - experiment with not building a unique bsp - made things far worse [X] - we need a new overall plan to make this thing fast. I think it would be reasonable to break the world up into a bunch of meshes which are basically just bounding boxes (basically sort into a bunch of bounding boxes) then have each of these bounding boxes be a brush and process it that way. [X] - be able to just build a bounding box worth of mesh data - scratch that, really just want to be able to add a brush to the sequence and we can keep our previous state so that it's fast to modify something I think. [-] - do the csg on verts that are just position and then interpolate later based on srcindex for more potential speed [X] - it looks like to do my thing to break the world up into pieces I might need to supply fake split planes for the geometry of the ground. I can generate these directly but it might also be interesting to do some work on manifold to figure out which ones are the open edges. [X] - add profiling data to mesh building [X] - save final state of brushes and continue from there [X] - Don't forget to max the bbox tighter for the wedge pieces by slamming it somehow or something [X] - the next big problem to solve is getting the brushes laid out nicely - need more perf for that - so figure out what's slow! - then do some optimization (BVH?) - can we build a subset of the planet? [X] - get incremental showing again and then show biomes, then preview, then final a bit at a time [X] - create some intersection functions that can be used to test the geometry [X] - implement bvh inside of a manifold so we can find individual triangles [X] - use this for the edge testing stuff [X] - spray features onto the terrain using this technology on the final mesh [X] - fix lighting normals [ ] - feature layout [X] - create a spec for a clump of features or an individual feature with density/spacing w/rules [ ] - layout features after csg phase as we need actual geometry [X] - write/find triangle intersect (use a bvh?) so we can place the feature on the actual stuff - still need to do the bvh part [ ] - need a guassian function for the clumps. [ ] - layout rules - create a rule class then have brushes and features access it since they can both use similar(ish) stuff - clump, single, on terrain vs final mesh, or on brush surface if it's a brush? [ ] - terrain textures - simple repeating or a shader projection even easier [ ] - clean up manifold geometry so you can't use it wrong by accident (e.g. track which data structures need to be updated correctly) [X] - use the bvh during the generation phase [ ] - nuke triangles within bbox then replace and re-gen - actually need to break out meshes based on the base meshes and only recalculate those ones so we can erase some specific bits or something erm... might need to edit the mesh in place or something or use the lod system should have some kind of bvh of all the brushes so we can quickly figure out what has changed as well [X] - need to add a bvh of the final mesh positions for the raycasting [ ] - save off the "base" planet and be able to rebuild pieces of it at a time [ ] - force entire biome for testing? [ ] - look at variable sized layers [X] - threshold / weight code needs to be more accurate [X] - implement room around things - use scale and have an "agreesive flag" [X] - add in 4 weight layers [X] - check all 4 layers (and decide which matter by having 4 thresholds) [ ] - try all returned brushes this will be cool [X] - look into simplex noise range [X] - add_previous seems broken in the sampling part [X] - also inherit previous noise function or specify a seed [X] - use the bias to figure out the order of the brushes for placing them down - ended up adding order and not using bias for now [X] - think about different orderings to iterate the cube for the layers - by noise value? - iterate the cube once per biome per layer? could work algorithm: - generate noise values - sort surfels by noise value - iterate in sorted order from highest to lowest noise - check multiple brushes that fit in order of bias [X] - make bias randomly select the order amongst brushes that have an equal ordering [X] - add a simplex to the layer if we want a totally new one [X] - think I broke the heightscale in the biome selection - turns out I was just a moron and I had the simplex displayed [Z] - count largest number of layers in any biome and use that for our layercount [X] - make features work so we can do tanks [ ] - look at erosion pass [ ] - more work on features [X] - clumping mechanism? Gradient sampling? something else? - gaussian looks pretty [ ] - control clump size, density should be based on noise - attachment to brushes (and brushes to brushes?) - selection similar to brushes order/bias - randomness in angle from ground [X] - potential height randomization default to off [X] - layers for features with controls [X] - featurespec (that is a brushspec?) - just added fields to brushspec [X] - use bias to select from list. [X] - clump size control/falloff [ ] - should clumps randomize within themselves? Probably...? [ ] - some sort of "maxdensity" to prevent too many features? maybe just a count? [X] - add in features into the brush layers [X] - for features add an offset random amount from their position for the "Starting pos" of the clump [ ] - think about procedural functions other than simplex like swirls for things like trees/rocks etc - also clumps of brushes than do more complex csg ops like canyon's / trenches / erosion using csg cubes? other cool **** we are just scratching the surface.. [ ] - what about the idea of unifying brush/feature code and adding clumps to features [ ] - try to fix our cubemap by repeating the border texel around it or something and then copying this later? - maybe make the actual cubemap index map just point to the surfels to the edges can point to the same ones.. this could work [ ] - blur weightmap and make available to features to select from [ ] - look at some kind of distance related brush function. - maybe some kind of edge detection where we have splits... go along the edges some kind of function - ambient occlusion map / edge detection - do some simple ray casts for ambient occlusion -- Texturing stuff -- [ ] - get textures sucked up and accessible [ ] - add texture stuff to biome [ ] - get basic texturing per biome showing up [ ] - put together plan for virtual texturing. [ ] - what about the idea of unifying brush/feature code and adding clumps to features efficient GPU roundtrip: - create an FBO and map it use glmapbuffer (or something) - have 4 of these buffers - render to the buffer at lower resolution - sync object for current frame - sync to sync object from 4 frames ago - then read back the buffer from 4 frames ago using the mapped buffer (we leave them mapped) - don't forget asyncbit for ati - then we can update our pages using some out of date data but it should be fast. - can we do part of the framebuffer so we don't need to render everything? UV unwrapping plan: - use a projection for the repeating texture, don't worry about the edges for now, projection can be centre of box or something - have artists unwrap other things so they are in the 0..1 range (e.g. unique texels everywhere) - is this cool? - should be consistent with surface area and unwrapped as nicely as possible - we then remap into virtual space using a surface area multiplier to give us an even amount of texture space - figure out surface area of mesh so we can allocate that much texture space to it's 0..1 range? - keep track of these which are a position, scale using a quadtree? - Could I do some kind of utterly bizarre triplanar virtual mapping? Where I render into the virtual texture from 3 different views and blend them together are runtime. Lots of texture samples and lots more rendering but no texture seems ever?? Triplanar has 3 textures combined to give us a final value. However, we are pre-rendering can't we render from all 3 sides and combine to give a value for that texel? Then we just need to have 3x the surface area possibly just by making the texture by 3x deep (e.g. 3 cache textures) one for each dominant angle oh crap that's 3 sets of tex coords though. can we do some sort of paramaterization in the shader itself that's per pixel or something? Like an index thing or something innovative? texture geometry image type thing? What about 3d texture mapping where we don't need to chart in 2d? hmmmmm that might be an interesting thing to have regardless write some renderer debug stuff that uses the console for funsies [ ] - verify brushes are manifold on load and ignore broken ones [ ] - replace textured shader with our mip calculation shader. [ ] - fill in a texture with the page addresses of all of the pages [ ] - this is actually a shader so get that working with an override - make renderer aware of this and get round trip thing going... ugh - get it rendering using current charts - lay out charts and get it rendering - now read back the framebuffer to get a list of active pages General renderer: [ ] - implement and use a cube map for lighting the planet or a spherical harmonic [ ] - debug modes like wireframe, show splits etc. [ ] - work on the planet camera [ ] - optimization and some perf measuring of all this nonsense (moving bsp and making it internal will help we we'll move less data) - consider re-writing the cubemap stuff in terms of a mesh surface paramaterization (e.g. we use a manifold geoemtry-ish style mesh and then iterate over it's entire surface generation stuff everwhere, this way the gen algorithms could work on any shape?) ================== ------------------ ================== - calculate the bounding rectangle inside the full mip by using the correct size and shifting by mip level - go through the brushes and find ones that are mapping into that triangle (touches rect function?) - create a scene that contains the necessary meshes for the requested page - pass that scene into ras ren using the make resident api - render these into a fixed page size framebuffer (show this buffer on the screen) - copy into our physical texture ---------- - decals and trees - water flow issues... */
Thanks for this neutrino - makes a very interesting read EDIT: Perhaps some of our more tech-savvy forum members can decipher this a bit for us?
In short: optimizing map making map making lighting map making TODO: swirl function optimizing map making It's all under the hood stuff that, if it works properly, you won't even know is there.
Swirl function is for procedural generation I'd guess. Can't wait to see that, though I'm curious as to how path finding will handle it. Hopefully units will be able to travel through shallow water by then.
Eugh man ive still go along way to go as a programmer just looking at that made my head hurt XD ha ha definitely going to need a few more years experience before I apply for a job at Uber
I'm really glad that this is still being looked at. I think it was one of the more interesting things of the aesthetic style of PA, and I do hope it can make it to the game. I'm also glad you were able to take the time to throw us that todo list. It's possible that I'm imagining it, but I feel that some of the more interesting discussions on the forums have dried up of late. The biggest aspect of this comes from Alpha feedback. See, most people here are generally helpful, and want to give useful feedback to the devs about the game they are playing. The issue is that they often don't know how. In my experience, there are four types of feedback that a player can give. The first is the most irritating. It's describing a feature or bug that the devs are already aware of, and simply haven't got round to looking at yet. This is the endless hordes of posts which used to wonder why metal spots didn't work, and simply didn't understand that games aren't built in one go, but are developed incrementally. The second are those who just don't get it. They are suggesting a feature that isn't just something that the dev's haven't got around to yet. It's something they are never going to get around to, because it's beyond the scope of the game. The third is the most obvious and useful. It's hunting down bugs that the devs might not have found yet, or don't know the full scope of. Most of this will be on the bug tracker. In my experience, I've yet to find a bug that's not already on the tracker when I go to look. And finally comes the one in a million post where the poster puts forth a interesting idea that the dev's hadn't thought of yet, and they decide it might be with thinking about. I don't know if there have been any of these on these forums, but they tend to be pretty rare compared to the other three categories. In my experience, the best way to get less of the first two, and more of the last two is community interaction. I know it's a pain sometimes telling people what you are doing, rather than getting on and actually doing it, but it really does help improve the signal-to-noise ratios of the forums.
