BSP Size
A few things you can do to reduce bsp size:
Optimising
Lowering lightmap scales
Nodrawing not-seen brush faces. This applies to func_details the most.
Converting sounds to mp3's and compressing custom textures (best compression allowed with a vtf)
Minimising custom textures and content as much as possible. If there is a similar default texture, use it!
Reducing the amount of brushes in your map
Using LOD model versions for distant models
Use cheap textures for distant textures (if available)
Use dynamic lighting (or real-time lighting) in appropriate places.
Remove unnecessary props and brushes.
Thanks for the tips, I'll try to cut some things down.
EDIT: Do displacements increase the filesize loads? That'd be one cause for the problem
ChickenMobile wrote:
Whoa. How big can your map be to get to 70mb in just a compile? Mine is a large map and it is only 20MB at the current moment (full compile including packed custom content)
My maps use to be that around too... when I include the custom content (around 17MB when not...)
Just a question, is the 100mb limit imposed by the Workshop or by TWP?
So I think this is both beneficial for us and the community.
How complicated are your visleafs - are they a real mess?
Do you have lots of named light entities?
Are you using loads of cubemaps?
It's difficult to say why your BSP is huge without much extra information. Could you post a compile log? There should be one in the same folder as your VMF.
You might want to try compiling the map with various visgroups (for example, World Details) turned off. Perhaps that will identify a cause for the filesize. Is there a means to compile with displacements turned off? That would test your theory that they are responsible. Try using 'H' to quick-hide selected objects for a compile (you can unhide with U).
Visleaf wise it doesn't seem too bad... I'll try turning more things into details (being an underground map there are lots of skinny panels, but i know that turning EVERYTHING to detail can make it worse, so i left a load of the bigger panels as world brushes).
There are two named light entites, but they have the same name. Might replace this with a ProjTex later and see if it improves much.
EDIT: As it takes a while to compile fully, i'll do Fast for now.
Without Disp Area: 31mb
materialPath: c:\program files (x86)\steam\steamapps\common\portal 2\portal2\materials
Loading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.vmf
ConVarRef gpu_level doesn't point to an existing ConVar
Error! Variable "$envmap" is multiply defined in material "metal/underground_wall_metal004a_bottom"!
nummapplanes: ( 3332 / 65536 )
nummapbrushes: ( 401 / 8192 )
nummapbrushsides: ( 3207 / 65536 )
num_map_overlays: ( 18 / 512 )
nummodels: ( 0 / 1024 )
fixing up env_cubemap materials on brush sides...
0...1...2...3...4...5...6...7...8...9...100...1...2...3...4...5...6...7...8...9...10Processing areas...done (0)
Building Faces...done (0)
Chop Details...done (0)
Find Visible Detail Sides...
Merged 184 detail faces...done (0)
Merging details...done (0)
FixTjuncs...
PruneNodes...
WriteBSP...
done (1)
writing c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.prt...Building visibility clusters...
done (0)
Finding displacement neighbors...
Finding lightmap sample positions...
Displacement Alpha : 0...1...2...3...4...5...6...7...8...9...10
Building Physics collision data...
done (0) (161218 bytes)
Placing detail props : 0...1...2...3...4...5...6...7...8...9...10
Water found with no water_lod_control entity, creating a default one.
Compacting texture/material tables...
Reduced 959 texinfos to 562
Reduced 58 texdatas to 50 (2181 bytes to 1874)
Writing c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
4 seconds elapsed
6 threads
reading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
reading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.prt
329 portalclusters
1264 numportals
0...1...2...3...4...5...6...7...8...9...100...1...2...3...4...5...6...7...8...9...
materialPath: c:\program files (x86)\steam\steamapps\common\portal 2\portal2\materials
Loading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.vmf
ConVarRef gpu_level doesn't point to an existing ConVar
Error! Variable "$envmap" is multiply defined in material "metal/underground_wall_metal004a_bottom"!
nummapplanes: ( 3332 / 65536 )
nummapbrushes: ( 401 / 8192 )
nummapbrushsides: ( 3207 / 65536 )
num_map_overlays: ( 18 / 512 )
nummodels: ( 0 / 1024 )
fixing up env_cubemap materials on brush sides...
0...1...2...3...4...5...6...7...8...9...100...1...2...3...4...5...6...7...8...9...10Processing areas...done (0)
Building Faces...done (0)
Chop Details...done (0)
Find Visible Detail Sides...
Merged 184 detail faces...done (0)
Merging details...done (0)
FixTjuncs...
PruneNodes...
WriteBSP...
done (0)
writing c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.prt...Building visibility clusters...
done (0)
Finding displacement neighbors...
Finding lightmap sample positions...
Displacement Alpha : 0...1...2...3...4...5...6...7...8...9...10
Building Physics collision data...
done (0) (161218 bytes)
Placing detail props : 0...1...2...3...4...5...6...7...8...9...10
Water found with no water_lod_control entity, creating a default one.
Compacting texture/material tables...
Reduced 959 texinfos to 562
Reduced 58 texdatas to 50 (2181 bytes to 1874)
Writing c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
7 seconds elapsed
6 threads
reading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
reading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.prt
329 portalclusters
1264 numportals
0...1...2...3...4...5...6...7...8...9...10Optimized: 26 visible clusters (0.00%)
Total clusters visible: 79545
Average clusters visible: 241
Building PAS...
Average clusters audible: 243
visdatasize:24724 compressed from 31584
writing c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
0 seconds elapsed
[Reading texlights from 'lights.rad']
[59 texlights parsed from 'lights.rad']
Loading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
3586 faces
2343470 square feet [337459776.00 square inches]
0 Displacements
0 Square Feet [0.00 Square Inches]
3586 patches before subdivision
232258 patches after subdivision
46 direct lights
0...1...2...3...4...5...6...7...8...9...100...1...2...3...4...5...6...7...8...9...10transfers 11209151, max 1170
transfer lists: 85.5 megs
0...1...2...3...4...5...6...7...8...9...10 Bounce #1 added RGB(12330, 10147, 5582)
0...1...2...3...4...5...6...7...8...9...10 Bounce #2 added RGB(2594, 1805, 669)
Build Patch/Sample Hash Table(s).....Done<0.4452 sec>
0...1...2...3...4...5...6...7...8...9...10FinalLightFace Done
0 of 0 (0% of) surface lights went in leaf ambient cubes.
0...1...2...3...4...5...6...7...8...9...10Writing leaf ambient...done
Ready to Finish
Object names Objects/Maxobjs Memory / Maxmem Fullness
------------ --------------- --------------- --------
models 46/1024 2208/49152 ( 4.5%)
brushes 401/8192 4812/98304 ( 4.9%)
brushsides 3208/65536 25664/524288 ( 4.9%)
planes 3374/65536 67480/1310720 ( 5.1%)
vertexes 3676/65536 44112/786432 ( 5.6%)
nodes 1060/65536 33920/2097152 ( 1.6%)
texinfos 562/12288 40464/884736 ( 4.6%)
texdata 50/2048 1600/65536 ( 2.4%)
dispinfos 0/0 0/0 ( 0.0%)
disp_verts 0/0 0/0 ( 0.0%)
disp_tris 0/0 0/0 ( 0.0%)
disp_multiblend 0/0 0/0 ( 0.0%)
disp_lmsamples 0/0 0/0 ( 0.0%)
faces 3586/65536 200816/3670016 ( 5.5%)
hdr faces 0/65536 0/3670016 ( 0.0%)
origfaces 1255/65536 70280/3670016 ( 1.9%)
facebrushes 124/0 248/0 ( 0.0%)
facebrushlists 3586/0 14344/0 ( 0.0%)
leaves 1107/65536 35424/2097152 ( 1.7%)
leaffaces 7137/65536 14274/131072 (10.9%)
leafbrushes 833/65536 1666/131072 ( 1.3%)
areas 4/256 32/2048 ( 1.6%)
surfedges 20472/512000 81888/2048000 ( 4.0%)
edges 11443/256000 45772/1024000 ( 4.5%)
LDR worldlights 46/8192 4600/819200 ( 0.6%)
HDR worldlights 0/8192 0/819200 ( 0.0%)
leafwaterdata 2/32768 24/393216 ( 0.0%)
waterstrips 182/32768 1820/327680 ( 0.6%)
waterverts 0/65536 0/786432 ( 0.0%)
waterindices 2883/65536 5766/131072 ( 4.4%)
cubemapsamples 10/1024 160/16384 ( 1.0%)
overlays 18/512 6336/180224 ( 3.5%)
LDR lightdata [variable] 29703832/0 ( 0.0%)
HDR lightdata [variable] 0/0 ( 0.0%)
visdata [variable] 24724/16777216 ( 0.1%)
entdata [variable] 95302/393216 (24.2%)
LDR ambient table 1107/65536 4428/262144 ( 1.7%)
HDR ambient table 1107/65536 4428/262144 ( 1.7%)
LDR leaf ambient 4334/65536 121352/1835008 ( 6.6%)
HDR leaf ambient 1107/65536 30996/1835008 ( 1.7%)
occluders 0/0 0/0 ( 0.0%)
occluder polygons 0/0 0/0 ( 0.0%)
occluder vert ind 0/0 0/0 ( 0.0%)
detail props [variable] 1/12 ( 8.3%)
static props [variable] 1/51166 ( 0.0%)
pakfile [variable] 822354/0 ( 0.0%)
physics [variable] 161218/4194304 ( 3.8%)
physics terrain [variable] 2/1048576 ( 0.0%)
Level flags = 0
Total triangle count: 8256
Writing c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
50 seconds elapsed
With Disp Area: 34.8mb
materialPath: c:\program files (x86)\steam\steamapps\common\portal 2\portal2\materials
Loading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.vmf
ConVarRef gpu_level doesn't point to an existing ConVar
Error! Variable "$envmap" is multiply defined in material "metal/underground_wall_metal004a_bottom"!
nummapplanes: ( 3332 / 65536 )
nummapbrushes: ( 401 / 8192 )
nummapbrushsides: ( 3207 / 65536 )
num_map_overlays: ( 18 / 512 )
nummodels: ( 0 / 1024 )
fixing up env_cubemap materials on brush sides...
0...1...2...3...4...5...6...7...8...9...100...1...2...3...4...5...6...7...8...9...10Processing areas...done (0)
Building Faces...done (0)
Chop Details...done (0)
Find Visible Detail Sides...
Merged 184 detail faces...done (0)
Merging details...done (0)
FixTjuncs...
PruneNodes...
WriteBSP...
done (0)
writing c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.prt...Building visibility clusters...
done (0)
Finding displacement neighbors...
Finding lightmap sample positions...
Displacement Alpha : 0...1...2...3...4...5...6...7...8...9...10
Building Physics collision data...
done (1) (161218 bytes)
Placing detail props : 0...1...2...3...4...5...6...7...8...9...10
Water found with no water_lod_control entity, creating a default one.
Compacting texture/material tables...
Reduced 959 texinfos to 562
Reduced 58 texdatas to 50 (2181 bytes to 1874)
Writing c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
3 seconds elapsed
6 threads
reading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
reading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.prt
329 portalclusters
1264 numportals
0...1...2...3...4...5...6...7...8...9...10Optimized: 26 visible clusters (0.00%)
Total clusters visible: 79545
Average clusters visible: 241
Building PAS...
Average clusters audible: 243
visdatasize:24724 compressed from 31584
writing c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
0 seconds elapsed
[Reading texlights from 'lights.rad']
[59 texlights parsed from 'lights.rad']
Loading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
materialPath: c:\program files (x86)\steam\steamapps\common\portal 2\portal2\materials
Loading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.vmf
ConVarRef gpu_level doesn't point to an existing ConVar
Error! Variable "$envmap" is multiply defined in material "metal/underground_wall_metal004a_bottom"!
nummapplanes: ( 4946 / 65536 )
nummapbrushes: ( 548 / 8192 )
nummapbrushsides: ( 4721 / 65536 )
num_map_overlays: ( 21 / 512 )
nummodels: ( 0 / 1024 )
fixing up env_cubemap materials on brush sides...
0...1...2...3...4...5...6...7...8...9...100...1...2...3...4...5...6...7...8...9...10Processing areas...done (0)
Building Faces...done (0)
Chop Details...done (0)
Find Visible Detail Sides...
Merged 216 detail faces...done (0)
Merging details...done (0)
FixTjuncs...
PruneNodes...
WriteBSP...
done (0)
writing c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.prt...Building visibility clusters...
done (0)
Finding displacement neighbors...
Finding lightmap sample positions...
Displacement Alpha : 0...1...2...3...4...5...6...7...8...9...10
Building Physics collision data...
done (1) (224485 bytes)
Placing detail props : 0...1...2...3...4...5...6...7...8...9...10
Water found with no water_lod_control entity, creating a default one.
Compacting texture/material tables...
Reduced 1588 texinfos to 760
Reduced 76 texdatas to 67 (2878 bytes to 2520)
Writing c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
5 seconds elapsed
6 threads
reading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
reading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.prt
425 portalclusters
1495 numportals
0...1...2...3...4...5...6...7...8...9...10Optimized: 34 visible clusters (0.00%)
Total clusters visible: 88393
Average clusters visible: 207
Building PAS...
Average clusters audible: 209
visdatasize:29108 compressed from 47600
writing c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
0 seconds elapsed
[Reading texlights from 'lights.rad']
[59 texlights parsed from 'lights.rad']
Loading c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
4053 faces
2708297 square feet [389994816.00 square inches]
14 Displacements
476273 Square Feet [68583336.00 Square Inches]
4053 patches before subdivision
264625 patches after subdivision
65 direct lights
0...1...2...3...4...5...6...7...8...9...100...1...2...3...4...5...6...7...8...9...10transfers 12615146, max 1170
transfer lists: 96.2 megs
0...1...2...3...4...5...6...7...8...9...10 Bounce #1 added RGB(36815, 37624, 28172)
0...1...2...3...4...5...6...7...8...9...10 Bounce #2 added RGB(9058, 7523, 3419)
Build Patch/Sample Hash Table(s).....Done<1.0861 sec>
0...1...2...3...4...5...6...7...8...9...10FinalLightFace Done
0 of 1 (0% of) surface lights went in leaf ambient cubes.
0...1...2...3...4...5...6...7...8...9...10Writing leaf ambient...done
Ready to Finish
Object names Objects/Maxobjs Memory / Maxmem Fullness
------------ --------------- --------------- --------
models 57/1024 2736/49152 ( 5.6%)
brushes 548/8192 6576/98304 ( 6.7%)
brushsides 4641/65536 37128/524288 ( 7.1%)
planes 4998/65536 99960/1310720 ( 7.6%)
vertexes 4681/65536 56172/786432 ( 7.1%)
nodes 1390/65536 44480/2097152 ( 2.1%)
texinfos 760/12288 54720/884736 ( 6.2%)
texdata 67/2048 2144/65536 ( 3.3%)
dispinfos 14/0 2464/0 ( 0.0%)
disp_verts 4046/0 80920/0 ( 0.0%)
disp_tris 7168/0 14336/0 ( 0.0%)
disp_multiblend 0/0 0/0 ( 0.0%)
disp_lmsamples 374290/0 374290/0 ( 0.0%)
faces 4053/65536 226968/3670016 ( 6.2%)
hdr faces 0/65536 0/3670016 ( 0.0%)
origfaces 1576/65536 88256/3670016 ( 2.4%)
facebrushes 130/0 260/0 ( 0.0%)
facebrushlists 4053/0 16212/0 ( 0.0%)
leaves 1448/65536 46336/2097152 ( 2.2%)
leaffaces 7661/65536 15322/131072 (11.7%)
leafbrushes 1178/65536 2356/131072 ( 1.8%)
areas 5/256 40/2048 ( 2.0%)
surfedges 23988/512000 95952/2048000 ( 4.7%)
edges 13733/256000 54932/1024000 ( 5.4%)
LDR worldlights 65/8192 6500/819200 ( 0.8%)
HDR worldlights 0/8192 0/819200 ( 0.0%)
leafwaterdata 3/32768 36/393216 ( 0.0%)
waterstrips 212/32768 2120/327680 ( 0.6%)
waterverts 0/65536 0/786432 ( 0.0%)
waterindices 3381/65536 6762/131072 ( 5.2%)
cubemapsamples 12/1024 192/16384 ( 1.2%)
overlays 21/512 7392/180224 ( 4.1%)
LDR lightdata [variable] 32623032/0 ( 0.0%)
HDR lightdata [variable] 0/0 ( 0.0%)
visdata [variable] 29108/16777216 ( 0.2%)
entdata [variable] 116104/393216 (29.5%)
LDR ambient table 1448/65536 5792/262144 ( 2.2%)
HDR ambient table 1448/65536 5792/262144 ( 2.2%)
LDR leaf ambient 5798/65536 162344/1835008 ( 8.8%)
HDR leaf ambient 1448/65536 40544/1835008 ( 2.2%)
occluders 0/0 0/0 ( 0.0%)
occluder polygons 0/0 0/0 ( 0.0%)
occluder vert ind 0/0 0/0 ( 0.0%)
detail props [variable] 1/12 ( 8.3%)
static props [variable] 1/55636 ( 0.0%)
pakfile [variable] 972954/0 ( 0.0%)
physics [variable] 224485/4194304 ( 5.4%)
physics terrain [variable] 11790/1048576 ( 1.1%)
Level flags = 0
Total triangle count: 9442
Writing c:\program files (x86)\steam\steamapps\common\portal 2\sdk_content\maps\sp_cavescaves.bsp
56 seconds elapsed
Not a gigantic jump but that could very well be the cause. I'll run on full tomorrow and test to see if the same rings true in Full, if not i'll delete that area and scale it down a tad.
SpAM_CAN wrote:
Thing is, it doesn't even have cubemaps yet (when i tried to build them i got access denied errors and things, so i'll try that again later.)
In order to be able to build cubemaps you have to have a copy of the map into portal2_dlc2 folder (it's a bug since Valve released the first DLC) because that folder is where the game searches into. In fact you can modify your settings in hammer in order to make it ALWAYS save a copy of your recently compiled map into that folder so you don't have to do it manually each time.
ChickenMobile wrote:
...blah blah blah...
Nodrawing not-seen brush faces. This applies to func_details the most.
...blah blah blah...
I'm a little late, but I read somewhere that with normal brushes that are outside the map (I mean the "outer" side of the map) it doesn't matter if they are nodraw or not since hammer doesn't compile(idk if that's the right word here) them anyways.
But yeah you probably meant brushes that are inside the map but not visible. And the later tip is very important especially if you have many func_details.
Villethemina wrote:
I'm a little late, but I read somewhere that with normal brushes that are outside the map (I mean the "outer" side of the map) it doesn't matter if they are nodraw or not since hammer doesn't compile(idk if that's the right word here) them anyways.
That be true. This is also the case for world brushes faces which are sealed by other world brushes. Because func_details do not split leafs the engine may still render and create lighting for faces on not seen func_detail's sides.
If you ever go through Valve's maps. They usually are sloppy with the nodraw on world brushes, however very careful about where they put it on details.
The fact that your fast compile produces a bsp of half the size compared to your full compile, points in this direction, I think.
(A fast compile creates lower lightmap resolutions than a standard or full one IIRC.)
SpAM_CAN wrote:
Visleaf wise it doesn't seem too bad... I'll try turning more things into details (being an underground map there are lots of skinny panels, but i know that turning EVERYTHING to detail can make it worse, so i left a load of the bigger panels as world brushes).
Just turning things into func_details does not fix your visleaves. You need to use other methods such as hint+skip, visclusters for large open areas, and (for BTS and Old Aperture especially) plenty of areaportals in the right places. Also make sure your nodraw behind the displacements is placed optimally for visleaves, but hint+skip should fix most of that.
Quote:
There is so much misinformation going on in this thread that I almost started to doubt my own knowledge on the subject.
Okay a few things. Beard: What info did you get from IRC exactly? Because if someone told you that the number of visleaves would increase the filesize, they were wrong. I'm assuming you misunderstood what someone (me?) said.
Also:
ChickenMobile wrote:
A few things you can do to reduce bsp size:
Lowering lightmap scales This is the most useful; lightmaps take up a huge chunk of your BSP size
Nodrawing not-seen brush faces. This applies to func_details the most. This will have a negligible effect on file size. This only helps your in-game budget. And as was said previously, any non-detail brush face that is either facing the void or is completely covered (IE sides of brushes facing another wall) will be culled
Reducing the amount of brushes in your map This is only a partial solution; the reason it helps is because of fewer lightmap data
Using LOD model versions for distant models This has nothing to do with file size, again this just affects your in-game budget
Use cheap textures for distant textures (if available) Ditto
Use dynamic lighting (or real-time lighting) in appropriate places. Ditto
Remove unnecessary props and brushes. Ditto
Try compiling your map without VRAD and see if it still jumps up the filesize. And as others have said before, take some time to optimize your lightmaps. Anywhere that the player won't be able to access, or anywhere that doesn't need a sharp shadow being drawn on, lower the lightmap scale down.
Vordwann wrote:
Just turning things into func_details does not fix your visleaves. You need to use other methods such as hint+skip, visclusters for large open areas, and (for BTS and Old Aperture especially) plenty of areaportals in the right places. Also make sure your nodraw behind the displacements is placed optimally for visleaves, but hint+skip should fix most of that.
Hint+skip brushes and areaportals are great for optimizing framerate (when used correctly), but I've never heard of them being used to lower bsp size. Visclusters speed up compile time by telling the compiler that all the visleafs inside the cluster can see each other, but again this data must be stored in the bsp, and I don't see how it would lower the bsp size. On the other hand, maybe I'm just oversimplifying everything and these actually are factors. If so, please provide source material so that I can get educated. 
I think the biggest impacts you can have are:
1. Reduce the overall number of drawn faces in the map (Edit: this helps mostly due to removing lightmaps (#2))
2. Optimize lightmap scales (which it sounds like you are already working on)
3. Lower the quality and count of cubemaps (which you haven't added yet)