Dev Diary PMC Ukraine Kharkiv ArmA 3 Terrain

This is dev diary for PMC Ukraine Kharkiv ArmA 3 real world data terrain by PMC.

We have been writing development or developer diaries to PMC Tactical Forums for two decades, these addon dev diaries are spread to many forum topics. One example PMC Tactical Forums: Vietnam: The Experience Neverending development diary 2007-10-13. On 2020 things changed in Farming Simulator 19 editing community when we started to write detailed dev diaries as web html pages because we had no forum, now in 2022 the same dev diary story telling continues in ofp/arma community.

These dev diaries are my current on the fly thoughts and emotions, if I feel shitty I will clearly state that. I try to avoid it but sometimes there might be hilarious profanity rants when things don't go exactly as I planned, so if you are "offended" by few curse words you probably should stop reading now Smiley ;)

Also these dev diaries are written to many addon projects and sometimes repeating the same lines of text or taking exact same screenshots feels stupid so there might be text just listing boring "I did XYZ and then ZYX" type of info without further details. And remember; this is NOT a tutorial, this is entertainment Smiley ;)

Time stamps are Universal Time Coordinated (UTC).

2022-02-27T16:47:00Z Project Started

2022-02-27T16:47:00Z This page created, this project started Smiley ;)

Got the Global Mapper project started, UTM 37N zone (UTM Projection) switched and then it was decision time for how large terrain will this be, is it going to be a compact 40km just around the city or larger 81.9km with plenty of ukrainian and even some russian landscape around it.

Larger terrain means more floating point precision issue jitter.

2022-02-27T17:00:00Z Made a decision, 81.9km. Go big or go home, bitch.

Dev Diary PMC Ukraine Kharkiv ArmA 3 Screenshot

First thing after locking that user created grid into position, I exported KMZ Rectangle file and started Terra Incognita download. This way I can work on other things while terra incognita is slowly downloading.

2022-02-27T17:09:00Z Initial burst of tasks done. As said KMZ exported, terra incognita grid section 1 satellite imagery download started, then I added SRTM 1-arc-sec online source and exported ASCII Grid heightmap.asc file.

Then, well then I jumped back to PMC Ukraine Kyiv editing while terra incognita was doing its thing Smiley :)

2022-02-27T20:15:00Z Terra incognita grid section 1 finished downloading. Starting grid section 2.

2022-02-28 Satellite Texture Downloading

2022-02-28T03:27:00Z New editing day has started, just continued terra incognita downloading grid section 3.

2022-02-28T05:14:00Z Got PMC Ukraine Kyiv edits wrapped up, now its time to switch into kharkiv full time.

Global mapper loaded all OpenStreetMap (OSM) shape files for ukraine. Then did the whole shuffle of cropping OSM shapes, reducing road features and exported cropped shapes.

Then calculated the UTM 31N projection shift coordinates and Terrain Builder'ized heightmap.asc header.

Next assigned RGB colors to global mapper OSM areas for forests and such, then exported Sat_Mask_??.png in 8 x 8 grid tiles.

2022-02-28T06:38:00Z Got OSM areas, points and roads all UTM 31N transformed and exported. Also ran PMC OSM Places CSV2HPP Converter tool for the transformed_places.csv file to generate pmc_names.hpp config.cpp include file.

Then used GraphicsMagick and ran _run_graphicsmagick_convert_Sat_Mask_PNG_to_BMP_with_white_fix_8x8.bat file to fix them satellite mask tiles all ready for terrain builder importing.

2022-02-28T06:49:00Z Terrain builder project created Smiley :)

2022-02-28T07:20:00Z Finished Terrain Processor smoothing roads and generating random forest tree objects using Terrain Processor TPP Templates, which is 6.452 million heh.

Then started to terrain builder import terra processor generated forest tree objects. ETA for this was over 40min, so not exactly a quick task. More waiting.

Half of terrain editing is waiting for old outdated piece of trash 32bit single threaded windows tools to finish their tasks Smiley :)

2022-02-28T08:12:00Z Terrain builder finished importing over six million tree objects.

2022-02-28T09:19:00Z Started to global mapper export satellite texture in 8 x 8 grid tiles.

2022-02-28T12:28:00Z Global mapper finished exporting satellite texture, uuh.

GIMP satellite texture fixing, clone brushing clouds off.

Satellite texture tiles with clouds: A1, B3, B2, C4, C3, C1, D4, D8, E8, F8, G3, H7

2022-02-28T15:00:00Z Finished GIMP clone brushing clouds off.

Then just Darken Merge Picturemap Satellite Texture Tiles pass before these satellite texture tiles are ready for terrain builder.

2022-02-28T15:14:00Z Terrain builder started to import satellite mask tiles.

2022-02-28T15:34:00Z Finished and now started to import satellite texture tiles.

2022-02-28T16:17:00Z Started terrain builder generate layers.

2022-02-28T19:01:00Z It probably finished some time ago, was watching tv heh, but yeah now saved project and CTRL-E exported WRP.

2022-03-01 Smooth Airfield Runways

2022-03-01T10:56:00Z New editing day has started, this project got really close to binarization last night but still have some small edits to get done.

Used L3DT to import heightmap.asc and L3DT_texture_map.png, then in 3d sapphire mode went and smoothed out all airfield runways. This terrain has three airfields. used 10 pixel radius brush (I think?) and 10 cycles or whatever power of smoothing.

Then edited the ASC file as L3DT doesn't automatically put in 200,000/0 coords (I never bothered to learn which geo settings make that happen), terrain builder imported ASC and rebuild terrain. Then just exported WRP.

2022-03-23 Initial Release Day

2022-03-23T01:20:00Z New editing day has started, I don't want to be too ambitious but shall I say... initial release day, maybe if I can get some good edits in, we'll see Smiley :)

Bugs/issues list shows PMC Ukraine Kharkiv "airfields class names missing" and "heightmap roads, are they really smoothed as I found some jagged locations?" hmm okay lets take a look at those two.

Hmm my 00.txt which is the root text file for arma3 / fs19 / fs22 real world data tool pipeline, shows that kharkiv is ready for arma3 in-game test, so wonder if these bug reports were from the in-game test and I just forgot to remove that tool pipeline step. Well doesn't really matter, going to do in-game test now right away.

2022-03-23T01:29:00Z Okay in-game arma3 check done, airfield runways looks like they are heightmap.asc smoothed but yes they are missing config.cpp class names{} entry, nothing there. And roads overall do not look like they have been heightmap.asc smoothed yet.

Copied terrain processor TPP files and roads copying bat file into Source/TerrainProcessor/ dir, edited and ran the roads bat file, then ran the both terrain processor TPP files.

Dev Diary PMC Ukraine Kharkiv ArmA 3 Screenshot

Deleted terrain builder obj_forest layers file, new objects being generated so we don't need old ones and its faster to delete it from windows file explorer than in terrain builder.

Terrain builder imported the roads smoothed heightmap.asc and rebuilt terrain, then was just waiting time until terrain processor finishes generating those new forest tree line objects. It feels like its taking a long time generating them, I'm kind of scared this will generate a lot of objects same as PMC Ukraine Luhansk did yesterday (nine point some million).

Hopefully I don't need to split the obj_forest.shp into 2 x 2 grid tile setup. Not that its a big deal, but just another step in the process.

And terrain processor saving finished, 12 million something objects, uh oh that will not work, terrain builder will not import that many objects into a single layer.

Dev Diary PMC Ukraine Kharkiv ArmA 3 Screenshot

Hmm hmm.

I am going to do two things next, I'll reduce the forest hectare density to 100 and re-generate the objects and in the mean time using global mapper to export obj_forest in 2 x 2 grid tile setup just in case the object hectare density reduction doesn't work.

Even with hectare density 100 I could just feel that its going to be too many objects, more than 10 million, before the saving was finished, it just took so long time that after generating hundreds object-sets with terrain processor you learn to spot when something goes wrong.

And I was right, 12.854 million objects, actually it didn't reduce at all. So yeah lets swap to use that 2 x 2 grid tile setup for obj_forest.shp file.

Looking at PMC Ukraine Kharkiv in global mapper using OSM areas view, the dark brown is farm land and bright green is forests, guess in 81.9km terrain that bright green area mounts to 12 million objects heh.

Dev Diary PMC Ukraine Kharkiv ArmA 3 Screenshot

Took 7-zip backup of obj_forest.* files just in case I might need them after all, to my surprised they were 605mb in size, whoah six hundred megs for a shapefile, ouch! Must be some detailed forest data there with that kind of megabytes Smiley :o

I deleted the original 605mb obj_forest.* shape file setup, then copied over the obj_A1_forest.*, obj_A2_forest.*, obj_B1_forest.* and obj_B2_forest.* shape files.

Then changed the original Forest_CUP_Chernarus_2022-03-20.tpp file to Forest_CUP_Chernarus_2022-03-20_A1.tpp and also copied it over to A2, A3, A4 ... which obviously was a mistake, it needs to be A1, A2, B1 and B2 so fixed that mistake heh.

Edited those TPP files, set the big forest generation task to hectare density 1 to just give a dry run through first. Then ran all the terrain processor projects to see everything gets generated OK.

Tests completed OK, no issues in object generation.

However I sort of got reminder of how terrain processor works, thinking about it more I realize I knew this already but just didn't remember when doing this dry run. So when I dropped the actual forest object task's hectare density from original 125 to 1 so it runs fast, I left all the bushes and rocks tasks to their normal settings, which you can see below:

Dev Diary PMC Ukraine Kharkiv ArmA 3 Screenshot

So that means now as forest object density is all but none, the "little guys" like bushes and rocks start to take it over like infestation as there is room to breathe for them. All of this resulted into 7, 8 even 9 million object LBT export files heh.

Who knows, maybe I need to still reduce these settings because I cannot imagine how the object density would be any lower per shape grid tile file when I run it with full forests, it still would be over 20 million objects.

But I need to run it with forest object hectare density to 125, ok lets make it 100 for now, to see how many objects each tile shape file will create, if its insane amounts total, then I simply need to do heavy handed reduction for every object type hectare density. Probably need to drop bushes and rocks to some silly 1 and increase their MINDIST etc, there are various ways this object generation "ecosystem" if affected by settings.

2022-03-23T02:39:00Z All four terrain processor grid tile projects running.

Dev Diary PMC Ukraine Kharkiv ArmA 3 Screenshot

Busy busy, busy...

2022-03-23T02:45:00Z Done, wow that was quick. Guess its the settings.

Dev Diary PMC Ukraine Kharkiv ArmA 3 Screenshot

So to sum it up in text, object generated in millions by A1: 1.511, A2: 3.781, B1: 4.309 and B2: 3.320.

Umm actually I guess A2 and some other got mixed up but it doesnt matter, its the overall amount and all combined, anyways.

2022-03-23T02:48:00Z Started to import all LBT export files into terrain builder.

Was monitoring ukraine news while terrain builder was importing a long long time... About half way through (I thought it was that) was thinking maybe I should have done some other tasks in the background while this process takes so long time.

2022-03-23T04:21:00Z Terrain builder object import finished, finally. Oh my over one hour straight, almost two.

Exported WRP and it came out as 12.922 million objects, that is quite large amount of trees but still doable.

Dev Diary PMC Ukraine Kharkiv ArmA 3 Screenshot

Saved terrain builder project, closed and exited. Just to be sure after such heavy object import. Then loaded terrain builder project back up again. This loading took extremely long time, was surprising to just sit and wait while it was loading and loading ...

2022-03-23T04:41:00Z Hmm I didn't timestamp the starting time, but at moment of writing this paragraph the loading had been going on, I already read all news, checked new youtube videos (nothing interesting live or as in new videos) and terrain builder is still loading... I'm getting that creepy feeling that this project might have broken up although terrain builder task manager shows about 4% CPU usage, currently 1,800mb RAM usage which fluctuates, so it is doing something. I don't recall VTE Laos Ban Pakha with 16 million objects or PMC Poland Suwalki with 18 million objects loading this long, I mean this is... ridiculous time.

2022-03-23T04:51:00Z Still loading, even from my "it has already been loading a long time" timestamp its now been TEN minutes...

Lets assume that the project is not broken and it truly takes this long to load it (while writing this its already 04:58), I don't think I'm comfortable to wait this long for a single project to load up. Hmm I cannot remember how long my other projects loaded up so maybe its just fuzzy muscle memory, but I don't recall it being this ridiculously long.

2022-03-23T05:02:00Z Its been now 21 minutes since my first timestamp of "loading...", so yeah this is absolutely crazy assuming the project is not broken.

I have some faint recollection that some huge PMC terrain with 15+ million objects, cant remember which, had a long loading time and I actually moved the terrain builder object layers FILES away from the layers dir when I was editing something which did not require objects to be present, then moved the files back for WRP export. But dunno, maybe I don't recall that correctly. I don't think it was any of the new 2021 terrains which already have dev diaries so I could string search those story texts for clues Smiley :)

Task manager shows terrain builder exe with same CPU usage with RAM usage now slightly higher on the 1,900 - 2,000mb range with constant fluctuation.

If this truly is the case that terrain builder project with 10+ million objects loads for HALF AN HOUR, yeah guess I could live with that if its a known issue, but life is going to be a real bitch when we have more of such terrains... IF this is the case. I truly hope this project is just broken or its now a fluke freeze or some-such issue which will not be there if I kill the task and retry.

I'm going to give it... lets say 45min loading time, just to be absolutely sure.

2022-03-23T05:26:00Z Okay unless I miscalculated its now been 45min, ok time to kill terrain builder task. Its currently using same CPU percentage and RAM fluctuates between 2,000-2,100mb so it does technically look like its loading... but man, fourty five fucking minutes, are you kidding me?

Heh.

After killing the task, restarting terrain builder and loading the project, it loaded instantly. I then 7-zip backup packed the WRP which had all the objects because, well read on...

Turned visibility on for satellite texture, shapes and keypoints, then chose keypoints creation mode.

Dev Diary PMC Ukraine Kharkiv ArmA 3 Screenshot

And created first airfield keypoint. Then created the two other airfield keypoints. Exported WRP, loaded pmc_ukraine_kharkiv.hpp in text editor, copy pasted the config.cpp lines into pmc_ukraine_kharkiv_cfg/pmc_names.hpp and there we go, airfield config done.

Next restored backup .7z of the WRP file so the 12-some million objects vegetation is back.

Now bugs/issues list is empty and I cant think of anything else to do, so its time to binarize this biatch.

2022-03-23T05:42:00Z Started pboProject binarization.

Dev Diary PMC Ukraine Kharkiv ArmA 3 Screenshot

While binarize (PMC Editing Wiki: search "binarize") was chewing on the WRP addon, I took a break to cook and eat food.

2022-03-23T07:40:00Z Was surprised to see binarization still underway after finishing cooking and eating food.

2022-03-23T08:40:00Z One hour later binarizing is still going on, this is now... how many hours of process already, wow this terrain is really taking its sweet time to binarize.

2022-03-23T11:05:00Z Still binarizing, this is not normal, not at all hehe. But I'll let it run because just in case it works eventually I hopefully then have release ready terrain to go.

2022-03-23T13:48:00Z Heh... still binarizing, I've just pretty much idled here rest of the day, monitoring ukraine war news and did few web html tweaks. Task manager shows that binarize (arma3 exe) uses now 4.1gb oh ram and its slowly increasing. I'm going to let it run overnight to see if its finished when I wake up tomorrow, assuming it doesnt finish before I go to bed in about four hours from now. If its not finished tomorrow morning then I am absolutely going to kill the task and go medievil debugging what is causing this issue. But yeah lets hope it finished soon or at least before tomorrow morning.

2022-03-23 Actual Release Day

2022-03-23T23:52:00Z New editing day has started. I woke up and was glad to see pboProject binarization had finished, all OK.

Dev Diary PMC Ukraine Kharkiv ArmA 3 Screenshot

Start time: 2022-03-23T0542
End time: 2022-03-23T2112

2022-03-24T01:35:00Z Just did arma3 in-game check and results are extreme disappointment. First of all before I even noticed anything wrong I realized that I made a mistake on heightmap.asc shuffle, the airfield runway heightmap smoothing got erased because new terrain processor road smoothing used the ASC file which did not include airfield runway smoothing, so while roads got fixed airfields got broken, left hand doesnt know what right hand is doing hehe.

And then the bad part... no objects, not a single vegetation object in sight, nothing, nada, zip Smiley :o

Err well what the hell was I binarizing for over 12hrs straight then?

Oh man what a shit show Smiley :(

So guess I'll go back and first fix the airfield runway smoothing, maybe I can find the old heightmap.asc that was OK, then run terrain processor again and import that heightmap to terrain builder. Once that is fully working and binarized (without objects) I can go and do another set of terrain processor vegetation generation.

Ugh.

But today we work smart, if it looks... even smells like that something is going wrong with binarize, like well over one hour binarization time, or ANY other editing process seems to be off, I'm calling it and simply task manager kill everything and retry. I am done sitting on my hands waiting for no reason Smiley :(

2022-03-24T02:06:00Z Time to continue editing. It kind of feels like I'm dragging my feet here, there are so many terrains to finish and release but like yesterday feels like I got nothing done, all day was spent binarizing. Lets make today a better editing day.

Used L3DT and exported heightmap_airfields_smoothed.asc heightmap, I have to assume this is the already smoothed airfield runways in the L3DT project. Then imported this ASC into terrain builder, rebuilt terrain, exported WRP and saved project. Ran moveObject WRP re-pathing and then started pboProject binarization. I want to check that binarization works on this object-less terrain and airfield runways are smoothed OK.

While binarize was chewing on the WRP addon, I started thinking... the global mapper exported obj_forest OSM shape files... hmm, were those in UTM 31N or in the original real world data location which is UTM 37N, it just might be that I forgot to export the TRANSFORMED UTM 31N shapes, it would make sense that terrain processor just did what it was told, generated objects on shapes regardless of the coordinates, then terrain builder imported the objects in their coordinates which were who-the-fuck-knows where "off the terrain grid" heh.

After quickly checking p:\temp\pmc\pmc_ukraine_kharkiv\source\terrainprocessor\ where pboProject moved the dir (sigh), opened up the smallest LBT export file and sure enough, coordinates were all off like this:

objects
0.776957 0 0.306238 0 0.835131 0 -0.306238 0 0.776957 265118.429365 0 5566231.014020 0 t_picea2s
-0.941162 0 -0.678515 0 1.160245 0 0.678515 0 -0.941162 265119.373258 0 5566263.852511 0 Misc_trunk_torzo
-1.062710 0 0.506012 0 1.177030 0 -0.506012 0 -1.062710 265119.137285 0 5566281.837275 0 t_pinusn2s

In short coordinates should be 200,000/0 starting but right now we got the second coord 5 MILLION number, which clearly is off the terrain grid.

So binarize did its thing, trying to optimize the WRP and guess it did several checks or "got lost" or whatever when it encountered an P3D object so far off the terrain coordinate range. No wonder it took over 12hrs to binarize and final result in-game was 0 objects in the terrain Smiley :)

Oh well, mistake was made with global mapper export UTM zone, that was the root cause. Such a small mistake evolved into what I would summarize as one day of wasted time.

But you know what... this mistake gained some more knowledge and I wont be making the same mistake ever again. Well I didn't actually learn anything new, of course I know about UTM zone but now I'm making sure when exporting shapes its done from the transformed ones instead of the gloabl mapper "original source" shape layer. Uhm kind of difficult to explain but yeah anyways.

Making mistakes is fine if you always gain knowledge and experience from them, one step forward always, never back, if you fall down you WILL get up dust yourself off and act better the next time. That's how we roll bitches Smiley :D

2022-03-24T02:43:00Z So while binarize was running I used global mapper to load the transformed OSM area shapes and exported them as 2 x 2 grid tile setup.

Then had to wait for binarize to finish so Source/ dir is back on its rightful place before proceeding to generate properly geo located objects. Binarization should not take a long time without any objects, but as I said above not going to waste any time, I'm going to start slowly putting together the web pages while waiting. Web page work will definitely be the background slow cooking process, I'm assuming binarize will finish any moment now anyways.

2022-03-24T02:53:00Z And as expected, binarize finished just as I got 2 php files copy pasted heh.

Full console log for pboProject binarization, for time reference:

        1 file(s) copied.
*** pmc ukraine_kharkiv
Start time: 2022-03-24T0213
pboproject pmc_ukraine_kharkiv_data_layers_04
pboproject pmc_ukraine_kharkiv_data_layers_03
pboproject pmc_ukraine_kharkiv_data_layers_02
pboproject pmc_ukraine_kharkiv_data_layers_01
pboproject pmc_ukraine_kharkiv_data_layers_00
pboproject pmc_ukraine_kharkiv_data_layers
pboproject pmc_ukraine_kharkiv_data
pboproject pmc_ukraine_kharkiv_cfg
pboproject pmc_ukraine_kharkiv
End time: 2022-03-24T0252

In-game check confirms airfield runways are now smoothed, good. At least some progress today. Renamed the previous heightmap ASC file to just "heightmap.asc" now as our main source. Then used terrain processor to smooth roads again.

Copied the new global mapper UTM 31N zone exported OSM area shapes for obj_forest_* and terrain processor generated brand new proper coordinated vegetation objects Smiley :)

Grid A1 generated 1.497 million objects, A2 3.309 million, B1 4.291 million and B2 3.765 million. Pretty nice set, now lets see if these import to terrain builder OK Smiley :)

2022-03-24T03:09:00Z Started terrain builder importing the new vegetation LBT export object files. This will take a few moments again, good time for me to create more of those web pages for PMC Ukraine Kharkiv Smiley :)

2022-03-24T03:23:00Z Got this dev diary web page created until this point in the time line (could I have said it any more awkwardly heh). What I mean is new web page php created, dev diary TEXT moved from my localhost dev dir and proper HTML code tags added etc. Now all ukraine-kharkiv.php pages are created, next is to actually add "content" like changelog, screenshots, etc, then proof read it all and make sure finer detail formatting beautify is in.

Oh and lets finish editing the actual terrain as well Smiley :p

2022-03-24T04:47:00Z Terrain builder finished importing objects. Saved project, worked ok (pfewf), exported WRP, came out as 12.865 million objects. Closed terrain builder, all good here.

Dev Diary PMC Ukraine Kharkiv ArmA 3 Screenshot

Then moveObject re-pathed WRP file and started pboProject binarization. Now we wait again. Hopefully once this process is done, we have fully release ready brand new ukraine terrain in arma3 Smiley :)

2022-03-24T05:32:00Z Finished binarizing.

Start time: 2022-03-24T0450
End time: 2022-03-24T0532

Before next arma3 in-game test went to grab a bite to eat. After feeding was done got to test kharkiv in-game, everything looked pretty good, airfields and roads are all smooth, nice forests which definitely eat some object space. Looks nice.

2022-03-24T07:08:00Z Finished taking, choosing, renaming and editing screenshots for web.

2022-03-24T07:45:00Z It is time start creating the release package and related items.

PMC Ukraine Kharkiv: Changelog, Dev Diary, Screenshots. Back to PMC Terrain's page.