Dev Diary PMC Taiwan Chiayi ArmA 3 Terrain

This is dev diary for PMC Taiwan Chiayi ArmA 3 real world data terrain by PMC.

We have been writing development or developer diaries to PMC Tactical Forums for two decades, these addon dev diaries are spread to many forum topics. One example PMC Tactical Forums: Vietnam: The Experience Neverending development diary 2007-10-13. On 2020 things changed in Farming Simulator 19 editing community when we started to write detailed dev diaries as web html pages because we had no forum, now in 2022 the same dev diary story telling continues in ofp/arma community.

These dev diaries are my current on the fly thoughts and emotions, if I feel shitty I will clearly state that. I try to avoid it but sometimes there might be hilarious profanity rants when things don't go exactly as I planned, so if you are "offended" by few curse words you probably should stop reading now Smiley ;)

Also these dev diaries are written to many addon projects and sometimes repeating the same lines of text or taking exact same screenshots feels stupid so there might be text just listing boring "I did XYZ and then ZYX" type of info without further details. And remember; this is NOT a tutorial, this is entertainment Smiley ;)

Time stamps are Universal Time Coordinated (UTC).

2021-10-09 Project Started

2021-10-09T22:11:00Z Created this page Smiley :)

First created real world data terrain tool pipeline copy-paste text to guide me.

Created Global Mapper project, imported already existing shape of user feature grid of our terrain area, created new grid on top of it (heh yep) and deleted the old, I just dont like the filename.shp layer in global mapper which does not behave same as regular user created feature grid.

Then exported KMZ coordinate file, saved project and exited.

Next started Terra Incognita and went through all the usual procedure to load KMZ coordinates in and creating 4 corner piece coordinates for our 81.9km terrain satellite imagery area.

2021-10-09T22:27:00Z Then just started to download the NW first corner piece, this will now take approximately 8hrs straight up downloading, interesting to see what time its all finished, hopefully I remember to time stamp it Smiley :)

2021-10-10T00:15:00Z NW corner piece finished and started downloading NE piece.

2021-10-10T01:56:00Z NE corner piece finished and started downloading SW piece.

2021-10-10T03:18:00Z SW corner piece finished and started downloading SE piece which is the final download.

2021-10-10T04:37:00Z Few minutes ago download finished, aayeah, we are done leeching satellite Smiley ;)

2021-10-10T06:23:00Z Loaded OpenStreetMap (OSM) shape files and cropped them into grid.

2021-10-10T09:09:00Z Assigned RGB colors to OSM shape areas and then started to export satellite mask.

Exportin' ...

PMC Taiwan Chiayi ArmA 3 Screenshot

When that was done (it was satellite mask btw, just to be clear), then it was satellite texture export time, another nice wait.

Its interesting that even though I'm editing three terrains at the same time now, I feel handicapped with these dev diary writings and well also watching twitch on the side heh, but if I didn't write anything down, didn't need to be ready to take screenshots or videos, arrange window dialogs certain way etc, I could get much more work done if it was just "me alone editing" here heh, difficult to explain again. Anyways.

Overall I feel its been slow going today when I cant just blast off full speed without stopping to "diary it" Smiley :)

2021-10-10T10:41:00Z Global mapper finished exporting satellite texture. And with this, I'm going to call it a night, a bit early today but just somehow feel... kind of exhausted, or something.

2021-10-13 GIMP Ocean Coastline

2021-10-13T09:04:00Z Back at Chiayi editing, well actually I'm editing taipei but pboProject is taking forever to do its thing so I decided to make chiayi satellite texture tile catalog while waiting.

Started to load sat_ images into GIMP and inspect them while writing down all full ocean, coastline and tiles with clouds in them. Discovered one Black Tile... err tile, so started terra incognita and re-saved Taiwan_Chiayi_81km_2_arcgis_17_level in order to fix the black tile issue.

2021-10-13T10:40:00Z Finished cataloging satellite texture tiles, this included cooking food and eating break heh.

2021-10-13T10:44:00Z Started to global mapper export satellite texture, lets get that black tile fixed. Kind of excessive to export full 81,920 x 81,920 pixel resolution imagery for one single spot, but oh well. I'm happy that now this knowledge has been written down to local notes and in my tool pipeline, there is not going to be any more black tiles because I'm going to always irFanview check all fresh terra incognita exports and if any are found, I'll re-save the corner piece and its fixed.

2021-10-13T11:52:00Z Global mapper finished exporting satellite and sat_A8.png is without a black tile, nice.

2021-10-13T14:16:00Z Was working on PMC Africa Niger Tongo terrain a little but now its waiting for p:\pmc\ mount so we're back working on taiwan chiayi a bit.

Used global mapper to export preview-arcgis-17-level.png image with 50 meter / pixel scale.

Then did all OSM shape files for that UTM 31N transformation, something went wrong with the transformed_places.csv location place names because they came out all messed up. But they were in taiwanese/chinese at source level so noone would understand them anyways hehe.

2021-10-13T16:21:00Z Finished GIMP clone brushing all clouds off, man that is tedious job. Its very deceiving because painting in GIMP feels "fun" when you start it... but after you've done first half hour of it... the fun is not there anymore, then its just painful tedious job and feels crappy to destroy satellite image.

2021-10-14 Terrain Builder

2021-10-14T02:39:00Z New editing day has started. Well actually I already did some work for PMC Africa Niger Tongo (tongo) terrain and now switched over to taiwan chiayi. Basically today we bring this baby to Terrain Builder.

Used GraphicsMagick to Darken Merge Picturemap Satellite Texture Tiles, while that was processing I created terrain builder project.

Then I was importing heightmap but ASCII Grid heightmap.asc had no ocean seabed data, then I was investigating whats going on and concluded that its not done even though tool pipeline does not include this step anymore which means I did it already. Well I started to do it ... and about half way through I realized that there already was heightmap done with ocean seabed data, tool pipeline was correct, I should have just searched for the ASC file and sure enough it was there when I did. Heh what a mess Smiley ;)

So imported heightmap.asc into terrain builder with ocean seabed data, then was ready to import satellite mask but the BMP images were not done yet, so had to run graphicsmagick bat for them first.

Got satellite mask imported.

PMC Taiwan Chiayi ArmA 3 Screenshot

And satellite texture imported, we got a full house.

PMC Taiwan Chiayi ArmA 3 Screenshot

2021-10-14T04:09:00Z Terrain builder generate layers started. Also on semi related note E:\ drive has only 132gb free space at the moment, it will soon run out when processing these huge terrains. I could already delete taiwan taipei and kaohsiung satellite work files, but I want to see them in-game first to make sure they are fine.

2021-10-14T04:51:00Z Was going through real world data terrain tool pipeline and saw that I need to separate OSM vegetation areas into individual files ... well I started to do that and much later figured out that yeah, I already did this heh. Oh man, trust the fucking tool pipeline text, if its not listed there then YOU HAVE ALREADY DONE IT! hehe

Ran Terrain Processor forest task with 1 hectare density and temporary roads.shp file, to get the base line number of objects, it was 663k so again with quick calculation we could do 15 hectare density, I set it this time to 16 and generated new objects. This generated 7.975 million objects, hmph, should have cranked the density up some more.

2021-10-14T06:10:00Z Terrain builder generate layers finished. Next I imported Terrain Builder Template Libraries and roads smoothed heightmap. Then started to import those 7.975 million vegetation objects.

Then just Convert Layers PNG to PAA Fast bat, Separate Data Layers for smaller PBOs with _run_Separate_WRP_Dir.bat and moveObject re-pathing of layers RVMAT's.

2021-10-14T07:06:00Z All done, terrain project is ready for pboProject binarization (PMC Editing Wiki: search "binarize"). However I wont start that now as it could be another 10hrs task.

2021-10-14T07:50:00Z After having a lunch and looking at list ot stuff to do, I just somehow felt like nah... that is it for today, I'm going to watch TV and chill rest of the day.

2021-10-14T15:19:00Z Well decided to run pboProject binarization before going to sleep, so just started it. Actually did taiwan kaohsiung just now which took 40min, odd as taiwan taipei took 10hrs yesterday.

2021-10-14T15:52:00Z Binarization finished, heh, nice.

2021-10-21 Ocean Coastline Editing

2021-10-21T11:08:00Z Back editing taiwan stuff. Added RoadsLib.cfg from root master location using symlink.

Next bug was that ocean coastline regions landmass had water creeping into it, meaning like really low land farm fields were under water, these had to be raised above arma3 water level which is 0 meters elevation. First I exported latest heightmap.asc from terrain builder, then imported it to L3DT and saved a new project, then imported texture map, then went into 3D sapphire with heightmap tools and started to check out the coastline. Well it was peppered with 0m or below values across the low land farm castal areas... so this task would be very tedious.

Tried it a bit and indeed its stupid to do this task, I need to come up with a tool pipeline method to handle these underwater land areas.

I'm thinking of using PMC Heightmap Clipper to set all 0m areas to 1m, then in global mapper merge it with GEBCO ocean seabed data. Hmm, hopefully that would work.

In the mean while I ran heightmap data through global mapper to get the GEBCO ocean seabed data in, that was also missing from the WRP, dunno why. Imported it to terrain builder and exported new WRP file, ready for binarization.

2021-10-21T12:58:00Z Had to start PMC Taiwan Kinmen dev diary page and terra incognita downloads Smiley ;)

Fixed latitude/longitude config coordinates where lon was negative, putting it to US west coast hehe. Then fixed roads.shp ID which was narrow dirt/gravel road to 10m width asphalt road, hopefully wont get too wide on some areas, should be fine.

2022-03-28 New Editing Year

2022-03-28T07:36:00Z New editing day has started, err I mean YEAR! Heh yeah it was october 21st 2021 last time when I was editing PMC Taiwan Chiayi terrain, but here we are back at it again.

World has changed since oct 2021, Russia invaded Ukraine and everything is going to HELL Smiley :(

I have released sevral PMC Ukraine terrains in recent days (hmm is it week+ already?) and now its time to mix stuff up a bit so decided to edit and release the alphabetically first taiwan terrain and that is chiayi.

Just created taiwan arma3 startup bat and sent it to desktop, loaded arma3 up with LOW video preset and went to check out chiayi.

First thing I see is that roads.shp is broken, was looking at this dev diary from 2021 and I wrote that I fixed the narrow dirt/gravel roads, but was that just the same issue I'm seeing today which is caused by global mapper saving roads.shp which breaks them, I have to import/export roads.shp in terrain builder in order to fix them for arma3.

Heightmap near the coastline has the water bleeding through, I wrote some remarks about it in 2021 but nothing definitive, so the problem remains. Besides the roads.shp this is my first priority for sure as its the most obvious bug/issue on this and all coastline terrains.

So the game-plan for today is to first move terrain builder object layers files into some "JAIL" dir so no objects show up, they are just making things heavy and long time loading, so while I'm working on heightmap there is no need for any objects, if I have to do many binarization passes it makes no sense to have millions of objects present.

Global mapper dir was empty as kind of expected. Moved terrain builder objects layer file (only one) into other dir.

In terrain builder created airfield_runways shape layer and keypoints for the only airfield / airport in this terrain.

PMC Taiwan Chiayi ArmA 3 Screenshot

Exported airfield_runways.shp to _data\roads\ dir and WRP so that I can get the config.cpp class names{} for airfield. Then copy pasted the config values into pmc_taiwan_chiayi_cfg/pmc_names.hpp aka class names{} thingy.

Used global mapper to load airfield_runways.shp, set ID 7 and ORDER 0, merged it with roads.shp. Then started terrain builder and imported roads.shp and exported it out right away, then deleted roads shape layer and exited.

Then deleted layers PAA images and RVMATs, then deleted pmc_taiwan_chiayi_data_layers_05 through _08 because those will not be used anymore after I do the satellite texture/mask generation with surface mask as 2048 resolution.

2022-03-28T08:21:00Z Terrain builder started generate layers, this will take few hours I believe. In the mean time I'll try to get some other work done which doesn't require terrain builder, maybe create the web pages already, optimizing time usage.

Or maybe I just watch TV Smiley ;)

Was searching this dev diary for info, gotta love that such info is available, of how many objects the previous Terrain Processor TPP Templates created and looks like it was 7.975 million objects, this is quite a lot of a single layer, I want to spread the load around a bit.

On global mapper loaded chiayi project, went to export OSM area shapes but noticed they were lacking, several of my 2022 standard OSM color assignment areas were missing, so I added those.

But on short term, didn't do much good as I'm just generating layers so they wont help at all, on the long term OSM areas are now colored properly, next time I import satellite texture/mask it will be nice.

Then exported cropped_areas.shp, unloaded data and loaded the new shp, then transformed it to terrain builder UTM 31N 200,000/0 projection coords. Then finally exported "obj" with 2 x 2 grid tiles setup which created nice and compact obj_A1_forest.shp etc named files.

These will be used in terrain processor and eventually can have nice and small object layers in terrain builder.

Created _run_terrain_processor_obj_Ax_Forest_sorting.bat file which had:

@echo off
if not exist TerrainProcessor md TerrainProcessor
move obj_??_forest.* TerrainProcessor
del obj_??_*.*
pause
exit

So I don't have to repeat these same boring commands each and every time I generate 2 x 2 grid tile OSM forest shape files Smiley :)

Then updated mod.cpp description with homepage link and slightly better text, also made "name" as first line in mod.cpp file.

2022-03-28T09:09:00Z Terrain processor A1, A2, B1 and B2 TPP files are done and running.

2022-03-28T09:52:00Z Finished having lunch, terrain builder is still generating satellite mask for about 12min.

Going to start creating chiayi web pages now as I have nothing else to do especially when binarize starts to run.

2022-03-28T10:08:00Z Terrain builder finished generating layers.

Then ran convert layers PNG to PAA fast bat, 14 core short burst version Smiley :)

Not going to import vegetation objects for this/first binarization, I want to see what can be done for the heightmap ocean coastline before adding all those heavy objects into play.

Then separated WRP and Data addon dirs and moveObject re-pathed RVMATs. While these were running I added full history to changelog page.

2022-03-28T10:44:00Z Started binarizing with pboProject.

Tested in-game and roads.shp is fixed along with nicely done airfield runwaysand class names{}, so far so good.

2022-03-28T12:10:00Z Terrain builder started importing objects, first layer was super small it went within 10sec. However the full set is well over 10 million objects so it will take probably, hmm an hour to import all of it.

PMC Taiwan Chiayi ArmA 3 Screenshot

Humm "well over" heh, not so sure about the well part, but yeah technically just over 10 million Smiley ;)

Then ran moveObject re-pathing of WRP and had to take a break, was one of those things that just lost all motivation and sort of got tired, yawning a lot even though I wasn't "physically tired". Hopefully this doesn't mean that next dev diary entry is from tomorrow ...

2022-03-28T20:27:00Z I did some "off the books" editing, experimented in global mapper, L3DT, PMC Heightmap Clipper and terrain builder how could I do the coastline water 0m elevations for arma3, in short; no progress, could not find a solution yet.

What I mean off the books is that I didn't write dev diary and didn't capture video. I was just experimenting, its not really a good thing to watch as dev diary. In general dev diary videos should be me showing you how I edit, its not meant to be live stream alike where mistakes are made and other wandering around aimlessly pressing buttons type of activity.

2022-03-29 Coastline Gets Cracked

2022-03-29T07:47:00Z New editing day has started... with "off the books" operations. Yeah continuing where I left off last night, trying to come up with a new tool pipeline to get SRTM online source heightmap data into arma3 so that the stupid fucking water creeping through land will not happen.

Once I get this sorted out or give up on it, I'll update this dev diary.

Just for reference, binarization time with 10 million objects:

*** pmc taiwan_chiayi
Start time: 2022-03-29T0735
pboproject pmc_taiwan_chiayi_data_layers_04
pboproject pmc_taiwan_chiayi_data_layers_03
pboproject pmc_taiwan_chiayi_data_layers_02
pboproject pmc_taiwan_chiayi_data_layers_01
pboproject pmc_taiwan_chiayi_data_layers_00
pboproject pmc_taiwan_chiayi_data_layers
pboproject pmc_taiwan_chiayi_data
pboproject pmc_taiwan_chiayi_cfg
pboproject pmc_taiwan_chiayi
End time: 2022-03-29T0817

Binarization time without objects:

Start time: 2022-03-29T0911
...
End time: 2022-03-29T0950

2022-03-29T09:56:00Z Just finished experimenting with the tool pipeline to get proper ocean coastline done without any annoying water creeping into land areas.

I'm not going to list everything needed to be done, but lets just sum it up that now it works in-game, taiwan chiayi terrain is the first one to have accurate coastline done from GIMP satellite texture tile selecting and painting coastlines.

The quality accuracy of the coastline now depends solely how carefully GIMP selection was done.

Right now taiwan chiayi heightmap is as good as it gets. Next going to add objects back in, smooth out airfield runways, then roads and I believe that is it before first release.

Its been too many hours without releases, I'm starting to get the shakes Smiley ;)

2022-03-29T10:15:00Z L3DT smoothed heightmap for the only airfield runways. Then started terrain processor road smoothing on top of that new heightmap.asc file.

Then restored the object layer files into terrain builder dir, started the project up and imported the new roads smoothed heightmap, rebuilt terrain, deleted heightmap layer, exported WRP, saved project and exited. Then moveObject re-pathed WRP as usual. We are ready to binarize.

PMC Taiwan Chiayi ArmA 3 Terrain Screenshot

Objects in the WRP was 10,729,528 total. Hopefully they look OK, not too dense or sparse heh.

2022-03-29 Release Day

2022-03-29T10:19:00Z Started pboProject binarization, lets hope this will be the release version compile Smiley :)

2022-03-29T11:02:00Z Binarization finished, I decided to start release procedures now. Obviously have to cancel those if there is something wrong at this compile at last minute.

Added chiayi to PMC "worldSize" mission pack terrains list and compiled new pack.

2022-03-29T11:15:00Z In-game check complete and everything looks good for a release. Took few screenshots too for web pages. And now its my lunch break.

2022-03-29T12:02:00Z Release package and torrent was ready to go. Now started to add content to web pages. This will take a while with all this dev diary links etc.

2022-03-29T13:33:00Z Finished adding links and screenshots to this dev diary page, also proof read it along the way.

2022-03-29T13:50:00Z PMC Taiwan Chiayi v0.1 has been released! Smiley :D

PMC Taiwan Chiayi: Changelog, Dev Diary, Screenshots. Back to PMC Terrain's page.