This is dev diary for PMC Ukraine kyiv ArmA 3 real world data terrain by PMC.
We have been writing development or developer diaries to PMC Tactical Forums for two decades, these addon dev diaries are spread to many forum topics. One example PMC Tactical Forums: Vietnam: The Experience Neverending development diary 2007-10-13. On 2020 things changed in Farming Simulator 19 editing community when we started to write detailed dev diaries as web html pages because we had no forum, now in 2022 the same dev diary story telling continues in ofp/arma community.
These dev diaries are my current on the fly thoughts and emotions, if I feel shitty I will clearly state that. I try to avoid it but sometimes there might be hilarious profanity rants when things don't go exactly as I planned, so if you are "offended" by few curse words you probably should stop reading now
Also these dev diaries are written to many addon projects and sometimes repeating the same lines of text or taking exact same screenshots feels stupid so there might be text just listing boring "I did XYZ and then ZYX" type of info without further details. And remember; this is NOT a tutorial, this is entertainment
Time stamps are Universal Time Coordinated (UTC).
2022-02-26T06:07:00Z This dev diary page created.
All my life I've known the city of Kiev, I bet you have heard saying of "Chicken Kiev" etc, so it was odd to me see many web sites calling Kiev with some odd "Kyiv" name in 2022. Doing a little googling shows that the new name is the legit Ukrainian spelling of the name.
Kyiv is Ukrainian spelling, Kiev is old soviet union communist era name.
PMC will call Ukrainian capital city "Kyiv" from now on. Glory to Ukraine!, Слава Україні!, Slava Ukraini!
2022-02-26T06:23:00Z Had the 00.txt real world data terrain tool pipeline text file setup, but its less than 10min before global mapper finishes exporting PMC Ukraine Mariupol satellite texture and then I've got my hands busy in GIMP editing the images, so decided to wait these few minutes watching live streams and news from Russia Ukraine War.
2022-02-27T09:45:00Z New editing day has started, okay yesterday after creating this html page I still was working on PMC Ukraine Mariupol but today we start to make those brand new terrains.
Decided to create 61.4km terrain for Ukraine Kyiv. Initially I was checking out larger but it included dymer city (google/maps) north of this location so I wanted to keep Kyiv compact and create different terrain for those Russian approach routes in the north.
Idea was to make even more compact maybe 40km terrain for dymer and ivankiv (google/maps). But that is for another terrain project.
Then used SRTM online source and exported 15m cell size ASCII Grid heightmap.asc.
2022-02-27T10:29:00Z Started to global mapper import OpenStreetMap (OSM) shapes for areas, lines and points. This is natural areas, roads, and locations.
Reduced road count by Remove Bad OSM Roads, this was usual ukraine style road feature type reduction, it was well above 10k when doing "track" type it plummeted to 4223. I bet with some studying there could be room for more roads to be in roads.shp file, but this will do for now.
Exported cropped areas, roads and points.
2022-02-27T10:55:00Z Finished RGB coloring global mapper OSM shape areas for satellite mask, then started to export it in 8 x 8 grid tiles.
It finished exporting really fast and that raised some questions. Well yeah its fast because 61440 pixel resolution image divided by 8 is only 7680 pixels per image, so no wonder its fast. Hmm this brings up another question, should I use 4 x 4 tiles instead as the tile size drops below 10k magical limit (there is no such magic limit heh).
The whole reason us terrain geeks split huge ArmA 3 satellite texture/mask images into smaller pieces is because satellites are huge, for example right now PMC is not going to do less than 40,960 x 40,960 pixel resolution satellites unless its highly unique special terrain. So imagine loading 40k res image into GIMP... yeah its not very nice to work with.
The other end of this is too small, it would make no sense to split lets say the mentioned 40k res image into 16 x 16 grid tiles because it would make the tiles ridiculous 2560 x 2560 pixel resolution. That would be just silly waste of time to handle all those images.
Overall today at 2022-02-27T11:04:00Z when I'm typing this PMC DevSurf computer (our editing computer) is running windows 8.1, has AMD Ryzen 9 3950x 3.5ghz (boost 4.7ghz) 16cores (32 threads) CPU and Corsair 64GB (4 x 16GB) Vengeance LPX DDR4 3200MHz RAM. This allows us to GIMP load images ... well I just realized that I have not even tested what is the limit of RAM/GIMP with this setup. I do remember from previous 32gb RAM computer that 20480 x 20480 res image with 9 layers in earlier GIMP version was extremely sluggish. So if we use that as guideline, splitting satellite texture/mask into below 20k res tiles would be enough.
So I am reconsidering this 8 x 8 tiles for 61.4km terrain... if I chose 4 x 4 tiles that would make 15360 x 15360 res per tile. I think that would be doable and is much nicer number of images to work with.
2022-02-27T11:36:00Z Decided to go with 4 x 4 tile setup. Started global mapper export of satellite mask.
Copied GraphicsMagick etc bat files into global mapper project dir.
2022-02-27T11:50:00Z Terra incognita satellite imagery download finished for grid 1 (NW), now started to download grid 2 (NE).
Also to my disapointment I noticed that global mapper OSM shape areas have few black outlines, these are most likely nature reserves or something like that. So looks like I have to go back and delete those and for the third time export this mask
There was "nature_reserve" and also "cliff" type features which I deleted.
2022-02-27T12:39:00Z Got roads.shp done with ID/ORDER edits plus the usual transformation coordinates. Same for areas to be used in Terrain Processor object generation, forests and tree lines mainly.
Then used Darken Merge Picturemap Satellite Texture Tiles method and ran _run_graphicsmagick_convert_Sat_Mask_PNG_to_BMP_with_white_fix_4x4.bat to get satellite mask ready for terrain builder importing.
2022-02-27T12:56:00Z Again hopped leaps ahead in dev diary writing; so I created Terrain Builder project directory, moved heightmap.asc, Roads/ and TerrainProcessor/ dirs there. Edited _cfg/config.cpp for the actual size and proper name. Hmm what else.
BTW this reminds me that I need to enhance my powergrep4 search and replace strings because I could easily do the name and terrain size properly already at that level, although it doesn't save much time but well at least it would skip the step of editing config.cpp separately.
Wrote a issue report about this.
Terrain builder imported heightmap.asc, it needed to have .prj file deleted first because terrain builder uses it and tries to put it to UTM 36N which is the real world ukraine kyiv coordinates.
Copied PMC Ukraine Yenky Terrain Processor TPP Template files into this projects dir, ran the forest TPP with hectare density 1 to verify it works, it did and then brough it back to 125 and then ran it to generate full set of forest tree objects.
Terrain processor generated 4.902 million objects. That is a nice number, whould have plenty of trees but still nowhere near as much which would create problems in terrain builder.
Then ran smooth_roads in terrain processor, it generated heightmap_smoothed_roads.asc file which I imported to terrain builder.
2022-02-27T15:34:00Z Did latitude/longitude config while last terra incognita section is downloading.
Then terrain builder imported obj_forest.lbt which is the 4.902 million tree objects for forest tree lines.
2022-02-27T16:24:00Z Terra incognita satellite imagery download completed. Alright now hectic editing continues
Created global mapper map catalogue for the downloaded oziexplorer .map tiles, then started to export satellite texture in 4 x 4 grid tiles. Now we wait again.
To keep things rolling I'm going to start the next terrain project in the mean time, editing two terrains at the same time. This wont be easy, it gets confusing and messy especially with dev diary video capturing, but gotta optimize time, when I edit without any dev diary stuff I routinely edit 2-3 terrains at the same time, its piece of cake. Dev diaries ain't no joke
2022-02-27T17:17:00Z Started GIMP painting clouds off from satellite texture tiles.
Satellite tiles with clouds: A4, A1, B3, C3, C1, D3
2022-02-27T18:15:00Z Finished clone brushing clouds, hmm that went quite fast, of course its smaller terrain than previous ukraine ones but still, was pretty quick.
Busy busy busy ...
2022-02-27T18:26:00Z Terrain builder started to import satellite mask images.
Once done imported satellite texture tiles.
2022-02-27T19:08:00Z Terrain builder finished importing satellite texture. Started to generate layers right away.
2022-02-27T20:22:00Z And generate layers finished. Then started Convert Layers PNG to PAA Fast bat process.
Also edited the Separate Data Layers for smaller PBOs bat files for 03* range of layers file names.
2022-02-27T21:10:00Z Got WRP and data split, now I'm dead tired its time to call it a day.
2022-02-28T04:25:00Z New editing day has started, already got terrain builder up, created airfield_runways shape layer and drew runway poly lines, also assigned ID/ORDER to them.
This should have been done in global mapper I guess, it feels so stupid to edit ID/ORDER database properties in terrain builder.
Then imported roads.shp and exported it back out again due some global mapper config issue I'm having right now which generates bad shape files for arma3 to read on.
Below screenshot shows how SRTM 1-arc-second 30m resolution heightmap data often looks like.
Now you wouldn't want to land a aircraft on that runway now would you
2022-02-28T04:35:00Z Used L3DT to import heightmap.asc and L3DT_texture_map.png, then 3D sapphire heightmap tools smoothed out all airfield runways so arma3 aircrafts can easily operate there.
Then merged airfield_runways.shp with roads.shp in global mapper, now this most likely broke the roads.shp again for arma3, but well yeah I tried UTF-8 codepage in the save settings so lets hope that helped, if not I'll just do another terrain builder import/export shuffle.
2022-03-24T10:29:00Z New editing day has started, I just released PMC Ukraine Kharkiv v0.1 and will jump right away into PMC Ukraine Kyiv editing because its the next terrain on the release queue.
My bugs/issues website states few issues like airfields are not done and roads.shp needs terrain builder import/export. However reading this dev diary above (just double checking) looks like those are already done, so I'm going to pboProject binarize straight up so can do arma3 in-game test.
Was surprising to see in-game that all those issues are still present, umm if I already fixed them... why are they still broken, hmm.
2022-03-24T11:53:00Z Started terrain builder and loaded the project up, found that one of the airfields was missing runway poly line so created that, I believe this is the hostomel airport (google/maps) in north west of Kyiv.
Exported airfield_runways.shp next to the roads.shp in _data\roads\ dir. Then imported heightmap_smoothed_roads_airfields_smoothed.asc because that is what the name says, this logically should fix any smoothing issues as long as its done properly. Rebuilt terrain, of course.
Exported WRP ...
... saved project and exited terrain builder. Ran moveObject re-pathing for WRP file. Used global mapper to load airfield_runways.shp and added ID 7 and ORDER 0 to the newly created poly line, then exported it back out. Then loaded roads.shp so its merged with airfield_runways.shp and exported roads.shp back out which is now our full road poly line shape file. Then started terrain builder and did import/export for the newly saved roads.shp file, now its fixed for arma3 use.
2022-03-24T12:06:00Z Then another pboProject binarization, hopefully all the pending issues are now fixed. Just hate that have to wait another hmm good 30-45min for this to happen, oh well.
In-game test shows that apparently I actually DID smooth out airfield runways... just not all of them. NW hostomel airport is not smoothed out nor is the second one close to the city, but the SE big airbase or is it Kyiv International Airport I guess, is. So yeah I need to go through them once more. Only problem is to finding them all in crappy L3DT
But at least now I confirmed that roads.shp are fixed and airfield runway poly lines are done.
Just a little bit more fixes and then its good to go. One of the new fixes, well improvements, is going to be the new terrain processor generated vegetation objects with latest TPP files.
2022-03-24T13:01:00Z Leaped forward in dev diary writings again; deleted existing terrain builder object layers file and created Terrain Builder Keypoints for airfields. Then exported WRP so I can get the HPP file, I copied its content into pmc_names.hpp config.cpp file.
Then copied latest terrain processor TPP files into Kyiv dir, edited it a bit and then generated new objects with existing settings which should be, hmm hectare dencity 100.
Terrain processor generated 9.693 million objects, its just on the limit which I'm brave enough to import into terrain builder.
2022-03-24T13:10:00Z Started to import new vegetation objects to terrain builder, ETA is about 1hrs 15min
When importing finished I didn't notice it as was editing PMC Ukraine Luhansk dev diary video at the moment, just realized later that oh its already done. Then exported WRP and moveObject re-pathed it.
2022-03-24T14:38:00Z Went to back editing that Luhansk dev diary video a bit, just had a mood for it right now.
2022-03-24T15:47:00Z Finished that video editing and now its very late at night for me, been up almost 17hrs so will call it a night here.
2022-03-25T02:45:00Z New editing day has started, last nights final edits and the video editing left me a bit confused, cant recall if the smoothed airfields are in yet or not, heh this sucks
Brought up L3DT and loaded Kyiv project, then checked all airfield runways and indeed there were two or was it three (was kind of hard to see on heightmap mesh in 3d sapphire) that were not smoothed out, so I did that. Exported heightmap to "heightmap.asc" file, then ran terrain processor smooth roads which reads this "heightmap.asc" file and outputs "heightmap_smoothed_roads.asc" file. Finally terrain builder imported this last file, rebuilt terrain, removed the imported heightmap file, exported WRP, saved project and exited.
Then of course ran moveObject re-pathing the WRP.
And then it was pboProject binarization time again. Hopefully this will now generate the initial release version of PMC Ukraine Kyiv terrain.
2022-03-25T03:36:00Z Finished binarizing.
In-game arma3 check confirms ALL airfield runways have now smoothed out. However I found out that roads.shp got carved into heightmap on the edges. That is odd because overall the advice in the community has been to "cut / export roads.shp with global mapper" for this issue, but these roads are done with global mapper in the begin with.
I faintly remember talking about this in PMC discord 2020 or 2021, I used some roads.shp cutting method of few pixels/meters inside the actual terrain grid, hmm I need to go search and find that chat as cant remember not how its done.
Of course the carving issue is not a showstopper, it only appears on the far edges of the terrain where you rarely operate.
There is something about it in PMC Editing Wiki: Terrain Processor, Road Smoothing page.
Searched PMC discord and found the chat where I spoke about this issue... however I'm just saying "shrunk roads.shp -5 meters to fix it." and now I'm like "yeah, but HOW you .. HRRR!?" hehe
Yes its very annoying to encounter this kind of issue, "just do it" advice without anykind of info of actually HOW. I do not remember at all how to cut OSM shapes -5m in global mapper, best guess right now would be to just create another user created feature grid which is 5 meters smaller than the actual terrain grid. But dunno. Maybe I'll remember it when browsing around global mapper a bit, muscle memory comes back quickly.
And this time I'm sure to document it, thoroughly, step by step
2022-03-25T04:15:00Z Had to call it here, just dead tired going into that zombi mode without being able to get anything done.
2022-03-25T09:59:00Z Yawn, stupid nap. Already been up, got a bite to eat, documented some arma3 stuff, signed the addons, 7-zip packed the release package and just now took basic screenshots from PMC Ukraine Kyiv for the official web page.
2022-03-25T10:37:00Z Release 7-zip archive and .torrent files are done. Currently uploading to PMC website Virtual Private Server (VPS).
Release file name: pmc_ukraine_kyiv_v0.1.7z and size: 2.16gb
This time I'm experimenting of only using two UDP torrent trackers as I'm studying torrent tech more indepth and looks like there is no need for so many trackers I've been using on PMC arma3 torrents so far. Some people say (I am unable to verify the claim) that for public torrents you don't need ANY trackers that DHT is enough, that trackers only help in the startup to speed up finding seeds. Hopefully I fully learn how the tech works soon, but for now I am not going to use any http trackers, just the bare minimum of two of the most old and reliable trackers I know of today (opentrackr.org and openbittorrent.com).
Actually the VPS has some problems running rTorrent right now, it freezes. Not sure if this is because Linux server uptime is 1135 days, yes one THOUSAND one hundred and thirty five fucking days
uptime 10:43:39 up 1135 days, 3:06, 4 users, load average: 0.12, 0.21, 0.14
Anyways. A little background for release making process
2022-03-25T11:34:00Z Web pages created along with screenshots up to this point, meaning still need to proof read and add links etc smaller details.
2022-03-25T12:29:00Z This dev diary page got all links added and images verified etc. We are looking pretty good here.
But now its my lunch time, I'm starvin'
2022-03-25T13:10:00Z Feeding is done, back to release procedures...
Was looking at my release procedure "tool pipeline" heh and I cant think of what else to to, everything is good to go for release, just need to write the news and forum post.
2022-03-25T14:05:00Z PMC Ukraine Kyiv v0.1 released!