This is a follow up to my prior post on AI LiDAR Ground Segmentation. If you have not read it, go here.
I want a well trained model and so I will be feeding it constantly. To start we are training on the Commonwealth of Virginia. I need data so I wrote a script that broke the state into 408 cells. Next it queried USGS for how many tiles are in each cell then logged the URLs. Should you do something like this, throttle it and don’t slam a site with the request all at once. Be nice! My script works slow and steady to not impact the backend. This takes a while to complete. When it did I had a CSV of the 312,330 LiDAR files I need.
Here’s what the indexing script does under the hood:
- Divides Virginia into 408 grid cells to keep queries small and manageable
- Sends a bbox query to USGS 3DEP for each cell to retrieve matching LiDAR products
- Extracts tile IDs, bounding boxes, and download links from each API response
- Deduplicates everything (since tiles overlap multiple cells)
- Writes a single consolidated CSV containing metadata and URLs for the full dataset
[info] writing incremental index to usgs_va_3dep_index.csv (records=308321)
[info] cell 400 bbox=-79.2,39.375,-78.7,39.5 added 585 new product(s); total unique tiles 308906
[info] cell 401 bbox=-78.7,39.375,-78.2,39.5 added 401 new product(s); total unique tiles 309307
[info] cell 402 bbox=-78.2,39.375,-77.7,39.5 added 741 new product(s); total unique tiles 310048
[info] cell 403 bbox=-77.7,39.375,-77.2,39.5 added 549 new product(s); total unique tiles 310597
[info] cell 404 bbox=-77.2,39.375,-76.7,39.5 added 229 new product(s); total unique tiles 310826
[info] cell 405 bbox=-76.7,39.375,-76.2,39.5 added 345 new product(s); total unique tiles 311171
[info] cell 406 bbox=-76.2,39.375,-75.7,39.5 added 644 new product(s); total unique tiles 311815
[info] cell 407 bbox=-75.7,39.375,-75.2,39.5 added 515 new product(s); total unique tiles 312330
[info] processed 408/408 cells; total unique tiles 312330
Visited 408 cells; found 312330 unique tiles.Once I had the full statewide index built, the next challenge was obvious.
The download problem
So I now have thousands and thousands of files to download from the cloud. I have the bandwidth and storage necessary to accommodate my goal. Just because one can download everything at once doesn’t mean one should. Slamming a public API with thousands of simultaneous requests is a great way to get your IP rate-limited into oblivion—or worse, disrupt access for everyone else. So onto the next script to politely download terabytes of data while not overwhelming my network or theirs. This will take a while… it’s a lot.
Again, please don’t just slam a site when doing something like this.
So the next step was to build a polite, well-behaved, resume-friendly download pipeline. At a high level, here’s what it does:
- It loads the statewide index CSV—basically a giant checklist of every tile I need.
- Downloads are handled in small, controlled parallel batches so I can take advantage of gigabit fiber without triggering rate limits.
- Each file is streamed in manageable chunks, which keeps memory stable regardless of the files size.
- The script automatically skips files already downloaded and can resume after interruptions, which is critical for multi-day runs.
- Any failures get logged for later retries so the process never stalls.
When complete, I will have a clean, local, fully organized archive of Virginia’s LiDAR. Perfectly ready for tiling, training, and everything that comes next.
[progress] completed 23000/244853 tasks | succeeded=22208, skipped=690, errors=102
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_AG169_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_AG169_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y149_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y149_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y150_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y150_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y151_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y151_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y152_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y152_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y153_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y153_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y154_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y154_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y155_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y155_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y156_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y156_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y157_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y157_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y158_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y158_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y159_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y159_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y160_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y160_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y161_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y161_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y162_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y162_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y163_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y163_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y164_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y164_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y165_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y165_LAS_2018.laz
[download] USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y166_LAS_2018.laz -> /AI/LiDARClassification/data/raw_usgs_va/USGS_LPC_VA_FEMA_R3_Southwest_B_2016_Y166_LAS_2018.laz
[progress] completed 23020/244853 tasks | succeeded=22228, skipped=690, errors=102Not so fast… Sanity check the files downloading
Early into my downloads I noticed a lot of files that did not look like they were in VA. I expect some of border state data, but it was more than that. I did some analysis and then I made another script to sanity check everything. After the statewide index run, I ran a boundary validation pass against Virginia’s official footprint. That sanity check trimmed the list from 300k+ candidate tiles down to about 85k tiles that actually intersect Virginia, which saves a ton of download time and storage and keeps the training data from drifting outside the environment I’m trying to model.
Incremental training for AI LiDAR Ground Segmentation
Training non-stop and in a way that produces incremental results for validation is a must for me. Throwing all the tiles at this at once would take an unreasonable amount of time. If there is an error somewhere far too much time will have passed and be lost. I need to make logic to manage the tile creation and training. Time for autotrain.
davidbarton@Mac-Studio LiDARClassification % PYTHONPATH=. python -m kpunet autotrain \
--raw-dir Source_Data/data/raw \
--tiles-root data/tiles/usgs_va_stream \
--output-dir models/kpunet_va_stream \
--config configs/usgs_va.yaml \
--num-shards-per-round 2 \
--epochs-per-round 15 \
--batch-size 2 \
--num-workers 0
Starting round 1 with 2 shard(s).
[Round 1] Epoch 1: train_loss=2.8792 seg=0.6434 regP=2193.6357 regF=42.1767
[Round 1] Epoch 2: train_loss=0.6376 seg=0.6294 regP=5.6065 regF=2.5594
[Round 1] Epoch 3: train_loss=0.6330 seg=0.6208 regP=8.3046 regF=3.8861
[Round 1] Epoch 4: train_loss=0.6788 seg=0.6289 regP=47.0715 regF=2.8719
[Round 1] Epoch 5: train_loss=0.6253 seg=0.6068 regP=17.4530 regF=1.0699
[Round 1] Epoch 6: train_loss=0.6268 seg=0.5967 regP=28.6346 regF=1.4969
[Round 1] Epoch 7: train_loss=0.5469 seg=0.5283 regP=17.1870 regF=1.4535
[Round 1] Epoch 8: train_loss=0.5621 seg=0.5416 regP=18.1027 regF=2.4274
[Round 1] Epoch 9: train_loss=0.5068 seg=0.4901 regP=16.5289 regF=0.2410
[Round 1] Epoch 10: train_loss=0.4857 seg=0.4724 regP=13.1144 regF=0.2053
[Round 1] Epoch 11: train_loss=0.4741 seg=0.4618 regP=11.9627 regF=0.3578
[Round 1] Epoch 12: train_loss=0.4777 seg=0.4523 regP=24.3338 regF=1.0749
[Round 1] Epoch 13: train_loss=0.4532 seg=0.4457 regP=6.8338 regF=0.6764
[Round 1] Epoch 14: train_loss=0.4427 seg=0.4357 regP=6.8018 regF=0.2656
[Round 1] Epoch 15: train_loss=0.4332 seg=0.4266 regP=6.3267 regF=0.2918
Completed round 1 with status=ok
Starting round 2 with 2 shard(s).
Resumed from models/kpunet_va_stream/latest.pt at epoch 15
[Round 2] Epoch 16: train_loss=0.4271 seg=0.4210 regP=5.8294 regF=0.3360
[Round 2] Epoch 17: train_loss=0.4297 seg=0.4206 regP=6.2544 regF=2.8113
[Round 2] Epoch 18: train_loss=0.4465 seg=0.4243 regP=6.9861 regF=15.2416
[Round 2] Epoch 19: train_loss=0.4362 seg=0.4208 regP=6.7896 regF=8.6342
[Round 2] Epoch 20: train_loss=0.4303 seg=0.4130 regP=6.0805 regF=11.1899
[Round 2] Epoch 21: train_loss=0.4126 seg=0.4058 regP=6.2279 regF=0.5299
[Round 2] Epoch 22: train_loss=0.4555 seg=0.4387 regP=6.7268 regF=10.0982
[Round 2] Epoch 23: train_loss=0.4526 seg=0.4292 regP=20.0388 regF=3.3928
[Round 2] Epoch 24: train_loss=0.4127 seg=0.4063 regP=4.6487 regF=1.7545
[Round 2] Epoch 25: train_loss=0.4069 seg=0.4015 regP=4.8230 regF=0.5086
[Round 2] Epoch 26: train_loss=0.3992 seg=0.3948 regP=4.3714 regF=0.0194
[Round 2] Epoch 27: train_loss=0.3962 seg=0.3920 regP=4.1049 regF=0.0058
[Round 2] Epoch 28: train_loss=0.3972 seg=0.3932 regP=3.9627 regF=0.0470
Days and days of GPU usage on the Mac Studio. These are the moments you pray your battery back up is healthy. I did implement resume so worst case scenario I should only loose the current training epoch vs everything I have done. It is noteworthy to highlight the power efficiency of this machine. Sure, an Nvidia RTX5090 would be faster for training, but it would certainly consume more power. I don’t have an RTK5090, if you want to contribute one to this series feel free to do so. It will be use well, however the Mac Studio is not making my power meter spin like Clark Griswold’s in National Lampoons Christmas Vacation. I am grateful for that.
Early LiDAR Ground Segmentation Training Results
david@Mac-Studio LiDARClassification % PYTHONPATH=. python -m kpunet evaluate \
--tiles data/tiles/usgs_va_stream \
--model models/kpunet_va_stream/latest.pt \
--batch-size 2
Accuracy: 0.4912, mIoU: 0.3232, acc_non_ground=0.4303, acc_ground=0.6163
Confusion matrix:
tensor([[ 8387641, 11103750],
[ 3635276, 5840009]])This is still the very beginning of the training process and the model has only seen four LiDAR shards so far, just a tiny fraction of what will eventually be a statewide dataset. Even with that small sample, the early checkpoint is already showing promising signs. The model is correctly identifying ground points about 62% of the time and non-ground points about 43%, with an overall accuracy just under 50% and an mIoU of 0.32. Those numbers may sound modest, but for a model that has barely begun to learn what Virginia looks like, mountains, flatlands, forests, marshes, and urban areas, this is exactly what I expect at this stage. Early rounds are about giving the model enough examples to understand the concept of “ground,” not about achieving perfect accuracy. I am happy the pipeline and workflow are working very well.
The confusion matrix tells the story: the model is a little too confident and tends to over-label points as ground, resulting in about 11 million non-ground points being misclassified as ground in this checkpoint. That’s normal at this point with so few examples, it hasn’t yet learned the subtle patterns that separate ground from low vegetation, roadside slopes, or detailed man-made features. I will add more shards into the training pipeline then the model will encounter far more variation, and those boundaries will tighten up quickly. This is the warm-up phase, but the model is learning exactly what it should be learning right now.
What is next for the training
These early results are a solid start, but they’re nowhere close to where this model ultimately needs to be. With only four shards under its belt, the model simply hasn’t seen enough of Virginia’s diverse terrain to generalize well. That will change quickly as I begin feeding it thousands of additional tiles from mountains, piedmont, coastal plain, and everything in between. The accuracy and mIoU will climb, the confusion matrix will tighten, and the model will gradually shift from “guessing the general idea” to making consistent, reliable classifications across an entire state. This checkpoint is just a snapshot of the first steps, not the destination.
The “Chew, Check, Tweak” Loop
This part is less “set it and forget it” and more like training a stubborn dog. I let the GPU chew on a chunk of data long enough to actually learn something (not just one batch, but enough epochs that the loss starts to settle). Then I stop and run evaluation on tiles the model hasn’t memorized. Either a held-out subset, or just “fresh” shards it hasn’t seen yet. That validation pass is the reality check. It tells me if the model is genuinely learning ground vs non-ground, or if it’s just leaning too hard in one direction and calling everything “ground” because the class balance nudged it that way.
Validate With Numbers, Not Vibes
Every evaluation produces the same set of scoreboard stats: overall accuracy, mIoU, per-class accuracy, and the confusion matrix. Those numbers are what drive the next decision. If ground accuracy is high but non-ground collapses (or vice versa), that’s usually a sign the class weights are off, the sampling is biased, or the training shards don’t represent enough terrain variety yet. If the confusion matrix shows a big “everything becomes ground” tendency, that’s not a reason to panic—it’s a reason to adjust the training knobs and expand what the model is exposed to.
Tweak the Inputs, Not Just the Model
The fastest improvements often come from changing what the model sees, not rewriting the network. That means updating class weights, tuning weighted sampling thresholds (like minimum ground points), changing the shard mix, or injecting more geographic variety so the model doesn’t overfit to a single region or project style. Then it’s back to training, same pipeline, new parameters, new shards, new round. Over time, the cycle gets tighter: train → evaluate → adjust → repeat, and the numbers steadily stop swinging wildly and start trending in the right direction.
My next post will be a deeper dive into the challenges I faced while training.