Overview

An end-to-end deep-learning pipeline for drone- and camera-based target identification. The system unifies heterogeneous public image datasets, trains a multi-output CNN that predicts both a fine-grained object class and a coarse valid/invalid label, and embeds the trained classifier behind a YOLO segmentation front-end for real-time video.

The work emphasizes practical concerns: dataset unification, label consistency, and computational efficiency, so the classifier can be dropped into a streaming application running on drone footage or fixed surveillance cameras.

Pipeline

Three modular stages, each independently upgradable.

Dataset

Source datasets use inconsistent directory layouts, label columns, and category names. The builder normalizes all of this into one corpus suitable for training.

Each fine label (e.g. soldier, civilian_vehicle, tank_av) is mapped onto one of three coarse buckets. Labels are derived per source via folder names, CSV columns, or filename regex, then aliased onto a canonical taxonomy. A per-class cap keeps dominant categories from swamping training, and each image is assigned at random to a 70 / 15 / 15 train/val/test split. Unreadable files are skipped, and hashing each re-encoded image catches the same picture appearing across multiple sources.

Model architecture

A compact multi-output ConvNet shared between two classification heads.

The backbone is three Conv → ReLU → MaxPool blocks with growing channel depth (32 → 64 → 128) and shrinking spatial resolution. The final feature map is flattened into a 256-unit dense layer with dropout, which feeds both heads:

Both heads train jointly with sparse categorical cross-entropy, weighted equally. L2 regularization on every kernel and 50% dropout mitigate overfitting; AdamW with a plateau-triggered LR schedule rounds out the optimizer setup, and the checkpoint with the lowest validation loss is kept.

Training

Implemented in TensorFlow with tf.data pipelines built directly from metadata.csv. Each row loads its image via out_relpath, decodes to RGB, resizes to a fixed input resolution, and scales pixel values to [0, 1]. No augmentation is applied.

Every sample carries equal weight. An optional flag controls whether invalid examples contribute to the fine-head loss: when it is set, invalids train only the coarse head, freeing the fine head to focus on examples with meaningful fine labels. Both heads train together from the first epoch.

Training and validation loss/accuracy curves for the multi-output drone CNN across 50 epochs at 128×128 input resolution.
Training and validation curves at 128×128 input resolution over 50 epochs. Validation accuracy levels off around epoch 20 while validation loss climbs, most on the fine head, so the network overfits late in training; the kept checkpoint is the one with the lowest validation loss. In the loss panel the dashed lines are validation loss, though the legend repeats the training labels.

Results

Held-out test set: 27,362 samples across 11 fine classes. Naïve single-class baselines reach 23.5% (fine) and 34.5% (coarse), so both heads must learn real structure to be useful.

Fine confusion matrix

Test set confusion matrix for the 11-class fine head. Strong diagonal dominance, with errors spread across many pairs rather than clustered: the largest, civilian / random_animal, is 310 of 3,534 misclassifications, and flying_target is the most common wrong prediction (952).

Fine-head confusion matrix across 11 classes: civilian_vehicle, tank_av, aerial_landscape, cloud_blanksky, tree_shrub, flying_target, bird, random_animal, civilian, soldier, rock_debris.
Held-out test set, 27,362 samples, from the 256×256 run the headline figures come from. Diagonal dominance confirms the 87.1% accuracy; the brightest off-diagonals match the qualitative error patterns documented in the writeup.

Per-class precision

The fine head is strongest on visually distinct categories (vehicles, landscape) and weakest where classes share appearance (rock debris, soldier, civilian). Recall is lowest on soldiers: 436 of 789 are found, with 148 of the misses going to flying_target and 131 to civilian.

Fine head: test precision per class
Class Precision
civilian_vehicle 0.9538
tank_av 0.9426
aerial_landscape 0.9306
cloud_blanksky 0.9174
tree_shrub 0.8885
flying_target 0.8629
bird 0.8123
random_animal 0.8027
civilian 0.7882
soldier 0.7730
rock_debris 0.7547
Coarse head: test precision per class
Class Precision
invalid_nontarget 0.9277
valid 0.9165
invalid_background 0.8941

Error analysis

Confusion matrices and qualitative review highlight three recurring failure modes.

Input resolution trade-off

Bumping inputs from 64×64 to 128×128 significantly improved both fine and coarse accuracy; 256×256 added very little. For offline classification, 128×128 is the sweet spot. For the live video pipeline, lower-resolution crops actually helped: they kept the YOLO + CNN loop closer to real time and improved detection of small, distant targets thanks to the larger effective receptive field.

Real-time utilization

The deployment program runs against video files (with webcam and live drone feed as future targets). YOLO produces per-frame instance masks; each mask is cropped from the source frame, resized to the CNN's input size, and normalized identically to training. Only the coarse head is consulted in the live overlay: the operational question is "valid target?", and the three-way decision is fast and easy to render:

Optional frame skipping keeps throughput up. Decoupling detection from classification means a newer YOLO model can drop in without touching the CNN, and adding new fine classes only requires retraining the classifier.

Live overlay demo: YOLO segmentation drives per-frame crops and the coarse head marks valid targets in green. These clips predate a label-order fix in the source code: their red overlays, labelled invalid_nontarget, are crops the model put in invalid_background, which the corrected script hides. Lower-resolution CNN input crops (32×32 in this run) keep the loop close to real time without sacrificing detection of small distant targets. Prefer the longer demo? Download the extended cut (≈7 MB).

Limitations & future work