A modular, production-grade computer vision pipeline designed and evaluated across classical machine learning baselines, custom deep convolutional neural networks (CNNs) trained from scratch, transfer learning regimes (frozen vs. partial fine-tuning), data augmentation ablations, multi-seed statistical variance testing, subgroup slice audits, and inference-time decision threshold interventions.
The pipeline enforces strict split isolation, in-memory RAM caching for fast I/O throughput, reproducible seed control, and comprehensive post-training diagnostic profiling.
+-------------------------------------------------------------------------------+
| Raw CIFAR-10 Dataset (32x32 RGB) |
+---------------------------------------+---------------------------------------+
|
v
+-------------------------------------------------------------------------------+
| create_manifest.py (Stratified Hash Indexing & Integrity Checks) |
+---------------------------------------+---------------------------------------+
|
v
+-------------------------------------------------------------------------------+
| data_manifest.csv (40k Train / 10k Val / 10k Test) |
+--------------------+------------------+-------------------+-------------------+
| | |
v v v
+-----------------------+ +--------------------+ +--------------------+
| Classical ML Baseline | | Custom CNN Scratch | | Pretrained Transfer|
| Logistic Regression | | 89.9k Parameters | | MobileNetV3-Small |
| Color HSV / Moments | | Native 32x32 Input | | 224x224 Upsampled |
+-----------+-----------+ +---------+----------+ +---------+----------+
| | |
| +-----------+----------+
| |
v v
+-----------------------+ +---------------------------------+
| Scikit-Learn Model | | train.py Engine |
| evaluate_classical.py | | Adam + Cosine LR Scheduler |
+-----------+-----------+ +-----------------+---------------+
| |
v v
+-------------------------------------------------------------+
| checkpoints/ (*.pth Checkpoints) |
+------------------------------+------------------------------+
|
v
+-------------------------------------------------------------+
| evaluate.py (Locked Test Partition Evaluation) |
+------------------------------+------------------------------+
|
+-------------------------+-------------------------+
| | |
v v v
+--------------------------+ +--------------------+ +--------------------------+
| Diagnostic Visualizations| | Subgroup Audits | | Threshold Optimization |
| - Confusion Matrices | | - Lighting Slices | | - threshold_intervention |
| - PR Operating Curves | | - Contrast Slices | | - Critical Class Recall |
| - Top-10 Failure Mosaic | | - Focus Slices | | - Zero-Retraining Tuning |
+--------------------------+ +--------------------+ +--------------------------+
This project benchmarks on the CIFAR-10 Dataset (Alex Krizhevsky, University of Toronto), consisting of 60,000 airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.
The dataset images must be structured relative to the vision_classifier repository as follows:
Level2/
├── images/
│ ├── train/
│ │ ├── airplane/
│ │ │ ├── 0001.jpg
│ │ │ └── ...
│ │ ├── automobile/
│ │ └── ... (all 10 classes, 50,000 total training images)
│ └── test/
│ ├── airplane/
│ │ ├── 0001.jpg
│ │ └── ...
│ ├── automobile/
│ └── ... (all 10 classes, 10,000 total locked test images)
└── vision_classifier/ # Main project root
├── data_manifest.csv # Unified dataset index with train/val/test splits
├── create_manifest.py # Manifest creation script
├── checkpoints/ # Model checkpoints (.pth)
├── configs/ # YAML experiment configurations
├── results/ # Visualizations, metrics, and model cards
└── ...
To generate the stratified data_manifest.csv index with deterministic hash integrity checks:
python create_manifest.py- Train Split: 40,000 images (4,000 per class)
- Validation Split: 10,000 images (1,000 per class) — utilized exclusively for model selection, early stopping, and threshold tuning.
- Test Split: 10,000 images (1,000 per class) — strictly locked until final evaluation.
All models were evaluated on the locked test partition (
| Experiment | Model / Method | Input Size | Augmentation | Trainable Params | Total Params | Test Accuracy | Macro F1 | CPU Latency (p50) | CPU Latency (p95) |
|---|---|---|---|---|---|---|---|---|---|
| 1a. Majority Baseline | Dummy Classifier (Most Frequent) | N/A | None | 0 | 0 | 10.00% | 1.82% | N/A | N/A |
| 1b. Classical ML | Logistic Regression + HSV/Moments | 32x32 | None | 5,830 | 5,830 | 39.80% | 39.47% | 1.68 ms | 3.08 ms |
| 2. Custom CNN Baseline | CustomCNN (Trained from Scratch) | 32x32 | Moderate | 89,946 | 89,946 | 71.76% | 71.41% | 3.29 ms | 9.25 ms |
| 3. Frozen Backbone | MobileNetV3-Small (Frozen Backbone) | 224x224 | Moderate | 10,250 | 1,528,106 | 66.69% | 66.83% | 102.40 ms | 224.62 ms |
| 4. Partial Fine-Tuning | MobileNetV3-Small (Top Blocks Tuned) | 224x224 | Moderate | 1,337,586 | 1,528,106 | 89.96% | 89.95% | 46.08 ms | 62.26 ms |
| 5a. Augmentation: None | CustomCNN (No Augmentation) | 32x32 | None | 89,946 | 89,946 | 65.35% | 64.83% | 4.20 ms | 9.04 ms |
| 5b. Augmentation: Excessive | CustomCNN (Heavy Rotation+Jitter+Erasing) | 32x32 | Excessive | 89,946 | 89,946 | 49.06% | 46.96% | 14.47 ms | 107.34 ms |
| 6. Threshold Intervention | MobileNetV3-Small ( |
224x224 | Moderate | 1,337,586 | 1,528,106 | 89.97% | 89.96% | 46.08 ms | 62.26 ms |
| 7. Multi-Seed Robustness | CustomCNN (Seeds 42, 123, 999) | 32x32 | Moderate | 89,946 | 89,946 | 73.78 ± 0.39% | 73.59 ± 0.40% | 3.29 ms | 9.25 ms |
-
Pretrained Features vs. Native Resolution:
- The frozen MobileNetV3 (66.69%) performed below the Custom CNN (71.76%) because upsampling
$32\times 32$ to$224\times 224$ introduces interpolation artifacts that generic ImageNet features struggle to adapt to without retraining. - Unfreezing the top inverted residual blocks (Partial Fine-Tuning) allowed domain adaptation, unlocking 89.96% test accuracy (+18.20% gain over scratch CNN).
- The frozen MobileNetV3 (66.69%) performed below the Custom CNN (71.76%) because upsampling
-
Data Augmentation Sweet Spot:
- No Augmentation (65.35%): Rapidly overfits training data; validation loss diverges after Epoch 3 (-6.41% drop vs moderate).
- Moderate Augmentation (71.76%): Reflection padding (4px) + Random Horizontal Flip provides the optimal spatial invariance.
- Excessive Augmentation (49.06%): Color jitter, 30-degree rotation, and random erasing destroy low-resolution semantic cues (-22.70% drop).
-
Inference Latency & Efficiency:
- CustomCNN achieves real-time CPU latency of 3.29 ms (p50), making it 14x faster than MobileNetV3 (46.08 ms) for compute-constrained embedded hardware.
The partial fine-tuned model demonstrates high diagonal concentration (>93% on vehicles and distinct animals). The primary residual confusion occurs between visually similar quadruped categories (Cat vs. Dog).
To address asymmetric operational penalties for false negatives on the lowest-recall category (Cat), decision threshold tuning was calibrated on the validation split (
Evaluating CustomCNN across three random seeds (Seeds 42, 123, 999) verifies that model convergence is statistically stable (
Inspection of the top 10 highest-loss test errors identifies key failure modes:
- Background Visual Shortcuts: Small birds flying against open blue skies misclassified as airplanes.
- Extreme Occlusion and Crouching Angles: Camouflaged animals against dense foliage.
- Perspective Distortion: Close-up harbor angles confusing ships with terrestrial vehicles.
To evaluate robustness under degraded conditions, test images were partitioned across environmental slices:
| Slice Dimension | Subgroup Category | Sample Count | Partial Fine-Tuning Acc | Custom CNN Acc | Accuracy Disparity |
|---|---|---|---|---|---|
| Lighting | Bright (Overexposed) | 3,334 | 91.66% | 73.61% | +1.70% |
| Normal Lighting | 3,333 | 90.25% | 72.61% | +0.29% | |
| Dark (Underexposed) | 3,333 | 87.97% | 69.07% | -1.99% | |
| Contrast | High Contrast | 3,333 | 91.18% | 73.57% | +1.22% |
| Medium Contrast | 3,333 | 90.31% | 71.77% | +0.35% | |
| Low Contrast | 3,334 | 88.39% | 69.95% | -1.57% | |
| Focus | Sharp / High-Edge | 3,333 | 91.96% | 75.10% | +2.00% |
| Medium Focus | 3,333 | 90.19% | 71.14% | +0.23% | |
| Blurry / Soft | 3,334 | 87.73% | 69.05% | -2.23% |
# 1. Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate # Linux/macOS
.venv\Scripts\activate # Windows
# 2. Install required dependencies
pip install -r requirements.txt# Baseline Custom CNN
python train.py --config configs/baseline.yaml
# MobileNetV3 Partial Fine-Tuning
python train.py --config configs/transfer.yaml
# Augmentation Ablations
python train.py --config configs/ablation_aug_none.yaml
python train.py --config configs/ablation_aug_excessive.yaml# Evaluate checkpoint on locked test set
python evaluate.py --checkpoint checkpoints/transfer_mobilenet_v3_partial.pth# Single image inference (returns formatted JSON)
python predict.py --checkpoint checkpoints/transfer_mobilenet_v3_partial.pth --image path/to/sample.jpg
# Batch inference on CSV manifest
python predict.py --checkpoint checkpoints/transfer_mobilenet_v3_partial.pth --csv data_manifest.csv --output results/predictions.csv# Optimize threshold for target class sensitivity
python threshold_intervention.py --checkpoint checkpoints/transfer_mobilenet_v3_partial.pth --critical_class cat --target_recall 0.90# Train and evaluate across Seeds 42, 123, 999
python multi_seed_evaluation.py# Extract highest-loss test misclassifications and generate mosaic
python generate_failure_gallery.py# Run automated verification suite
python -m unittest -v tests.pyvision_classifier/
├── checkpoints/ # Saved model weights
│ ├── baseline_custom_cnn.pth # Custom CNN scratch checkpoint (1.1 MB)
│ ├── transfer_mobilenet_v3_partial.pth # Best fine-tuned checkpoint (6.2 MB)
│ ├── transfer_mobilenet_v3_frozen.pth # Frozen backbone checkpoint (11.0 MB)
│ ├── ablation_aug_none.pth # No augmentation ablation checkpoint
│ ├── ablation_aug_excessive.pth # Excessive augmentation ablation checkpoint
│ ├── seed_123_custom_cnn.pth # Multi-seed checkpoint 123
│ └── seed_999_custom_cnn.pth # Multi-seed checkpoint 999
├── configs/ # Experiment YAML configurations
│ ├── baseline.yaml # Baseline Custom CNN configuration
│ ├── transfer.yaml # Partial fine-tuning configuration
│ ├── transfer_frozen.yaml # Frozen backbone configuration
│ ├── ablation_aug_none.yaml # Zero-augmentation ablation
│ ├── ablation_aug_excessive.yaml # Excessive augmentation ablation
│ ├── seed_123_custom_cnn.yaml # Seed 123 configuration
│ └── seed_999_custom_cnn.yaml # Seed 999 configuration
├── results/ # Evaluation outputs and artifacts
│ ├── experiment_table.csv # Complete benchmark summary table
│ ├── model_card.md # Detailed ML Model Card
│ ├── predictions.csv # Test set prediction logs
│ ├── seed_robustness_metrics.csv # Multi-seed statistical outputs
│ ├── seed_per_class_stability.csv # Per-class variance logs
│ ├── confusion_matrices/ # Generated normalized & raw confusion matrix heatmaps
│ │ ├── transfer_mobilenet_v3_partial_cm_normalized.png
│ │ ├── baseline_custom_cnn_cm_normalized.png
│ │ └── threshold_intervention_cat_cm.png
│ └── failure_gallery/ # Qualitative error analysis artifacts
│ ├── failure_mosaic.png # 2x5 mosaic of top 10 test errors
│ ├── multi_seed_robustness.png # Multi-seed accuracy comparison plot
│ ├── threshold_pr_curve_cat.png # Critical class PR operating curve
│ ├── top_10_failures.csv # Top 10 high-loss error metadata
│ └── failure_01.png ... failure_10.png # Individual error diagnostic cards
├── dataset.py # ManifestDataset & FastManifestDataset loaders
├── models.py # CustomCNN and TransferClassifier architectures
├── transforms.py # Augmentation regimes (None, Moderate, Excessive)
├── train.py # Training engine with Cosine scheduler & early stopping
├── evaluate.py # Comprehensive evaluation & slice profiling engine
├── predict.py # Production CLI prediction tool
├── baseline_classical.py # Logistic Regression + HSV/Moments baseline
├── threshold_intervention.py # Post-processing decision threshold optimizer
├── multi_seed_evaluation.py # Multi-seed training & ANOVA variance suite
├── generate_failure_gallery.py # Top-loss failure extractor & visualizer
├── create_manifest.py # Dataset manifest indexer & SHA-256 validator
├── tests.py # Automated unit test suite
├── requirements.txt # Pinned dependency requirements
├── .gitignore # Git ignore rules
└── README.md # Technical documentation




