Skip to content

About

End-to-end computer vision pipeline on CIFAR-10 with Custom CNN & MobileNetV3 backbones, data augmentation ablations, slice-based auditing, and threshold optimization.

Topics

Resources

Stars

4 stars

Watchers

0 watching

Forks

Repository files navigation

Production Vision Classifier: End-to-End Deep Learning Pipeline

A modular, production-grade computer vision pipeline designed and evaluated across classical machine learning baselines, custom deep convolutional neural networks (CNNs) trained from scratch, transfer learning regimes (frozen vs. partial fine-tuning), data augmentation ablations, multi-seed statistical variance testing, subgroup slice audits, and inference-time decision threshold interventions.


1. System Architecture

The pipeline enforces strict split isolation, in-memory RAM caching for fast I/O throughput, reproducible seed control, and comprehensive post-training diagnostic profiling.

+-------------------------------------------------------------------------------+
|                      Raw CIFAR-10 Dataset (32x32 RGB)                         |
+---------------------------------------+---------------------------------------+
                                        |
                                        v
+-------------------------------------------------------------------------------+
|         create_manifest.py (Stratified Hash Indexing & Integrity Checks)      |
+---------------------------------------+---------------------------------------+
                                        |
                                        v
+-------------------------------------------------------------------------------+
|              data_manifest.csv (40k Train / 10k Val / 10k Test)               |
+--------------------+------------------+-------------------+-------------------+
                     |                  |                   |
                     v                  v                   v
         +-----------------------+ +--------------------+ +--------------------+
         | Classical ML Baseline | | Custom CNN Scratch | | Pretrained Transfer|
         | Logistic Regression   | | 89.9k Parameters   | | MobileNetV3-Small  |
         | Color HSV / Moments   | | Native 32x32 Input | | 224x224 Upsampled  |
         +-----------+-----------+ +---------+----------+ +---------+----------+
                     |                       |                      |
                     |                       +-----------+----------+
                     |                                   |
                     v                                   v
         +-----------------------+           +---------------------------------+
         | Scikit-Learn Model    |           | train.py Engine                 |
         | evaluate_classical.py |           | Adam + Cosine LR Scheduler      |
         +-----------+-----------+           +-----------------+---------------+
                     |                                         |
                     v                                         v
         +-------------------------------------------------------------+
         |               checkpoints/ (*.pth Checkpoints)              |
         +------------------------------+------------------------------+
                                        |
                                        v
         +-------------------------------------------------------------+
         |        evaluate.py (Locked Test Partition Evaluation)       |
         +------------------------------+------------------------------+
                                        |
              +-------------------------+-------------------------+
              |                         |                         |
              v                         v                         v
+--------------------------+ +--------------------+ +--------------------------+
| Diagnostic Visualizations| |  Subgroup Audits   | |  Threshold Optimization  |
| - Confusion Matrices     | | - Lighting Slices  | | - threshold_intervention |
| - PR Operating Curves    | | - Contrast Slices  | | - Critical Class Recall  |
| - Top-10 Failure Mosaic  | | - Focus Slices     | | - Zero-Retraining Tuning |
+--------------------------+ +--------------------+ +--------------------------+

2. Dataset Setup and Relative Directory Structure

This project benchmarks on the CIFAR-10 Dataset (Alex Krizhevsky, University of Toronto), consisting of 60,000 $32 \times 32$ color images across 10 mutually exclusive classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.

Expected Directory Layout

The dataset images must be structured relative to the vision_classifier repository as follows:

Level2/
├── images/
│   ├── train/
│   │   ├── airplane/
│   │   │   ├── 0001.jpg
│   │   │   └── ...
│   │   ├── automobile/
│   │   └── ... (all 10 classes, 50,000 total training images)
│   └── test/
│       ├── airplane/
│       │   ├── 0001.jpg
│       │   └── ...
│       ├── automobile/
│       └── ... (all 10 classes, 10,000 total locked test images)
└── vision_classifier/          # Main project root
    ├── data_manifest.csv       # Unified dataset index with train/val/test splits
    ├── create_manifest.py      # Manifest creation script
    ├── checkpoints/            # Model checkpoints (.pth)
    ├── configs/                # YAML experiment configurations
    ├── results/                # Visualizations, metrics, and model cards
    └── ...

Dataset Manifest Generation

To generate the stratified data_manifest.csv index with deterministic hash integrity checks:

python create_manifest.py
  • Train Split: 40,000 images (4,000 per class)
  • Validation Split: 10,000 images (1,000 per class) — utilized exclusively for model selection, early stopping, and threshold tuning.
  • Test Split: 10,000 images (1,000 per class) — strictly locked until final evaluation.

3. Comprehensive Experimental Benchmark

All models were evaluated on the locked test partition ($N=10,000$) with CPU batch-1 latency profiled over 100 consecutive forward passes on an Intel/AMD CPU.

Primary Results Table

Experiment Model / Method Input Size Augmentation Trainable Params Total Params Test Accuracy Macro F1 CPU Latency (p50) CPU Latency (p95)
1a. Majority Baseline Dummy Classifier (Most Frequent) N/A None 0 0 10.00% 1.82% N/A N/A
1b. Classical ML Logistic Regression + HSV/Moments 32x32 None 5,830 5,830 39.80% 39.47% 1.68 ms 3.08 ms
2. Custom CNN Baseline CustomCNN (Trained from Scratch) 32x32 Moderate 89,946 89,946 71.76% 71.41% 3.29 ms 9.25 ms
3. Frozen Backbone MobileNetV3-Small (Frozen Backbone) 224x224 Moderate 10,250 1,528,106 66.69% 66.83% 102.40 ms 224.62 ms
4. Partial Fine-Tuning MobileNetV3-Small (Top Blocks Tuned) 224x224 Moderate 1,337,586 1,528,106 89.96% 89.95% 46.08 ms 62.26 ms
5a. Augmentation: None CustomCNN (No Augmentation) 32x32 None 89,946 89,946 65.35% 64.83% 4.20 ms 9.04 ms
5b. Augmentation: Excessive CustomCNN (Heavy Rotation+Jitter+Erasing) 32x32 Excessive 89,946 89,946 49.06% 46.96% 14.47 ms 107.34 ms
6. Threshold Intervention MobileNetV3-Small ($\tau^* = 0.468$) 224x224 Moderate 1,337,586 1,528,106 89.97% 89.96% 46.08 ms 62.26 ms
7. Multi-Seed Robustness CustomCNN (Seeds 42, 123, 999) 32x32 Moderate 89,946 89,946 73.78 ± 0.39% 73.59 ± 0.40% 3.29 ms 9.25 ms

Key Experimental Insights

  1. Pretrained Features vs. Native Resolution:

    • The frozen MobileNetV3 (66.69%) performed below the Custom CNN (71.76%) because upsampling $32\times 32$ to $224\times 224$ introduces interpolation artifacts that generic ImageNet features struggle to adapt to without retraining.
    • Unfreezing the top inverted residual blocks (Partial Fine-Tuning) allowed domain adaptation, unlocking 89.96% test accuracy (+18.20% gain over scratch CNN).
  2. Data Augmentation Sweet Spot:

    • No Augmentation (65.35%): Rapidly overfits training data; validation loss diverges after Epoch 3 (-6.41% drop vs moderate).
    • Moderate Augmentation (71.76%): Reflection padding (4px) + Random Horizontal Flip provides the optimal spatial invariance.
    • Excessive Augmentation (49.06%): Color jitter, 30-degree rotation, and random erasing destroy low-resolution semantic cues (-22.70% drop).
  3. Inference Latency & Efficiency:

    • CustomCNN achieves real-time CPU latency of 3.29 ms (p50), making it 14x faster than MobileNetV3 (46.08 ms) for compute-constrained embedded hardware.

4. Visualizations and Experimental Findings

A. Normalized Confusion Matrix (Best Model: MobileNetV3 Partial Fine-Tuned)

The partial fine-tuned model demonstrates high diagonal concentration (>93% on vehicles and distinct animals). The primary residual confusion occurs between visually similar quadruped categories (Cat vs. Dog).

Normalized Confusion Matrix


B. Critical-Class Decision Threshold Intervention (Experiment 6)

To address asymmetric operational penalties for false negatives on the lowest-recall category (Cat), decision threshold tuning was calibrated on the validation split ($\tau^* = 0.468$) and evaluated on the test set.

Side-by-Side Threshold Intervention Confusion Matrix

Critical Class Precision-Recall Operating Curve


C. Multi-Seed Robustness and Variance Analysis (Experiment 7)

Evaluating CustomCNN across three random seeds (Seeds 42, 123, 999) verifies that model convergence is statistically stable ($\sigma = 0.39%$).

Multi-Seed Robustness Plot


D. Qualitative Error Analysis & Top-10 Failure Gallery

Inspection of the top 10 highest-loss test errors identifies key failure modes:

  1. Background Visual Shortcuts: Small birds flying against open blue skies misclassified as airplanes.
  2. Extreme Occlusion and Crouching Angles: Camouflaged animals against dense foliage.
  3. Perspective Distortion: Close-up harbor angles confusing ships with terrestrial vehicles.

Top 10 Failure Mosaic


5. Subgroup Slice Reliability Analysis

To evaluate robustness under degraded conditions, test images were partitioned across environmental slices:

Slice Dimension Subgroup Category Sample Count Partial Fine-Tuning Acc Custom CNN Acc Accuracy Disparity
Lighting Bright (Overexposed) 3,334 91.66% 73.61% +1.70%
Normal Lighting 3,333 90.25% 72.61% +0.29%
Dark (Underexposed) 3,333 87.97% 69.07% -1.99%
Contrast High Contrast 3,333 91.18% 73.57% +1.22%
Medium Contrast 3,333 90.31% 71.77% +0.35%
Low Contrast 3,334 88.39% 69.95% -1.57%
Focus Sharp / High-Edge 3,333 91.96% 75.10% +2.00%
Medium Focus 3,333 90.19% 71.14% +0.23%
Blurry / Soft 3,334 87.73% 69.05% -2.23%

6. Command-Line Interface (CLI) Usage

Environment Setup

# 1. Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate   # Linux/macOS
.venv\Scripts\activate      # Windows

# 2. Install required dependencies
pip install -r requirements.txt

1. Training

# Baseline Custom CNN
python train.py --config configs/baseline.yaml

# MobileNetV3 Partial Fine-Tuning
python train.py --config configs/transfer.yaml

# Augmentation Ablations
python train.py --config configs/ablation_aug_none.yaml
python train.py --config configs/ablation_aug_excessive.yaml

2. Evaluation

# Evaluate checkpoint on locked test set
python evaluate.py --checkpoint checkpoints/transfer_mobilenet_v3_partial.pth

3. Inference

# Single image inference (returns formatted JSON)
python predict.py --checkpoint checkpoints/transfer_mobilenet_v3_partial.pth --image path/to/sample.jpg

# Batch inference on CSV manifest
python predict.py --checkpoint checkpoints/transfer_mobilenet_v3_partial.pth --csv data_manifest.csv --output results/predictions.csv

4. Decision Threshold Intervention

# Optimize threshold for target class sensitivity
python threshold_intervention.py --checkpoint checkpoints/transfer_mobilenet_v3_partial.pth --critical_class cat --target_recall 0.90

5. Multi-Seed Robustness Suite

# Train and evaluate across Seeds 42, 123, 999
python multi_seed_evaluation.py

6. Failure Gallery Extraction

# Extract highest-loss test misclassifications and generate mosaic
python generate_failure_gallery.py

7. Unit Tests

# Run automated verification suite
python -m unittest -v tests.py

7. Repository Structure

vision_classifier/
├── checkpoints/                              # Saved model weights
│   ├── baseline_custom_cnn.pth               # Custom CNN scratch checkpoint (1.1 MB)
│   ├── transfer_mobilenet_v3_partial.pth      # Best fine-tuned checkpoint (6.2 MB)
│   ├── transfer_mobilenet_v3_frozen.pth       # Frozen backbone checkpoint (11.0 MB)
│   ├── ablation_aug_none.pth                 # No augmentation ablation checkpoint
│   ├── ablation_aug_excessive.pth            # Excessive augmentation ablation checkpoint
│   ├── seed_123_custom_cnn.pth               # Multi-seed checkpoint 123
│   └── seed_999_custom_cnn.pth               # Multi-seed checkpoint 999
├── configs/                                  # Experiment YAML configurations
│   ├── baseline.yaml                         # Baseline Custom CNN configuration
│   ├── transfer.yaml                         # Partial fine-tuning configuration
│   ├── transfer_frozen.yaml                  # Frozen backbone configuration
│   ├── ablation_aug_none.yaml                # Zero-augmentation ablation
│   ├── ablation_aug_excessive.yaml           # Excessive augmentation ablation
│   ├── seed_123_custom_cnn.yaml              # Seed 123 configuration
│   └── seed_999_custom_cnn.yaml              # Seed 999 configuration
├── results/                                  # Evaluation outputs and artifacts
│   ├── experiment_table.csv                  # Complete benchmark summary table
│   ├── model_card.md                         # Detailed ML Model Card
│   ├── predictions.csv                       # Test set prediction logs
│   ├── seed_robustness_metrics.csv           # Multi-seed statistical outputs
│   ├── seed_per_class_stability.csv          # Per-class variance logs
│   ├── confusion_matrices/                   # Generated normalized & raw confusion matrix heatmaps
│   │   ├── transfer_mobilenet_v3_partial_cm_normalized.png
│   │   ├── baseline_custom_cnn_cm_normalized.png
│   │   └── threshold_intervention_cat_cm.png
│   └── failure_gallery/                      # Qualitative error analysis artifacts
│       ├── failure_mosaic.png                # 2x5 mosaic of top 10 test errors
│       ├── multi_seed_robustness.png         # Multi-seed accuracy comparison plot
│       ├── threshold_pr_curve_cat.png        # Critical class PR operating curve
│       ├── top_10_failures.csv               # Top 10 high-loss error metadata
│       └── failure_01.png ... failure_10.png # Individual error diagnostic cards
├── dataset.py                                # ManifestDataset & FastManifestDataset loaders
├── models.py                                 # CustomCNN and TransferClassifier architectures
├── transforms.py                             # Augmentation regimes (None, Moderate, Excessive)
├── train.py                                  # Training engine with Cosine scheduler & early stopping
├── evaluate.py                               # Comprehensive evaluation & slice profiling engine
├── predict.py                                # Production CLI prediction tool
├── baseline_classical.py                     # Logistic Regression + HSV/Moments baseline
├── threshold_intervention.py                 # Post-processing decision threshold optimizer
├── multi_seed_evaluation.py                  # Multi-seed training & ANOVA variance suite
├── generate_failure_gallery.py               # Top-loss failure extractor & visualizer
├── create_manifest.py                        # Dataset manifest indexer & SHA-256 validator
├── tests.py                                  # Automated unit test suite
├── requirements.txt                          # Pinned dependency requirements
├── .gitignore                                # Git ignore rules
└── README.md                                 # Technical documentation

About

End-to-end computer vision pipeline on CIFAR-10 with Custom CNN & MobileNetV3 backbones, data augmentation ablations, slice-based auditing, and threshold optimization.

Topics

Resources

Stars

4 stars

Watchers

0 watching

Forks

Contributors

Languages