ÃÛÌÒÊÓƵ

Using Generative AI to Find Weaknesses in Aerial-Image Detection Models

Diagnosing Aerial-View Object Detectors with Foundational Image Generative Models

Aerial object-detection models help identify buildings, vehicles, infrastructure, and other objects in satellite and drone images. These systems support important work in disaster response, national security, and urban planning.

However, even a model that performs well overall can struggle under certain conditions. For example, it may have difficulty identifying objects in dense city shadows or at industrial sites during winter. Because these situations may be rare in training datasets, the weaknesses can be hidden by strong average performance scores.

A new research framework uses generative AI to uncover these hidden weaknesses before a model is deployed.

The researchers use generative image models to create aerial scenes representing different environments—including urban, industrial, desert, forest, and residential areas—during each season. They then test object-detection models on these controlled variations to determine exactly where performance declines.

The process involves three main steps:

1. Generate targeted aerial scenes: Create images showing specific combinations of environments and seasons, such as an industrial area during winter or a residential neighborhood during summer.
2. Verify the generated images: Use multimodal AI models to confirm that each image contains the correct objects and environmental conditions.
3. Test for weaknesses: Evaluate object-detection models—including Faster R-CNN, YOLOv8, and ViTDet—to identify the conditions in which each model struggles.

The researchers evaluated the framework using three major real-world aerial-image datasets: DOTA-v2, LINZ, and UGRC.

They found that weaknesses identified using synthetic images closely matched the models’ performance problems on real aerial images, particularly in dense urban and industrial environments. The framework revealed which types of real-world images were missing from the training data, allowing researchers to address those gaps directly. By adding a small number of carefully selected real images from the weakest categories, they improved detection performance by as much as 13% AP50 while using roughly ten times fewer additional images than broader data-collection approaches.

Instead of collecting large amounts of new data and hoping it improves the model, researchers can first use generated images to identify specific weaknesses. They can then collect a much smaller, more focused set of real-world images to address those problems.

This approach could make aerial object-detection systems more reliable while reducing the cost and time required to improve them.

This research was conducted through a collaboration among:

• :, , , , , and
• : and
• :