SkyFind: A Large-Scale Benchmark Unveiling Referring Expression Comprehension for UAV

🌟 Highlights

• New Task Definition: We formally introduce UAV-based REC, focusing on human–UAV interaction via natural language.
• Large-Scale Dataset: We release SkyFind, a dataset with over 1M target–expression pairs across diverse UAV scenarios.
• Baseline Method: We propose AerialREC, a two-step localization framework tailored for aerial scenarios, effectively addressing the unique difficulties of UAV-based REC.

---

🚀 Motivation

Unmanned Aerial Vehicles (UAV) are increasingly used in surveillance, rescue, and delivery tasks. For effective collaboration, UAV need to understand human intentions and objectives, which is essential for subsequent actions. Since language is the most direct and natural medium through which humans convey their intentions, it becomes a critical source of information for UAV to understand task objectives. This motivates the study of UAV-based Referring Expression Comprehension (UAV-based REC), which connects linguistic expressions with specific targets within the visual scene, enabling UAV to interpret human intent and perform subsequent tasks.

---

⚡ Unique Challenges of UAV-based REC

Compared with general REC (RefCOCO, RefCOCO+, RefCOCOg), UAV-based REC introduces three distinct challenges:

1. Abundant Background Interference The wide field of view in UAV imagery often leads to scenes with rich semantic content, containing numerous non-target entities that resemble the target. Such distracting entities in the background increase the difficulty of localization.

2. Small Target Size UAV imagery typically follows a ”large scene, small object” pattern, where the target occupies only a small fraction of the frame. Such target often has blurred boundary and weak texture, which increase the difficulty of recognition.

3. Complex Referring Relations To identify a target in cluttered UAV imagery, referring expressions often incorporate fine-grained details, making them longer, characterized by more intricate reference relations, and consequently more difficult to comprehend.

---

📊 SkyFind Dataset

We construct SkyFind, the first large-scale dataset for UAV-based REC.

• Scale: 1,015,638 target–expression pairs, 35,599 images, 352,910 objects.
• Split: 331,364 (train), 5,000 (val), 16,546 (test). Test set sourced from distinct maritime datasets to ensure distribution shift.
• Diversity: Each original expression in the training set is augmented with two additional variants generated by GPT-4, expanding linguistic coverage with varied vocabulary and sentence structures. This triples the data volume (331,364 → 331,364 × 3).

---

📂 Resources

We are excited to announce that the SkyFind dataset is now fully open-sourced! You can access the data conveniently through HuggingFace or download the raw files manually.

🤗 HuggingFace (Recommended)

The dataset is hosted on Hugging Face as partitioned Parquet files for efficient loading. You can easily load it using the datasets library.

First, install the required libraries:

pip install datasets huggingface_hub pyarrow

Then, load the dataset in Python:

from datasets import load_dataset

# Load specific splits
train_data = load_dataset("kunyu241/SkyFind", split="train")
val_data = load_dataset("kunyu241/SkyFind", split="validation")
test_data = load_dataset("kunyu241/SkyFind", split="test")
train_aug_data = load_dataset("kunyu241/SkyFind", split="train_aug")

For more details, visit our HuggingFace Dataset Page.

📥 Direct Download (Raw Resources)

If you prefer to work with the raw images and JSON annotations locally, you can download them via the links below:

• Baidu Netdisk: Download Link (Extraction Code: vcnw)
• Google Drive: Download Link

Download Contents The released dataset consists of the following 5 files:

• images.zip: Contains all UAV images for the train, val, and test sets.
• Train.json: Annotations for the original training set.
• Val.json: Annotations for the validation set.
• Test.json: Annotations for the test set.
• Train_aug.json: Annotations for the GPT-4 augmented training set (includes the tripled linguistic variants).

---

🔧 Data Patch

During dataset checking, we identified a small number of bad images in the released files. We provide a patch file to replace them.

• Baidu Netdisk: bad_image.zip (Extraction Code: qrv8)

Please download bad_image.zip and replace the corresponding images in the original images.zip.

📄 License & Terms of Use

The SkyFind dataset is released under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

By downloading and using this dataset, you agree to the following terms:

1. Academic Use Only: The dataset is strictly for non-commercial, academic research purposes.
2. No Commercial Use: You may not use the material for commercial purposes without explicit written permission from the authors.
3. No Derivatives: You may not distribute modified, transformed, or built-upon versions of this dataset without prior authorization.
4. Citation: If you use this dataset in your research, you must cite our TPAMI paper.

Disclaimer: The dataset is provided "AS IS", without warranty of any kind, express or implied. The authors and their affiliated institutions assume no responsibility for any consequences, damages, or liabilities arising from the use of this dataset.

---