Learning More From Less: The Rise of Active Learning

There's a phrase that keeps circulating in AI circles: "Data is the new oil." But the truth is, unlike oil, data isn't always abundant and cheap. In fields like medical imaging, remote sensing, autonomous systems, and industrial quality control, producing labeled data requires expert time, money, and sometimes months of effort. A radiologist labeling thousands of MRI scans, an agricultural engineer marking weeds one by one in drone imagery — all of it is expensive.

This is exactly where Active Learning enters the picture. The idea is actually quite human: a good teacher doesn't ask a student random questions — they focus on the topics the student struggles with most, where uncertainty is highest. Active Learning does the same thing with a model: it picks the examples the model is "most confused about" and has only those labeled by an expert. The result: similar or better performance with far less labeled data — often 50–80% less than random sampling.

It's no coincidence that Active Learning is resurfacing today in large language model (LLM) data selection and fine-tuning pipelines, in medical image analysis, and in autonomous vehicle development — labeling cost is still one of the biggest bottlenecks in AI projects.

How does Active Learning work?

Formal definition: the quintuple model

In the literature, an active learning system is typically modeled as a quintuple: (G, Q, S, T, U)

• G — The supervised classifier trained on the labeled training set T.
• Q — The query function that selects the most "informative" samples from the unlabeled pool U.
• S — The supervisor / oracle who can assign the true class label to any selected sample.
• T — The labeled training set; it starts as a small seed set and grows as the loop progresses.
• U — The unlabeled, typically large pool of samples.

These five components feed into a closed loop: classifier G evaluates the samples in pool U; query function Q uses that evaluation to select the most valuable samples; supervisor S labels them; the newly labeled samples are added to T; G is retrained. This loop continues until a predefined number of iterations is reached, or a stopping criterion is satisfied.

Algorithm 1 — General Active Learning procedure

1. Train classifier G with the initial training set T
2. Classify the unlabeled samples in pool U
Repeat:
    3. Query a set of samples X from pool U using query function Q
    4. Supervisor S assigns true labels to the queried samples X
    5. Add the newly labeled samples to the training set T
    6. Retrain the classifier
Until a stopping criterion is satisfied

The Active Learning loop — the cyclical flow between the G, Q, S, T, U quintuple.

This pseudocode is short, but it forms the skeleton of every AL method — DeepAL, BALD, CEAL, and even the AdaRisk-Agent and UKM-AL studies I share below are, at their core, variations of this same five-part loop with a different Q (query function) and sometimes a different T update strategy. The real difference between methods is hidden in step 3: the function that decides which sample counts as "informative."

Put more intuitively, the loop works like this:

• A model is trained on a small, labeled initial dataset.
• The model makes predictions for every sample in the large unlabeled pool.
• A query strategy selects the most "informative" samples.
• Those samples are sent to an expert (oracle) to be labeled.
• The model is retrained and the loop repeats — until the labeling budget runs out.

The critical question here is: which sample is "informative"? This is where methods diverge.

Main Active Learning methods

1. Uncertainty sampling

Pick whichever sample the model is least confident about. The three most common criteria:

• Least confidence: Samples where the model's probability for the most likely class is low are selected.
• Margin sampling: Samples where the probability gap (margin) between the top two most likely classes is small are selected.
• Entropy sampling: Samples where the entropy of the probability distribution is highest are selected:

H(x) = − Σᵢ p(yᵢ|x) · log p(yᵢ|x)

The higher the entropy, the greater the model's "uncertainty" about that sample.

2. Query-by-Committee (QBC)

Instead of a single model, multiple models (a committee) are trained. The more the committee members disagree on a sample, the more valuable that sample is. Disagreement is usually measured with vote entropy or KL-divergence:

D_KL(Pᵢ ‖ P_consensus) = Σⱼ Pᵢ(yⱼ) · log [Pᵢ(yⱼ) / P_consensus(yⱼ)]

3. Expected model change

This estimates how large an update (gradient magnitude) labeling a given sample would produce in the model's parameters; the sample expected to cause the largest change is selected.

4. Diversity-based methods (diversity-based / core-set)

Looking only at uncertainty can sometimes lead to selecting similar, closely clustered samples ("redundancy"). The core-set approach aims to have the selected samples represent the data space as well as possible — typically formulated as a k-center problem:

min_S max_{x∈X} min_{s∈S} d(x, s)

This idea's roots actually predate deep learning by a long way, going back to classic SVM-based batch-mode AL work. Two techniques stand out in that literature: angle-based diversity (ABD), which uses cosine angle distance in kernel space to filter out samples that are too close (small angle) to each other; and clustering-based diversity (CBD), which first clusters uncertain samples and then picks a representative from each cluster to reduce redundancy. In remote sensing (hyperspectral/RS) image classification, these two approaches are typically combined with an uncertainty criterion (such as margin sampling) in a two-stage pipeline: first a broad candidate pool is narrowed down by uncertainty, then diversity is applied on top of that pool — the "vote entropy + kernel k-means" design in the UKM-AL study I mention below is a direct continuation of this tradition.

Comparison of uncertainty sampling versus diversity-aware batch selection.

5. Hybrid / Bayesian approaches (BALD and its variants)

Bayesian Active Learning by Disagreement (BALD) maximizes the mutual information between uncertainty and model uncertainty:

I(y; θ | x, D) = H[y | x, D] − E_{p(θ|D)}[H[y | x, θ]]

This formula tries to distinguish between "is the model's prediction uncertain" and "does this uncertainty come from a genuine lack of knowledge, or from natural noise" — a distinction that is critical especially in high-stakes fields like medical imaging.

Meeting deep learning: Deep Active Learning (DeepAL)

Classic Active Learning was mostly developed with "shallow" models such as SVMs or k-NN. The idea of combining deep learning's ability to automatically represent high-dimensional data (images, text, audio) with Active Learning's labeling efficiency is referred to in the literature as DeepAL (Deep Active Learning). But this combination is not as straightforward as it looks; it runs into three fundamental challenges:

• Unreliable model uncertainty: The softmax output of deep networks tends to be "overconfident," so when raw softmax probabilities are used directly as an uncertainty measure, performance can end up worse than random sampling.
• Insufficient labeled data: Classic AL proceeds by querying one sample at a time; but since deep networks are data-hungry, one-at-a-time querying doesn't work in practice. This is why DeepAL selects samples in batches.
• Pipeline mismatch: Classic AL is usually built on a fixed feature representation, whereas in deep learning, feature extraction and classifier training are optimized jointly. Treating the two as independent problems can cause the model to diverge.

Batch querying: Batch Mode DeepAL (BMDAL)

Since retraining after every single sample is computationally unsustainable for deep models, DeepAL selects a batch of samples at once. The problem is that selecting purely by informativeness (uncertainty) can lead to picking samples that are too similar to each other, and therefore "redundant." This is why modern methods account for both informativeness and diversity at the same time:

• BALD (Bayesian Active Learning by Disagreement): Selects samples that maximize the mutual information between model parameters and predictions — capturing both "the model is uncertain" and "this uncertainty stems from a genuine lack of knowledge."
• BatchBALD: Extends BALD from single-sample to batch querying by computing the joint mutual information across samples — preventing very similar samples from being selected into the same batch.
• Core-set approach: Aims for the selected labeled subset to represent the distribution of the entire dataset as well as possible; typically solved as a k-center optimization problem.
• BADGE (Batch Active learning by Diverse Gradient Embeddings): Implicitly balances uncertainty and diversity through samples' representations in a "hypothetical gradient space" — without needing manual hyperparameter tuning.
• VAAL (Variational Adversarial Active Learning) and its variants (TA-VAAL, ARAL): Learns the latent representation of labeled and unlabeled data using an adversarial network, then selects the most "informative" unlabeled samples based on that representation.

Batch Mode DeepAL: uncertainty scoring, candidate pool, diversity step, and the oracle loop.

"Expanding" the labeled data

Another important dimension of DeepAL, alongside improving the query strategy, is folding the existing unlabeled data into training. For example, the CEAL (Cost-Effective Active Learning) approach expands the training set without human intervention by assigning pseudo-labels to samples the model predicts with high confidence; GAN-based approaches bring in generative models for data augmentation. These kinds of strategies can sometimes yield larger performance gains than changing the query strategy itself.

Where is it used?

DeepAL's application areas are quite broad: image classification and object detection, medical image segmentation (such as lung nodule, lymph node, and finger bone segmentation), LiDAR-based 3D object detection in autonomous driving, drone imagery for wildlife counting, hyperspectral remote sensing classification, text classification and machine translation, EEG/ECG signal analysis, and speech emotion recognition, among many others. The common thread is always the same: expert labeling cost is high, and unlabeled data is abundant.

Why does it matter even more now?

With the rise of foundation models and LLMs, the assumption that "more data = a better model" has started to be questioned. Deciding which examples to show a human during fine-tuning and RLHF is, in essence, an Active Learning problem. Similarly, labeling cost remains high in areas such as satellite imagery, medical screening, and industrial anomaly detection — which is turning Active Learning from an academic curiosity into a practical necessity.

Case study: AdaRisk-Agent — LLM-assisted adaptive risk calibration for drone-based weed detection

To ground the theoretical framework described above, I want to share an example from my own published work: "AdaRisk-Agent: LLM-Orchestrated Adaptive Risk Calibration for Cost-Sensitive Active Learning in UAV Weed Detection" (Drones, 2026).

Problem: In precision agriculture, when detecting weeds with a drone (UAV), the two types of error are never equally costly: a missed weed patch (false negative) leads to yield loss, while a false alarm (false positive) only causes an unnecessary spot treatment. Cost-sensitive active learning (cAL) is used to address this asymmetry — a risk coefficient (r⁺) penalizes false negatives more heavily, shifting the model's decision boundary toward the "more cautious" side. The problem is that the right value for this coefficient varies by scene, and can't be determined in advance without a domain expert.

Method: In this study, I proposed AdaRisk-Agent, an agent that automatically tunes r⁺ at every active learning iteration. The system extracts a 7-dimensional context vector containing spectral uncertainty, a class separation index, budget consumption, and the false-negative rate (FNR) trend; it sends this vector to an LLM (Claude Haiku), which selects an r⁺ value from {1, 3, 5, 7}. For comparison, a deterministic rule-based version (AdaRisk-Rule) implementing the same threshold logic in code was also developed.

The AdaRisk-Agent architecture: context vector, LLM calibrator, r⁺ selection, and the cost-sensitive AL loop.

Data and experiments: The framework was tested on four UAV scenes drawn from two different public datasets (corn fields in Germany — WeedsGalore; rice paddies in Vietnam — WeedyRice), covering two different crop types, two sensor configurations, and weed density ranging from 3% to 30%.

Findings:

• Adaptive calibration reduced the false-negative rate by up to 80% across all scenes compared to fixed-cost (symmetric-cost) baselines.
• The deterministic rule-based version (AdaRisk-Rule), with no prior knowledge of scene difficulty, outperformed the fixed-policy reference method (r⁺=7) on two of the four scenes; it reduced the false-negative rate by 50% on the spectrally most challenging scene.
• The LLM-based calibrator and the rule-based version reached nearly identical r⁺ decisions independently of each other — serving as cross-validation that the thresholds are grounded in principled logic rather than arbitrary choices.
• Feature ablation analysis showed that the budget consumption signal was the most critical context feature for calibration, while signals such as the class separation index remained relatively less decisive for this rule set.
• The LLM's main contribution wasn't raw performance so much as the fact that every decision comes with an auditable, natural-language rationale — making the decision chain transparent for agricultural experts and regulators.

Takeaway: This study shows that Active Learning isn't limited to just the question of "which sample should we label" — the question of "how aggressively should we prioritize this sample" can also be automated dynamically and explainably. Integrating LLMs into the AL loop as a hyperparameter calibrator suggests a new design pattern, particularly for agricultural decision-support systems that require field expertise.

Case study 2: UKM-AL — climate-aware Geo-AI for label-efficient agricultural land-cover mapping

The second example is a different study that brings together remote sensing and climate data: "A Geo-AI Framework for Label-Efficient Agricultural Land-Cover Mapping with Climate-Aware Active Learning" (Land, 2026).

Problem: Global agricultural land-cover mapping is critical for food security and climate adaptation planning, but obtaining reliable reference labels — especially over fragmented land structures — is expensive because it requires fieldwork and expert interpretation. The question is: how much can Active Learning reduce the labeling need in crop/non-crop mapping, and does multi-source environmental data actually make a difference in this process?

Method — UKM-AL (Uncertainty-guided Kernel k-Means Active Learning): A two-stage query strategy was proposed:

1. The most uncertain samples are gathered into a candidate pool using vote entropy, which measures disagreement among the trees of a Random Forest ensemble (candidate pool size = α × batch_size, α=4).
2. This candidate pool is then clustered with kernel k-means, and the most uncertain sample from each cluster is selected — so that both informativeness and diversity are accounted for at the same time.

The UKM-AL two-stage query strategy: narrowing with vote entropy, then diversity via kernel k-means.

The features combine Sentinel-2 (optical), Sentinel-1 (SAR), ERA5 (climate), and topographic (DEM/slope) data — as a 12-month time series, drawn from the CropHarvest dataset (113,892 samples, 27 regional sub-collections, 6 continents).

Experimental design: On a balanced candidate pool of 10,000 samples, three query strategies (random sampling, vote entropy only, vote entropy + kernel k-means) were compared using a fixed Random Forest classifier; five different feature-group combinations (optical only → all features) and a regional transfer test on agricultural data in China were also carried out — all repeated across 10 different random seeds.

Findings:

• Uncertainty-guided active learning (vote entropy + kernel k-means) reached nearly the same performance (macro F1 = 0.827) as a fully supervised reference model trained on all 10,000 samples, using only 3,950 labels (less than 40% of the pool).
• Uncertainty-based strategies performed statistically significantly better than random sampling (Wilcoxon test, p=0.001); however, the contribution of the kernel k-means diversity step on top of vote entropy was limited — its main benefit wasn't final accuracy so much as lower variance across repeated runs (more stable learning).
• In the feature-group analysis, the largest performance jump came from adding climate variables (ERA5 temperature, precipitation, evapotranspiration) (macro F1: 0.780 → 0.822, +4.2 points) — the contribution of SAR data, on the other hand, was surprisingly limited (+0.3 points), because the monthly aggregated SAR features in CropHarvest largely smooth out the fine-grained temporal backscatter dynamics that are SAR's real strength.
• On the held-out regional test set in China, the model learned from the global pool showed competitive performance, but its error pattern was the exact opposite of the main benchmark — more false positives were seen in the non-crop class. This points to the global training pool not adequately representing China-specific non-crop land-cover types (dense urban fabric, certain shrub types) — concrete evidence that environmental context is not homogeneous under regional transfer.

Takeaway: This study questions the assumption in Active Learning that "diversity always helps": in a pool with global diversity, the uncertainty criterion may already spread samples sufficiently across the feature space, so the contribution of a clustering-based diversity layer can remain marginal under these conditions. At the same time, it shows that the contribution of climate context (like ERA5) to label efficiency, on top of the pure remote-sensing signal, can be more decisive than an "intuitively useful" data source like SAR — which suggests that feature selection in Geo-AI systems is just as critical as the query strategy.

Conclusion

Active Learning embraces the philosophy of "smarter data" rather than "more data." This family of methods — from uncertainty sampling to Bayesian approaches, from classic AL to DeepAL integrated with deep learning — is an invisible but critical component of modern AI systems, especially in fields where labeling cost is high. The field is still maturing — inconsistent results across different studies also point to the need for a standard evaluation protocol. Still, data will never always be abundant; systems that know how to ask the right question at the right time will keep making a difference with constrained resources.

This article offers a general overview of Active Learning methods. If you have questions or would like to discuss any of it further, please get in touch.