A significant challenge in selecting and refining artificial intelligence models, particularly for vision AI workloads, has been the inherent bias within evaluation datasets. Property Finder, a prominent digital real estate application serving the Middle East and North Africa (MENA) region, encountered this issue when aiming to migrate its vision AI tasks to a more cost-effective solution on Amazon Bedrock. The company discovered that a staggering 96.8 percent of their evaluation labels were derived directly from the predictions of their existing model, rather than independent human verification. This skewed data systematically favored the incumbent model, creating an artificial underperformance for any alternative, and masked underlying errors.
The Chenab Times has learned that this evaluation bias was a critical blocker for Property Finder, which processes tens of thousands of property images daily. Their system evaluates 11 quality attributes per image, including watermark detection, room classification, and furnishing status, for thousands of new property listings averaging 15 images each, all requiring quality assessment before publication.
Identifying and Addressing Evaluation Bias
The core problem was not the availability of better models, but the flawed methodology of evaluation itself. Property Finder’s engineering team, in collaboration with the AWS Generative AI Innovation Center, developed a strategy to identify and rectify this bias. They found that inherited labels contained not only errors but also systematic biases that prevented objective model comparisons. For instance, watermark text extraction had character-level mistakes that were wrongly marked as correct, and photo classification often assigned single labels to images depicting multiple room types.
To build a reliable ‘ground truth,’ Property Finder moved away from relying on existing model outputs. Instead, they initiated a process of fresh, independent annotations by human domain experts. This involved hand-annotating 356 UAE images and a separate set of Egypt images, carefully selected from unlabeled production listings across representative industry segments. A strategy and analytics expert with specialized knowledge of real estate listing quality standards performed these annotations, with the AWS Generative AI Innovation Center team providing systematic quality reviews and clarification sessions to ensure consistency.
A crucial adjustment was the transition from single-label to multi-label classification. Real estate images frequently display multiple rooms through doorways or overlap in characteristics. Allowing for multiple labels accurately reflects these scenarios, measuring what a model gets right rather than penalizing it for reasonable secondary predictions. The team also refined their metrics selection, utilizing Micro F1 as the primary metric for its equal weighting of predictions and alignment with the business goal of overall label correctness. Macro F1 and per-label precision and recall were also employed to identify specific failure modes and performance on rare attributes.
Prompt Optimization for Enhanced Accuracy
With an unbiased evaluation framework in place, Property Finder and AWS focused on prompt optimization, treating the process akin to reinforcement learning. Each prompt version acted as an ‘agent,’ with the Micro F1 score serving as the ‘reward signal’ and error analysis guiding improvements. Initially, prompts used generic classification instructions. Property Finder’s initial proprietary model A achieved a Micro F1 score of 0.55 on a development set using a tuned prompt (v22).
Through iterative refinement, incorporating structured output formatting and attribute-specific definitions, significant gains were achieved. Error analysis revealed that models consistently misclassified rooms visible through doorways in early single-label versions. Explicit instructions to prioritize the room occupying the majority of the frame, coupled with the switch to multi-label output, addressed this. Regional customizations, such as accounting for decorative Arabic text overlays in some regions that were not watermarks, further reduced false positives.
By version v21, the prompt adopted multi-label output for photo classification and room category, becoming a shared starting point for all models. Using this prompt, Meta’s Llama 4 Maverick 17B Instruct model achieved a Micro F1 of 0.75. Further tuning, including the use of the automated prompt optimization framework MIPROv2 from the DSPy library, pushed this score higher. The final prompt, v27_PF, incorporated Property Finder’s specific rules for renders, bordered images, and corridors, alongside the Arabic text rules and output format instructions. This final iteration resulted in Llama 4 Maverick 17B Instruct scoring a Micro F1 of 0.80 on the evaluation set, a substantial improvement over the proprietary model A’s score of 0.56.
Results: Significant Cost and Accuracy Improvements
The collaboration led to Property Finder selecting Llama 4 Maverick 17B Instruct for their production workload due to its superior accuracy across 11 attributes and cost efficiency. The model achieved exact-match accuracy of 90 percent or higher on six of the nine single-label attributes evaluated, and matched or exceeded the proprietary model A on all 11. On photo classification, both models performed similarly.
Crucially, the move to Llama 4 Maverick 17B Instruct on Amazon Bedrock resulted in an estimated cost of $2.35 per 1,000 images, a remarkable 86 percent reduction compared to their previous setup. This cost efficiency is attributed to the use of an open-weight model and an optimized single-call pipeline. Furthermore, Property Finder observed a 40-60 percent reduction in inference latency, enhancing the speed of their image processing operations.
The success highlights the importance of unbiased evaluation and tailored prompt engineering for unlocking the full potential of AI models. Property Finder’s experience demonstrates that by addressing evaluation bias and refining prompts, organizations can achieve both higher accuracy and significant cost savings when leveraging cloud-based AI platforms like Amazon Bedrock.
The Chenab Times News Desk