Methodology

Visual AI Lab compares AI image and video generation models under the same conditions, records reproducibility, and applies final human review to determine which models work in real Japan-market production.

Terminology

The aggregation units used across the site are defined as follows.

Term	Definition
Run	One generation execution for one model
Task	One input / prompt / production-purpose combination
3-Run task	A task generated three times under the same model and conditions
Comparison case	One task compared across multiple models
Theme	Subject unit such as ramen, uchiwa fan, or product rotation
Category	Capability, use case or Japan-market fit classification
Article	Editorial content that explains comparison results

How models are selected

We compare T2I, I2I, T2V and I2V models that are practically accessible via public API or official/representative platforms.
Only models with "ACTIVE" status appear in comparison pages and rankings.

Same-condition comparison

We generate outputs from each model with the same prompt, same key image and same settings.
For images, we align resolution, aspect ratio and quality parameters as closely as possible.
For video, we record duration, fps, resolution, inference steps and guidance scale.
Cost is estimated from each model's official pricing or measured API call, converted to USD per image or per second.

Evaluation types

Every comparison page shows its evaluation type.

Baseline / Common Prompt: models are compared on a shared prompt and shared settings to measure baseline capabilities.
Model-Optimized: each model uses recommended prompts or settings for that model. These are not labeled as a "fair test".
Production Workflow: includes post-processing, rerolls, upscaling, ComfyUI or other production steps.

Scoring criteria

Outputs are scored on prompt fidelity, subject consistency, motion quality, temporal stability and background integrity on a 5-point scale.
Commercial usability is reported separately and is not included in the overall quality score.
Final verdicts are classified as "first_run_usable", "reroll_needed" or "unusable".
Failure tags (face collapse, hand collapse, object morphing, ignored camera direction, background collapse, temporal flicker, prompt mismatch, etc.) visualize failure patterns.

Human review and automation

Automation is fine for production and first-pass sorting, but the following are always reviewed by humans:

Output images and video
Scores and failure tags
Article conclusions
Model names, pricing and versions
Legal/commercial terms
Sponsor disclosures
Comparison template consistency

Published evaluation scores come from manual evaluation in our admin tool. Articles containing images or video are not published until human review is complete.

Sample transparency

We disclose the number of prompts, generations per prompt and how representative outputs were selected.
We record failed outputs, API route, model version and generation date.
Resolution, duration, quality setting and reroll count are shown.
When only the best output is shown, the selection criteria are stated.

Reliability and calculation rules

Averages, rankings and win rates on this site are calculated as follows.

One prompt, one latest Run: For each model we use the latest Run for each prompt. If multiple attempts exist, the latest one is treated as the representative Run.
Average score: Quality scores are averaged per model across all scored prompts (commercial_usability is excluded).
Exclusion rule: Runs that failed, were rejected by content policy, are marked unusable, or have no quality score are excluded from averages and win rates.
Common-test ranking: The common-test ranking only uses prompts where every model has a scorable Run, ensuring a fair head-to-head comparison.
Win count: The model with the highest average score on a given prompt earns one win.
Cost efficiency: Average score divided by estimated cost per image or per 4-second clip. If pricing is unknown we use official pricing or measured API cost as an estimate.
Sample size: Category and criteria rankings with few prompts are labeled "Initial evaluation" and treated as reference values, not statistically settled conclusions.

Japan suitability for images vs video

Image and video models are checked against different Japan-market axes.

Image models: One frame is evaluated for Japanese text accuracy, Japanese people, Japanese products/contexts, background integrity and logo/trademark contamination. Temporal collapse does not apply to still images.
Video models: Standard 4-second clips (or longer clips where noted) are evaluated for temporal consistency, subject maintenance, Japanese text persistence and local-scene understanding. Motion, camera control and lip sync are also considered where relevant.
commercial_usability: A separate 1–5 score for whether the output can be used directly as an ad, EC, SNS or store asset. Rights, credits and terms are stated in the article.

Handling ambiguous categories

Categories such as "Japanese animals", "Japanese scenery" or "Japanese people" are vague and are therefore redefined as follows.

Concrete subjects: "Japanese animals" prompts specify species such as Shiba Inu, tanuki or Northern boobook, including breed, environment and shooting conditions.
Commercial scene connection: Prompts reflect real production scenes such as EC, ads, SNS, store PR and real estate rather than relying on a broad category label.
Cultural specifics: Japanese festivals, traditional clothing, urban landscapes and food expression are made into individual themes that foreign models are less likely to have learned.
Initial evaluation label: Categories with few themes are labeled "Initial evaluation" so readers know the comparison base is limited.

Sponsor and advertising policy

Sponsors, article production fees, free credits and affiliate links are clearly marked so readers can recognize them.
We do not change scores or rankings in exchange for advertising fees.
When a sponsor is being compared, we disclose it and apply the same generation conditions to non-sponsored competitors.
Exclusion and failure conditions are defined before testing.

Updates and corrections

Articles and comparison pages show the last updated date.
Pricing and terms show a confirmation date.
Corrections are recorded in a changelog.
Old comparisons are labeled as "old version" and articles that are no longer updated may be archived or set to noindex.

This site is not a rigorous academic benchmark; it is a practical comparison review based on production use. Published content may change as models are updated or prices change. Please check each provider's official documentation for the latest information.