AVTime-RL

AVTime-RL-BiTSC-GRPO-v1 is a merged, bf16 milestone for time-aware audio-video understanding. It starts from Qwen/Qwen3-Omni-30B-A3B-Instruct, uses one-round AVTime/LongVALE supervision, and is then optimized with multi-task GRPO for timestamped dense video captioning, temporal video grounding, and segment captioning.

Milestone identity

Field	Value
Release	AVTime-RL-BiTSC-GRPO-v1
Checkpoint	merged checkpoint 60
Precision	bfloat16
Serialization	15 safetensors shards, about 66 GB total
Architecture	Qwen3OmniMoeForConditionalGeneration
Release date	2026-08-11

This is a fully merged checkpoint; no LoRA adapter needs to be applied at inference time. Use a recent transformers or vLLM build that supports Qwen3-Omni MoE models and provide both video and audio when available.

Training

The GRPO mixture contains 12,484 examples: 70.0% dense video captioning (DVC), 15.0% temporal video grounding (TVG), and 15.0% segment captioning (SC). It combines 7,239 LongVALE DVC examples, 1,500 AVTime DVC examples, 1,872 LongVALE TVG examples, and 1,873 LongVALE SC examples.

The main DVC reward is BiTSC (bidirectional time-semantic consistency): a local RLAIF judge matches captions to events, while deterministic code measures the resulting temporal overlap in both Time-to-Event and Event-to-Time directions. TVG uses official-format parsing plus temporal IoU and boundary proximity; SC uses deterministic lexical coverage and length calibration.

LongVALE evaluation

The following results use the LongVALE test split (1,171 videos and 13,867 events) and the official metric implementation.

Task	Metric	Score
Omni-TVG	R@0.3	26.13
Omni-TVG	R@0.5	16.85
Omni-TVG	R@0.7	8.71
Omni-TVG	mIoU	17.76
Omni-SC	BLEU-4	7.79
Omni-SC	ROUGE-L	24.84
Omni-SC	CIDEr	31.28
Omni-SC	METEOR	13.20

On the clean 512-video DVC evaluation using the historical one-round generation protocol, the model achieves SODA-C 5.057, METEOR 7.033, and CIDEr 12.156.

For TVG, evaluation used constrained decoding to enforce LongVALE's official From XX to YY response syntax. Fair comparisons should use the same output constraint for every compared model; this checkpoint should not be interpreted as a claim about unconstrained free-form timestamp formatting.

Limitations and use

This is a research checkpoint, not a safety-validated production system. Its timestamps and captions can be inaccurate, especially for subtle audio events, very long videos, or videos outside the training distribution. The checkpoint does not include any LongVALE or AVTime media or annotations. Users must comply with the terms of the upstream base model and any input datasets.