Multi-Task Multi-Frame Visual Piano Transcription
Abstract
Audio-based piano transcription performs well on onset, pitch, and velocity, but the sustain pedal lets sound persist long after key release, so audio systems predict pedal-extended offsets rather than physical key release. Yet existing Visual Piano Transcription (VPT) systems focus on onset detection from short video windows, offset accuracy lags onset by a wide margin, and note-level velocity has not been reported. To address these gaps, we present V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center. Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy; longer temporal context yields further improvements. V2N sets new state-of-the-art results on PianoVAM and R3.
Community
We introduce V2N (Video to Notes), the first complete visual piano transcription (VPT) system. From top-view video alone, with no audio, it predicts onset, offset, key hold, and velocity for every note.
Why video? When the sustain pedal is down, a note keeps sounding long after the key lifts, so audio systems predict pedal-extended offsets rather than the physical key release. The camera sees the key itself, including how hard it is struck. Prior visual systems, though, focus on onset from short windows, so offset accuracy lags onset by a wide margin and note-level velocity has not been reported. V2N closes both gaps with a shared temporal backbone and task-specific heads for onset, offset, key hold, and velocity, trained with per-frame supervision instead of only at the window center.
Ablations show multi-task supervision enables offset and velocity while improving onset, and longer temporal context helps further. V2N sets new state of the art on PianoVAM and R3. To appear at ISMIR 2026.
🎬 Demo: https://huggingface.co/spaces/PianoVAM/V2N
📄 Paper: https://arxiv.org/abs/2608.03419
💻 Code: https://github.com/yonghyunk1m/V2N
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- FATE: Frame-Level Audio-Visual Temporal Embedding (2026)
- Snapping Matters: Context-Aware Onset Refinement for Automatic Music Transcription (2026)
- SKY-Piano: A Multimodal Piano Performance Dataset (2026)
- Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space (2026)
- P-MUSE: Prompt-MIDI-Optional Model for Unified Instrumental Music Synthesis and Editing (2026)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (2026)
- OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2608.03419 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper