SENSE

This SENSE team has created a report that defines the platform's requirements by surveying the state of the art across its four-stage pipeline: perceptual decomposition, semantic/structural understanding, multimodal fusion, and sensory mapping. For the decomposition stage, the team evaluated leading music source separation models (SCNet, BS-RoFormer, Moises-Light) and the newer prompt-based SAM-Audio approach, comparing them by signal-to-distortion ratio to determine how best to isolate vocals, drums, bass, and other instruments from a mix.

​

The semantic layer review covers speech recognition (Whisper, Omnilingual ASR, Samba-ASR) for extracting lyrics, beat and rhythm tracking (BEAST, multi-resolution and dual-path Transformer architectures) for tempo and structure, chord recognition (ChordFormer) for harmonic content, and automatic music transcription tools (Omnizart and lightweight polyphonic models) for note-level detail, all feeding into how a song's musical and emotional character can be captured.

The report closes with an early look at haptic translation methods (Sound2Hap, HapticGen, HapticCap, HapticLLaMA) that map this extracted musical information onto vibrotactile stimuli,. The multimodal fusion module, which will combine these signals into a unified, personalized output, is flagged as a topic for the next report.