MuseGran: Multi-Granularity Conditioning with Music-Aware Position Encoding for Long-Form Music Generation

MuseGran is a DiT-based latent diffusion system for long-form music generation (up to ~180s, 44.1kHz stereo) with multi-granularity conditioning at global, segment, and frame scales. Key components include: (1) granularity-matched conditioning pathways for text, key, BPM, beat, and per-segment structure; (2) MusicRoPE, a hierarchical position encoding that separately encodes bar, beat, and sequence positions and adds a periodic bar prior for phrase-level recurrence; and (3) SegAlign, a segment-level representation alignment strategy that regularizes musicality while preserving condition semantics. A fine-tuned LLM (StructPlanner) converts free-form user prompts into structured generation specifications.

Music Demos

1. Overall Comparison

Prompt	MusicGen-Large-3.3B ~47s	MuseControlLite (melody) ~47s	Stable Audio Open (fine-tuned on our data) ~47s	MuseGran (Ours) ~120s
Country-pop relaxing song with happy mood and acoustic guitar, ideal for young girls going out shopping or dreaming about their love
This is a slow but deliberate jazz song, featuring two guitars (one acoustic, one bass), with a bass drum being used in points to keep timing
Pop solo piano instrumental song. Simple harmony and emotional theme. Makes you feel nostalgic and wanting a cup of warm tea sitting on the couch
An electronic music track which has an upbeat melody, gets increasingly complex and evokes a sense of positive excitement
Bluesy guitar with a slow repetitive rhythm in a smoky room in latin america
Smooth chill electronic music featuring calming flute and techno beats
Rock song starting with guitar line, with a catchy refrain.
A hip-hop repetitive beat featuring a keyboards and synths.
This song starts with a piano and guitar duet going through an emotional phase and then joined with a dramatic electric guitar.

2. MusicRoPE Effect

Same prompt, key, and beat conditioning. Standard RoPE tends to produce more rigid beats compared to MusicRoPE.

Prompt	Standard RoPE	MusicRoPE
Relaxing music played mostly on the piano that can be used while studying, meditating or just relaxing
In this song, a tropic back-beat and a lush piano chord is then accompanied with a salient organ to give way into a joyful melody and a beat.
Alternative pop rock that is appealing to the 2010s audience.
Slow-paced guitar ballad with clean guitar solos interspersed throughout the composition
1990s techno with cliche minor key progression and tacky synthetic saxophone lead

3. Multi-Granularity Control

Each conditioning dimension demonstrated independently: tempo, key, duration, and segment-level structure.

3.1 Beat & Tempo Control

Same prompt at different target tempos. Format: input condition → extracted from generated audio.

Prompt	Case 1	Case 2	Case 3
Peaceful sounding lo-fi, it gives a feeling of security	75 BPM → 75.0	105 BPM → 105.3	140 BPM → 139.5
An upbeat and celebratory folk tune with lively instrumentation and a driving rhythm	75 BPM → 150.0	105 BPM → 105.3	140 BPM → 139.5

* BPM extracted via madmom beat tracker. Octave errors (2× or 0.5×) are a known limitation of beat tracking algorithms and do not indicate incorrect tempo generation.

3.2 Key Control

Same prompt at different target keys. Format: input condition → extracted from generated audio.

Prompt	Case 1	Case 2	Case 3
Calming instrumental played on bass violin and flute that can be put in the background while doing some work or study	E minor → E minor	G minor → G minor	B minor → B minor
Ambient electronic music with layered synth textures and atmospheric pads	D major → D major	F minor → F minor	F major → F major

3.3 Duration Control (Natural Endings)

Same prompt at different target durations. The model produces natural endings rather than abrupt cuts.

Prompt	Case 1	Case 2	Case 3
A vibrant fusion of traditional Middle Eastern and electronic music	~30s	~90s	~180s
A relaxing jazz song, for focusing, that evokes sentimental feeling	~30s	~90s	~180s

3.4 Segment-Level Structural Control

Per-segment dynamics and style descriptions control the musical structure over time.

3.4.1 A/B Comparison: With vs. Without Segment Conditioning

A jazzy piano is playing in the foreground. A bass is doing chromatic and moving lines behind.
C major · 172 BPM · 120s

▸ Segment Conditions · 5 seg

Seg 1 0.0–26.3s pp

Style: Smooth jazz instrumental featuring piano as the lead instrument.
Rhythm: Steady 4/4 time signature with a moderate tempo around 85 BPM.
Instruments: piano.
Mood: relaxed, slightly melancholic.

Seg 2 26.3–41.8s p

Style: Jazz instrumental piece featuring piano in C major.
Rhythm: Steady 4/4 time signature with a moderate tempo around 85 BPM.
Instruments: piano.
Mood: calm, slightly melancholic.

Seg 3 41.8–60.0s mf

Style: Instrumental jazz piece featuring piano in C major.
Rhythm: Steady 4/4 time signature with moderate tempo around 167 BPM.
Instruments: piano.
Mood: calm, slightly melancholic.

Seg 4 60.0–83.1s p

Style: Smooth jazz instrumental featuring piano as lead.
Rhythm: Steady 4/4 time signature with moderate tempo around 176 BPM.
Instruments: piano.
Mood: calm, slightly melancholic.

Seg 5 83.1–120.0s mf

Style: Jazz instrumental piece featuring piano as lead instrument.
Rhythm: Steady 4/4 time signature with moderate tempo around 167 BPM.
Instruments: piano.
Mood: relaxed, engaging.

Without Segment Control

With Segment Control

90s hip hop with a moody synth which gives a very ominous but danceable vibe to it.
D major · 90 BPM · 120s

▸ Segment Conditions (right) · 5 seg

Seg 1 0.0–17.5s pp

Style: Ambient electronic instrumental piece.
Rhythm: No discernible rhythmic pattern.
Instruments: synthesizer, bass guitar.
Mood: calm, sustained tone.

Seg 2 17.5–60.0s p

Style: Electronic downtempo blending beats with subtle hip-hop influences.
Rhythm: Steady 4/4 beat around 90 BPM, relaxed yet engaging groove.
Instruments: synthesizers, drum machine, bassline.
Mood: calmness, introspection.

Seg 3 60.0–80.3s mf

Style: Chillout instrumental blending ambient electronic textures with downtempo beats.
Rhythm: Steady 4/4 beat around 90 BPM.
Instruments: synth pads, soft drums, subtle bassline.
Mood: calmness, smooth textures.

Seg 4 80.3–117.8s mf

Style: Electronic instrumental blending ambient elements with chillout vibes.
Rhythm: Steady 4/4 beat typical of electronic dance music.
Instruments: synthesizers, drum machine beats.
Mood: energy and excitement, driving tempo.

Seg 5 117.8–120.0s pp

Without Segment Control

With Segment Control

3.4.2 Interactive Segment Visualization

Play the audio to see real-time segment tracking with spectrogram overlay. Click on the spectrogram to seek. Segment info updates in real-time during playback.

The song is a melancholic and emotionally rich classical composition that blends piano, strings, and woodwinds to evoke nostalgia and introspection.
A minor · 109 BPM · 139s

A serene and emotionally resonant classical composition that blends calm introspection with moments of grandeur and nostalgic beauty.
G major · 85 BPM · 161s

Synth-y, spacey art-pop track with a catchy beat
G major · 125 BPM · 120s

4. StructPlanner: Prompt → Structure → Music

A fine-tuned LLM converts free-form user prompts into full structured specifications, which are then consumed by the generation system. End-to-end examples below.

Write me a soothing piano piece in 3/4 time

A neoclassical piano piece in 3/4 time, blending gentle classical influence with modern minimalist touches. Serene, elegant, and contemplative.

Bb major 103.4 BPM 3/4 73s Classical, Neoclassical

Seg 10:00–0:19mpPiano — Gentle, melodic solo with classical influence
Seg 20:19–0:38mfPiano — Modern classical, minimalist touch
Seg 30:38–0:56mfPiano — Serene, elegant, touch of romanticism
Seg 40:56–1:13mpPiano — Slow, contemplative solo, romantic touch

来一段中国古风音乐

The song blends lively and serene moments, transitioning between energetic and reflective moods, with a rich tapestry of traditional and classical instruments.

B minor 100 BPM 4/4 98s Traditional Chinese, Classical Instrumental

Seg 10:00–0:19mpGuzheng, Pipa, Erhu, Guqin — Lively and energetic, celebratory
Seg 20:19–0:38pPiano, Strings, Woodwinds — Serene, elegant, calm reflection
Seg 30:38–0:58mpPiano, Strings, Harp — Steady and flowing, gentle melancholy
Seg 40:58–1:17mpViolin, Piano, Cello — Fast-paced, dynamic, grandeur
Seg 51:17–1:38pPiano, Strings, Harp, Flute — Tranquil, nostalgic conclusion

生成一首动感电子舞曲，节奏强烈，适合蹦迪

An energetic and uplifting electronic dance track that blends cinematic elements with a driving rhythm, creating a sense of excitement and euphoria.

C# minor 120 BPM 4/4 143s Electronic, House, Progressive House

Seg 10:00–0:24pSynth, Ambient pads — Cinematic, atmospheric, mysterious
Seg 20:24–0:40mfSynth, Drum machine — Upbeat, danceable groove
Seg 30:40–0:56mfSynth, Bass — Driving four-on-the-floor, euphoric
Seg 40:56–1:14fFull mix — Fast-paced, maximum energy drop
Seg 51:14–1:28pAmbient pads, E-drums — Downtempo bridge, introspective
Seg 61:28–1:44mfSynth, Drum machine — Energy rebuild
Seg 71:44–2:00fFull mix — Euphoric climax, driving bassline
Seg 82:00–2:22fFull mix — Final push, high-energy resolution

Abstract