MGT-python

The Musical Gestures Toolbox for Python is a collection of tools for visualizing and analysing audio and video files.

📖 Documentation & Examples

Quick Start

Installation

pip install musicalgestures

musicalgestures installs its core Python dependencies automatically. You still need a working ffmpeg installation on your system for video processing.

Basic Usage

import musicalgestures as mg

# Load a video (mp4, avi, mov, … all supported)
v = mg.MgVideo('dance.mp4')

# Create visualizations — call .show() to display the result
v.grid().show()
v.videograms().show()
v.average().show()
v.history().show()
v.heatmap().show()              # where the video changes most

# Motion analysis
v.motion().show()
v.motiontempo().show()          # dominant movement tempo (Hz/BPM)
v.eulerian(mode='motion').show()  # amplify subtle motion (EVM)

# Audio analysis
v.audio.waveform().show()
v.audio.spectrogram().show()
v.audio.mfcc().show()
v.audio.tempo().show()          # tempo + beat tracking
v.sonomotiongram().show()       # sonify the motiongram

# Pose estimation (MediaPipe is GPU-capable on the standard pip OpenCV)
v.pose(model='mediapipe').show()

Display happens via .show() — analysis methods return result objects (MgVideo/MgImage/MgFigure) and do not auto-render.

Runtime Notes

• ffmpeg is required for video I/O and preprocessing.
• pose() defaults to the MediaPipe backend and downloads its weights on first use if they are missing; the OpenPose models ('body_25'/'coco'/'mpi') download their larger Caffe weights on first use instead.
• In notebooks and other non-interactive runs, missing pose weights are downloaded automatically when possible.
• If device='gpu' is requested but OpenCV CUDA support is unavailable, pose() falls back to CPU execution.
• flow.dense(), flow.sparse(), and blur_faces() use CPU by default (use_gpu=False). Set use_gpu=True to opt into CUDA acceleration with automatic CPU fallback.
• get_cuda_device_count() is available to quickly check whether OpenCV sees CUDA devices.
• blur_faces() returns the generated result object consistently, including when save_data=True.

Try Online

Quick Links

• Installation Guide
• Quick Start Tutorial
• API Reference
• Wiki & How-Tos
• Contributing

Features

• Video Analysis: Motion detection, optical flow, motion vectors, movement tempo, Eulerian Video Magnification
• Pose Estimation: MediaPipe (default; fast on plain CPU, GPU-capable) and OpenPose (multi-person) backends, with average-pose and trajectory summaries (per-marker quantity of motion + dominant frequency), optional marker motion trails, and a 3D pose waterfall
• Audio Processing: Waveforms, spectrograms, MFCC, chromagrams, tempo/beat tracking, spectral descriptors
• Visualizations: Motiongrams, videograms, motion history, heatmaps, sonomotiongrams (motion → sound)
• Space-time displays: Stroboscope (chronophotography), silhouette waterfall, Motion History Image, 3D space-time volume, combined motion SSM
• Integration: Works with NumPy, SciPy, librosa, and Matplotlib ecosystems
• Cross-platform: Linux, macOS, Windows support

Presentation

See this short video presentation made for the Nordic Sound and Music Computing Conference 2021:

Requirements

• Python 3.10+
• FFmpeg
• See installation guide for complete requirements

Research Background

This toolbox builds on the Musical Gestures Toolbox for Matlab, which again builds on the Musical Gestures Toolbox for Max. Many researchers and research assistants have helped its development over the years, including Balint Laczko, Joachim Poutaraud, Frida Furmyr, Marcus Widmer, Alexander Refsum Jensenius

The software is currently maintained by the fourMs lab at RITMO Centre for Interdisciplinary Studies in Rhythm, Time and Motion at the University of Oslo.

Reference

If you use this toolbox in your research, please cite this article:

• Laczkó, B., & Jensenius, A. R. (2021). Reflections on the Development of the Musical Gestures Toolbox for Python. Proceedings of the Nordic Sound and Music Computing Conference, Copenhagen.

@inproceedings{laczkoReflectionsDevelopmentMusical2021,
    title = {Reflections on the Development of the Musical Gestures Toolbox for Python},
    author = {Laczkó, Bálint and Jensenius, Alexander Refsum},
    booktitle = {Proceedings of the Nordic Sound and Music Computing Conference},
    year = {2021},
    address = {Copenhagen},
    url = {http://urn.nb.no/URN:NBN:no-91935}
}

License

This toolbox is released under the GNU General Public License 3.0 license.