This notebook shows how to setup a new project, train a keypoint-MoSeq model and visualize the resulting syllables.

Note

The calibration step below requires jupyterlab launched from the keypoint_moseq environment. It will not work in jupyter notebook.

Project setup

Create a new project directory with a keypoint-MoSeq config.yml file.

import keypoint_moseq as kpms

project_dir = 'demo_project'
config = lambda: kpms.load_config(project_dir)
Loading BokehJS ...
Setup from DeepLabCut
dlc_config = 'dlc_project/config.yaml'
kpms.setup_project(project_dir, deeplabcut_config=dlc_config)
Setup from SLEAP
sleap_file = 'XXX' # any .slp or .h5 file with predictions for a single video
kpms.setup_project(project_dir, sleap_file=sleap_file)
Custom setup
bodyparts=[
    'tail', 'spine4', 'spine3', 'spine2', 'spine1',
    'head', 'nose', 'right ear', 'left ear']

skeleton=[
    ['tail', 'spine4'],
    ['spine4', 'spine3'],
    ['spine3', 'spine2'],
    ['spine2', 'spine1'],
    ['spine1', 'head'],
    ['nose', 'head'],
    ['left ear', 'head'],
    ['right ear', 'head']]

video_dir='path/to/videos/'

kpms.setup_project(
    project_dir,
    video_dir=video_dir,
    bodyparts=bodyparts,
    skeleton=skeleton)

Edit the config file

The config can be edited in a text editor or using the function kpms.update_config, as shown below. In general, the following parameters should be specified for each project:

  • bodyparts (name of each keypoint; automatically imported from SLEAP/DeepLabCut)

  • use_bodyparts (subset of bodyparts to use for modeling, set to all bodyparts by default; for mice we recommend excluding the tail)

  • anterior_bodyparts and posterior_bodyparts (used for rotational alignment)

  • video_dir (directory with videos of each experiment)

Edit the config as follows for the example DeepLabCut dataset:

kpms.update_config(
    project_dir,
    video_dir='dlc_project/videos/',
    anterior_bodyparts=['nose'],
    posterior_bodyparts=['spine4'],
    use_bodyparts=[
        'spine4', 'spine3', 'spine2', 'spine1',
        'head', 'nose', 'right ear', 'left ear'])

Load data

The code below shows how to load keypoint detections from DeepLabCut. To load other formats, replace 'deeplabcut' in the example with one of 'sleap', 'anipose', 'sleap-anipose', 'nwb'. For other formats, see the FAQ.

# load data (e.g. from DeepLabCut)
keypoint_data_path = 'dlc_project/videos/' # can be a file, a directory, or a list of files
coordinates, confidences, bodyparts = kpms.load_keypoints(keypoint_data_path, 'deeplabcut')

# format data for modeling
data, metadata = kpms.format_data(coordinates, confidences, **config())
Loading keypoints: 100%|████████████████| 10/10 [00:00<00:00, 12.19it/s]

Calibration

This step requires jupyter lab. It will not work in jupyter notebook.

The purpose of calibration is to learn the relationship between keypoint error and confidence scores, which is stored in the config as a pair of slope and intercept coefficients. One can also adjust the confidence_threshold parameter at this step, which is used to define outlier keypoints for PCA and model initialization. This step can be skipped for the demo data.

  • Run the cell below. A widget should appear with a video frame on the left.

  • Annotate each frame with the correct location of the labeled bodypart

    • Left click to specify the correct location - an “X” should appear.

    • Use the arrow buttons to annotate additional frames.

    • Each annotation adds a point to the right-hand scatter plot.

    • Continue until the regression line stabilizes.

  • At any point, adjust the confidence threshold by clicking on the scatter plot.

  • Use the “save” button to update the config and store your annotations to disk.

kpms.noise_calibration(project_dir, coordinates, confidences, **config())
Loading sample frames: 100%|████████████| 90/90 [00:06<00:00, 14.25it/s]