This notebook shows how to setup a new project, train a keypoint-MoSeq model and visualize the resulting syllables.

Note

To ensure prevent errors during the calibration step below, make sure to launch jupyter from the keypoint_moseq environment.

Project setup

Create a new project directory with a keypoint-MoSeq config.yml file.

import keypoint_moseq as kpms
import matplotlib.pyplot as plt

project_dir = "demo_project"
config = lambda: kpms.load_config(project_dir)
Loading BokehJS ...

Setup from DeepLabCut

 
dlc_config = "dlc_project/config.yaml"
kpms.setup_project(project_dir, deeplabcut_config=dlc_config)

Setup from SLEAP

 
sleap_file = "XXX"  # any .slp or .h5 file with predictions for a single video
kpms.setup_project(project_dir, sleap_file=sleap_file)

Custom setup

 
bodyparts = [
    "tail",
    "spine4",
    "spine3",
    "spine2",
    "spine1",
    "head",
    "nose",
    "right ear",
    "left ear",
]

skeleton = [
    ["tail", "spine4"],
    ["spine4", "spine3"],
    ["spine3", "spine2"],
    ["spine2", "spine1"],
    ["spine1", "head"],
    ["nose", "head"],
    ["left ear", "head"],
    ["right ear", "head"],
]

video_dir = "path/to/videos/"

kpms.setup_project(project_dir, video_dir=video_dir, bodyparts=bodyparts, skeleton=skeleton)

Edit the config file

The config can be edited in a text editor or using the function kpms.update_config, as shown below. In general, the following parameters should be specified for each project:

  • bodyparts (name of each keypoint; automatically imported from SLEAP/DeepLabCut)

  • use_bodyparts (subset of bodyparts to use for modeling, set to all bodyparts by default; for mice we recommend excluding the tail)

  • anterior_bodyparts and posterior_bodyparts (used for rotational alignment)

  • video_dir (directory with videos of each experiment)

  • fps (frame per second of the input video)

Edit the config as follows for the example DeepLabCut dataset:

kpms.update_config(
    project_dir,
    video_dir="dlc_project/videos/",
    anterior_bodyparts=["nose"],
    posterior_bodyparts=["spine4"],
    use_bodyparts=["spine4", "spine3", "spine2", "spine1", "head", "nose", "right ear", "left ear"],
    fps=30,
)

Load data

The code below shows how to load keypoint detections from DeepLabCut. To load other formats, replace 'deeplabcut' in the example with one of 'sleap', 'anipose', 'sleap-anipose', 'nwb'. For other formats, see the FAQ.

# load data (e.g. from DeepLabCut)
keypoint_data_path = "dlc_project/videos/"  # can be a file, a directory, or a list of files
coordinates, confidences, bodyparts = kpms.load_keypoints(keypoint_data_path, "deeplabcut")

Remove outlier keypoints

Removing large outliers can improve the robustness of model fitting. A common type of outlier is a keypoint which briefly moves very far away from the animal as the result of a tracking error. The following cell classifies keypoints as outliers based on their distance to the animal’s medoid. The outlier keypoints are then interpolated and their confidences are set to 0 so that they are interpolated for modeling as well.

  • Use outlier_scale_factor to adjust the stringency of outlier detection (higher values -> more stringent)

  • Plots showing distance to medoid before and after outlier interpolation are saved to {project_dir}/QA/plots/

  • Plotting can take a few minutes, so by default plots will not be regenerated when re-running this cell. To experiment with the effects of setting different values for outlier_scale_factor, set overwrite=True in outlier_removal.

kpms.update_config(project_dir, outlier_scale_factor=6.0)

coordinates, confidences = kpms.outlier_removal(
    coordinates,
    confidences,
    project_dir,
    overwrite=False,
    **config()
)

Format data for modeling

data, metadata = kpms.format_data(coordinates, confidences, **config())

Calibration

The purpose of calibration is to learn the relationship between keypoint errors and confidence scores. The results are stored using the slope and intercept parameters in the config.

  • Run the cell below. A widget should appear with a video frame and the name of a bodypart. A yellow marker denotes the detected location of the bodypart.

  • Annotate each frame with the correct location of the labeled bodypart

    • Click on the image at the correct location - an “X” should appear.

    • Use the prev/next buttons to annotate additional frames.

    • Click and drag the bottom-right shaded corner of the widget to adjust image size.

    • Use the toolbar to the left of the figure to pan and zoom.

  • We suggest annotating at least 50 frames.

  • Annotations will be automatically saved once you’ve completed at least 20 annotations. Each new annotation after that will trigger an auto-save of all your work. The message at the top of the widget will indicate when your annotations are being saved.

%matplotlib widget
kpms.noise_calibration(project_dir, coordinates, confidences, **config())

Loading sample frames: 100%|████████████| 90/90 [00:06<00:00, 14.25it/s]

WARNING:param.OverlayPlot01356: Tool of type 'pan' could not be found and could not be activated by default.
WARNING:param.OverlayPlot01356:Tool of type 'pan' could not be found and could not be activated by default.
WARNING:param.OverlayPlot01356: Tool of type 'wheel_zoom' could not be found and could not be activated by default.
WARNING:param.OverlayPlot01356:Tool of type 'wheel_zoom' could not be found and could not be activated by default.