Brain data ingestion processing steps

Prerequisites

  • Note: These instructions assume you previously completed imaging scans and have created a folder for your .czi files. This files should be stored in birdstore file server in directory:

  • Windows Example: Z:Active_Atlas_Datadata_rootpipeline_data{ANIMAL_ID}czi

  • Linux Example: /net/birdstore/Active_Atlas_Data/data_root/pipeline_data/{ANIMAL_ID}/czi/ where {ANIMAL_ID} is given name of animal subject. Contact immediate supervisor for available names/sequences.

  • N.B. You will need to create czi folder under animal name and copy all .czi files into this directory

  1. Login

  • Navigate to BrainSharer Admin portal (brainsharer.org)

    1. Login in upper-right-hand corner

    2. After authentication, click username and choose ‘Admin’ from drop down menu

../_images/login.png

1. Enter experimental subject information

Create new ANIMAL_ID in the database using Add link under BRAIN -> Animals

../_images/brain.categories.png

ANIMAL_ID is same name used in PREREQUISITE above (e.g. name of folder on birdstore) Only BOLD fields are required (Prep id); other fields are optional

Click save at the bottom of the form to proceed.

2. Enter scanning information

Add scanning info for ANIMAL_ID using ‘Add’ link under BRAIN > Scan runs

../_images/scan_run.png

Notes:

  • XY resolution is required (likely .325 µm on Axios)

  • Z resolution is required (likely 20 µm on Axios)

  • Number of slides (directory listing count for number of slides)

  • Width (pixels) (may leave at 0 – autodetect later)

  • Height (pixels) (may leave at 0 – autodetect later)

Choose ANIMAL_ID in dropdown then enter details of scan.

Click save at the bottom of the form to proceed.

3. Enter histology information

Choose ANIMAL_ID in dropdown then enter histology.

../_images/histology.png

Enter as much information as possible. The bold fields are required.

Congratulations! - The brain is now ready for processing

Back-end steps for processing brains - Overview

../_images/pipeline.flow.png

Figure 1. Summary description of data ingestion pipeline. This describes the automated and manual steps and total automated time for representative brain with 3 channels (34 hours).

Note: above steps are controlled through terminal python application, accessible on Linux. All files should be installed under /data/pipeline folder but may need updating if you encounter any errors. If script is not in this location, or you encounter errors, contact Duane or Ed for assistance.

Login to Linux:

Kleinfeld lab has 3 primary workstations used for brain processing with varying capacities: muralis (most powerful in cores/RAM), ratto, basalis. Brains will process faster on muralis, however other lab members may already be using the workstation.

Note: Use ActiveDirectory login to access all workstations.

Activate python virtual environment

source /usr/local/share/pipeline/bin/activate

Run each requested step in sequence

python src/pipeline/scripts/create_pipeline.py –task {TASK} –animal {ANIMAL_ID}

  • Use ANIMAL_ID where czi images are located.

  • Default channel is 1

  • Available tasks are:

extract | mask | clean | align | histogram | neuroglancer | cell_labels | status | extra_channel

(but may need multiple runs if multiple channels, downsampled version is required or errors occur)

Note: When run with –-debug True , a log file will be created in root folder of processing folder on birdstore (pipeline-process.log). Check this file for status and/or errors.