Biorobotics Laboratory BioRob

Powerful deep-learning based tracking method for animal behaviors requires large-scale curated and annotated data. Several recent papers [1,2] revealed the possibility to leverage the data requirement by rendering animated synthetic animals such as mice and ants.

In this project, the student will work on an existing biomechanical model of Salamander Pleurodeles Waltl to create a synthetic dataset for marklerless keypoint tracking tasks. The dataset would help improve the performance of a salamander tracking network, which would ultimately provide invaluable kinematics data for designing muscle models, neural controllers and validating neuroscience hypothesis.

For a PdM, this project involves:

• Improve the realism of the current salamander model in Blender and add diversity with procedurally generated noise and domain randomization.
• Generate a synthetic image dataset for markerless tracking tasks.
• Train an Image Domain Translator (e.g. U-GAT-IT[3]) to increase the dataset fidelity and reduce the reality gap
• Bonus: evaluate the dataset power on a markerless tracking network (e.g. DLC[4])

For a semester project, work packages will be optionally dropped and tailored according to student’s skills and interests.

The student is expected to have good programming skills and previous experience/knowledge in deep learning. Knowledge in 3D modeling and computer graphics is a plus but not required. If interested, please send an email to Chuanfang Ning with your motivation, CV, transcripts and most relevant experience.

[1] Bolaños, Luis A., et al. "A three-dimensional virtual mouse generates synthetic training data for behavioral analysis."

[2] Plum, Fabian, et al. "replicAnt: a pipeline for generating annotated images of animals in complex environments using Unreal Engine." 

[3] Kim, Junho, et al. "U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation." 

[4] Mathis, Alexander, et al. "DeepLabCut: markerless pose estimation of user-defined body parts with deep learning."