Hello all,
I wanted to start a discussion about SymForce + PyTorch project ideas and find use-cases from the community that might improve/enable new functionality in robotics applications.
For conversation starters, here are some of the ideas that I find interesting:
-
Taking inspiration from GradSLAM: https://gradslam.github.io/, it would be a nice experiment to add factor-graph based cost functions to a fully differentiable slam pipeline. Something like this can allow to fully flow the geometric residues and covariances all the way up to the feature extraction layers. Maybe adding custom geometric factors can lead to better convergences than chamfer or quadric losses. One could essentially learn features using factors as a training signal.
-
We can also take a look at some massive sfm problems that can benefit from parallelization on gpu. Pytorch can possibly be a (simpler) lower hanging fruit than going down the CUDA route.
-
We can also come up with a couple of slam + prediction problems that may benefit from a common graphical representation in torch. Something on the lines of prediction aided slam.
-
I am also curious about gaussian belief propagation: https://gaussianbp.github.io/ and it's parallels with factor graphs and transformers. Can be an interesting idea to leverage that to guide the gradient descent in slam and sfm problems.
I am very curious to learn about even more ideas and applications from the symforce community that would benefit from the above integration. Looking forward to gathering a pool of ideas here and then we can use this space to roadmap them according to feasibility, impact and added value.
CC: @hmartiro @aaron-skydio
Hello all,
I wanted to start a discussion about SymForce + PyTorch project ideas and find use-cases from the community that might improve/enable new functionality in robotics applications.
For conversation starters, here are some of the ideas that I find interesting:
Taking inspiration from GradSLAM: https://gradslam.github.io/, it would be a nice experiment to add factor-graph based cost functions to a fully differentiable slam pipeline. Something like this can allow to fully flow the geometric residues and covariances all the way up to the feature extraction layers. Maybe adding custom geometric factors can lead to better convergences than chamfer or quadric losses. One could essentially learn features using factors as a training signal.
We can also take a look at some massive sfm problems that can benefit from parallelization on gpu. Pytorch can possibly be a (simpler) lower hanging fruit than going down the CUDA route.
We can also come up with a couple of slam + prediction problems that may benefit from a common graphical representation in torch. Something on the lines of prediction aided slam.
I am also curious about gaussian belief propagation: https://gaussianbp.github.io/ and it's parallels with factor graphs and transformers. Can be an interesting idea to leverage that to guide the gradient descent in slam and sfm problems.
I am very curious to learn about even more ideas and applications from the symforce community that would benefit from the above integration. Looking forward to gathering a pool of ideas here and then we can use this space to roadmap them according to feasibility, impact and added value.
CC: @hmartiro @aaron-skydio