Dobb·E

Project webpage · Documentation (gitbooks) · Paper

Authors: Mahi Shafiullah*, Anant Rai*, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, Lerrel Pinto

Open-source repository of the hardware and software components of Dobb·E and the associated paper, On Bringing Robots Home

https://github.com/notmahi/dobb-e/assets/3000253/d332fa42-d351-4ad0-a305-4b6664bd7170

What's on this repo

Dobb·E is made out of four major components:

1. A hardware tool, called The Stick, to comfortably collect robotic demonstrations in homes.
2. A dataset, called Homes of New York (HoNY), with 1.5 million RGB-D frames. collected with the Stick across 22 homes and 216 environments of New York City.
3. A pretrained lightweight foundational vision model called Home Pretrained Representations (HPR), trained on the HoNY dataset.
4. Finally, the platform to tie it all together to deploy it in novel homes, where with only five minutes of training data and 15 minutes of fine-tuning HPR, Dobb·E can solve many simple household tasks.

Reflecting this structure, there are four folders in this repo, where:

1. hardware contains our 3D printable STL files, as well as instructions on how to set up the Stick.
2. stick-data-collection contains all the necessary software for processing any data you collect on the Stick.
3. imitation-in-homes contains our code for training a policy on your collected data using our pretrained models, and also the code to pretrain a new model yourself.
4. robot-server contains the code to be run on the robot to deploy the learned policies.

The primary documentation source is gitbooks at https://docs.dobb-e.com. There are also associated documentations inside each folder's READMEs.

Paper

Get it from ArXiv or our website.

Citation

If you find any of our work useful, please cite us!

@article{shafiullah2023bringing,
  title={On bringing robots home},
  author={Shafiullah, Nur Muhammad Mahi and Rai, Anant and Etukuru, Haritheja and Liu, Yiqian and Misra, Ishan and Chintala, Soumith and Pinto, Lerrel},
  journal={arXiv preprint arXiv:2311.16098},
  year={2023}
}