Berkeley released BridgeData V2 dataset for robot learning at scale

Hearken to this text

The preliminary and last state of a trajectory with the pure language annotation “shut cupboard”. | Supply: UC Berkeley

A analysis group at UC Berkeley has launched BridgeData V2, a big and various dataset of robotic manipulation behaviors. The dataset goals to facilitate analysis in scalable robotic studying. 

This up to date dataset is suitable with open-vocabulary and multi-task studying strategies conditioned on objective photos or pure language directions. The abilities discovered from the dataset could be generalized to novel objects and environments and throughout establishments. 

The Berkeley group collected information from a variety of duties in many alternative environments with variations in objects, digicam poses, and workspace positioning. All of those variations are to raised help broad generalization. 

The dataset contains 60,096 trajectories, 50,365 teleoperated demonstrations, 9,731 rollouts, 24 environments, and 13 abilities. Every trajectory is labeled with a pure language instruction similar to the duty the robotic is performing. 

The 24 environments included within the dataset are grouped into 4 totally different classes. Many of the information comes from seven distinct toy kitchens, which all embrace some mixture of sinks, stoves, and microwaves. 

Many of the 13 abilities included in BridgeData V2 come from foundational object manipulation duties like pick-and-place, pushing, and sweeping. Some information comes from setting manipulation, which incorporates issues like opening and shutting doorways and drawers. The remainder of the info comes from extra advanced duties, like stacking blocks, folding garments, and sweeping granular media. Some elements of the info come from mixtures of those classes. 

The group evaluated a number of state-of-the-art offline studying strategies utilizing the dataset. First, they evaluated the dataset on duties which are seen within the coaching information. Whereas the duties had been seen in coaching, many strategies nonetheless required the group to generalize to novel object positions, distractor objects, and lighting. The group then evaluated the dataset on duties that require generalizing abilities within the information to novel objects and environments. 

The information was collected on a WidowX 250 6DOF robotic arm. The group collected the demonstrations by teleoperating the robotic with a VR controller. The management frequency is 5 Hz and the typical trajectory size is 38 timesteps. 

For sensing, the group used an RGBD digicam that’s mounted in an over-the-shoulder view, two RGB cameras with poses which are randomized throughout information assortment, and an RGB digicam hooked up to the robotic’s wrist. The entire photos are saved at a 640×480 decision. 

The dataset could be downloaded here. The information for teleoperated demonstrations and from the scripted pick-and-place coverage are supplied as separate zip recordsdata. The group offers each mannequin coaching code and pre-trained weights for getting began with the dataset.

This repository offers code and directions for coaching on the dataset and evaluating insurance policies. This guide offers directions for organising the robotic {hardware}.