Scooter project
2018-2025
Introduction
The project I was first involved in at the UMass Lowell HRI Lab was the development of an assistive manipulator for a mobility scooter. Imagine something you could ride around a grocery store and have pick items off of shelves for you. Prior to my involvement, my colleagues at the HRI Lab, together with Northeastern University’s Helping Hands Lab, had built this system to be operated with a laser pointer [1] based off of the work of Kemp et al. [2] on laser designation of objects. It was, interestingly enough, built upon a single disembodied arm from a Rethink Robotics Baxter.
The next version of this system used a Universal Robots UR5 in place of a Baxter arm, and was capable of moving a lot faster and had algorithms that could better segment objects in clutter. To augment the laser pointer, the system had a screen in the armrest that could give the user text prompts and show a pointcloud from the the system’s depth cameras. This was awesome, but a frequent pain point was that it was difficult for the user to understand what items were actually within the arm’s reach and what it could actually pick up.
During a visit to the Helping Hands Lab, we were brainstorming ways to mitigate this issue. Several ideas were brought up, including one I found particularly interesting — using a laser to display a ring of light around the scooter, showing the radius of the arm. I really liked this idea, but we noted that this solution would be mostly only applicable were the robot’s workspace limited to a flat plane of a given height. Anything more complex — a series of shelves, tables of varying heights, angled surfaces, etc. — the laser would need to update to account for this geometry.
I brought up the fact that we did in fact have complete knowledge of the geometry of the robot’s workspace already from the system’s five depth cameras, and this, with some math, would let us use a projector to project light onto surfaces indicating whether they were in range of the arm.
Projection mapping
The somewhat hacky but entirely functional solution I landed on to enable us to project information from the robot’s perception back into the real world was to treat the projector as an “inverse camera.” We were already rendering a visualization of the pointcloud in RViz, the 3D visualization tool in ROS. If I placed a virtual camera in the rendered scene with the same lens characteristics as those of the projector, and sent that image back to the projector, anything I placed in the rendered environment would end up back at the coordinates it came from when this image was projected back through the projector.
I used this technique to prototype a few different types of functionality for the scooter. By selecting and projecting subsets of the pointcloud captured by the 5 depth cameras, I could, for example, highlight specific objects, or highlight surfaces within reach of the arm.
 |  |
|---|---|
| Highlighting object | Showing reach of arm |
For our initial implementation, I calculated the projector’s intrinsics manually, using a jig that held the projector’s optical axis orthogonal to a piece of posterboard at a known fixed distance. I marked the corners of the image and used the pinhole camera model to calculate the focal length and principal point. These values were then packed into the projection/camera matrix P in the ROS sensor_msgs/CameraInfo message that the virtual camera subscribed to. Later iterations of our projection mapping system used automatic projector-camera calibration methods such as ProCamCalib [3].
 |
|---|
| Jig used to manually calibrate projector |
The projection mapping system was included in [4], where we presented our work on the assistive manipulation system, demonstrating an improvement in runtime and success rate relative to a baseline from literature. We further describe the implementation of our projection mapping system in [5]. Our work implementing a similar system for a mobile robot is described in [6], [7].
Graphical/Tangible user interfaces
 |  |
|---|---|
| GUI | TUI |
In the summer of 2019, I led a team to develop two different user interfaces for the scooter. The two UIs were designed to be contrasting, with one keeping all of its feedback to the user on a touchscreen, and the other in the real world itself. Each of these were designed to allow the user to designate an object that the arm would pick up and either hand to the user or place elsewhere.
Both UIs were designed to use the same workflow and state diagram despite using radically different modalities — see [8] for more detail on the design of the two UIs.
GUI
The GUI was implemented in Qt, and the workflow for designating an object involved tapping the object in an image of the workspace shown on the touchscreen.
|  |  |
|---|---|---|
| GUI touchscreen | Selection | Confirmation |
TUI
The TUI was implemented using a projector and voice lines to provide feedback to the user. Selections were made by using the joystick to manipulate a “cursor” projected into the world.
|  |
|---|---|
| TUI control panel | TUI “cursor” |
User study
With a key difference between the two UIs being the modality through which the system provides feedback to the user, we hypothesized that the TUI’s ability to project information directly into the real world could give it a unique advantage in certain scenarios. Specifically, we thought that the GUI was taxing on a user’s spatial visualization ability in a way that the TUI was not, since with the GUI the user is sometimes required to mentally map between objects in the real world and objects on the touchscreen. We also hypothesized that this effect would scale with higher levels of clutter in the workspace — the GUI would be more affected by increased clutter than the TUI.
To test these hypotheses, we ran a within-subjects study, recruiting 27 participants over the age of 60, asking them to complete a series of tasks with each UI [9]. There were two UI conditions (GUI and TUI), and two clutter conditions (low clutter and high clutter). We measured task completion time, success rate, and usability of each UI, as well as each participant’s preference between the two UIs, and their spatial visualization ability. We found some evidence supporting the idea that TUIs could mitigate difficulty caused by increased environmental clutter and differences in individual spatial visualization ability, but further research is needed.
ICRA 2025 presentation
This video was presented during the ICRA 2025 Assistive Human-Robot Interaction session.