Recreating image(s) from one paper on another requires our system to be capable of sensing, planning, and actuation. For sensing, the project requires scanning the designated workspace for a paper with the image to reproduce, and subsequently for AR tags marking the paper to draw on. Planning for the Robo Artist begins with image analysis that identifies individual contours, followed by transforming them into contour paths, calculating an optimized path traversing them, simplifying and approximating them with fewer waypoints, and ending with transforming those points into real-world base frame coordinates for the Sawyer to traverse to. The final step is actuation, in which the robot arm is commanded to draw out the contours using a marker.
To achieve the goal of drawing with a marker, we had concerns about the Sawyer's ability to consistently translate a marker tip perfectly on the table plane, as to allow the marker to deposit ink, without crushing its tip. To ensure the pen was consistently located relative to the Sawyer wrist, we opted to fix it onto the wrist directly as the primary attachment method. We also wanted still to leave an option to use the grippers for holding the pen holder in case there were issues with attaching directly, or if we wanted to have multiple holders to swap colors, so we made the base of the gripper rectangular rather than cylindrical to facilitate grabbing.
These requirements compose the criteria to achieve the desired functionality of commanding the Sawyer arm to recreate images presented to it at arbitrary positions on a blank sheet also presented at an arbitrary position. One final, less technical requirement, is that any designs we made must fit within the available workspace, a small table near the Sawyer arm.
To sense the papers presented to it, we used the built-in cameras on the Sawyer robot. The robot arm begins by taking an image of the table space with the paper containing an image on it with its wrist camera. The Sawyer then switches to its head camera to scan the workspace for the AR tags designating the blank sheet for drawing. We had originally planned on using the wrist camera for both tasks, but the small field of view, low resolution, and short focal length made the camera insufficient for performing the latter task. We had also planned on having both the source and target papers on the table at the same time, but due to table space and camera sensing constraints, we ended up having only one on the table at a time, which made our sensing more robust by increasing AR detection reliability and ensuring the source image is isolated, but reduced convenience for the user as they have to manually switch cameras (due to limited bandwidth on the Sawyer for both cameras) and needing to manually replace the papers.
To plan for drawing, we elected to build our image processing sequence around using contour detection and skeletonization to produce the contours for the robot arm to trace out. It also transforms the robot arm path to the location and orientation of the target paper. This makes our controller more robust as its planned output is less sensitive to the target paper orientation and location, and increases user convenience by increasing the freedom they have to place the output paper. The planning algorithm also attempts to pick adjacent contours to draw consecutively, which increases final actuation efficiency.
Finally, for actuation, calculations for the output image coordinates using homogenous transformations and MoveIt are used to command the robot arm to produce the final image. Our actuation speed is 20% of the robot's maximum movement speed using MoveIt, which ensures motion accuracy and precision at the cost of drawing speed.
On the mechanical design side, we made a pen holder mechanism that attaches to the Sawyer arm. This mechanism uses a spring to ensure the marker touches the paper to produce lines, without applying excessive pressure. The spring also handles small errors in the Z-direction, allowing the pen height to remain constant while drawing even when the Sawyer arm is unable to maintain a consistent wrist joint height or accurately detect the depth of the table. The pen holder incorporates a robust but simple locking mechanism, which allows for easy integration with the robot. Additionally, to ensure consistent detection and easy setup, we printed a couple AR tags of known size and designed a small base to paste them onto that lines up against the output paper.
Key Design Choices
Some of the key design choices for our hardware include spring selection and geometric complexity. We initially had a stiff spring with 1mm wire diameter, which was determined to be too stiff, and any significant deflection crushed the marker tip. We got much softer springs, which are enough to push the marker against the marker tube portion of the holder, but required very little force for high deflection. For the geometry of the pen holder, we aimed to create a design that was easy to manufacture, and easy to assemble and disassemble. This led us to the locking mechanism with a spring canister, which was strong and very simple to interface with the robot while allowing the Robo Artist to maintain correct pressure without damaging the markers.
Trade-offs
Due to the short project timeline and resource constraints we had to make some trade-offs in our design. For our custom marker attachment, we used PLA filament for 3D printing in the Jacobs Maker Space. While this material was accessible and low-cost, it let to durability issues -- the spring canister design protected our markers, but a few of the PLA pen holders we printed were cracked during testing. Furthermore, we chose to use the integrated cameras on the Sawyer arm to reduce cost and free up resources for others, which ended up costing us many headaches dealing with issues caused by the lower quality cameras. In printing our own AR tags, we used printers around campus that were free to use for students. In using these printers, we were constrained to using the default ink they were loaded with, which was glossy and led to detection issues.
Efficiency & Robustness
Despite the destruction of a few pen holders due to weaker prototyping material, our spring based design allowed a single marker to survive all of our testing, demonstrating the long term durability of this design, which may be useful in industrial applications. Furthermore, during early testing, we noted that the limitations of the Sawyer arm's sensing ability and accuracy caused difficulty in maintaining z-height. Our spring based design handled this deficiency, maintaining marker contact with the paper even when the robot's wrist height was visibly fluctuating. Although the decision to use the integrated Sawyer cameras caused many issues due to the lower image quality, it led to a design choice that could potentially increase the efficiency of repeated use of the Robo Artist. Because the Sawyer system's low bandwidth only allowed using one camera at a time, switching from the wrist to the head camera required breaking up the sensing phase from the planning and actuation -- that is, images are processed into contours and points before being handed off to command the arm. This allows for the repeated execution of a single drawing and stringing together multiple drawings in sequence without the need to recompute contours every iteration, which would greatly increase the systems efficiency in a real engineering application.