"The primary use of projection technology today is for creating large flat displays that provide a shared viewing experience for presentations or entertainment. While the field of computer graphics has explored the power projected light has to create illusions that can reshape our perception of and interaction with surfaces in the environment, very few of these systems have had success in terms of commercial and consumer adoption. Part of this market place failure can be attributed to the lack of practicality in the cost-of-operation due to their complexity of installation and reliability of execution. Often these systems require expert knowledge to perform system setup and calibration between the projected and the physical surfaces to make these illusions effective. Additionally, costly external tracking systems may be needed to support input. In this thesis, I present a technique for inherently adding object location discovery and tracking capabilities to commercial projectors. This is accomplished by introducing light sensors into the projection area and then spatially encoding the image area using a series of structured light patterns. This delivers a unique pattern of light to every pixel in the projector's screen space directly encoding the location data using the projector itself.
By unifying the image projection and location tracking technologies, many of the difficult calibration and alignment issues related to projector-based augmented reality applications can be eliminated simplifying their implementation and execution. Furthermore, by creating a hybrid visible light and infrared light projector, a single calibration-free device can perform invisible location tracking of input devices while simultaneously presenting visible application content. In this thesis, I present a detailed description of the projector-based location discovery and tracking technique, a description of three prototype implementations I created, and a demonstration the effectiveness of this simplification by implementing several location-sensitive projector applications."
The following are video demonstrations of the components of my thesis work on projector-based location discovery and tracking.
Automatic Projector Calibration
The fundamental concept of my thesis is to: 1) Embed optical sensors into the projection surface. 2) Project a series of Gray-coded binary patterns. 3) Decode the location of the sensors for use in a projected application. This video demonstrates this idea in the form of a target screen fitting application. It goes on to demonstrate how this approach can be used in multi-projector applications such as stitching (creating a large display using tiled projection) or layering (multiple versions of content on the same area for view dependent displays). Additionally, it can be used to automatically register the orientation of 3D surfaces for augmenting the appearance of physical objects.
This technique is also useful for performing automatic touch calibration of interactive whiteboards or touch-tables.
Lee, J., Dietz, P., Aminzade, D., and Hudson, S. "Automatic Projector Calibration using Embedded Light Sensors", Proceedings of the ACM Symposium on User Interface Software and Technology, October 2004. [pdf][mov][ppt]
RFIG Lamps: Interacting with a Self-Describing World via Photosensing Wireless Tags and Projectors
By using photosensitive RFID tags, physical objects in the world can be interacted with using a handheld projector to annotate and augment thier appearance. The projector uses light patterns to allow the tags to discover thier own locations and thier physical relationship to other tags. Projector tracking in this early system was accomplished using computer vision techniques. In my later work, you will see how tracking can be done using the projector itself.
Raskar, R.; Beardsley, P.; van Baar, J.; Wang, Y.; Dietz, P.H.; Lee, J.; Leigh, D.; Willwacher, T., "RFIG Lamps: Interacting with a Self-Describing World via Photosensing Wireless Tags and Projectors", Proceedings of the ACM SIGGRAPH, August 2004.
Moveable Interactive Projected Displays Using Projector Based Tracking
By projecting smaller patterns over the discovered locations of the sensors, we can obtain location updates much faster sufficient to do interactive tracking of hand-held surfaces and objects. Additionally, small patterns free up pixels that can be used for application content. This work also describes a technique for projecting the patterns in a frequency modulated (FM) manner such that they are imperceptible to the human eye. The result is that the patterns appear as solid gray squares eliminating the high-contrast black and white patterns from before.
Lee, J., Hudson, S., Summet, J., and Dietz, P. "Moveable Interactive Projected Displays Using Projector Based Tracking", Proceedings of the ACM Symposium on User Interface Software and Technology, October 2005. pp. 63-72 [pdf][mov][ppt]
Hybrid Infrared and Visible Light Projection
A device capable of projecting both visible and infrared light would be a valuable tool for computer-vision systems as well as projector-based location discovery. Full screen location discovery patterns could be projected in infrared light while visible application content is presented to human viewers. The benefit of using a single device to do both is that the correspondence between location data and the application image pixels is inherent and thus calibration free and optically robust. This hybrid projector is a custom DLP projector using a LED light source. The Digital Micro-mirror Device is capable of generating over 50,000 binary images per second allowing very fast location discovery. The LED array allows us to rapidly alternate between visible and infrared light illumination allowing tracking and image projection in a real-time interactive manner.
This capability would provide inherent multi-stylus input tracking in any projection that had this feature as well a the applcations shown in the previous videos. Additionally, this capability could be achieved with only a minor change in upcoming LED illuminated DLP projection systems.
Lee, J., Hudson, S, Dietz, P. "Hybrid Infrared and Visible Light Projection for Location Tracking", Proceedings of the ACM Symposium on User Interface Software and Technology, October 2007. [pdf] [ppt]
Foldable Interactive Displays
This is a concept demo of what would be possible once a high-speed hybrid infrared and visible light projector were available. As mentioned above, DLP technology has the potential to perform high-speed tracking simultaneously with image projection. With this type of location tracking, we could track points on non-rigid geometries and project accurately onto flexible and foldable surfaces as well as obtain stylus input. This achieves a vision commonly found in science-fiction films where an individual can summon a large display from a pocket-sized device.
[Currently In Review] Lee, J., Hudson, S, "Foldable Interactive Displays"