We created a 20x5 foot multi-touch, multi-user interactive wall for the Northwestern University Visitor Center. The nodes in the scene represent stories and 'hashtags', which provide information to prospective students and their families in a playful manner. The application is built as both a user interface filled with panels and scroll views, driven by a physics simulation based on fluids and a network of mass-springs.
We installed the software at two locations: the main NYU campus in Evanston, IL, and their satellite space in San Francisco, CA.
The UI was built with Cinder-View, with this being an ideal use case for testing and developing its multi-touch abilities. The interface is designed so that nine people can interact with the application at the same time (one per portrait display).
When you tap on a node, a story panel appears that consists of an image / video carousel, styled text, and related story nodes. The image carousel is built as a customized ScrollView, where each page stacks on top of the other. The transition between pages ended up being quite fun, since there were no enforced size or aspect requirements. Performance-wise, the difficulty was mostly in that it was possible for nine 1080p (max resolution) videos to be playing simultaneously, while users are scrolling through the carousel. I ended up using a pool of video players that were built on Cinder-WMFPlayer, which solved some of the problems around loads causing hitches in the UI.
To handle anti-aliasing at this high resolution, most things were rendered with SDF's (signed-distance fields), like the simple geometric shapes (circles, rectangles, and lines connecting nodes). The text was also rendered using SDF's, and a static glyph map that was generated in a preprocess.
We had many places with text on top of images and other aspects of the scene, so drop-shadows were a natural solution to make these more legible. For the story image carousel, I managed this with a custom shader since the pages are rectangular. For text, it's implemented as a Cinder-View Filter, which renders and caches an image of the drop-shadow for any semi-transparent content.
The main concept behind the scene was that of an aquarium filled with interlinked 'story nodes' you could interact with, evolving over time to present new information. I modeled the node connections as an N to N mass-spring network, where resting distances were fixed when a node was spawned. The physics update was implemented using OpenGL transform feedback on a fixed set of instanced geometry, which required limiting the number of connections in some way. A maximum of four links seemed to be enough, although for stability there is also an additional attribute for 'reverse links'.
The node movement is driven by the velocity buffer of a 2D fluid simulation (I used Hai Nguyen's implementation CinderFx) applied to the 'surface level' of the nodes, or in other words what was closest to the screen. Touches are injected into the velocity field so that a person's swipe causes the nodes to sway in that direction, and also get pulled and rotated in 3D by their linked nodes.
The nodes are managed spatially with a grid system that tallies whether there are two many nodes in a given area, and also whether there are too few of a specific type. When there are too many, a 'repelling magnet' is placed, which causes the nodes in that area to rotate around a randomized axis, outwards until they are off-screen. This caused a nice rolling effect, which really improved the 'underwater' feeling of the scene. If the nodes are really bunched up, I just kill them off by shooting the node backward in the Z direction, causing it to fade out in the fog. Similarly, if there aren't enough nodes of a given type within a region, a suitable node is spawned and faded in from far away along the z-axis, so that it fades in naturally. This spawning also happens for story nodes needed for the UI mode if there isn't a particular one close enough.
The nodes live freely within the fluid simulation, yet they are also used to populate the Story and Category UI pannels. In order to move from the free-moving physics world into a usable interface, constraints were introduced to control the positions. When a node is marked as in use for a UI panel, it gets an anchor position, yet it is still allowed to act as it usually would within the simulation. Maximum and minimum positions are set that restrict the node to reasonable distortions within the UI, but they are otherwise still updated within the physics loop. Allowing the physics loop to always update nodes resulted in a much more fluid and playful experience, very different from much of what we see on the web or other touch devices today.
Kelp Room at Power Station of Art, Shanghaiinstallation
Flow ParticlesGPU Particles with the Intel Realsense Depth camera and optical flow.
Intel RealSense Tunnelinstallation
Samsung Centerstagetouch wall
Glimpse Twitter Visualizationinterative visualization
Cinder Audioreal-time, modular, node-based audio library