UNSW Centre for Interactive Cinema Research

AVIE

AVIE: interior view showing T_Visionarium II

AVIE is the world’s first mobile and scalable 360-degree 3D immersive and interactive visualisation system. It measures 7–10 metres in diameter and 3–3.5 metres in height. It includes a motion tracking system and a multi-channel audio system. Driven by an experimental framework, it uses a suite of cutting-edge hardware and software resources that enable the development of applications in AI visualisation, immersive environments and networked collaboration. With a number of these systems available, it is deployable on-site with industry and project partners for research development and application.

AVIE has been installed as core visualisation infrastructure at: ArtLAB ÉPFL (Switzerland); Chengdu Research Institute, Chengdu (China); CityU Hong Kong (Hong Kong); China Coal Technology & Engineering Group, Fushun (China); Experimental Media Performing Arts Centre, Rensselaer University, New York (USA); ZKM Media Museum (Germany); Mines Rescue Pty Ltd; Museums Victoria; Museum of Applied Arts & Sciences; UTS Sydney (all Australia).

It has been commercialised for the mine safety training industry by: Mines Rescue Pty Ltd at its Newcastle, Wollongong, Lithgow and Singleton facilities, Australia; Shenyang Research Institute of China Coal Technology & Engineering Group (CCTEG), Fushun, China.
AVIE: exterior view showing Nebula

AVIE has been developed though three iterations, with further development on-going. AVIE1 (2004-07) facilitated audio-visual data to respond kinaesthetically to the user's body across the full breadth of the projected space; AVIE2 (2008-2010) has augmented this environment by delivering the ability to recompose digital files in real-time; AVIE3 (2011-2017) has delivered a base-line AI Database system that enables digital agents to interact with users,based on their ability to interpret and respond to the user’s behaviour in space. AVIE provides a platform in which 3D audio-visual experiences are created by a combination of multiple projectors and multiple sound sources, driven by a cluster of computers, running a highly versatile real-time 3D VR engine. It comprises custom designed technologies: a Motion Tracking system, a Mixed Reality system and an AI Reasoning system. The AI system coordinates the actions of the visualisation system by allowing it to autonomously reason about its response to user behaviour. The Motion Tracking system tracks the physical location of users in AVIE, while the Mixed Reality system allows the creation of 'scenarios' that exhibit a feedback loop between the world of the user and of the audio-visual data.

AVIE visualisation forms
High-resolution panoramic stereo images with augmented 3D graphics overlays
High-resolution panoramic videos, including stereo videos
Immersive multimedia data visualisation
Real-time interactive 3D environments supported by artificial intelligence

AVIE interaction platforms

AVIE utilises two types of interaction:

  1. remote device: a wireless tablet device allowing real-time selection and navigation within interactive applications
  2. motion tracking: real time software that detects, captures, tracks and interprets human movement and gestures, allowing capture and audience participation in the direct cinematic experience, interplay between users and digital characters and analysis of user behaviour

Below: Interior views showing Mining VR

AVIE Components

  • AVIE consists of a cylindrical silvered screen 4 metres high and 10 metres in diameter on which 3D panoramic images sequences and videos are projected. The 6 projectors use active shuttering technology to deliver stereoscopic 3D to end users, and the images from the individual projectors are seamlessly blended by custom software developed in-house at iCinema. 6 high-performance image generators, capable of processing images and videos as well as rendering large virtual environments in real time, are linked to the projectors to deliver the full immersive 3D experience. It accommodates up to 30 people at any one time.
    External view
    Projector set up

  • This provides a multi-purpose sensory environment for unencumbered tracking of viewer movement and gesture, offering a large range of motion capture and interaction design functions. Covering the entire AVIE area, the system is able to accurately track the spatial position and movements of up to thirty people. When applied to smaller groups of up to five people, it generates high-resolution three dimensional voxel models of their body movements and gestures, enabling real time motion tracking and some gesture recognition.
    Volumetric voxel tracking

  • Directly complementing the unique visualisation and interaction features of AVIE is its spatialised audio system. This is a twentyfour channel system with custom surround audio application software. This system enables fully immersive 360 degree placement of sound anywhere around the viewers. As with visual projection, the audio system can be coupled with the Intelligent Vision-Based Interaction and Motion Tracking Systems to allow voices to ‘follow’ projected characters, or to be activated only when viewers move within ‘earshot’ of the virtual audio sources, or correctly perform some task.
    Spatialised Audio System

Project Directors: Jeffrey Shaw, Dennis Del Favero

Programmers: Ardrian Hardjorno, Volker Kuchelmeister, Matthew McGinity
(Additional Software and Hardware Engineering: Jared Berghold, Marc Chee, Robin Chow, Alex Kuptsov, Alex Ong & Xin Guan)

Project Funding: ARC DP0209550, ARC DP0345547, ARC LE0453517
2004–2020

AVIE has been commissioned as a curated visualisation platform by numerous premier international venues for biennales, exhibitions and festivals, including:

  • Fo Guang Shan Buddha Museum, Dashu, Taiwan, 2020
  • École Polytechnique Fédérale de Lausanne, Switzerland, 2020
  • CityU Galleries, City University of Hong Kong, 2020
  • Maritime Museum 360, Hong Kong, 2020
  • Centre for Applied Computing & Interactive Media, Chengdu, 2020
  • Art of Immersion: Virtual Frontiers, ZKM, Karlsruhe, 2017
  • Chengdu Research Institute, Chengdu, 2017
  • Digitalife, Palazzo delle Esposizione, Rome, 2017
  • Sydney Film Festival, Sydney, 2016
  • Bonlieu Scène Nationale, Annecy, 2016
  • International Symposium of Electronic Art 2016 (ISEA16), Hong Kong, 2016
  • GLOBALE, ZKM, Karlsruhe, 2016.
  • Seoul Mediacity Biennale, Seoul Museum of Art, Seoul, 2016
  • Montreal Museum of Fine Art, Montreal, 2016
  • Chronus Art Center, Shanghai, 2014-15
  • Child, Nation and World Cinema Symposium, Sydney, 2014
  • 20th International Conference on Virtual Systems and Multimedia, Hong Kong, 2014
  • Data Arena, UTS, Sydney, 2014
  • International Symposium of Electronic Art 2013 (ISEA13), Sydney, 2013
  • 10th Shanghai Biennale, Power Station of Art, Shanghai, 2013
  • SIGGRAPH Asia, Hong Kong, 2013
  • Hong Kong Book Fair, Hong Kong, 2013
  • Limited Knowledge, Hong Kong, 2013
  • Festival d’Avignon, Avignon, 2013
  • Business of Design Week, Hong Kong, 2012
  • NODEM, Hong Kong, 2012
  • Grand Opening of Run Run Shaw Creative Media Centre, Hong Kong, 2012
  • Smithsonian Institution, Washington DC, 2012
  • Matter-Light, Bethune, 2011
  • Sydney Film Festival, Sydney, 2011
  • Festival of Avignon, Gymnase Paul Giera, Avignon, 2011
  • National Institute of Experimental Arts double conference, Sydney, 2011
  • ALiVE – Applied Laboratory for Interactive Visualization & Embodiment, CityU, Hong Kong, 2011
  • Digitalife, La Pelanda, Rome, 2010
  • 15th Biennial Film & History Conference, UNSW, Sydney
  • Dynamic Earth, Museum Victoria, Melbourne, 2010-2020
  • Inaugural Exhibition, ALiVE – Applied Laboratory for Interactive Visualization & Embodiment, Hong Kong, 2010
  • STRP Festival, Strijp R Klokgebouw, Eindhoven, 2010
  • APPEA Conference and Exhibition, Convention and Exhibition Centre, Brisbane, 2010
  • Energy Field Days, Showground, Sydney, 2010
  • Un Volcan Numérique, Le Volcan Scène Nationale du Havre, Le Harve, 2009
  • Seconde Nature, Espace Vasarély, Aix-en-Provence, 2009
  • International Documentary Film Festival Amsterdam, Zuiderkerk, Amsterdam, 2009
  • International Architecture Biennale, Erasmus Bridge Tower, Rotterdam, 2009
  • Imagining Media at ZKM, Karlsruhe, 2009
  • Immigration Museum, Melbourne, 2009
  • Sydney Festival, Sydney, 2008
  • Biennale of Seville, Carlos V Alambra Palace, Granada, 2008
  • eARTS Festival: eLandscapes, Zendai Museum of Modern Art, Shanghai, 2008
  • Mines Rescue Pty Ltd Australian Facilities in Woonona, Argenton, Lithgow & Singleton, 2008
  • YOUser: the Century of the Consumer, YOU_niverse & Panorama Festival, ZKM Media Museum, Karlsruhe, 2007
  • EMPAC, Rensselaer University, New York, 2007
  • i.Future Festival, Singapore Science Centre, 2007
  • VSMM Conference, UNSW, Sydney, 2007
  • Australian Council of University Art & Design Schools Conference, UNSW, Sydney, 2007
  • AIMEX Australian Mining Industry Conference, Sydney, 2007
  • From Spark to Pixel, Martin Gropius Bau, Berlin, 2007
  • Kaladham Museum, Karnataka, (permanent installation since 2006)
  • La Rivoluzione Algoritmica, Artescienza: Spazio Deformato, Casa dell’Architettura, Rome, 2006
  • Lille3000 Festival, Bombaysers de Lille, Rotonde de L’Opera, Lille, 2006
  • Festival d’Avignon, Avignon, 2005
  • ZKM, Karlsruhe, 2004
  • Cinémas du Futur, Lille 2004 Capitale Européenne de la Culture, Centre Euralille, Lille, 2004

Concept: Jeffrey Shaw

Project Directors: Jeffrey Shaw, Dennis Del Favero

Lead Software Engineer: Matthew McGinity

Software and Hardware Engineering: Ardrian Hardjono, Jared Berghold, Alex Kuptsov, Marc Chee, Robin Chow, Xin Guan

Project Co-ordination and Management: Damian Leonard, Ardrian Hardjono, Volker Kuchelmeister, Densan Obst, Kate Dennis, Sue Midgley, Joann Bowers

Projection system: F20SX+ with the generous support of Projectiondesign

 

This project was financially supported under the Australian Research Council’s Discovery and Linkage funding schemes, the UNSW University Capital Infrastructure Grants Scheme and the UNSW School of Mining Engineering

Dimensions 

Screen Structure: 360-degrees, scalable 3.5m–4m height

Screen Diameter: Scalable 7.5m–10m

Screen Height: Scalable 3m–3.5m

Space Requirement without Projector Truss: 10.4m x 10.5m

Space Requirement with Projector Truss: 11m x 12.5m

Ceiling Height without Projector Truss Mount: 4.6m (Vibration risk if direct ceiling mount)

Ceiling Height with Projector Mount: 5.3m

Server Space: 2.5m x 3m for 2 computer stations and server racks

 

Hardware

Computing Software: Unity 3D Cluster

Projection system: 6 x Barco F35 projectors

Sound system: 5.1 channel sound receiver and speakers

User Interfaces: Tablet, Omnitrack full body motion capture
AVIE: Axonometric View