Howdy!

I'm

Saad A Akash

Today,

We'll be

talking

about

Mixed Reality

and

Artificial

Intelligence

and

Their

potential

in

future

computing

paradigm

Software Engineer,

Interspeed Digital

---

AI Practitioner,

Intelligent Machines

Former Developer

& Roboticist,

Team Spark

---

Tech Instructor,

Nerd Community

Founding Member,

MakerShare

The official

Global community

Make:Magazine & Intel

Contents

  • Future Technology Trends
  • Virtual Reality (VR)
    • Molecular Data Visualization
    • Architectural Simulation
  • Augmented Reality (AR)
  • Mixed Reality (MR)
  • Cognitive Navigation
  • Contextual Intelligence & AI
  • THE BIG NEWS!
  • Deep Learning for Computer Vision
  • Projects related to MR and AI
  • Platforms, Tools and SDKs
  • Coding a small AR Demo App
  • Related Research Works, Plugins & Resources

Down arrow

AR & VR

Down arrow

AI

Down arrow

So...What's VR?

.

.

.

.

.

.

.

Virtual Reality (VR)

  • The use of computer technology to create a simulated environment
  • Immersive multimedia or computer-simulated reality
  • Implies a complete immersion experience that shuts out the physical world
  • Replicates an environment that simulates a physical presence allowing the user to interact in that world

Immersive

Simulated

The Tool for Future Education

Down arrow

No, VR is not restricted to Video Games only!

VR is being used in

  • Data Visualization
  • Medical Assistance
  • Aviation Training
  • Architectural Simulation
    • ....and so on!

Molecular Visualization

"Required ~700 lines code (500 in Python and 200 in C++) interfaced to the Oculus SDK library"
Ref: http://www.cgl.ucsf.edu/chimera/data/oculus-jan2014/oculus.html

Molecular Visualization

“The most plausible use is to show others features of 3-dimensional data sets from an immersive viewpoint.”

Architectural Simulation

Architectural Simulation VR

IRIS VR

So..

What

Is

AR

?

Augmented Reality (AR)

  • Integration of digital information with the user's environment in real time
  • Adds digital elements to a live view often by using the camera on a smartphone
  • Examples:
    1. Snapchat lenses
    2. Pokemon Go

Visual Overlay

What

Snapchat

AR

Feels

Like

Live View

What

Snapchat

AR

Feels

Like

Mixed Reality (MR)

  • Combines elements of both AR and VR
  • Interaction among real-world and digital objects
  • The merging of real & virtual worlds to produce new environments and visualizations where physical & digital objects co-exist and interact in real time

VR & AR @ Google I/O 17

  • MR: DayDream
  • VR:
    1. Tango - 3D scanning camera system
    2. WorldSense
    3. SLAM - Simultaneous localization and mapping
  • AR:
    1. Google ARCore
    2. Expeditions AR

Google ARCore

  • Motion Tracking
  • Plane Finding
  • Light Estimation


So yes, we need to incorporate cognitive technologies for AR & VR...or, MR!

An interesting research paper on Cognitive Navigation, published on MITCSAIL just yesterday!

Ref: Autonomous Vehicle Navigation in Rural Environments without Detailed Prior Maps, Teddy Ort, Liam Paull, Daniela Rus , ICRA2018

From

Cognitive

Systems

to

AI

The Big News!

4 days ago, Google has open sourced Seurat for bringing high-fidelity scenes to mobile VR

“Today, we're open sourcing Seurat to the developer community. You can now use Seurat to bring visually stunning scenes to your own VR applications and have the flexibility to customize the tool for your own workflows.”

So how can we combine contextual Intelligence with Computer Vision?

See a demo Maybe?

Deep Learning Frameworks for Computer Vision

  • Caffe: Focused on expression, speed & modularity
  • Theano: Can optimize a graph, can generate customized C code or machine instructions and uses GPU for faster computation
  • Keras: Runs on top of TensorFlow, Theano, Microsoft Cognitive Toolkit (CNTK)
  • Deepgaze: Uses CNN and Backprojection for estimating Head Pose, Motion, Gaze Direction etc
  • Torch7: Used by Google Deepmind

    • And yes, TensorFlow.

Let's look at some projects!

Platforms, Tools & SDK

    • Unity
    • Android Studio

    • Google AR Core
    • EasyAR
    • Vuforia
    • DeepAR.ai
    • ARToolkit
    • Aurasma
      • ....and so on!

Let's see a code for a small easy AR app!

We'll be using Android Studio, simple Java codes and implement one of the aforementioned SDKs.

Open up a New Project on Android Studio and head over to AndroidManifest.xml

Steps

  • Download EasyAR library from Github repo
  • Place it in app/libs/EasyAR.jar
  • Open up a directory named "assets" in app/src/main
  • Primarily, put an image (ie .jpg, .png) and a video (ie .mp4) inside "assets"
  • We need some Java classes from provided samples and only 1 XML layout file
  • Open a FrameLayout inside a RelativeLayout on that only XML layout file

Head over to build.gradle of the app and add codes to compile the library files included in the proojects

  • EasyAR also has iOS, Windows, Mac, Unity 3D SDKs available
  • The free version provides Java, Swift, Objective-C. QR code Scanning and Unity3D 4.x & 5.x support!
  • The free version provides Java, Swift, Objective-C. QR code Scanning and Unity3D 4.x & 5.x support!
  • Free version also allows upto 1000 offline targets!
  • However, Pro version allows SLAM & 3D Object Tracking.

Ref: https://www.easyar.com/doc_sdk/en/Getting-StarThe freeed/Setting-up-EasyAR-Android-SDK.html

  • EasyAR also has iOS, Windows, Mac, Unity 3D SDKs available
  • The free version provides Java, Swift, Objective-C. QR code Scanning and Unity3D 4.x & 5.x support!
  • The free version provides Java, Swift, Objective-C. QR code Scanning and Unity3D 4.x & 5.x support!
  • Free version also allows upto 1000 offline targets!
  • However, Pro version allows SLAM & 3D Object Tracking.

Ref: https://www.easyar.com/doc_sdk/en/Getting-StarThe freeed/Setting-up-EasyAR-Android-SDK.html

Related Research Papers

  • An Adaptive Mixed Reality Training System for Stroke Rehabilitation, Margaret Duff; Yinpeng Chen; Suneth Attygalle; Janice Herman; Hari Sundaram; Gang Qian; Jiping He; Thanassis Rikakis, IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2010, Vol: 18, Issue: 5
  • Integrating Artificial Intelligence and Virtual Reality – A Feasibility Study, Rajani S Kamath, Rajanish Kamat , Conference: National Conference on Latest Advances, Trends in Electronic Science and Technology (LATEST - 2014)
  • Study of Artificial Intelligence(AI) with Augmented Reality(AR), Apoorva Jain
  • http://technav.ieee.org/tag/2188/virtual-reality

Resources

Out!