Mixed Reality Guide for MasterpieceVR

Written By jeff ross

Part 1: MasterpieceVR Mixed Reality Setup

‍

So you’ve decided to jump into Mixed Reality (MR) - Welcome!  

MasterpieceVR is the most full-featured creative app for MR and non-MR performance available. Once you have your first basic setup running smoothly, be sure to checkout the features below and level up your MR! If you are new to this, the first time through can take some patience, diligence and extra equipment but don’t despair - you’ll be an MR Master in no time!  With the latest tools it’s much easier than it used to be - believe me.

Download: MasterpieceVR Mixed Reality Guide (pdf)

Mixed Reality Definition (for our purposes): The ability to show a 2D view of a real user interacting, in whole or in part, in a Virtual Reality (VR) space

What are the Elements of a Mixed Reality Scene?

An MR scene is generated by sandwiching (compositing) two or more visual layers together to create the 2D MR view:

‍

Overlay (optional)

  1. Shows typical overlay info like watermarks, chat feeds, PIP, and other content.

  2. Useful for live streaming or branding. Eg. twitch, facebook, conference demo, etc.

Foreground

  1. Extracted from the VR environment.

  2. Contains elements from the VR environment that are in front of the subject.

Subject

  1. Video of the person seen to be interacting with the VR environment.

  2. If you’d like to see the subject appear inside the VR environment, you need to separate them in the video feed from the real background by one of the following techniques:

  3. a real world green screen is placed behind and around the subject. The compositing software then uses chroma-keying. This is the most common approach.

  4. Automatic background removal by compositing software. Imposes some restrictions: eg. background must be static, well lit and can’t move the camera position.

  5. 3D camera depth culling (eg. Zed camera). This is experimental and not yet widely available.

Background (optional)

  1. Contains all VR elements behind the subject.

  2. Extracted from the VR environment.

  3. Can be dropped if no green-screen is desired for subject shooting.

Backdrop (optional)

  1. Used to relocate the scene to a place other than the VR environment or subject’s current environment. Eg. sculpting in the Louvre/on Mars.

  2. Can be static or dynamic.

  3. Requires that both the Subject and Background layers be chroma-keyed.

Mixed Reality Style Options

There are two basic MR styles:

1. Subject in VR space

  • Most common style.

  • Shows the subject as if they were inside the VR environment.

  • Great for games where the VR environment is a big part of the action.

  • Requires 3 layers - foreground, subject and background.

2. VR elements in Subject (real) space

  • Shows the subject in a real environment but with elements appearing from the VR world.

  • This is good for creation/performance apps where the surroundings are less important.

There are two very different techniques for this:

1. Simple (no background layer)

  • Background layer is not used, only foreground and subject layers.

Pros:

  • Simplest of all MR setups.

  • No green screen needed for subject layer.

  • Least processing load.

Cons:

  • Forces the subject to stay beside or behind all Foreground elements.

  • Elements that pass behind the subject will typically disappear which is not natural looking.

‍

2. Advanced (masked background)

  • Background layer must consist of only objects of interest with the rest of the VR environment masked/chroma-keyed.

  • Requires VR app to support internal chroma-keying and very flexible MR compositing software.

  • Allows for objects behind the subject to appear naturally in real-time

  • Uses the addition of a 4th backdrop layer to provide context. Eg. the subject can be placed somewhere other than the VR or current real environment. eg. Mars, the Louvre, Tatooine, etc.

‍

What You Will Need

Basic setup

1. Camera calibration/setup software options:

MixCast/LIV:VIVR

  • Can be used to generate the camera calibration file for use with OBS custom/advanced compositing.

Oculus Camera Tool

2. Compositing software options:

Turnkey

  • MixCast/LIV client for turnkey compositing

Custom

  • OBS - free open source for custom compositing

  • XSplit

3. Webcam

  • e.g. Logitech C920 works well

4. Green screen

  • Required to show the subject in the VR environment.

  • Required to properly show objects behind the subject

  • Not required if you only want to show the subject in their real environment and keep the creations in front of the subject

‍

Advanced Setups

1. Third controller (for moving cameras)

  • Tracker for Vive

  • Third Oculus Touch controller for VRObject for Rift (Only works with MasterpieceVR from Oculus Store)

2. DLSR camera

  • Can provide better quality image than webcam but requires an HDMI capture card (eg. Avermedia LGX)

3. Green Screen Room

  • Useful for moving cameras and wider FOV’s

4. Second PC

  • Compositing and streaming can take a fair bit of additional processing power so a dedicated PC for this purpose is useful - especially for complex setups or complex creations in MasterpieceVR.

  • Use it to capture the camera feed and intercept the full screen monitor output from the VR PC. Use third party video system (eg. OBS/XSplit) to do the compositing.

  • The downside is that you will probably need a producer to manage the feed.

  • Do the camera calibration/alignment on the VR PC first, then plug it in to the Compositing PC.

The real trick to setting up MR is calibrating the real camera with the virtual camera.  

Note: If you installed MasterpieceVR from Steam you can use either MixCast or LIV:VIVR to do camera setup/calibration. Currently, the Oculus Store app requires the Oculus MR tools to setup. The free version of MixCast or the free LIV:VIVR will give you the calibration file (externalcamera.cfg) that you will need for more complex OBS setups.

‍

Part 2: MasterpieceVR Mixed Reality Capabilities

‍

MasterpieceVR is a powerful platform for Mixed Reality performances.  From simple webcam setups to moving camera, real-time collaboration from anywhere in the world, and real time streaming rigs, MasterpieceVR is the most full featured Mixed Reality creative app available!

‍

Platform Support

MasterpieceVR provides full MR support when purchased from Steam, Viveport or Oculus store platforms. It is integrated with MixCast, compatible with the LIV MR client (SDK integration coming soon) and all external compositors (eg. OBS, XSplit).

SteamVR & Viveport Apps

  • MixCast integration

  • LIV client compatible (integration coming soon)

  • Standard Steam Quadrant mode for custom external compositing (eg. OBS, Xsplit, etc.)

  • Supports Vive Tracker integration for optional moving cameras with Vive headsets

  • Supports static cameras for Oculus Rift and Microsoft MR headsets

(Note: Third controllers to allow camera tracking for Oculus and Microsoft headsets are not currently supported by SteamVR)

‍

Oculus Store Apps

  • Basic 2 layer Oculus MR for custom external compositing (eg. OBS)

  • Supports VRObjects for 3 controller, moving camera Rift rigs

‍

Mixed Reality In-Game Features

Think of MasterpieceVR as your recording studio space.  Once the cameras are setup and calibrated, MasterpieceVR has several advanced features that help you to customize your MR and non-MR performance/streaming setups well beyond the basics.  A single user can completely control the simplest to the most full-on pro MR live stream without outside help.

‍

1. Enable MR from in-game

All mixed reality modes can be enabled and disabled via the in-game user interface (See options/Mixed Reality).  No need to install virtual third controllers etc. or take off your headset to start/stop.

2. Hide/Show controllers

You can choose to hide or show the in-game controllers depending on what look you are going for.  Eg. Show the palette choices for instructional videos or hide all but the tool tip for clutter-free sculpting.

3. Multiple Angle Display Cameras

MasterpieceVR allows the user to create multiple in-game cameras that replace the standard HUD display on the main console window.  They can be used for MR or non-MR setups to provide user controlled, multi-angle feed switching.  Eg. For the Steam quad mode, the user can change the HUD quadrant to show a side/top down/bottom up shot instead of the HUD display any time during the session.  This gives unheard of flexibility to live streamers. If you do have help, these feeds can also be controlled by an external producer via the keyboard. You can create your own video studio!

4. In-game Desktop

You can use the in-game desktop to display and control your recording and compositing app (eg. transitions, start/stop recording, etc) Yes - you can do it all live in-game without help!  Position the desktop viewer above the virtual camera for reference and use it to control OBS directly to switch between MR views, HUD scenes, side views, b-roll, intro/outros etc. all without taking your headset off!

5. In-game Browser

Want to play a Youtube video in the background while you’re sculpting? No problem!  The in-game browser can be used to show anything you want to your audience while you work without having to setup a separate compositing layer or screen overlay.

6. Reference Images

Importing reference images is a powerful and simple way to give your VR studio a custom look.  Just import a picture and scale it to fill the in-game camera FOV and poof - you’re on Mars!

7. In-game Chroma-keying

Need a more powerful/flexible compositing capability?  No problem - MasterpieceVR also allows you to set the VR background to green (or any other color) so it can be chroma-keyed out in your compositing software.  This can allow for much more complex setups like adding a second PC to do the compositing and streaming while putting minimal load on the sculpting machine.

Part 3: Setting Up your Mixed Reality Rig

Links to help you in setting up your rig:

MixCast:

LIV:

Oculus:

OBS: See below for basics

Generic MR Setup Process

The basic process for setting up for MR is:

  • Setup and calibrate your VR rig as per your headset/store

  • Connect your camera

  • Recommend starting with a webcam (eg. Logitech C920) because it’s much cheaper and easier than a DSLR or camcorder + capture card. You can decide if you want to upgrade once you’ve had a chance to experiment.

  • A DSLR combo we use is Sony Alpha 6000 + Avermedia LGX hdmi capture card. Can conveniently mount a Vive tracker on a cold shoe.

  • Optionally install and configure your third controller as per Steam or Oculus instructions.

  • Only needed for moving cameras

  • Can also be useful for static sessions - if the camera and controller are hard-mounted together then the camera can be stored/moved/repositioned without recalibrating.

  • Calibrate real and virtual cameras (use LIV:VIVR or MixCast studio for Steam)

  • Use Oculus CameraTool for Rift on Oculus store

  • Setup your compositor software

  • MasterpieceVR has integrated support for MixCast

  • OBS is recommended as a free external compositor for advanced use (See below for external compositing with OBS on Steam)

The Simplest Setup

LIV

  • Go to Steam and find LIV:VIVR and LIV Client and install both.

  • Setup your camera and optionally your green screen

  • Run LIV:VIVR and follow the instructions on how to calibrate the camera

  • Copy the externalcamera.cfg file to the root directory of MasterpieceVR (from Steam, right-click on MasterpieceVR, select properties->browse local files)

  • Launch LIV Client, then use it to launch MasterpieceVR - Done!

MixCast

  • Go to Mixcast.me, download MixCast Studio and sign up for a free account.

  • Setup your camera and optionally your green screen

  • Run MixCast Studio and follow the instructions/videos on how to calibrate the camera.

  • Once the setup is complete, launch MasterpieceVR from Steam and enable MixCast from the UI (under Options>MixedReality>MixCast).

  • Done!

Setup Tips:

  • You can use OBS or XSplit Broadcaster to capture the MixCast/LIV Client window for streaming to your favourite site eg. Twitch, Youtube, Facebook live, etc.

  • Completing the setup and calibration in MixCast Studio/LIV:VIVR creates the required externalcamera.cfg file that you will need if you want to use a more complex setup like OBS for direct Steam quadrant compositing. (See below) To get it from VIVR, press the "Export Config File" button. From MixCast, click on the “About” box after calibration is complete.

  • Use LIV Client or MixCast Studio Compositing>Display mode>Buffered to discover and adjust the best time delay for the game feed to match camera feed timing. Ie. if you see the virtual controllers leading the real controllers, try adding a buffer time delay. (Around 200ms works for us in several webcam and DSLR setups)

  • If MixCast Studio or LIV Client doesn’t support your camera feed natively (ie. when you select it, it doesn’t show anything), download XSplit Broadcaster and get the free version (required for some camera feeds)

  • Run XSplit Broadcaster and add camera source (hdmi capture card, webcam)

  • Select Xsplit Broadcaster from the input device list

  • We recommend hard mounting your third controller to your camera so that the offset and rotation doesn’t change while in use or between uses.

  • If you are not using a webcam, try to get a camera lens with a fixed FOV or set it to its max or min zoom range.

  • Warning! Adjusting the zoom, or changing the position of the camera relative to the third controller destroys the calibration! Hard mounting them together and using a fixed zoom lens is recommended. A ¼” shoe mount is a great way to mount a tracker on a DSLR.

OBS Compositing

MasterpieceVR can also be used directly with more advanced (read complicated) setups like OBS or XSplit Broadcaster for compositing and streaming.  It’s definitely not as simple as MixCast/LIV Client but allows for more flexibility and control for the pro or motivated/curious enthusiast.  Below we will use OBS since it’s quite capable and free.  We still recommend using LIV:VIVR/MixCast Studio to generate the required externalcamera.cfg file, since it’s definitely the easiest way we’ve found.  You will also need all of the hardware components from the above setup (camera, third real/emulated controller, green screen, lights, etc.)  In fact, we’d recommend running through the VIVR/MixCast setup process above before attempting this setup since it will solve all of the tricky issues of Mixed Reality and only leave one's specific to OBS.

Caveats:

  • Not all setups, screens, cameras, controllers, capture cards etc. are the same. Using OBS should only be attempted if you’re fairly familiar with it.

  • This setup assumes

  • a main display of 1920x1080

  • MasterpieceVR running in full screen mode

  • A 1920x1080 video feed from your webcam, DSLR, video cam, etc

  • Some tweaking may be required for your specific setup so please take this as a starting point, not a definitive guide!

  • Watch your controller power on order. Getting this wrong can attach the camera to the wrong controller. Restart SteamVR with the correct order.

Preliminary Steps

  • Use MixCast/VIVR as above to calibrate your setup. Ie. align your virtual and real cameras and your camera/controller offset/rotation settings. When finished, this generates a local copy of the externalcamera.cfg file. Redo this step anytime you change your camera/tracker positions or settings.

  • The externalcamera.cfg file is where all of the camera/controller placement and calibration goodness lives. You will need to put this file in the MasterpieceVR root directory once MixCast/VIVR creates it for you. (In VIVR, press the "Export Config File" button. In MixCast go to the “About” menu) and get the externalcamera.cfg file and copy it to the MasterpieceVR root folder. You can find the MasterpieceVR root folder by right clicking on MasterpieceVR in the Steam library, selecting properties from the drop down, click on the “local files” tab and select “browse local files”

OBS Setup

Setup OBS with the following sources in the following order:

  • MasterpieceVR foreground (Game capture or Window Capture MasterpieceVR)

  • Camera feed (Video Capture Device - select your device)

  • MasterpieceVR background (Game Capture or Window Capture MasterpieceVR)

\Expand these to 1920x1080 and apply the following filters and transforms:

OBS Settings for Steam Quadrant mode on MasterpieceVR

1. Foreground transform

2. Background Transform

3. Foreground crop filter

4. Foreground Colour key (removes black without alpha mask)

5. Foreground Delay (adjust as needed)

6. Background Crop

7. Background Delay (adjust to match foreground)

8. Camera Transform

9. Camera Chroma Key (adjust as needed for your lighting)

Tips:

When running, set OBS to use the “Full screen projector (Preview)” then, In-game you can use the Desktop viewer to see what is being recorded/streamed in OBS (assuming OBS is currently on top of the desktop)

Conclusion

If everything went well, the sculptures and controllers should align and you should see a similar image to the one seen in the MixCast/LIV Client display. If not, some of the cropping and chroma-keying may need adjusting.  

If it seems good in LIV/MixCast and you’ve copied over the externalcamera.cfg file, your troubles are likely with tweaking the settings in OBS.  You’re on your own from here though since each setup has unique challenges and it’s impossible for us to test all possibilities. Hopefully you’re close and on your way to advanced Mixed Reality!

Do send us any comments and suggestions though to devops@masterpiecevr.com. We’re keen to pass along any lessons learned and keep this document updated.

Happy Mixed Reality!

jeff ross
Previous

MIXER Update
Next

MasterpieceVR 3G Update - Grid, Guides & Gizmo