Buying your first green screen became a rite of passage for mixed reality content creators. It was that first hit, and before you knew it you were upgrading your stream with new lights, GPUs, and painting an entire room in your house green for Beat Saber streams.
This week, we’ve eliminated the need for a green screen to make mixed reality content: Segmentive Keying is here!
In short, Segmentive Keying removes your background for you, with no green screen required. Using machine learning, it is able to detect humans and keep them in the shot, so everything around you is removed and becomes your VR space!
How in the world?
We were able to pull this off thanks to the help from Segmentive, a team from Germany that works on people segmentation machine learning models. The LIV team and Segmentive have teamed up to provide a segmentation model that works well for mixed reality.
If you’ve tried the LIV.tv iOS app already, you’ve already used Segmentive! We anticipated the high demand for an Android mixed reality app, so one of the reasons we’re using Segmentive is that they’ll make a great solution for the LIV.tv Android app in the future.
Fake it ‘til you make it!
If you don’t have room for a green screen, or you want to try out mixed reality streaming before committing to buying tons of equipment, Segmentive Keying is for you.
Although a green screen will always provide the best, studio quality chroma keying, we wanted to release machine learning-based keying on PCVR to give everyone a way to capture your favorite VR games in mixed reality.
With that being said though, here are a few tips to help give the LIV PCVR app an easier time cutting you out of the background with Segmentive:
1. Just like a real green screen, make sure your room is evenly lit with lots of light! Try not to mix indoor and outdoor lighting. Both kinds of light look the same to our eyes, but they actually have different color temperatures, which we’ve found can trip the app up.
2. Have as plain of a background as possible. Segmentive still works great if you have bookshelves, your bed, or a couch in the background, but if you can get in front of a clean mono-colored wall, it will have an easier time detecting what’s not a person.
3. Similar to tip 1, try not to have too many shadows on your body. Flowy dresses get cut out on accident a bit more compared to t-shirts. My plain t-shirts were detected perfectly and kept in the mixed reality shot.
4. You’ll need a stronger PC! Rendering mixed reality, running a VR game, and OBS for streaming and recording all at once is already a huge ask for a gaming PC. Segmentive adds additional CPU strain, so you’ll need a more modern CPU for smooth video in mixed reality.
If you’re using Quest-on-PC however, your PC is less important!
If you're really trying to squeeze out a bit more performance, lowering your output frame rate from LIV to 30 or 60 should help.
Don’t have a beast PC? That’s okay, with Segmentive Keying available on PCVR and the LIV.tv iOS app, you can use a gaming PC or iPhone to capture your mixed reality gameplay, and now neither solution requires a green screen!
We're always trying to make LIV accessible to more people. Our work on Segmentation is more of the same: enabling those without green screens to use LIV, regardless of what camera they have! - Dr. Doom
Segmentive works independently of LIV too! It can remove backgrounds in real-time for anything, we just happen to use it for mixed reality. If you want to learn more about them, check out their website!
With Segmentive Keying now available on PCVR, you now capture yourself with or without a green screen in mixed reality, avatars, with a PCVR headset or Quest using Quest-on-PC, or ditch the entire PC setup and use an iOS device with LIV.tv!
We’d love to see what you capture. Make something cool and show it to us @LIV on Twitter, or on join our Discord to show the entire LIV team what you’ve made!