There is a fundamental shift in how we interact and experience the world due to the nonstop digital progress. Aside from mixed reality games, VR headsets, and 5G networks, one of the biggest changes these past few years is augmented reality. As the real world continues to combine seamlessly with the virtual space, what does the future really hold? David Rose, author of Enchanted Objects and SuperSight, sits down with Penny Zenker to discuss both the advantages and dangers of augmented reality and related technologies. He also talks about Better Choice, an app he created to filter out products when you are shopping. Discover what to expect looking forward into the digital future by tuning into this episode.

—

Listen to the podcast here

SuperSight: How We See The Future Through Augmented Reality With David Rose

I am super excited to talk about this augmented reality with David Rose. A lot of people are talking about the future of work and what the future looks like. We are going to get a glimpse of that. Let me share a little bit about David’s background. He’s an MIT lecturer, author, and serial entrepreneur. He is a five-time entrepreneur, who draws on culture, design, travel, and music to envision the future products and businesses that are sparked by the next generation of technology.

His book, the Enchanted Objects, is a definitive book on designing the internet of things. He has a number of other very interesting books out there. He’s also written a seminal patent on photo sharing and founded an AI company focused on computer vision and was also a VP of Vision Technologies at Warby Parker. We are in for a super treat with David Rose. David, welcome to the show.

Thanks for having me on.

We are talking about the future and what the future looks like. I find this very interesting. A lot of people are talking about this the future of work and the fear of so many jobs that are going to be eliminated because of AI. It feels like there’s a lot of fear about what’s coming in the future. I get a lot of excitement from you. I get to feel that there are a lot of interesting things for us to think about that can replace that fear. Are you positive about where the future is going and how technology is guiding us? Are you gloom and doom where this is all going to take us over if we are not careful?

I try to be balanced, although I’m naturally optimistic. What I write in my book called SuperSight is about how these technologies will help evolve how we see. In the same way that we have been using glasses forever to correct our reading prescription or visual acuity, we are starting to be able to mix what we see with the naked eye with virtual layers that allow us to see differently and visualize faster.

One of the projects that I talk about in the book that I’m working on is something that helps us see sustainable landscapes in a way that we wouldn’t be able to see now. If you look at a lot of homes, they have wall-to-wall carpeting in the grass. It’s not good for the environment. People use a lot of chemicals to take care of their grass. It doesn’t capture a lot of carbon. It requires a lot of water.

A lot of states are getting people to pull up the grass and put something else in but they don’t do it because they don’t have the imagination to see how the lawn could be different. We are using augmented reality so that you can hold up your phone and see shade trees, naturally pollinating bushes, outdoor furniture, and lighting. There are companies like Home Depot and Lowe’s that love that ability to have an instant redesign of some part of the world and then have people become inspired to make a change that’s both good for the value of their home and also good for those retailers that want to sell you things.

That’s hugely valuable. Envisioning a house needs work. I know a lot of people will not buy a house because they can’t envision what it could look like if they put a little bit of work into it. A fresh coat of paint, take a wall down here, and things like that. It’s very interesting what you are saying is that we could put a phone up over some of these different applications to help us to see what we are not seeing to get the best out of the environment. What are some other applications of that? That’s fascinating to me.

What’s happening with the company home outside that’s doing these algorithmically composed reimaginings of people’s landscapes is you need a system that perceives the world. Phone cameras and increasingly LiDAR which is built into the phone, and then a 3D engine that does the imagination part for you and composes a scene in a new way to give you this vivid data layer on top of the real world.

Those technologies are built into phones now. One of the reasons that I’m excited about the category is that increasingly they are being built into glasses. When I was at Warby Parker, my job was to think about what the future of technology-enabled glasses could look like that would not only change and become sunglasses when you needed sunglasses but also have AirPods embedded in the temples.

Bose makes some, and then also do the mixed reality of the future where the new information is glued and grounded on the real world. What we are seeing now is in terms of efficiency. The simplest use case for this is navigation. Have you tried the new Google AR view where you hold up the phone and it puts big signs on the world and allows you to see through the camera and then says like, “You might be disoriented because you popped out of the subway, but this is the way you need to go in order to get to your next meeting?”

That’s a very situational awareness application of the technology. I think the next one that’s coming is the collaboration where how many times have you held up your phone to share your view with somebody else so that they could say like, “This is how you fix that. This is how you adjust the thermostat.”

I saw a plumber that came to my house and he was doing that with a more experienced plumber who is not at the house saying like, “How do you fix this Kohler faucet?” That remote collaboration and telepresence certainly is coming into healthcare for dermatology, but also to so many situations where you can share your view of the world with someone. Then ideally, they can annotate the world and their marks stick to the world.

It’s real-time. We have YouTube where people share their screens of how they are doing stuff or whatever, but that’s seen after the fact. This is live and collaborative in nature. That’s the difference. These little tweaks as to how things change. Am I understanding that correctly? Before the thing, we were talking about ways of collaboration where people are sharing information, so multiple views even of the same thing can be coming in at the same time.

That brings a lot of efficiency for people because not all the experts have to be present in order for you to get advice on that 5% of the project where you need a couple of other eyes on whatever you are designing, fixing, or repairing. It might be the most dangerous part of the job when you are up on the pole or down in the water or whatever you are doing. That’s how it promises to reinvent the future of work. You can distribute expertise in a more ad hoc way.

To better understand that like talking about how collaboration grew and morphed. Wikipedia isn’t obvious but I’m thinking one step earlier than that there was a reference that I remember somebody had given, and maybe you have some thoughts about this. You’ve got Wikipedia where everybody’s bringing in their knowledge.

You had Unix and all these like open-source type of software that people are creating. There’s a lot of interest in this free collaboration space. How do you see those types of collaborations or companies’ major projects? How do you see them opening up to collaborate outside of their own company? Is that something that you see here or is it more of a public sharing?

I do see that there are many new businesses that are parallel with what’s going on in healthcare where you are getting a consult from a dermatologist that used to make moving your body into the healthcare system and waiting in a room. Now you can it do in less than five minutes. You can get a diagnosis and a prescription for whatever you need very quickly.

That same fast remote telepresence use case works in a gig economy as well. It’s like you say where it’s not only people within your organization, but if you want to do some pair programming with somebody, if you want somebody to beam into your situation to help you with a repair or a shopping experience, that help will be more available in a way that’s in time and the friction costs are much lower when someone can beam in.

What came up for me is when you talk about it in the glasses and getting this real-time is the idea of, “How does AI play a role?” I could envision that. What comes up for me is you’ve got multiple views. How will AI be integrated into this to supercharge it from what we already see?

AI is required for powering the perceptual system where when you look through your glasses, you automatically are scanning the scene and you have a scene understanding of what’s happening in front of you. You have a depth map. You know what’s where. In order for a computer to be able to see what you see and perceive what you perceive, there’s a computer vision technique called semantic segmentation, which means every pixel in the scene is labeled by the computer.

For example, we are on a boating application right now, and it knows in the scene in front of you as you are looking out of the front of the boat like what water is and what sky is. What is a small canoe that you should not hit a kayak or a jet ski? Not only what is what, but also what is heading in which direction and what represents risk.

All of the self-driving car technology is trying to create a real-time seeing and understanding of what’s around you to figure out where to navigate and to go. What people and bikers to continue to avoid is coming to our glasses as well. That will help us read other people’s emotions. It will help us know everyone’s name. It will help us to quickly come to a conference to understand what the commonalities are between why you are here, why I’m here, and how to approach you. We’ll have rapport faster.

For a sales perspective, you are saying that I could like look at you and I could see in my glasses your personality type and some word suggestions on things that would create rapport between us.

It doesn’t have to get all of that from your face. It could know that you are Penny and look it up on LinkedIn and say that the bizarre hobby that you both have is you both grew up in rural North Carolina and you are both into kiteboarding. Nobody else in the room is into those things. There’s this idea from data science called least common commonalities and it turns out that the things that create the most rapport the fastest are the things that exquisite party host does. When they say, “David, you should meet Penny. Both of you guys are interested in this.”

Augmented Reality: There’s this idea from data science called least common commonalities. It turns out that the things that create the most rapport the fastest are the things that an exquisite party host does.

It’s like an obscure thing that draws us both together. In terms of conversational scaffolding too, it can give you tips like you are dominating the conversation or asking her more about this. Here’s a good empathetic follow-up question or reframing. My wife is a psychotherapist and has lots of good techniques for people in conversation.

I have always wondered. Do you feel like you are being analyzed all the time?

I am.

This has incredible potential in a lot of areas to help us when we come back to the idea of productivity to get to the core of what we are looking to focus on and get there faster. Achieve rapport faster. Achieve our creative ideas faster and so forth. What are the side effects? That sounds good but maybe I feel like nobody’s genuine anymore because maybe you are seeing this to manipulate me. Do we then live in a society where we don’t feel like people are authentic or what are some of the side effects?

The definitions of authenticity are already sliding. When you remember people’s birthdays, or when you answer emails using Crystal on LinkedIn that tells you to like, “Phrase this in a certain way because this person responds to data more than emotional stories.” We already have lots of tools that help us to engage on the right channels with the right types of messages. I feel like it’s a big level up for a lot of us. Is it unfair to get to a party with a GPS? No. You are more likely to get there on time. That doesn’t seem like an unfair advantage.

You are not manipulating anyone. I do think that as a society and I have no proof about this, so I’m speaking out of my butt on this. I feel like people are more critical because of all the ads that follow you. Is this person genuine? People are contacting me left and right on LinkedIn saying these things. They don’t know me and they don’t care to know me. They just want to pitch me. The people are more critical. They are more apprehensive. I’m curious, does this make it worse?

It gives us more tools and those tools can be abused. ChatGPT can help us get to a first draft faster so that we can then make our own. It lifts all boats to a C-level, and hopefully, that will lift the A-levels into higher levels as well. There is this notion in computer vision that’s the opposite of augmented reality, which is diminished reality. This is when the AI can see what’s in a scene and automatically mask or swap out certain things from the scene.

In many situations, we need a little bit of diminished reality to apply some focus. I have two kids in my family that have allergies. If I go to the grocery store, I do not want to see 90% of what’s at the grocery store because it has gluten, it has dairy, or it’s not kosher. There are a bunch of filters that I have to apply and I would love to not see the things that I shouldn’t be buying because it’s going run into somebody’s allergies. That’s a good use case for diminished reality and I can think of lots of negative use cases for diminished reality too. It will give us the strategic blinders that a lot of us need in order to focus on the right things at the right time.

What would they look like? The shelves would be empty except for the one thing that I’m looking for. Is that what it is? As I walk in, the shelves are empty. I look like I’m in a place by myself.

Nobody has designed this yet. That’s the exciting part. I created an app that allows you to pick up something. It’s called Better Choice and you put your phone camera on it, and then it swaps out the one you should be holding in that category like the granola that you should be holding because it matches your allergy profile. That’s one way of searching and replacing in the world.

Augmented Reality: Better Choice allows you to pick up something. You put your phone camera on a product, then swap out the one you should be holding in a specific category.

I love that. It’s called Better Choice.

Yes. It’s a prototype. We are doing it with Whole Foods.

Keep us posted on that. I love that idea because that has so many positive applications in terms of what your goals are if you might have allergies or it might be that you can swap out and find something that’s within your allergic range, but also the calories, healthy, and milk substitute. That’s fantastic. I love it.

It’s a faster way to get to the right thing.

Could it also be when I’m watching TV that it says, “Better Choice. Get out and do something. Exercise.”

Maybe. We build a lot of those nudges so they have to be tastefully done. The meta-comment about that is if you are a product manager or interested in innovation and design, it’s this whole field of how do you mix the real world with virtual layers? How do you augment? How do you diminish? All of that is a brand-new design space that is exciting for a lot of people. Maybe you are a web developer or a mobile developer. This will be the next platform for computing that will be the next several years, and there are lots of tools to start prototyping. Adobe Aero is one. Apple’s Reality Composer is another one.

Are these for developers?

Tools and techniques. I teach workshops and those tools are simple enough that there’s no coding required. You go to 3D asset stores. You anchor this over images, videos, or objects. In GPS, GPS locations in the world. You can start to experiment with how to morph the world around you.

This is so interesting. We could go on for a long time here. Let me back up a second and then, we wrap it back around productivity and maybe that’s around collaboration. How do you define, with your augmented reality brain, productivity and why?

For me, as I’m working on projects that involve diverse teams of people that are making physical hardware. Oftentimes we are doing a camera system. We are doing a camera system right now for boating that sits on top of a boat. It has hardware, software, computer vision, algorithms, and some connections to people that make boats and people that understand GIS software. I want to get to the team that has the right knowledge in order to come together to make something extraordinary fast. The biggest variable there is finding the right people who have the right expertise and finding the right tools and the right data all at the same time and all quickly.

That’s the definition. That’s how we find the right team, tools, and data at the right time to create something extraordinary. How do we do that? Maybe you’ve already said it and we are summing it up. What do you think is the best way going forward to achieve that?

We still need a better expertise location. LinkedIn and GitHub are a way, but there still needs to be better tools that help organize the who knows what and how do I find that team member that I need for these two weeks in order to build a special computer vision classifier that’s good at differentiating between waves and boats. That’s what I’m looking for. The way that I find them is I look at academic papers, conference presentations, LinkedIn, and who knows who. I would love a faster way of navigating that tree.

There needs to be some platform dedicated to people who want to work on projects and categorize and classify their skills and these major projects.

Fiverr is trying to do that a little bit but usually, it’s people that have computer vision skills.

Take one of those types of platforms. There are some other ones that are out there too that have contractors. I don’t know if they have this level that they have 3D capabilities and that stuff, but I could see that. I could see that’s where we are headed as there will be those specialized platforms where people will come to collaborate. It looks like a project for you.

Once you find people, it’s hard to have them jump into a project, understand the goals, and understand scrum methodologies. It takes a long time to spin up a Trello board with a team as people are coming in.

Getting them onboarded quickly.

That’s the challenge.

We are up for the challenge. Anyone who runs major projects that are global comes with those types of challenges as team members come and go, keeping everybody up to date and moving forward and projects are complex. Anyway, thank you so much for being here. Is there anything that I missed that you feel we should cover before we sign off?

One of my hopes and visions for SuperSight is that other people can more quickly and more agilely beam in to see what you see and understand in a more intimate way of what you are doing, what you are challenged with, and where you are in the world. I seek this holographic projection that will change meetings. You’ll be able to rather than scheduling a 45-minute thing or a half an hour check-in, it’ll be more like taking a walk with me or observing what I’m observing, and we won’t stick to this calendar in the same rigid way that we stick to it now. Meetings will be more like a few minutes, but many more times a day, and where you can pop in and out of other people’s heads.

SuperSight: What Augmented Reality Means for Our Lives, Our Work, and the Way We Imagine the Future

If you don’t want to go there because I’m going to tell you that’s all sorts of worse than what we have. Structure helps people to focus. If people are popping in and out of other people’s focus time, we already have that problem with multiple chat platforms and everything. I think the burnout that you see now is going to be much worse in the future because there’s no turning things off. There are no stopping people. I don’t want to say depth to meetings because meetings provide a structure. I don’t mean to play the combative role.

It’s true. I do believe the meetings provide a structure, but it depends on what you are trying to achieve. If you are stuck, if you need somebody else to beam in and look at your code, look at the mechanical engineering problem that you are having right now and you need their advice, the gig economy model to get somebody to look at that or five people to look at that in parallel and tell them what they think will help unstick you faster.

What you said was the switching cost and the transition cost comes down because you are very quickly into what someone else sees. I could see the value in that. “I’m going to Slack you. I need you to have another set of eyes on this.” The other set of eyes then is there. I could see that as expediting the process that already might exist now and making it a lot faster.

I hope that it’s used wisely and that we don’t end up in a structureless type of usage where it creates something. In the pandemic, the multiple chat platforms are part of what’s burning people out because they don’t know which platform they should be on. There are no rules about what’s communicated where. That’s part of what’s left people burnt out and constantly distracted.

It’s true. If you are in a situation where you are up on a pole trying to fix something or you are boating through a narrow passage and you can’t control the time, you need a little bit of expertise right now. That’s very helpful.

This was great. I’m excited because you talked about seeing what you see. Thank you for helping me see what you see. It wasn’t something that was on my radar of thinking about the possibilities with AI. I was looking at different things and how we use our devices and glasses. I got glasses so now I’m part of the club. Thank you for opening up my eyes and everybody else who’s reading now.

I hope you have a good focus in the next weeks.

Thank you. Where can people find out more about you and take a look at the books that you have?

SuperSight.word is where people can learn about my background. They can look at the workshops that I do, see some of the projects, and see some design principles if they are interested in designing something with AR.

Thank you for being here and sharing all that you’ve shared now. I’m excited about all the projects you’ve got going on.

You are welcome. Thanks for the conversation.

Thank you all for being here because, without you, we wouldn’t have a show. I hope that your eyes are open now, and that you see some of the possibilities of the future and that you see also what you can do with it and how you can create it. Also, how you need to use it wisely and in a way that’s going to create more productivity versus creating more distraction like anything that we look at in the future. Thank you all for being here. We’ll see you in the next episode.

Important Links

• David Rose
• Enchanted Objects – Amazon
• Warby Parker
• SuperSight
• https://Amzn.to/3XU4fyG

About David Rose

David Rose, MIT lecturer, inventor, and five-time entrepreneur, draws on culture, design, travel, and music to envision future products and businesses sparked by the next generation of technology. His last book, Enchanted Objects, is the definitive book on designing the Internet of Things. David wrote the seminal patent on photo sharing, founded an AI company focused on computer vision, and was VP of Vision Technology at Warby Parker. He is known for translating complex technologies into delightfully intuitive new products and consulting with businesses on how to thrive during digital disruption.