Queen's Gazette | Queen's University

The Magazine Of Queen's University

2017 Issue 2: The Technology Issue

Search form

Living in the future

Living in the future

Arriving at Jackson Hall is an unremarkable experience. The building’s foyer is dreary: it’s a space to pass through, but no place to linger. If you take the stairs up, you’ll land in another hallway, equally ordinary. If you’re paying attention, you’ll notice the row of boots and shoes lined up outside the first door at the top of the stairs in this otherwise unadorned space. But short of a modest wooden sign that reads “Human Media Lab,” you’ll get very few clues that you’re about to step into the future. Or at least, a space designed to inspire those dreaming about the future...

[team with BitDrones]
An idea made real: BitDrone prototypes hover around Calvin Rubens, Roel Vertigaal, Xujing Zhang, and Sean Braley. The creation of BitDrones by the Human Media Lab team is the first step toward building interactive self-levitating programmable matter. (Photo by bernard Clark)

Pushing through that door takes you into a different world, one in which all the hard edges have been done away with and every surface – from the floor tiles to the wallpaper – has been re-imagined in colourful, computer-rendered patterns. Designed by Karim Rashid, the internationally acclaimed industrial designer, this is a space that feels more like a futuristic showroom than an academic workspace. A large circular couch in fuchsia anchors the bright space. It has enough seating to accommodate the entire Human Media Lab team at once, making it ideal for brainstorming sessions.

“Because we live in the future in our thoughts, so we need to live in the future physically,” says Director Roel Vertegaal simply, as if it’s the most ordinary thing in the world. A professor of human-computer interaction, Dr. Vertegaal spends a lot of his time thinking about the future – specifically, about the ways in which humans will interact with technology in the years and decades to come. Although he’s a computer scientist, Dr. Vertegaal, who first came to Queen’s from Leiden 17 years ago, has always been more interested in people than in machines. At the heart of everything he does is a desire to make technology that is better suited to respond to the way in which humans actually think and move, rather than the status quo, which sees us adapting to our devices instead (watch people hunched over their tiny smartphone screens for a vivid example).

“The human is always front and centre in what we do,” says Dr. Vertegaal, “which is not a classic computer science approach.”

Since opening the Human Media Lab 17 years ago, Dr. Vertegaal and his team – he currently has eight graduate students at different levels and from different disciplines working with him – have captured their share of headlines in the international media, with inventions ranging from the first flexible smartphone and a foldable paper computer, among other organic user interfaces (OUIs or non-flat computer screens), to sophisticated eye input sensors that allow users to interact with devices by looking at them. He has overseen the development of the first “pseudoholographic 3D cylindrical telepresence display,” essentially a human-sized cylinder onto which you can project and interact with an image of a person from a different space (Star Trek, anyone?), and is now fascinated by BitDrones – small quadcopters that function like self-levitating Lego bricks that can be rendered into different shapes in mid-air depending on how they’ve been programmed. Dr. Vertegaal ultimately imagines them as tiny, fly-sized devices capable of locking together to create shape-shifting objects in mid-air.

[inside the Human Media Lab]
Inside the Human Media Lab

As he and master’s student Sean Braley send a group of BitDrones up into the air, it’s clear that Dr. Vertegaal is genuinely enchanted by the products he and his students have devised over the years. He bats a Bitdrone playfully with his hand, only to have it rebound to exactly where it’s been programmed to hang in space. “It really makes me amazed by mosquitos,” Dr. Vertegaal says to the room, his eyes trained on one of the flying devices. “It’s fascinating that they can fly so well, given that they experience air as a viscous substance.” Mr. Braley, who has clearly absorbed some of Dr. Vertegaal’s appetite for inquiry, agrees. “Especially when there’s only so much energy they can store,” he adds, illuminating their own prototype quandaries. “But at least we have laid the groundwork.”

[pseudo-holographic 3D cylinder]
Pseudo-holographic 3D cylinder

For Dr. Vertegaal, whose brain works nimbly and playfully, nothing – not even a mosquito – is too small to provide inspiration. Unlike many in the tech world, Dr. Vertegaal is deeply committed to the physical, in the sense that he envisions a future where specialized technology is adapted to suit and improve life in our existing 360° world, rather than one where humans are expected to live more and more virtually.

“Imagine that you could read the news on your leather purse,” he says suddenly, his mind whirring, “but it would also behave like a purse. Or this couch could be a display, and its primary function would be to be pink one day and blue the next, and that could be related to a wristband that could sense your mood and choose the right colour to suit it.”

Finding form for new ideas

While coming up with ideas is clearly never a problem, the challenge is in giving them physical form, whether it’s finding a way to make manipulating a flexible screen feel like you’re flipping the pages of a book (a very different action from swiping between pages on a flat device), or devising a cylindrically shaped screen you could hold in your hand.

“There has been a drive to move away from the physical to the digital because digital is malleable,” he says, citing newspapers, which are expensive and environmentally problematic to produce, as an example. “But there’s simply nothing like the tactility of being able to leaf through a paper.” For Dr. Vertegaal, the ideal solution is always in finding a way to retain the pleasures of the physical object while embracing the efficiencies of the digital – using, say, an electronic paper you could unfold and read, just as you would a newspaper.

The ideal solution is to retain the pleasures of the physical object while embracing the efficiencies of the digital.

[prototype of phone]
The current prototype of the flexible phone allows users to see objects on-screen in 3D, without having to wear glasses. (Photo: Human Media Lab)

But Dr. Vertegaal is quick to point out that it’s hard to come up with new ways of doing things until you really understand what a product’s existing limitations are. “If you don’t have flying cars, it is difficult to think of new kinds of traffic jams they might cause, because you don’t know their properties,” he says. “But if someone shows you one, then all of a sudden you can start thinking about completely new ways of organizing highways.”

Looking at things in reverse

The key to coming up with inventive solutions, says Dr. Vertegaal – and this is what his students learn to do as members of his team – is in looking at things in reverse. Until he learned of the existence of flexible screens, for example, it was challenging to imagine how such a thing might be used. As soon as he was able to simulate one through projection, however, everything changed. “Then the problems start dawning on you because before that you were living in Flatland,” he says with a laugh. “I realized all software to date was limited to two dimensions. You have to try a prototype, and then you get an intuition for what it can do.”

Flatland: an 1884 novella in which two-dimensional characters grapple with the notion of a three-dimensional universe.

In the case of the flexible phone, Dr. Vertegaal first began experimenting with the technology in 2004. The first iteration was projected on paper. In 2010, he started working with a real four-inch black-and-white display that used e-ink, while the latest boasts a very thin, full-colour touch display. Ultimately, he argues, flexible phones would not only be thinner, less inclined to shatter when dropped, and less expensive to produce, they would be more natural to use, allowing users to bend the display to manipulate on-screen objects – a more ergonomic motion than using a finger to swipe and tap. Current prototypes also allow users to see objects on-screen in 3D, without having to wear glasses. Here, bend is used to point into the screen – the third dimension.

But as remarkable as his lab’s many inventions have been to date, Dr. Vertegaal admits he will be lucky to see most of them applied in the real world in his lifetime. “For us to be competitive in the research domain, we have to think at least 20 years ahead,” he explains. “And we do.” In many cases, however, the marketplace simply isn’t ready for them yet. “Everything we’re using right now is already 40 to 50 years old and has been tested for three decades,” he says, laughing.

Learning to think BIG

When she was looking to continue her education in virtual reality, master’s student Xujing Zhang did her research. “I looked up labs all over the world,” says the student, who already holds a master’s degree in software engineering from Beijing’s Peking University. “I liked the projects in this lab.”

Standing in front of a large screen at a standing desk in a dark blue room, Ms. Zhang taps away at a keyboard. A few feet away stands the top half of a male mannequin dressed in a sweater (its bottom half, wearing pants, stands forlornly in the front foyer), with three small, high-end cameras turned on it. It is this body that gets the honour of being “beamed” onto a cylinder in the lab’s main space. Ms. Zhang, who has studied artificial intelligence, robotics, and augmented reality, is currently fascinated by telepresence and holograms.

Like the other students, Ms. Zhang isn’t here simply for another degree. She is passionate about her research and motivated by the chance to learn new things and complete her projects – no matter how long that takes. She has been at Queen’s for three years now, and loves Kingston’s small size and comfortable pace, which, she says, frees her up for thinking. “What I’ve learned from this lab is that I need to think big,” she says with a smile. “You never know what you can do until you try. If you don’t take the first step, the second will never happen.”

Her enthusiasm is shared by Dr. Vertegaal. “The beautiful moments are when you get the idea, and then when you see the thing work for the first time,” he says. “That’s why I do this. Those experiences are out of this world.”

Taking risks

The Human Media Lab, whose reputation for innovation has enabled it to compete with larger institutions for high-calibre students, is entirely oriented toward experimentation and risk-taking. Students are encouraged to find new ways of doing things, ask each other questions, and help one another explore new ways to solve problems. Dr. Vertegaal sees the group dynamic as “the Socratic method in action,” as his students find a common understanding and teach each other, both at formal brainstorming sessions and more spontaneously by asking each other questions.

Socratic method: the method of teaching inn which the teacher asks a sequence of questions, and by answering them the pupil eventually comes to the desired knowledge.

“Inevitably, two people will get together and start a conversation and then everyone will gravitate toward it,” explains Calvin Rubens, who first joined the lab three years ago as an undergraduate summer student but is now pursuing a master’s degree in electrical engineering. “Everyone plays off one another. The common places [within the lab] are nice for that, because you can hear people talking and then you get ideas.”

Mr. Rubens, who was first drawn to the Human Media Lab because of an interest in drones and has since had the opportunity to help build the BitDrones, describes Dr. Vertegaal as the “visionary” who directs the projects and gives them meaning, while he and the other students do much of the hands-on work, with plenty of room for creativity and autonomy.

[BitDrone]

“It’s exciting work because it’s all new stuff,” he says, adding that he loves being able to apply what he’s learning as an engineering student to coming up with real-world applications. “You get to do the research and have that level of exploration without being a top-level scientist at Microsoft or Google or NASA.”

Turning on the lights

Indeed, it’s training that serves graduates of the Human Media Lab well: many of Dr. Vertegaal’s students become so adept at solving complicated problems in new ways that they often end up designing their own jobs, giving them a leg up in the tech industry. They venture into the world with a future- focus, and an ability to take what they’ve learned and apply it in new areas, just as Dr. Vertegaal did in setting out to bring his idea for a visionary lab to life.

“It’s cool that it has all panned out,” he says, “although I never really had any doubt. I had a confidence that I think comes with experience: as you shift fields, you take something you’ve learned and apply it to that new field. The confidence comes from knowing it already works in another context. Then all you have to do is turn on the lights – and it will work.”

[cover of Queen's Alumni Review, Issue 2, 2017]