Queen's Gazette | Queen's University

The Magazine Of Queen's University

Search form

The magic of everyday objects

The magic of everyday objects

Dr. Sara Nabil and her team look at physical objects – including technology – in a different, more inclusive way.

Sarah Nabil sews on a sewing machine
Photography by Johnny C. Y. Lam

Imagine you could monitor your household’s water consumption by glancing at a painting of a waterfall and noting how much water was cascading onto the rocks below. No longer just a painting, this interactive home aesthetic might also encourage your household to think more about water usage − perhaps resulting in behavioural changes.

And what if a lamp could do the same, but with regard to energy consumption? A glance at its interactive shade could inform those living in the home about how much the lamp had been used and, consequently, provide a visual indication of the household’s energy consumption.

Aside from prompting us to think about our use of precious resources such as water or energy, what potential could these items have for the psychological state of our homes? What if, for example, a rug could sense the level of noise in the house – particularly the screaming, crying, loud or harsh language – and, over time, visualize this information by changing its shape? A flatter rug might reflect an atmosphere of tranquility, while a crinkled-up, contorted rug might reflect a more turbulent household. Could this provide a visual prompt for the household to modify noise levels?

These are just a few examples drawn from an IKEA-like catalogue filled with household objects that “sense and interact in order to visualize unseen data and behaviour of family members,” says Dr. Sara Nabil, who has a PhD in human computer interactions (HCI) computing and is an assistant professor at Queen’s University’s School of Computing as well as head of the university’s iStudio research lab.

Decoraction: A Catalogue for Interactive Decor of the Nearest-Future is a collaborative project between Dr. Nabil and Dr. David Kirk, professor of digital living at Newcastle University in the U.K. with the advice of Northumbria Design School’s Dr. Julie Trueman. The collaboration took place between 2017 and 2020 and was published in February 2021 at the International Conference of Tangible, Embedded, and Embodied Interaction (TEI’21).

The catalogue explores how interaction can occur between humans and “interactive materials that have sensing and actuating properties,” Dr. Nabil explains − the name itself referring to “decor” and interaction or action of the decor.

“My work with Sara has been hugely engaging and has given us the opportunity to really push creative thinking around interactive futures,” says Dr. Kirk. “The work on Decoraction is truly groundbreaking, opening up discussions around how we might design interactivity into everyday objects and materials. This really opens up the possibilities of how we might interact with technology in the future.”

Decoraction uses “pre-existing technology,” often aesthetic objects in the home, such as wall hangings, lampshades, objets d’art, furniture and rugs, and “intersects them with recent aesthetic IoT devices [internet of things devices, such as smart appliances],” explains Dr. Nabil. This might include using sensors, conductive thread that transmits data/energy, or thermochromic pigments and photochromic paints that change properties with heat and light.

What Decoraction does not include? Gears, motors, or bulky wires. Rather, the technology providing the interactivity is woven seamlessly into the objects.

For example, in Decoraction, Dr. Nabil and Dr. Kirk designed Despot, a lamp with spots painted on the shade’s fabric in blue thermochromic paint, which disappear when heat emits from the lamp’s bulb. As the catalogue notes, “the more spots vanish from the lampshade, the more it raises occupants’ awareness of their energy usage.”

The aforementioned painting that responds to the home’s water use – this is drawn from the Decoraction catalogue as well, referred to as Waterfall, which is, as one might expect, a picture of water cascading onto rocks. As Dr. Nabil explains, the artwork could be connected to a smart meter in the home that measures household water consumption. The readings from the meter are accessed by actuators in the painting, which alter the cascading water displayed through the use of thermochromic paint. The meter’s readings relay information to the painting; this data heats or cools the wall art, and the heat-sensitive paint responds to the change in temperature. In doing so, the painting displays more or less water.

One might be curious as to the style of the painting, since the aesthetic value of the object in Dr. Nabil and Dr. David Kirk’s approach in Decoraction is not overlooked. As the catalogue states, “This multifaceted painting relies on an impressionistic approach that is better suited to interior aesthetic experiences.” Think French impressionist Claude Monet.

“By weaving interactivity into these artifacts, we have to think about aesthetics; otherwise they wouldn’t be acceptable to the potential owners, but also importantly the interactivity seems to enhance and extend the aesthetic potential of the artifacts,” explains Dr. Kirk about the relationship between interactivity and aesthetics.

As for the rug that reacts to noises, it’s called Lither in the Decoraction catalogue, which explains that it “responds to ambient sounds, specifically high pitches of loud voices or noises … deforms as a whole … then relaxes, leaving behind small parts that are kept deformed.” The rug’s ability to react to noise is achieved through the use of shape memory alloy wires sewn underneath it. These thread-like wires correspond to degrees of sound pitch. In this example, the rug is programmed to respond to noise through high pitch detected by a sound sensor, causing the wires to heat and deform, while a lack of noise causes the wires to relax and the rug to slowly return to its original shape. The possibility that Despot, Waterfall, and Lither can support self-reflection and self-care touches on another aspect of Dr. Nabil’s research, which is present throughout the Decoraction catalogue, but is also one that she is researching at Queen’s University: “What would all of this seamless interactivity with everyday objects be useful for?”

Since she was young, growing up in Cairo, Egypt, Dr. Nabil has asked herself, “Can decorative objects do more than simply play an aesthetic role?”

“Since I was a child, I wanted to be a designer. I ended up studying computing instead,” she says with just a hint of regret, noting that both her parents are architects and that they discouraged her from the field, claiming it “wasn’t the thing now.” Instead, she took up computer science, eventually working with various tech companies.

“I loved coding, but there was something empty in my heart. I was considered by my peers to be successful, but I was not happy. I always loved the design of decorative things. At some point I wanted to bring my two selves together – my designer and my developer selves.”

Dr. Nabil eventually formed a PhD proposal to blend technology with the physical environment of interior design, which was accepted and fully funded by Newcastle University. This produced Decoraction.

“People see decorative objects as beautiful things… I see much more than that. They support our well-being; they express who we are and our identity and our personalities; they tell stories and shape memories; they work for the remembrance of loved ones,” she explains.

This may have been especially acute during COVID-19, when the planet was confined during lockdowns. In these environments, without social contact, she says people had to rely on the objects around them for emotional support.

Dr. Kirk expands on that idea, saying, “As a species, humans love things. We fill our lives and our homes with them. We have deep and meaningful relationships to the things we own. Why would we not want to hybridize them, making them part physical and part digital? They could offer so much more with a layer of digital interactivity woven through them, enhancing our relationship to them, personalizing them further, giving them memory and the dynamic ability to respond to us. There is a real possibility of being able to instill a little bit of everyday magic into the objects around us – a little bit of enchantment makes the world a better place.”

“There is a real possibility of being able to instill a little bit of everyday magic into the objects around us - a little bit of enchantment makes the world a better place.”

As Decoraction demonstrates, household objects can do more – should do more – and not just for those living in a household, either. Dr. Nabil’s research includes designing with marginalized groups in order to assess how objects, innovations, and even fabrication technologies (think 3D printers, laser scanners, etc.) might impact them and their community.

“A lot of our work explores empowering underrepresented people and communities,” she explains. “Diversity and inclusion is required for this process to bring to our team new members who have a widened perspective. … Those with different identities and backgrounds.”

Presently, Dr. Nabil and her team are exploring advanced techniques for everyday embodied interaction thanks to a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery grant received in 2021, in addition to a generous Discovery Launch Supplement grant.

The university was especially helpful in acquiring this grant, says Dr. Nabil, adding that Queen’s is very supportive in helping faculty apply for funding grants from federal and provincial governments.

“These grants… require different kinds of writing,” she explains, citing the differences between writing grant applications for NSERC compared with those for the Social Sciences and Humanities Research Council (SSHRC). “Otherwise, I would not have been able to get all of those funding applications approved,” she says.

One application of this research could be related to museums, says Dr. Nabil. Currently, the focus is on everyday objects, particularly in relation to inclusive communities – that is communities that provide equal opportunities for everyone.

Her latest research draws on previous work with people with visual impairment, who have been left out of the conversation regarding handheld devices and security.

A solution? Bendable phones.

“Instead of typing in a password, what if you could bend the edges of your phone in a certain pattern and that could be your PIN?” she asks.

Working with 16 participants with visual impairment, the team created and tested a prototype called BendyPass. Roughly the size of an iPhone, it’s made from silicone and has five sensors. Users create passwords by bending or folding the four corners, and/or pushing a button in the centre of the BendyPass. The team found that not only were bendy passwords as easy to input and as secure as regular PINs, but they were faster to put in.

While Dr. Nabil may produce possible solutions to problems, that’s not her end goal.

“Most of my projects involve experimenting with things or exploring new ideas. It’s not necessarily presenting end products or solutions, but opening up design spaces for the design community,” she explains, like her work with people with visual impairment, a community often underrepresented in today’s technological marketplace. She gives the example of a student of hers, an immigrant to North America, who told her that Alexa – virtual assistant technology that answers questions through voice recognition – could not understand him or his wife, only his children. In this example, the immigrant parents have had to adapt to the technology, not the other way around. She points out that this example illustrates that the relationship between humans and technology is currently tipped in favour of the technology.

“I’ve always believed that technology should be able to change to adapt to us … I shouldn’t be changing my accent so that Alexa can understand what I’m saying,” says Dr. Nabil, who says her team aims to “empower people” by looking at physical objects in a different, more inclusive way.

“I’ve always believed that technology should be able to change to adapt to us… I shouldn’t be changing my accent so that alexa can understand what I’m saying.” - Dr. Nabil.

“[Technology] should be designed to support people’s well-being, their self-care, their self-reflection, self-connection, and social engagement; to feel good about themselves, not focus-demanding, not intrusive. The current technology that helps people to stay connected actually causes them stress. What if something’s more touching than alarming, more emotional than intrusive?” she asks.

It’s another question in a series of questions – but it’s these questions that propel Dr. Nabil and her team as they explore the relationship between humans and technology, between us and everyday objects. In doing so, she asks that we demand more from these relationships, and wonders what possibilities and potential may be revealed if we do so.

Dr. Sara Nabil’s project Soft Speakers: Interaction Design with Digitally Embroidered Fabric Actuators was one of 10 shortlisted for the Interaction Award for the “Expressing” category in the February 2021 global Interaction Design Competition. It is a great example of her exploration of interactivity with physical objects in a seamless way.

Headphones – whether the bulky old-school ones or the new wireless buds – are ubiquitous nowadays, yet not all users’ experiences with them – even with wireless ones – is the same. During her post-doctoral fellowship at Carleton’s School of Computer Science, Dr. Nabil, along with Dr. Audrey Girouard, her supervising professor, and a graduate student, Lee Jones, explored the potential of weaving conductive threads into fabrics to transmit sound.

  • Seamless technology – Soft Speakers use “innovative digital fabrication methods to make them seamless within wearables. We designed fabric speakers using conductive thread, eliminating as much as possible the rigid parts [of sound systems],” explained Dr. Nabil. Dr. David Kirk, Dr. Nabil’s collaborator on Decoraction, emphasizes how important seamless interactivity is, explaining that “conceptually, it forces us to reconsider the materiality of interaction, allowing us to think more expansively about how we actually make interactive artifacts, where digital interaction occurs and the kinds of digital interaction opportunities this affords.”
  • Inclusive Design – “A user who wears the hijab said that it was hard for her to use headphones or earpieces inside [because] they get all tangled,” Dr. Nabil explained about the head covering worn by many Muslim women among other headwear used by people in different cultures and religions. Since Soft Speakers can be customized in versatile everyday things, this innovative technology supports the diversity of users’ gender, ability, needs, and beliefs more than conventional commercial speakers. For example, the design for an interactive hijab prototype she created features two Soft Speakers, one on each ear, embroidered within the fabric pattern, making the sound system almost undetectable. According to Dr. Nabil, this can be expanded to utilize the tactility of its fabric membrane, inspiring applications for accessibility and assistive technology such as hearing and vision impairment.
  • T-shirts that vibrate – One user wanted the fabric actuators – the speakers – to vibrate as he played video games. “Because the speaker has haptic capabilities [technology that stimulates the senses of touch and motion], he wanted the speakers to vibrate on his upper back; his muscles interacting with the game,” Dr. Nabil explains. Soft Speakers, designed in this way, allowed this gamer to feel in his body the explosions he was seeing and hearing as he played.
  • Furniture speakers – A pattern, using conductive thread of a hummingbird hovering near a flower (the disguised speaker) was sewn onto an armchair, allowing a person sitting in the chair to listen to music or podcasts free of headphones or a conventional audio system. “This application is limited to only the person sitting on the chair, not heard by others around, while still allowing the individual to be aware of the shared aural space of their surroundings,” she explains. She adds that the speakers are not loud enough to be heard more than a few metres away. “Contrary to current technology, Soft Speakers are not only aesthetically designed as part of the material or garment, they are ideal for non-bubbling, private interaction in both mobile and stationary scenarios, [since] only the individual(s) wanting/needing the audio output can hear/feel it,” she says.
Queen's Alumni Review 2021 Issue 4 cover