Internet of Things and People

The research area interaction technology deals with how users interact with the connected devices. Key questions in this research area include:

• How can people interact with hybrid things?
• How do you design place-specific interactions?
• How do you leverage the co-creation of hybrid things?
• How much of the infrastructures should be accessible to different users?

Interaction design, as well as computer science, has a legacy of addressing primarily virtual interaction surfaces. Innovative interaction models such as gestural interaction, gaze tracking, and augmented reality are established elements of seminal IoT research. They form important parts of the design repertoire being explored in Interaction Technology research. Moreover, in a broader perspective, the emergent infrastructures of the Internet of Things imply that the hardware increasingly becomes part of what can be designed and modified in the context of new products and services – in other words, the “design material” changes from being a predominantly virtual one to a hybrid physical/virtual one, cf. embodied interaction.

The products and services being designed and developed become hybrid things, where physical form and digital hardware – as well as the software – are open for change and experimentation.

This in turn raises a whole range of research questions at the borders of interaction design and computer science:

How can people interact with hybrid things?

At the most general level, it is clear that new hybrid materials enable new interaction models that need to be explored and assessed. Specific technologies, such as motion and gesture-based interaction and tangible interfaces, are merely indications of this new area of design-led inquiry that addresses embodied interaction in various ways. Mobile devices, such as smartphones, are expected to serve as important transitional interaction points to hybrid things. However, insights regarding their significance in the long term is that new hybrid materials actually drive innovation in terms of new interaction models.

How do you design place-specific interactions?

When designing interaction surfaces in software on generic mobile devices, there was an understandable focus on anytime-anywhere interaction. Since the hybrid materials make it possible to design physical form and digital hardware, it becomes relevant to consider products and services that are not generically mobile but rather embedded in architecture and urban space, tied to specific places and to physical structures. Previous work in ubiquitous computing and mobile services demonstrates that a great deal of design and research remains to be done in order to better understand the potential of designing place-specific hybrid things.

How to leverage the co-creation of hybrid things?

IoT is not a simple one-way operation of connecting existing physical things to the Internet. Rather, recent developments in 3D printing, open-source hardware, and other collaborative-creation frameworks illustrate how physical things can be open-sourced, disseminated, and transformed to degrees that only software was previously capable of. What this means is that the power of grassroots creation and open source will have an increasing impact on physical products and hybrid things. Research is needed to explore the potential of this development for product development, building and construction, urban planning, and a range of other domains into which IoT infrastructure extends.

How much of the infrastructures should be accessible to different users?

Traditionally, in the physical world as well as the virtual, infrastructures have been considered part of preconditions, receding more or less into the background. The hybrid physical/virtual nature of the IoT implies that infrastructure becomes customisable, visible, and generally much more present in the design and use of products and services. This has positive as well as negative consequences. On one hand, it empowers users to tailor their environments to their needs, but on the other hand it sometimes entails unreasonable requirements on users to install, set up, update, and tweak. Such trade-offs and other consequences of the emerging infrastructures must be explored from an interaction design perspective.

The research area embedded intelligence investigates how intelligence embedded in the devices can improve the usability and functionality of IoT services and products. Key issues in this research area include:

• intelligence of the individual devices
• interaction between entities
• management of data

Intelligence of the individual devices

Since large volumes of data will be generated in the IoT, it seems reasonable to embed mechanisms for data processing and decision-making in the devices (or close to the devices, e.g., in a gateway) to enable decentralised processing of information. In some applications, there are real-time requirements that may be difficult to handle by remote processing at a server or in a cloud. Also, the robustness of the system can be increased if processing is distributed, as there often is no critical single point of failure. Thus, the scalability can be enhanced in several ways if intelligence is embedded in the individual devices. The research can be characterised as 'modern' Artificial Intelligence (AI) where the intelligence is embodied in an 'intelligent agent'.

IOTAP focusses on the following crucial issues concerning the intelligence of the individual devices:

• What are the appropriate models for context-awareness in IoT applications? A device should be able to identify in which situation the device and/or its user currently is, and use this to bring added value. For example, context-awareness can be used to adjust the level of autonomy so that the degree of automation depends on the current situation and user preference or ability, or to adapt the interaction surface.
• How can the behaviour of the devices be improved through learning from experience? This could be achieved by adapting the behaviour based on direct feedback provided by the user. Another approach is to improve decision-making by learning from sensory input in a more autonomous manner. Moreover, devices may learn from each other, for example, by sharing their experiences. In some situations, it is important that the algorithms comprising the intelligence of the devices are efficient, e.g., when a large amount of data needs to be processed and the response time is crucial. Also, the power supply limitations of many devices place further requirements on efficient processing and communication.

Interaction between entities

There are a number of interesting research questions concerning the interaction between entities. Agreement technologies refer to computer systems in which entities, often called (software) agents, negotiate with one another, typically on behalf of humans, in order to come to mutually acceptable agreements. Semantic web and ontologies are two important concepts to realise such systems. Some other key concepts used to specify and verify such systems are semantic alignment, negotiation, virtual organisations, norms, and obligations. Agents can be seen as encapsulating functionality (possibly implemented as web services), and can be orchestrated to create new, higher-level functionality, which could be deployed and executed in a distributed fashion. Such orchestrations are often referred to as multi-agent systems.

Important research questions that IOTAP addresses in relation to interaction between entities are:

• How can relevant devices and services automatically be discovered by other devices and services? When more and more devices are connected to the Internet, the problem of finding the relevant one for a particular situation becomes more difficult, especially in new environments. To reduce the cognitive load of the user, service discovery of this kind may be carried out by software agents that do not need continuous user intervention.
• How can we make the devices, services, and users understand each other in an IoT context? The units need to be semantically interoperable, for example by using ontologies and other semantic web technologies. This is important, for instance, to implement service discovery functionality, as well as for making use of data that have been made publicly available. Also, in order to achieve a task, an agent (a device or a service) often needs to collaborate, or at least coordinate, with other agents. An interaction protocol defines a set of rules that guides the interaction between agents for supporting structured collaboration and negotiation in order to achieve a common or individual goal.
• What are appropriate architectural designs for heterogeneous distributed IoT systems? What is the best distribution of functionality between smart devices and the supporting infrastructure? How can we make complex open IoT system of systems more robust? For instance, reliability may be increased through redundancies that can cover up for the loss of entities, or norm systems that guide the behaviour of the entities can be introduced. Moreover, risk analysis can be made beforehand, as well as validation through simulation. Another challenge is how to identify and avoid potential undesired effects of embedded intelligence with respect to resource utilization and user behaviour. For instance, bad system design could cause unnecessary system resource utilization peaks or undesirable influence of user behaviour.

Management of data

As the number of connected devices grows at a fast rate, the amount of data they generate will explode, in particular different types of sensor data. Regarding the management of data, we will address the following questions:

• What are the best ways of utilising the large amounts of data that becomes available? Different methods for achieving this will be studied. For instance, we will investigate data mining methods for extracting relevant data and discovering patterns, data fusion methods for combining different data sources, and visualization techniques to make the data more comprehensible for human users.
• How can individual user privacy be sustained in a super-connected and data-intense surrounding? In principle, it is desired that as much as possible of the data generated is made available to other services and devices. However, the ways in which user-related data can be collected, analysed, and utilised for various purposes – benign and commercial, as well as selfish and malicious – in a digital ecosystem are uncountable. It is thus important to study the need for and the design of privacy-enhancing mechanisms in order to ensure the privacy of the users. In this respect, other aspects directly linked with human interactions, such as the management of data ownership, the gathering of consent from users, and the robustness of data security methods are also important to study.

This research area explores how users can be involved in the development of new IoT services and products. Key questions in this research area include:

• How can existing methods for user-centred development be adapted to the IoT context?
• How can user feedback be efficiently integrated into the different development phases of products and services?
• What are the most efficient techniques for continuing to improve IoT products and services also after deployment to users?

As IoT will become widely deployed, is important to better understand the needs and requirements of users and how these can be efficiently integrated throughout the development process. Traditionally, users have been actively involved in pre-development phases, i.e., in requirements engineering and in prototyping activities, and there is extensive research within areas such as participatory design, practice-based design, contextual design, and collaborative design on how to involve users as co-designers. This phase is critical and with new areas of deployment for IoT, as well as emergence of new technologies, there is a need to explore new and innovative ways to involve users in the early stages of development.

Moreover, embedded software with sensor technology and communication and processing capabilities allow for users to be actively involved also in the post-development phase, i.e., after the product and/or service has been deployed. During this phase there is the opportunity to continuously collect data on user behaviour and, as a result, continuously improve the quality of new products and services based on real-time user feedback. Thus, there is the need to find mechanisms to collect, analyse, and experiment based on user feedback.

In this way, the area of user-centred development will involve both the pre-development phases in which users provide feedback based on active collaboration with designers, and the post-development phase in which users provide feedback based on their real-time usage of products and services.

There are a number of interesting research questions related to user-centred development:

• How can existing methods for user-centred development be adapted to the IoT context? IOTAP studies how they relate to (1) contemporary developments in Living Labs methodology and other forms of participatory design for heterogeneous constituencies, and (2) to the increasingly hybrid physical/virtual nature of new products and services (i.e., new design materials). IOTAP also studies how established methods for rapid prototyping and collaborative exploratory design can be adapted to the new materials. One aim is to continuously learn from user groups and user behaviour patterns. With an increasing number of digital products and services it will be possible to improve our ways of learning about user behaviour and how products and services contribute to specific contexts.
• How can user feedback be efficiently integrated into the different development phases of products and services? Solutions include ways in which user feedback can be collected before any development starts. What are the feedback mechanisms necessary for continuously collecting user data? Solutions may include tools, techniques, and organizational processes that allow the organization to more rapidly receive user feedback on functionality before the product/service is fully developed. How can agile and lean development practices facilitate active user involvement in pre-development of IoT products/services, during development, and after product/service deployment? A specific question of interest is how insights from Living Labs and other participatory design practices can inform existing development methods and approaches.
• What are the most efficient techniques for continuing to improve IoT products and services also after deployment to users? How can user feedback be efficiently translated into improvement of current functionality, as well as future innovative products and services? Specifically, how can the deep levels of sustained user participation associated with Living Labs and related practices, and related concepts such as continuing-design-in-use and infrastructuring, inform existing notions of feedback and continuous improvement in contemporary deployment-driven methodologies?

The integration of Internet of Things technologies and urban development places Smart Cities in a good position to address contemporary societal challenges. Applications include sustainable energy consumption, traffic management, waste management, public safety and security, and other city-level issues. But equally important are other types of value creation, such as including ICTs in government systems and bringing ICTs and people together to enhance the innovation and knowledge that they offer.

Projects include ECOS, CoSIS, SHINE, UseIT, Smart Public Environments

Contact: Per Linde

Health is the most valued aspect of life. By using IoT technologies to measure biometrics as well as other context cues, Smart Health can empower people to proactively engage in their health as well as to manage their recovery from illness or injury. In addition, Smart Health can be used in healthcare settings to involve patients in their treatment and to share data to improve outcomes. Thus, Smart Health is about using IoT technologies to enable and improve health-related services using a network of context-aware things.

Projects include TagOn, AppSam, Framtidens sport, CACT

Contact: Nancy Russo

For a more detailed introduction to the Smart Health area, please see Nancy Russo's article Smart Health and IoT: Opportunities and Challenges.

What IOTAP is doing in this area

At Malmö University, we have a track record of research in the Smart Health area. Here are a few examples of initiatives we are or have been part of:

• A Smart Health Check-Up | We hosted a Smart Health workshop where we focused on how IoT-based technologies can play a role in future healthcare development.
• Serious games for rehabilitation: Walk the Ward | This project was about encouraging patients to activity and movement during hospitalisation. The project created an application which motivates the patient to be active and involved in their care. The application consists of a program on a tablet and sensors and QR codes set in different locations in the care unit.
• IoT for dementia care: AppSam | The project explored IoT concepts that can contribute to the improvement of the quality of life for the elderly, relatives, and support health professionals.

Researchers

The following researchers are involved in the Smart Health application area:

• Nancy Russo (application area coordinator)
• Suzan Boztepe
• Jonas Christensen
• Jeanette Eriksson
• Åse Jevinger
• Magnus Johnsson
• Carl Johan Orre

The connected tools for life-wide learning provide new opportunities for bringing together technology, data and people to provide creative and personalized education. Society is connected through the use of personal technology across diverse landscapes that provide opportunities for IoT services that provoke curiosity and playfulness while engaging people in relevant learning activities at school, home, work, and on the go. Making sense about how to understand, use, and share data between people and services that provides privacy, control, and security that empower people is at the centre of our research.

Projects include Makerspaces for Lifelong Learning, PELARS, Mobile Language Learning Tools for Supporting Asylum Seekers

Contact: Daniel Spikol

The connected lifestyle of people today, through their use of mobile and wearable technology for tracking and various interactive and self-adapting IoT services, places Smart Living as a natural point of connection between everyday life and special events or activities, as well as between spare time and work. This includes technology for activity and self tracking as well as infotainment services that promote convenience, safety and entertainment in a secure and privacy-preserving manner.

Projects include CACT, iSMASH, ISHPA, DynahMat, Framtidens sport

Contact: Carl Magnus Olsson

A vital aspect of the society is the movement of people and goods. By using IoT in transport systems for planning, guiding users, and support autonomous acting (e.g. self-driving cars) etc., the transports can be improved. The improvement concerns efficiency, experiences and comfort for the user, reduced environmental impact, as well as increased security. In particular, individualizing transport services and providing better control and information of transported goods are areas where IoT can make a difference.

Projects include Data Innovation Arena, EcoTell, Information-based Disturbance Management for Public Transport

Contact: Jan Persson

Who is it for?

The lab is open to our researchers, students working in IOTAP-related projects, and to IOTAP partners, both  commercial and academic.

Do you want to spend time in the lab?

We provide access to the lab depending on relevance to the IOTAP research center and on currently available space.

What can the lab be used for?

The lab can be used for rapid development of interactive and embedded IoT systems. The lab provides a suitable environment for demonstrating new concepts and prototypes.

What kind of equipment is there?

The lab offers a variety of equipment for IoT prototyping and rapid development including cloud deployment with support for virtual and augmented reality (VR and AR). We also offer a physical space for your project to happen in.

We have a strong focus on the Arduino platform with a large selection of diverse shields to extend the projects’ connectivity. Additionally, we work with Raspberry Pi and other microcontroller boards. We offer a wide variety of sensors and actuators for both platforms that include off-the-shelf products like Philips Hue kits and commercial products from our partners.

We also offer 3D printing capabilities. For exploring interaction and visualization, we work with different gesture, haptic, voice, VR, and AR platforms to extend data beyond the 2D screen.

We offer a dedicated network for all things IoT, wireless as well as wired. We also offer several ways of connecting your projects through e.g. Zigbee, WiFi, GPRS, as well as more passive technologies such as Bluetooth beacons and NFC/RFID.

See full inventory

What kind of help can I get?

The lab offers help with most aspects of prototype development. We can get you started with designing your IoT artefact and we are happy to help in choosing and getting up to speed with a relevant development platform, e.g. Arduino, Raspberry Pi, or Hololens.

Our network of competencies includes people experienced in:

• Interaction design and technology
• Sensor network architecture
• Data gathering and analysis
• Connectivity
• Rapid prototyping and digital fabrication

• Johan Eker, Ericsson, Lund University (Chair)
• Anna Gillquist, Region Skåne
• Andreas Jacobsson, Malmö University (Faculty of Technology and Society)
• Anders Kottorp, Malmö University (Faculty of Health and Society)
• Niklas Malmros, Sigma Technology
• Annika Olsson, Malmö University (School of Arts and Communications)
• Andrej Petef, Sony Mobile
• Maria Stellinger Ernblad, City of Malmö

Current and past partners which have collaborated with IOTAP on various projects:

Networks

NMSA – Network for Mobile Services and Applications
Internet of Things Sweden
K2 National Knowledge Centre for Public Transport
ITS Postgraduate School

Companies

4IT
All Binary
Apptus Technologies
Arduino
Arla
Arvax Consulting
Aura Light
AutoIDExpert
Axelerate Motorsport
Axis Communications
Bergendahl Food
Biosync Technology
Blackberry Sweden
Boris Design
Bring
Bröderna Hanssons
Cybercom Sweden
Data Ductus
DSV Road
E.On Sweden
Electrolux
Ericsson
Flextrus
Fraktkedjan Väst
Fujitsu Sweden
GS1 Sweden
Health Guide IRL
HiQ Skåne
Husqvarna
IBM Svenska
IKEA
Media Evolution
Nobina
Odd Hill
Packbridge
Perch Dynamic Solutions
PosTrack Europe
Preem
Samtrafiken
Scan
Scandinavian CleanTech Group
Scania CV
Schneider Electric
Sensative
Sigma Connectivity
Sigma Technology
Sofiahemmet
Sony Mobile Communications
Svenska Retursystem
System Verification
TeliaSonera Sverige
TerraNet
Tetra Pak
Trivector
u-blox (connectBlue)
Verisure Innovation
Volvo Technology
Wememove
Wireless Maingate
Yanzi Networks
Ziggy Creative Colony
ÅF Technology
ÖGS Bolaget

Institutes 

Copenhagen Institute of Interaction Design
IVL – Swedish Environmental Research Institute
SICS – Swedish ICT
SIK – the Swedish Institute for Food and Biotechnology
SP – Technical Research Institute of Sweden
Viktoria – Swedish ICT

Non-profit organizations 

European Network of Living Labs
Friskis & Svettis
Stockholm Consumer Cooperative Society
Mobile Heights
NTM – Network for Transport Measures
Stockholm Innovation & Growth

Public sector 

Blekingetrafiken
Citilab – Cornella
Municipality of Karlshamn
NetPort Science Park
City of Lund
City of Malmö
MKB
Region Skåne
Skånetrafiken
Trafikverket

Universities 

Blekinge Institute of Technology
Chalmers
Technical University of Denmark
Royal Institute of Technology
Lund University
National College of Art and Design (Dublin)
Scuola Superiore Sant`Anna
Stellenbosch University
Stockholm University
University of Bremen
University of Craiova
University of London, Institute of Education
Uppsala University