Interactive Systems: A Gentle Introduction
What we will cover
- What is an interactive system? (or IS for short)
- What are its component parts?
- How do these component parts communicate?
- Analyzing interactive systems
Interactive system
We normally understand interactive systems as some kind of technical system that responds to human action by eliciting a reaction, whether this reaction takes place on screen as a visual or sound cue or physically as package on our door after having placed an order in an online shop. The notion of an interactive system that we will be exploring in this introduction is fairly abstract and it emcompases everything from a button in a web form, to a character in a videogame, to an order in amazon.com.
Interactive systems can be aggregated to compose larger, more complex systems. In the example of the amazon order, we can think of the 1-click ordering system in amazon as an interactive system, that is part of a larger order placement & processing system, which in itself triggers an order fulfillment system, which then activates a logistics and delivery system that results in the package being delivered to your doorstep. You can think of that package in your doorstep as the output of your initial interaction with amazon.com's web shop.
As you can see this definition of interactive system is pretty broad and can be thought to include almost anything in the digital realm.
Interactive systems are a specific type of systems when we think of them from the optics of system's theory. Systems Theory is the study of systems, as interdependent and related parts in a larger whole, systems theory contemplates natural systems like fungi, weather or climate as well as human-made systems like aviation, government, agricultural policy or videogames. Interactive systems are a sub-discipline of systems thinking.
When we design and analyze interactive systems we think of them as being primarily composed of three parts.
Anatomy of an Interactive System
Where it says protocol in this diagram you can also think API, both amount to the same thing. A kind of contract between two parts of a system that agree to pass information in a specific manner.
You can think of a videogame character as an interactive system in itself, the character move when we gve it a new position through inputs in the form of mouse clicks, and its behaviour will be determined by an internal state determined by previous inputs and as outputs we get the range of possible representations of the character's state.
For example we trigger the shooting action in our character and depending on which weapon we selected before the character will shoot a cannon or wield a sword. The same input can result in a different behaviour if it is preceded by another input.
Interactive systems can be aware of time and can have internal states that modify their behaviour.
Inputs and outputs
Inputs
Inputs can be of the following types:
- User-generated interactions like for example a person pressed a button, or touched an interactive surface or tripped a door sensor or a webcam detects the user's face, etc. Basically any HCI (human-computer interface) can mediate a user-generated interaction. Some examples (this list is by no means complete):
| kind of datastream | devices |
|---|---|
| usb hci / bluetooth | keyboard, mouse, trackball |
| video & structured light | webcam, kinect, realsense, leap motion, LiDAR |
| physical computing | microcontrollers, arduino |
| radio-based identification | RFID/NFC |
| sensors | accelerometers, light sensors, gyroscopes, imu |
| radio signal | 3/4/5G, wifi, bluetooth |
| geopositioning | compass, GPS, barometer, wifi |
| presence | infrared, video blob-tracking, ultrasound proximity sensor |
| sound | microphone, clap sensor |
| music interfaces | MIDI controllers, pianos, kaoss pad |
- Events generated by our interactive system such as timeouts (e.g. trigger something 25 seconds after our game begins)
- Outputs from external agents from another system, for example a monitoring system of a gas tank let's out system know that gas is running low or a web API lets us know that an email has arrived.
Styles of processing and decision-making
The is the part of the interactive system that contains our behavioural logic, that is the aspect of our interaction that determines how the system will behave in response to the inputs. Most of the time we express this logic in code.
reading = cap.capacitiveSensor(30); // read our sensor value
threshold = analogRead(A5); // read threshold from potentiometer
// trigger the LED light if our reading is greater than our threshold
if(reading > threshold) {
digitalWrite(LED_BUILTIN, HIGH);
} else {
digitalWrite(LED_BUILTIN, LOW);
}In sensor-based systems this logic can sometimes be expressed in terms of thresholds, for example, do this when a specific sensor measures a specific level. Sometimes a particular decision can depend on multiple inputs, or multiple inputs along a period of time, for time-based inputs.
For example: in a plant irrigation system we might want to trigger an irrigation cycle but only if one wasn't triggered in the last 24h and the soil has been dry for at least one hour.
Machine learning for interaction
These days it is becoming increasingly common to use machine learning for this aspect of an interactive system. This is particularly useful for prototyping, when we might not yet have a fixed idea of how the system has to behave in response to user input, or the input is not yet fully understood.
An interactive system can learn from examples. We can use the inputs to provide those examples. We can for example have a bunch of sensors plugged to a plant and tell our irrigation system when the plant is in this specific condition (that's the example) then trigger an irrigation cycle (that's the expected output).
The advantage of using ML fto train interactive systems is that it allows us to work with the machine in a more intuitive way, without having to figure out threshold, proessing of sensor signals and other topics that would require advanced expertise to prototype.
Exercise: analyze an interactive system
Pick an interactive system that fits the definition laid out earlier in this chapter and try to identify its "inputs", "outputs" and it's decision making process.
Example: imagine that the city council has a network of IoT sensors measuring aspects such as noise pollution levels and air quality levels across the city of Amsterdam. At the present time they have a web-based dashboard where all this data is aggregated, a map with a visualization and a daily report that is emailed to community centers around the city.
Follow the rabbit 🐰
Voice UI authoring tools
- Voiceflow for voice user interfaces (VUIs)
- Botsociety for chatbots and conversational interfaces
Machine Learning for interaction
- Wekinator map example inputs to expected outputs, very simple and versatile tool to use machine learning for interactive systems
- Machine Learning for Musicians and Artists
- Runwayml machine learning for artists and designers
- Example-based Sensor Predictions (ESP) system applies machine learning to real-time sensor data
- Talk by David Mellis on Machine Learning for Makers