Apple has been experimenting lately with different types of input devices. In 2016, they tried the butterfly laptop keyboards which were a huge step backward in reliability and a teeny hop forward in "key stability". After 3 years of painful reliability issues, they went back to a well tested scissor design in their new 16" laptop. 2016 was also the year that Apple came out with their first touchscreen Mac; the touch bar MacBook Pro.
I guess the question is, what is the problem that a new input device would attempt to solve. Key stability is a very minor problem and could be argued it's really not a problem at all. I like the thinking that the touch bar could be more intuitive than mousing around menus, but in practice it's just impractical. It's hard to reach and hides previously hard-wired functionality like volume and brightness controls. So what's wrong with the standard keyboard and mouse that we have had for years? I don't think there is anything particularly bad about these input devices. They get us pretty much all the functionality we need on a computer and they are pretty ubiquitous. There is a learning curve to touch typing, but many schools teach it at a young age now. It's really not much more difficult to learn than driving a car. And the mouse, or alternatively trackpad? It's pretty intuitive to pick up, but may not be as efficient as the keyboard in a lot of cases. Then there are touchscreens. They are probably the most intuitive but quite imprecise. So we have:
- Keyboard: Fast and precise, but high learning curve and often leads to RSI
- Mouse: Intuitive and mostly precise, but limited input capability and often leads to RSI
- Touchscreen: Very intuitive, but limited input capability and not precise
If I were to imagine a new type of input device, I would start with areas that could be improved with our existing forms of input. It wouldn't be unreasonable to say that the areas of ergonomics, speed of input, intuitiveness, and precision of input would all be areas to consider for improvement. We've tried different forms of voice input to improve on ergonomics, but those are slow and imprecise. It's reasonable to assume that touch input of some sort has the most legs, so to speak. We can feel feedback with our fingers and be pretty precise with our wrists and arms if the input device allows us to be (keyboard and mouse for example). The big challenge with touch is ergonomics. The more precise we make the input, the more strain it puts on our bodies. The more functionality that is added to touch, the more the learning curve as well.
Let's take touch out of the mix and think about our other senses. We have taste, hearing, smell, and sight. Is there anything we can do with these other senses that would improve ergonomics, precision, speed, and is also intuitive? Taste and smell don't seem to offer much, but I think that hearing and sight have some interesting potential. I've been thinking about the Airpods Pro for instance. They are great for sound output and also use the microphone for input and sound filtering. What if they could also pick up on brain activity? I can imagine a flow something like: these things we stick in our ears monitor brain activity for a stream of "text" input. The text then goes through a filter very much like what we use today for speech-to-text (i.e. composing a message) and speech-to-command (i.e. the various "ladies in a tube"). This of course leads to imprecise input as we're playing a really bad game of telephone at this point. So, what if we brought back in touch and sight at this point. Imagine a real-time display of what is coming from the pipeline (mind reading Airpods to speech-to-text/command) with suggestions for quick adjustments and corrections on the fly. This could hit on all key areas of improvement - speed, intuitiveness, precision, and ergonomics. As the "mind reading" part of this is a bit far fetched at this point, I wonder what it would be like to replace that with speech. It's disruptive in situations with other people involved, but could be an interesting proof of concept to try right now.