Apple’s accessibility tools are changing the world

Translate everything…

Apple’s accessibility tools are changing the world. Now it has a huge opportunity to do something quite amazing. Here’s what I think it can do with some of the accessibility improvements it introduced at WWDC 2020.

FaceTime Sign Language

Starting in iOS 14, FaceTime will become smart enough to tell when someone on the call is using sign language during a Group FaceTime call. When it recognizes this, the system will make that person(s) more prominent in the call – essentially, once it recognizes that movement means the person is speaking, it will treat them more like any other speaker. That’s great I hope, as it should mean people who communicate using sign language will have a better chance to put their point across when the usual dominant shouty person tries to take over the chat.

Translation of things

Another fab improvement is Translation. Siri gets much smarter about this, and will be able to translate whole conversations between languages for you on your device. That’s cool.

Another thing that’s cool is the built-in translation we can look forward to in Safari. That means you’ll be able to visit a website and read it in your own natural language.

The translation elements only extend to a handful of languages at this time, but we know that number will increase over time. This will become so useful to travellers.

Hand and body pose detection

One interesting addition in iOS 14 and macOS 11 Big Sur is the introduction of an updated Apple Vision framework developers can use in order to detect body and hand poses in photos and videos. What’s this for? Apple suggests it might be used in fitness apps in order to automatically track how you exercise, safety training apps and more.

And now for something else

That’s great but putting these three items together it’s hard not to see another potential.

There’s a chance here for a developer (or Apple) to combine these three technologies:

Hand and body pose detection to read sign language, Siri to translate that language into some other form (words, language, different sign language dialects, or different languages), and make the translation available during the FaceTime call, or via the device, or in a movie, or as an email message.

This could cut both ways, of course, why not create an on-screen figure who can translate non-sign language into sign language?

How may truly smart people communicate in sign language but find themselves frustrated because they can’t easily put their thoughts across to family, friends, or colleagues?

I don’t know the answer to that, but it seems to me that Apple is just a few short steps away from a real accessibility advance with these three technologies.

That’s even before thinking about the Minority Report style gesture-based user interfaces we all think are coming.

But there’s more

These are just three of the many accessibility enhancements Apple introduced at WWDC 2020.

• Back Tap is an awesome feature in itself (read about that here).
• Another is Headphone Accomodations (Settings>Accessibility>Audio) which lets users adjust the sound they hear to better suit their audible range. The idea is that you reduce louder sounds and enhance softer ones to make them a little easier to hear.
• Last year’s big hit tech, VoiceOver, has also received some brilliant improvements, it will now identify key display elements and can provide VoiceOver access to apps and sites that don’t have their own accessibility tools, including custom rotors. It can even read text in images.
• Another improvement uses the iPhone’s microphone to listen out for sounds like babies crying, smoke alarms, running water or your front doorbell – if those sounds are heard you’ll be notified. Think how useful that might be if you have poor hearing but receive an alert on Apple Watch.
• Apple’s Magnifier and voice control options have also been updated – and Apple has even made a few tweaks to Xcode to make it more accessible, also.

But, somewhere out there, a developer may well be interested in exploring these technologies for sign language.

Please follow me on Twitter, or join me in the AppleHolic’s bar & grill and Apple Discussions groups on MeWe.