Isaac’s Little Movie Player
A veteran designer for HoloLens, Apple, Google Earth, Second Life, and Disney VR is making an iPad app for his autistic son
Leaving my full-time job at Apple was a hard choice, but the biggest benefit of returning to independent life was finally having more time to spend with and for my kids. It sounds like a bit of a cliché, but after two years of talking about it, I finally built a train table for my younger son, and I finally wrote this iPad app for my older son.
Isaac is autistic and, at age 12, still nonverbal. Experts advise that if autistic kids don’t start speaking by around age eight, they may never. So we’ve helped him learn how to use an iPad and press buttons to speak for him. He’s been doing that for about six years now and is steadily making progress.
But it’s one thing for Isaac to ask for things he wants or to say thank you, as appropriate. My deeper hope is that he’ll learn to read and write complex thoughts as well as anyone his age and unlock a whole new way to share his inner world.
Isaac loves all kinds of puzzles, as well as iPad games where he has to drag and drop letters or words in place to trigger animations. He also loves 3D animated movies (especially those by Pixar, Dreamworks, and Sony). Even at a young age, he had the patience to watch these for 90-plus minutes straight, through the end credits.
The idea was to combine these two very motivating activities to help him learn more language skills. So I made a game out of the closed captioning information on Isaac’s favorite movies.
There’s a hidden design challenge: These words fly by fast. I didn’t want to make his most rewarding activity frustrating by pausing the movie for input every few seconds. Ideally, if Isaac does well, the movie never even slows down, and he remains fully engaged.
After trying a few design ideas, I settled on this one:
The movie dialog first appears at the top of the screen and then “waterfalls” down to the bottom before the words get replaced by whatever comes next.
Occasionally, one word gets “stuck” up top, and Isaac must move it manually, drag and drop, which is where the game aspect comes in. Can he pick the right word and get it to the right position in the sentence?
I tried this at first with one word on top and a helpful ghosted copy at the bottom. He got it right away and was soon going at full speed.
Notice how the top words slowly turn yellow, giving Isaac a visual hint about the next choice-point coming up. He’s gotten very good at anticipating and rapidly dragging the words to the right spots, so the movie almost never pauses in practice, except when he occasionally gets the wrong word.
Soon, we moved on to having three words on top (a great suggestion from my younger son), with two false choices coming from elsewhere in the movie. Isaac got a few wrong answers at first and then got the hang of it.
I then randomized the locations of the three words up top and added speech synthesis for any words he touches so he could hear the same sounds multiple ways: once from the dialog as it’s spoken in the movie, once from his touching the word, and hopefully once more in his own head.
The app keeps statistics on the number of right and wrong answers. Isaac has moved from 75% correct to about 90% to 95% correct for the three-choice approach.
My next goal is to add an occasional second blank word at the bottom, so two of the three words up top are correct but not necessarily in order. Ideally, this should help him with sentence structure. Sequencing seems to be one of his challenges.
Since April last year, Isaac’s use of his iPad app for language has exploded. It could be that something naturally just clicked due to his age or via other efforts from teachers and aides. He’s even created a new kind of game for himself after my wife started testing him more at home.
Isaac uses his old iPad spelling game to prompt words and then goes to his school-supplied iPad to find or type (copying the spelling) the same words. We realized along the way that he could even read handwriting and match the word.
The biggest and most welcome change is that he gives us a look like, “Say this word,” and waits for us to participate in his game, giving him more positive social feedback for his effort. If you have experience with autism, you know that’s huge.
Every night since March, Isaac gets a few hours of solo movie time with his iPad playing this movie dialog game. I’ve had to expand the number of movies to about 12 to keep it fresh.
Some people have asked if I’m going to release this movie app for iOS. I’d love to! It might be helpful for younger kids to learn to read as well. I haven’t tested that at all yet. But there are a few technical challenges remaining.
First, we own a single digital copy of these movies for home use. I can’t ship an app with these movies bundled unless I get permission from the licensing groups at Disney, Sony, and Dreamworks. So, releasing this app requires me to do a lot more work to make it easy for any parent to drop their own movies onto their iPads (including format conversion).
Second, so far I can only partially decipher the closed captioning dialog track embedded in most movies. I’m still reverse-engineering the format from binary examples. I found it much easier to grab .srt files from the excellent OpenSubtitles.org site and parse those as text. But I’d still need to automate fetching these files or finish the decoding work to make that process painless for nontechnical users.
What do you think?
Let me know in the comments or email me (firstname.lastname@example.org) if you’re sufficiently interested in this work. I might push the priority up to improve and release the app sooner if there’s enough interest.