gina smithRobotRepublic — Pianist Jason Barnes lost part of his arm back in 2012. But thanks to new prosthetic tech from researchers at the Georgia Institute of Technology, this month he started tickling the keys again.

The innovation is built around Georgia Tech’s new ultrasonic sensor, which allows amputees like Barnes to control each prosthetic finger individually.

It’s a real first.

“Our prosthetic arm is powered by ultrasound signals,” says Gil Weinberg, the Georgia Tech researcher who is spearheading the project.”This new technology … can detect which fingers an amputee wants to move,” he added. And it works “even if they don’t have (real) fingers.”

Barnes, 28, lost his hand and most of his forearm five years ago after he was electrocuted in an accident at work. Doctors amputated his right arm just below the elbow. But despite the amputation, he still retains the muscles  that would control his fingers if they were present.

Algorithms and ultrasound

Like other commercially available ones, the prosthesis Barnes wears uses electromyogram (EMG) sensors. Once attached to his muscles, he can switch the arm into several different modes of operation by pressing buttons mounted on the arm.

It’s controlled by a flexing or contraction motion the amputee does with his muscles. For example, flexing allows Barnes to clamp together his index finger and thumb; contracting his muscles makes his fist close.

But that isn’t the precise individual finger control Georgia Tech researchers were going for.

“EMG sensors aren’t very accurate,” said Weinberg, who is the director of music technology at the university. “They can detect a muscle movement, but the signal is too noisy to infer which finger the person wants to move. We tried to improve the pattern detection from EMG for Jason but couldn’t get finger-by-finger control.”

Then came what Weinberg calls “the Eureka moment.”

The team started thinking about whether ultrasound technology could make a difference. It quickly linked up with researchers from sciences and mechanical engineering departments to work up an ultrasound probe to the arm. That’s exactly the same kind of probe obstetricians use to take a look at babies in the womb.

“It’s completely mind-blowing,” said Barnes. “This new arm allows me to do whatever grip I want, on the fly, without changing modes or pressing a button. I never thought we’d be able to do this.”

They then paired the tech with machine learning algorithms that can first detect which finger it is Barnes wants to move as well as the force he wants that movement to have.

It worked.

And “if this type of arm can work on music, something as subtle and expressive as playing the piano, this technology can also be used for many other types of fine motor activities such as bathing, grooming and feeding,” Weinberg points out.

“I also envision able-bodied persons being able to remotely control robotic arms and hands by simply moving their fingers.

If Barnes the amputee pianist looks strangely familiar to you, by the way, that’s no accident.

The breakthrough prosthetic Weinberg’s lab built for Barnes is his second.

Barnes, who also plays drums, received the first one back in 2014. It was a prosthetic arm with two drumsticks, one which Barnes controls and the other which moves on its own. Barnes and Weinberg have toured the world with that robotic device, playing at Moogfest, Kennedy Center in Washington D.C. and other high profile concerts. The robotic stick plays faster, they claim, than any drummer can in the world.

Check that video below. Below that, find other videos on the the lab’s robotic stick and more videos on its latest innovation, which has Barnes on the ivories again.

For RobotRepublic, I’m Gina Smith.