Technical aspects. The primary goal of most neuroprosthetics is either to restore brain function to its normal state or to substitute for lost function. This may entail a sensory prosthesis, such as a cochlear or retinal implant (ie, a device that substitutes for the defective part of the sensory organ and stimulates either the cochlear nerve or the retinal ganglion cells directly). Such sensory prostheses are “biomimetic,” in that they convert the environmental sensory input into a signal to the intact remaining sensory nervous system that mimics the latter. Even the sense of touch has been restored using a prosthetic arm plus a brain stimulation technique (80; 29).
Brain-computer interface (BCI). Unlike sensory neuroprostheses, a BCI will only “sense” (ie, monitor) brain activity, but with a view to specifically allow people to control some device or software. BCI systems have initially exploited “neurofeedback” paradigms (ie, operant conditioning with brain signals, where subjects learn to modulate their brain activity through feedback so as to optimize the control of a device) (31; 10). However, modern BCIs have fully transitioned to the artificial intelligence era and respect a pattern recognition and machine learning architecture. This means that all state-of-the-art systems comprise some form of the following processing modules: signal processing (eg, spatial and spectral filtering), feature extraction and selection, and classification or regression.
Clinical BCI applications rely on this prototypical design, as well as on two additional design choices: the type of neural interface or brain imaging method employed for capturing brain activity and the “BCI paradigm,” ie, the brain phenomenon that is exploited to enable interaction. The latter usually regards either evoked brain responses triggered by external stimuli (P300, Steady-State Visually Evoked Potentials (SSVEP), etc.) or some endogenous, self-paced and spontaneous modulation of brain activity (eg, task-specific sensorimotor rhythms regulation elicited by imagined or attempted movements).
A BCI may substitute for a defective motor output system, for example, bypassing the spinal cord in a patient who has suffered a spinal cord injury to stimulate the muscles of the arm directly (functional electrical stimulation). An alternative to functional electrical stimulation in a quadriplegic patient is to place an electrode in the motor or premotor cortex (usually an electrode array with 100 or more individual electrodes) to record the electrical activity of that part of the brain in order to guide a substitute motor output for the paralyzed arm or leg (13; 02). The original experiments involved patients learning to operate basic computer functions (allowing internet surfing, etc.); more recent BCIs allow patients to control a prosthetic arm with multiple degrees of freedom (see clinical vignette #1).
An attractive means of monitoring a patient’s brain electrical activity is by noninvasive scalp electrodes, such as those used for electroencephalography (EEG). The advantage of noninvasiveness may be offset by the poor resolution, which precludes recording electrical activity from small regions of the brain. However, for some applications the information from scalp electrodes is sufficient to guide the desired output. The P300 has already found successful clinical applications (75; 90). Other EEG-based interfaces provide end-users with more ecological, self-paced interaction (10; 62; 64; 46). Greater noninvasive precision can be obtained using magnetoencephalography (MEG) or functional magnetic resonance imaging (fMRI), but the need for the patient to be placed in a multimillion-dollar machine makes these techniques impractical for most brain-machine interface applications (12). Functional near-infrared spectroscopy (fNIRS) is a less expensive and more portable type of BCI (also based on the metabolic activity of the brain), which has been shown to provide a viable communication solution for locked-in syndrome (LIS) patients (20), as well as for enhanced gait rehabilitation when fNIRS is combined with deep neural networks (33).
An essential component of most BCIs is a sophisticated computer algorithm that converts one signal (or stream of information) into another signal in real time in order to bypass the nervous system function or component that is impaired or lost due to the brain disorder involved. In a sensory neuroprosthesis, the algorithm converts the sensory input (sound energy in the case of cochlear implants; light energy in the case of retinal prostheses) into electrical signals that are then carried via the usual channels to the appropriate cortical areas for processing--the “biomimetic” function noted above. However, to date, the biomimetic conversion or encoding is not perfect, and the brain must go through a learning phase (possibly many hours of training) before the prosthesis works as an effective functional substitute (see the clinical vignettes).
In a neuroprosthesis for a quadriplegic patient, either scalp electrodes, epidural or subdural surface contact electrode arrays, or (for the most precision) needle electrode arrays implanted in the cortex (the 100-electrode Utah array being one of the most common) are employed to collect brain electrical activity data (54). The algorithm converts that brain electrical activity into a signal to guide motor output--usually to a robotic arm or leg but potentially to the patient’s own extremity (in the case of spinal cord injury rather than amputation). The potential to use brain cortical electrical activity to guide a prosthetic or robotic arm was described decades ago (31). Advances in the ability to monitor and analyze cortical electrical activity have demonstrated the complexity of the relationship between that brain electrical activity and the motor activity required for fine control of a prosthetic arm. It has been shown that such neuroprostheses can result in the reorganization of brain electrical activity, eg, electrical activity between the cortex and subcortical structures such as the striatum, which is a demonstration of neuroplasticity in the mature brain (44; 89).
Brain-machine interface (BMI). Although the present deep-brain stimulation systems using a stimulating electrode greater than 1 millimeter in diameter have proven remarkably effective for movement disorders such as Parkinson disease, dystonia, and essential tremor, it has not been nearly as successful in treating disorders such as depression, obsessive-compulsive disorder, and many types of refractory epilepsy. The usefulness of more miniaturized, multielectrode microarrays (such as the Utah array noted above) for recording focal brain electrical activity with increased precision has suggested that much more attention needs to be placed on the actual tissue--electrode interaction in the brain, which is the initial link in BCI as well as the basis of the BMI. A number of groups have been working on improving the ability of electrodes to record and stimulate brain electrical activity down to the cellular level with minimal tissue trauma. Some groups are developing micron-level electrodes that can be inserted via the intracranial vasculature (down to the capillary level) to reduce the invasiveness of inserting electrodes through the brain parenchyma (58).
Because the regions of interest in optimizing brain tissue recording and stimulating involve such anatomical locations as the proximal axon and the synapse, the techniques must assume similar dimensions, ie, submicron or nanolevel. Fortunately, using nanotechniques (involving materials such as carbon nanotubes or nanofibers with conducting polymer coatings), electrode arrays with greatly reduced impedance and greatly enhanced capacitance can be realized (53; 41; 22; 16). The enhancements in charge transfer and signal-to-noise ratio of nanoarrays (in comparison with standard platinum electrodes) as well as the improvements in spatial and temporal resolution can reach several orders of magnitude. Such improvements in the brain tissue-electrode interface not only enhance the spatial and temporal resolution of the recording and stimulating device but also reduce remarkably the electrical current needed to drive the device. This may allow the device to be driven entirely by current generated from pressure changes in the brain (eg, cardiovascular pulsations) thanks to nanomaterials that capitalize on piezoelectric effects (93). This would obviate the need for the bulky subcutaneous battery required with the present deep-brain stimulation systems and eliminate the need for battery replacement operations in the future. Nanolevel techniques have now allowed the development of probes that can record activity from hundreds of sites simultaneously (37; 60). A needle 10 mm long and 20 x 70 μm cross-section has been fabricated with 960 12 x 12 μm recording sites, 384 of which recording sites may be used at one time. This technical feat has permitted gathering simultaneous electrical data in vivo (mouse and rat) from large neuronal populations for up to 8 weeks.
An alternative technique is a “living electrode” that utilizes living cortical neurons in a hydrogel microcolumn (about 300 μm diameter) implanted perpendicularly into the cortex to monitor/modulate brain electrical activity (01). Such a device, implanted in the cortex with one end of the column on the cortical surface and the other end at the desired depth (eg, layer IV), allows an LED array to optogenetically stimulate and a photodiode array to record—either array being on the cortical surface end of the column. Stereotactic insertion of such microcolumns into the rat cortex have been followed for up to one month in vivo; axonal growth and synaptic interaction with the host cortex have been documented (01).
Many, if not most, brain disorders are primarily disorders of brain chemistry (neurotransmitters) rather than disorders of brain electrical activity. The emphasis on brain electrical activity in brain-machine interface research has been an example of “measuring what we have the tools to measure, not necessarily measuring what is most important.” Even Parkinson disease, the most dramatic example of brain stimulation to treat a nervous system disorder, is the result of a loss of dopamine in the striatum (and not primarily a disorder of brain electrical activity). Indeed, it has been shown in a swine model that stimulation of the subthalamic nucleus (with parameters matching those used in subthalamic nucleus deep-brain stimulation in humans) produces an elevation of dopamine levels in the striatum (77). The same group has also monitored dopamine release in the caudate nucleus with subthalamic nucleus stimulation in a patient suffering from advanced Parkinson disease who was undergoing deep-brain stimulation surgery (39). An article has reviewed the role of neurotransmitter monitoring for closed-loop deep brain stimulation (67).
Fortunately, nanotechniques are also improving the chemical brain-machine interface--in addition to improving the electrical brain-machine interface. Fast-scan cyclic voltammetry (FSCV) and related techniques allow real-time in vivo measurement of neurotransmitter levels, including neurotransmitters (eg, dopamine and serotonin) that are involved in common mood disorders, such as severe depression and obsessive-compulsive disorder. Although FSCV using a standard microelectrode is unable to distinguish accurately the individual levels of dopamine and serotonin in mixtures simulating the brain in vivo situation, a carbon nanofiber electrode array has been shown capable of measuring individual levels of dopamine and serotonin in such a mixture (70). Another group has used microelectrode arrays to investigate electrical activity and glutamate levels in columnar microcircuits in the nonhuman primate brain in vivo (56). One group has developed a wireless system for in vivo neurotransmitter monitoring in humans (39). The same group has also monitored thalamic adenosine levels with thalamic electrical stimulation during surgery to implant deep-brain stimulation electrodes in eight patients suffering from essential tremor. A further step toward real-time monitoring of both electrical and chemical activity as well as electrical stimulation has been taken with a hybrid 1.2 mm diameter lead for deep brain stimulation that incorporates both fast-scan cyclic voltammetry for dopamine neurotransmitter recording and electrical recording (06).
One long-term goal for the BMI is actual “parts replacement,” ie, cellular and subcellular level devices to substitute for defective axons, synapses, neurons, glia, etc. One example is an artificial afferent nerve (43). Pressure sensors convert pressure information (from skin deformation) into action potentials, similar to a sensory nerve. The action potentials then stimulate the efferent (motor) nerve to result in muscle contraction and movement (in this case, in a cockroach).
Another example is a biomimetic synapse that replicates the plasticity of brain synapses (42). An organic neuromorphic device—a conducting polymer-coated gold on silicon microfluidic post-synaptic receptor—has been incorporated with presynaptic dopamine-secreting PC-12 cells to mimic a biological synapse. Such a device offers the possibility of going beyond precision monitoring and modulating brain electrical and chemical activity to the ability of creating an artificial neuromorphic synapse that can undergo the long-term conditioning and recovery of synaptic weight that is seen in the typical brain synapse where neurotransmitter oxidation and recycling (in this case, dopamine) takes place.
With a brain-machine interface that includes real-time continuous monitoring of both electrical and chemical activity in specific regions of the brain, the understanding of brain disorders ranging from depression to obsessive-compulsive disorder to epilepsy will improve dramatically. Subsequently, equally dramatic improvements in using brain-machine interfaces for treatment of such disorders will also be realized.
Clinical applications. The most common clinical neuroprosthetic applications to date have been sensory neuroprostheses, with well over 100,000 cochlear implants worldwide. The number of retinal implants for those with macular degeneration and other, less common forms of retinal disease (eg, retinitis pigmentosa) is much lower but will increase rapidly as the number of contact micro/nano electrodes in the implanted arrays increases--allowing vision enhancement beyond basic independent navigation and reading large print. The number of DBS implants over the past 20 years is also well over 100,000 worldwide, the majority used in patients with Parkinson disease and other movement disorders.
Clinical applications of BCI include brain-actuated spellers (10; 62; 84) and patients suffering from locked-in syndrome or late stage amyotrophic lateral sclerosis. Other applications include assistive mobility (46) and orthotic devices (65).
An increasingly important brain-machine interface application is stroke rehabilitation (68; 03; 66; 15; 30; 09; 17; 40), marking a shift in brain-machine interface research focus from motor substitution to motor recovery. Laboratory brain-machine interface research in rats has demonstrated that closed-loop brain-machine interface is superior to open-loop in fostering functional recovery after a focal head injury (32). It is reasonable to expect such closed-loop brain-machine interface systems to enhance functional recovery in humans with similar focal brain injuries (due to either trauma or stroke), as also demonstrated for spinal cord injury patients (24; 74). The role of brain-machine interface techniques for traumatic brain injury rehabilitation has been reviewed (83).
More complex brain-machine interface devices include those that allow a quadriplegic spinal cord injury patient, a limb amputee, or even a patient with the locked-in syndrome to interact with the environment (see clinical vignette #2). By virtue of recording brain electrical activity with scalp electrodes or arrays implanted intracranially (epidural, subdural, or cortical), this interaction may involve a computer for Internet surfing or driving a motorized wheelchair; it may be used, eg, to control a prosthetic limb.
A major breakthrough has been the ability of a complete locked-in syndrome patient to learn to control an invasive interface combining the benefits of invasive signals with the learning capacity of the brain, mostly exploited so far in noninvasive and primitive BCI designs (see vignette #5) (19).
Future directions. Future directions in brain-machine interface can be divided into techniques and applications.
Regarding future techniques, improvements in the physiological neural interface (using nanotechniques to enhance monitoring and modulating brain electrical and chemical activity) have been considered in the previous section. Organic and soft materials can more closely approximate the mechanical properties of brain tissue than metallic electrodes. Such biomimetic materials also can more accurately interact with brain electrochemical physiology. The topic of organic neuromorphic devices has been discussed (82).
The miniaturization of the biosensors may be taken a step further by eliminating the electrode altogether and replacing it with a large number of micron-sized “neural dust motes” distributed throughout the regions of interest in the brain (76; 52). These “neural dust motes” communicate ultrasonically with a subdural transceiver, which in turn communicates transcranially with the external transceiver. Because it was shown nearly two decades ago that it is possible to record brain electrical activity with an electrode inside a capillary as effectively as with an electrode in the brain parenchyma (47), for brain electrical activity monitoring such “neural dust motes” could remain within the capillary. Interventional neuroradiologists are becoming so adept at catheterizing small brain blood vessels that the possibility of implanting “neural dust motes” in the brain parenchyma for neurochemical monitoring by puncturing through the capillary wall is also feasible, similar to the implantation of endovascular stent electrodes noted above. The era of the truly minimally-invasive brain-machine interface or neuroprostheses for neuromodulation is close at hand.
Sophisticated computational analysis techniques will provide much more efficient brain-machine interfaces for applications such as deep-brain stimulation. One example comes from the modification of DBS in a primate model of Parkinson disease (81; 86). The therapeutic effects of DBS for movement disorders such as Parkinson disease usually are lost within minutes of discontinuing the stimulation. However, using algorithms that coax the abnormally synchronized brain electrical firing patterns in Parkinson disease back to more normal patterns--algorithms that also require much less energy (electrical current) than standard high-frequency DBS--it is possible to have sustained aftereffects of DBS, ie, the benefit on the abnormal Parkinsonian movements can last for up to 4 weeks following cessation of stimulation. When combined with the efficiencies noted in the discussion of technical aspects above, such computational analysis techniques promise to promote DBS from the ranks of a crude brain bludgeon to a subtle brain symphonic conductor (48). Furthermore, closed-loop DBS could reduce or eliminate adverse effects of conventional DBS and increase the battery life (59). Several research efforts are devoted to a better understanding of the still elusive mechanisms underlying the success of DBS (05).
The computational analysis of brain signals, made more complex as the number of electrodes monitoring brain electrochemical activity increases, will be markedly enhanced by techniques such a multi-threaded parallel processing (28). The lengthy learning curve for patients to master, for example, fine motor control of a prosthetic limb will be reduced by multi-tier distributed computing and linked data techniques, although coadaptation and learning and coping with nonstationarity (23) in brain-machine interface is likely to remain crucial for successful neuroprosthetic control. A pilot study used EEG-based brain-machine interface plus cloud servers, in-home fog servers, and smartphones (94). The “big data” generated by many patients will allow the development of personalized algorithms that should enhance the speed and efficacy with which a patient learns how to operate the neuroprosthesis (14; 73). However, the extent to which brain-machine interface can be treated as a purely decoding problem and the potential importance of the role of learning and of the coadaptation between user and machine need to be carefully investigated (64; 26; Perdikis and Millan 2020; 78).
Novel brain-machine interface techniques--what might be termed the software rather than the hardware of the actual electrodes themselves--are likely to be a major advance. One such technique has the subject judge whether the actions of the brain-machine interface or neuroprosthesis are correct or not (35). Thus, the subject “teaches” the BCI rather than the subject learning to adapt his/her thought processes to achieve the desired action. Another technique, neural dynamical modeling, may allow accurate predictions of future neural population activity based on single trials (38). This would shorten the often-lengthy learning curve for BCIs in real-world situations (14).
Future applications range from the obvious to the exotic. The obvious include refinements in the electrodes of the BCI/BMI to allow more sophisticated control of paralyzed limbs (mimicking intact hand and leg function) as well as, in the amputee, more realistic control of artificial or robotic limbs. Similar improvements will occur in sensory prostheses, notably cochlear and retinal implants. It is conceivable that such sensory prostheses may achieve a resolution exceeding that of normal intact humans--subject, of course, to the degree to which sensation is limited by the central brain processing rather than the peripheral sensory-neural interface (cochlea or retina).
Although the focus of clinical neuroprostheses to date has been primarily the replacement of lost nervous system function, the use of neuroprostheses to enhance brain function is an increasingly important application of the brain-machine interface. One group that has been quite active in this area is the United States’ Defense Advanced Research Projects Agency (DARPA). Although DARPA has funded brain-machine interface research for over 40 years, recent efforts have included using brain-machine interface techniques to enhance brain function beyond normal capabilities (rather than replace lost nervous system function) (50). These have generally taken the form of a noninvasive brain-machine interfaces (such as a scalp EEG) to improve human function beyond baseline. One such closed-loop BCI system, the Adaptive Peak Performance Trainer (APPT), was able to more than double the learning rate for novices undergoing rifle marksmanship training (50):
| || |
The APPT® system incorporates knowledge of EEG, electrocardiography (ECG), respiration rate, and eye tracking signatures of learning stages. The system can provide continuous physiological monitoring and feedback (visual, auditory, or haptic) to the trainee in real-time through integration of algorithms that derive physiological state changes based on sensor inputs.
Another DARPA project uses the P300 of the EEG to enhance an individual’s ability to detect a target when analyzing images. In testing with over 40 professional imagery analysts, such a brain-machine interface system “resulted in up to a 10-fold increase in analysis throughput (area of imagery analyzed per unit time) with no loss of target detection sensitivity, as compared to the analysts’ performance using their standard imagery analysis approaches” (50). Although DARPA’s goals are understandably influenced by the needs of the military, the potential applications of such performance-enhancing noninvasive brain-machine interfaces to fields ranging from education to job productivity are endless.
Given the devastating conditions for which brain-machine interfaces are often used, an overlooked aspect regarding ultimate success is the patient’s acceptance of a specific BCI/BMI (device and technique, hardware and software, invasive vs. noninvasive, etc.). Fortunately, efforts at understanding patient acceptance are being made (72). In addition, practical problems in the clinical application of the BCI/BMI are increasingly taken into account. Rapid advances in dry electrode technology for EEG-based prostheses are expected to make the use of brain-machine interface less obtrusive.
A final, possibly disquieting, application of brain-machine interfaces has been demonstrated by a group at the University of Washington, with the goal of enhancing interpersonal communication—brain-to-brain interface (BBI). Given the limitations of verbal communication (eg, the inability of a virtuoso violinist to “tell” a novice how to master the violin), a direct brain-to-brain interface has been proposed (71). The demonstration involved the collaboration of two people to perform a simple computer game, the goal of the game being to press a button to fire a cannon that would shoot down a rocket fired to destroy a besieged city--but not shoot down a supply airplane headed for the city. The “sender” could view the game and imagine pressing the “destroy” button when a rocket (but not an airplane) appeared; the “receiver” had one finger placed close to the button. EEG recording from the “sender” was transmitted via the internet to a transcranial magnetic stimulation device placed over the motor cortex of the “receiver” (who was located in a separate building). The EEG activity of the “sender” was able to correctly command the “receiver” (via transcranial magnetic stimulation of the motor cortex) to press the “destroy” button when a rocket appeared over 80% of the time for one pair of participants. For another pair, the success rate was 37.5%, and for a third it was 25%. Analysis of the EEG during rocket versus airplane trials showed that the “sender” in pair one had much greater separation of the EEG log power between the rocket and airplane trials than the “sender” in pair 2, and the separation of the EEG log power for the “sender” in pair three was minimal--demonstrating that the “weak link” was the ability of the “sender” to differentiate between the rocket and airplane trials (in terms of EEG activity) and not the communication between the EEG activity, the transcranial magnetic stimulation device, and the response of the “receiver” (ie, the finger movement, resulting from the transcranial magnetic stimulation, by the “receiver” to push the “destroy” button). Interestingly, in another iteration of the technique dubbed “BrainNet,” 2 “senders” playing a Tetris-like game instruct the “receiver” through transcranial magnetic stimulation of the occipital (visual) cortex of the “receiver” (36). Additional rounds of information can be sent from “senders” to the “receiver” to improve performance. The authors conclude: “Our results point the way to future brain-to-brain interfaces that enable cooperative problem solving by humans using a “social network” of connected brains.”
This final application, the brain-brain interface, emphasizes that along with the technological advances in the BCIs/BMIs that are occurring at virtual “warp speed,” we will need to keep ethical standards evolving with similar diligence to maintain the BCI/BMI as a benefit rather than a curse for humanity.