HWK Workshop 2023

Michael Beyeler

Human-in-the-Loop Optimization of Simulated Prosthetic Vision

Bionic Vision Lab, UC Santa Barbara

To improve the quality of restored vision generated by current visual prostheses, recent studies used deep learning to predict the electrode activation patterns required to elicit a desired visual percept. However, these models assume perfect knowledge of the user-specific mapping from stimulation to perception for arbitrary electrode combinations. Clinical experience with existing devices has highlighted drastic individual differences, making personalized stimulus optimization a key challenge.

Here we propose a practical method for obtaining a personalized stimulus encoding strategy by integrating state-of-the-art deep learning into a Bayesian optimization framework. We first trained a deep stimulus encoder (DSE) to optimize stimuli assuming perfect knowledge of phosphene perception. We then embedded the DSE within a human-in-the-loop optimization (HILO) strategy. HILO iteratively learns the best DSE configuration through a series of duels, where the user is asked their preference between two candidate stimuli. The resulting DSE can then be deployed as a personalized stimulation strategy.

As a proof-of-concept, we tested our framework on 100 simulated patients. HILO led to dramatic improvements in visual outcomes, outperforming a conventional naive encoder and the DSE without personalization after as few as 20 duels. HILO converged to favorable encoders even when 2 out of 3 user preferences were random or when the assumed phosphene model did not match the patient’s true model of perception. Our framework enables optimization across thousands of electrodes in real time and could be applicable to a variety of devices. Overall, this is a first step towards creating personalized stimulus encoders for real users of future prosthetic devices.

Avi Caspi

Eye tracking in cortical visual prosthesis - from theory to practice

Jerusalem College of Technology, Jerusalem, Israel

Eye movements dominate the perceived location of cortical stimulation-evoked phosphenes, even after years of blindness. Hence, by accounting for eye positions, we can mimic retinal mapping as in natural sight in future cortical implants aim to restore sight.

Specifically, eye tracking can be used as a marker to construct the spatial map of the implanted cortical electrodes. This was done by instructing patients to conduct an eye movement toward the phosphene. Furthermore, eye trackers are essential to perform a combined eye-head scanning mode. Head movements will shift the entire field-of-view of the head-mounted camera and eye movements will shift the region of interest within the wide field-of-view of the camera. This will be done by sampling eye positions in real-time and shifting the instantaneous visual field of each electrode based on eye position.

In the current presentation we will review various techniques to calibrate an eye tracker for the above procedure, We will also present results from recent eye tracking experiments from blind subjects with cortical visual implants.

Gislin Dagnelie

Early functional outcomes for the first human with the Intracortical Visual Prosthesis (ICVP)

Gislin Dagnelie, Michael P. Barry, Roksana Sadeghi, Vernon L. Towle, Kelsey Stipp, Hannah Puhov, William Diaz, Patricia Grant, Frederick T. Collison, Frank John Lane, Janet P. Szlyk, Philip R. Troyk

Purpose: The Intracortical Visual Prosthesis (ICVP) is a novel device for creating visual percepts in adventitiously blind individuals. Data on human visual performance using a camera with this device have not been previously reported. We tested the hypothesis that the implantee could discriminate horizontal and vertical orientations of the smallest gratings of the Berkeley Rudimentary Vision Test (BRVT).

Methods: The ICVP consists of multiple wireless floating microelectrode arrays (WFMAs), each with 16 stimulating electrodes. Twenty-five WFMAs were implanted in the right occipital visual cortex of a participant with only bare light perception in an FDA-approved Phase 1 clinical trial (NCT04634383). Groups of electrodes within WFMAs were evaluated across 9 months to measure current thresholds and phosphene positions and persistence. Stimulation for each electrode was provided at 200 Hz, 200 µs cathodic phase duration, and up to 60 µA cathodic current. An off-the-shelf pair of glasses with an integrated camera and USB connection was used to drive stimulation with 6 selected WFMAs, using 4 electrodes per WFMA. During 2 days of exploratory testing, the 50 M BRVT grating was tested at 25 cm in front of the participant (0.15 cycles/degree) in either horizontal or vertical orientation, with 30 balanced, randomized forced-choice trials.

Results: Thresholds under 60 µA were found, individually or in combinations, for 233 electrodes. Thresholds varied across days but remained stable on average. Of the 6 WFMAs used for camera testing, 4 produced phosphenes within a 4° cluster centered 4° below and left of fixation, and the others located 4° below and 20° left of the central cluster, respectively. Phosphene sizes varied within 0.3–6° across, generally increasing in size with distance from fixation. The participant correctly determined the orientation of the 50 M grating in 27/30 trials (p < 10−5, binomial test), corresponding to an acuity of 2.3 logMAR or better, responding within 24 s on average (range: 6–91 s). In more recent tests, the participant successfully placed checkers pieces on a magnetic board, located small objects and walked a short mobility course.

Conclusions: Using only a small subset of the 233 percept-generating electrodes with the first human-implanted ICVP system and minimal training, the participant, otherwise having only bare light perception, was able to successfully score a grating acuity of 2.3 logMAR or better. We continue investigating performance with stimuli at higher acuity levels and more extensive stimulation strategies.

Theo Doll

Additively Fabricated Cortical Electrodes – a materials challenge

Theodor Doll^1,2, Adrian Onken¹, Anna Sophie Heumann¹, Kristina Lachmann², Christopher Reiche³, Thomas Stieglitz⁴, Verena Scheper¹

1 Biomaterial Engineering, ENT, Hannover Medical School, Hannover Germany

2 Fraunhofer Leistungszentrum Biomedical and Pharmaceutical Technology Braunschweig - Hannover - Lübeck, Germany

3 Electrical and Computer Engineering, University of Utah, Salt Lake City, USA

4 Biomedical Micro Technology, IMTEK & Brain Links – Brain Tools Center, University of Freiburg, Germany

Cortical surface electrodes are research instruments but are used by some physicians for differential diagnosis trials in epilepsy. A key quality feature is their electrode density and their mechanical flexibility with which they nestle against the neocortex or dura, because the closer the electrodes come to the electrically active regions, the higher the effective channel separation. Therefore, there are also approaches to individually adapt these electrode mats to the topography (sulci) of the neocortex through additive manufac­turing, which would mean abandoning the previous gold standard of platinum electrodes and wires. The goal of such an endeavor would therefore have to be printing of the electrode mat substrate plus printing of electrical connections and electrode surfaces. When considering this, it is not far to rethink the whole architecture of the wiring: Why lay 1000 wires when you could multiplex them?

Well, the whole thing boils down to fundamental material and processing problems according to the current state of science and technology. In particular, the spectrum of materials available today does not offer a proper answer for a neurotechnology that takes the clinically relevant aspects of biocompatibility and conformity seriously: The lecture will talk about material systems for substrates, own preliminary work, missing solutions and approaches to completely new architectures, which admittedly again lead to new material and process questions.

Udo Ernst

I-See - Evaluating novel approaches for constructing visual cortical prostheses

Udo Ernst, David Rotermund

University of Bremen, Institute for Theoretical Physics

Present studies concentrate on constructing prostheses with many electrodes to interact with the primary visual cortex inside the brain. The objective of these devices is to create an image using light blobs, known as phosphenes. Our "I-See" project aims to assess extensions or alternative approaches that can enhance the standard of visual perceptions for blind participants. Specifically, it's imperative to comprehend the brain's language and utilize neurocomputational models to devise electrical activation patterns to yield precise visual percepts.

Our team aims to monitor the brain's background activity carefully, so we can deliver stimulation at the most opportune and successful moment, requiring minimal electric current to trigger the neurons. We also plan to utilize the complex neural representations of higher visual areas, such as V4, enabling us to transfer more information per electrode stimulation than with evoking phosphenes in V1. Our objective is to enhance communication with the brain and increase the functional abilities, safety, and longevity of visual cortex prostheses. Our consortium employs a combination of different approaches, tackling these goals from various angles: electrophysiology in non-human primates, stimulation and wide-field imaging in mice, psychophysical studies in humans, functional and structural imaging in normal-vision and vision-impaired subjects, as well as computational studies, simulations and modelling. This presentation offers an overview of the goals, ideas, approaches, and activities of I-See. The following presentations, in contrast, will focus on specific topics.

Authors are funded by ERA-NET NEURON. We are also funded by the “Iris und Hartmut Jürgens Stiftung: Chance auf ein neues Leben” and the “Stiftung Bremer Wertpapierbörse (SBWB)”.

Eduardo Fernandez Jover

Towards an advanced cortical visual neuroprosthesis based on intracortical microelectrodes

Cellular Biology at the Department of Histology and Anatomy of the University Miguel Hernández

We have proposed that arrays of intracortical microelectrodes, such as the Utah Electrode Array (UEA), might form the foundation for the restoration of a limited but useful visual sense to the profoundly blind. We will present our recent results regarding the implantation and explantation of intracortical microelectrodes in three blind volunteers (ClinicalTrials.gov identifier NCT02983370). We consistently obtained high-quality recordings, and the stimulation parameters remained stable over time. Microstimulation evoked simple and complex phosphenes at stable locations in visual space. The evoked perceptions allowed the participants to identify some letters, recognize object boundaries, and were of significant help in performing orientation and mobility tasks. No adverse effects have been reported to date. These results validate and support our previous findings, demonstrate the safety and efficacy of intracortical microstimulation via a large number of electrodes in humans, and suggest that several arrays of penetrating electrodes might form the basis for a cortically based solution for sight restoration in some individuals with profound blindness. However, there are still a relevant number of open questions, and more experiments should be done to achieve the clinical goals envisioned by this technology.

Ione Fine

Pulse trains to percepts: Using virtual patients to describe the perceptual effects of human visual cortical stimulation

Vison and cognition group, University of Washington

The field of cortical sight restoration prostheses is making rapid progress. However, as yet, we have only limited insight into the perceptual experiences likely to be produced by these implants. Here we describe a computational model or ‘virtual patient’, based on the neurophysiological architecture of V1, which successfully predicts the perceptual experience of participants across a wide range of previously published cortical stimulation studies describing the location, size, brightness and spatiotemporal shape of electrically induced percepts in humans. Our simulations suggest that, in the foreseeable future the perceptual quality of cortical prosthetic devices is likely to be limited by the neurophysiological organization of visual cortex, rather than engineering constraints.

Shelley I. Fried

Towards the development of a micro-coil based cortical visual prosthesis

Shelley I. Fried^{1, 2}, Sang Baek Ryu², Justin Tanner³, Bradley Greger³, Andrew Whalen⁴, John S. Pezaris², Seung Woo Lee⁵

1 Boston VA Healthcare System, Boston, MA, USA

2 Department of Neurosurgery, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA

3 Department of Biological and Health Systems Engineering, Arizona State University, Tempe, AZ, USA

4 Department of Neurosurgery, Yale University

5 Department of Brain and Cognitive Sciences, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea

Purpose: We are exploring the use of implantable microcoils as an alternative to conventional microelectrodes in the development of a cortical visual prosthesis. The use of magnetic stimulation to drive cortical neurons is attractive because the induced fields from microcoils are spatially asymmetric and therefore, can be harnessed to selectively target or avoid specific neuronal sub-populations. For example, implanted coils can activate local pyramidal neurons while avoiding activation of passing axons, thus confining activation to a focal region around each coil. Also, because there is no direct contact between coils and cortical tissue, many of the concerns about stability of the interface associated with microelectrodes can be avoided.

Methods: A series of in vitro, in vivo, and behavioral experiments have been performed in both rodents and non-human primates (NHPs) to evaluate the viability of microcoils. The in vitro and in vivo experiments in mice consistently show responses from magnetic stimulation are better confined to focal regions around each coil vs. those from electrodes. Psychophysical behavioral experiments in NHPs use a detection task to evaluate efficacy for eliciting phosphenes and determining activation thresholds. Additional experiments are underway to evaluate the safety and stability of implanted microcoils. Parallel efforts have been performed to assess the stability of implantable electrodes for stimulation of cortex.

Results: Chronic implantation of micro-electrodes into S1 of rats revealed changes in threshold over the course of implantation with some variability depending upon which layer was stimulated. Acute testing of magnetic and electric stimulation (implanted into V1 of mice, layers 2/3 and 5) both elicited strong surface (ECoG) responses, although the area activated by micro-coils was confined focally within a ~300-µm in diameter region, and that of micro-electrodes was more spatially expansive, often extending more than 1-mm from the stimulation site. In vitro testing of human cortical tissue (resected during medically necessary neurosurgical procedures) revealed comparable sensitivity of individual neurons to those measured in mice. A limited amount of psychophysical testing in non-human primates provides encouraging preliminary indications that the animal can detect magnetic stimulation from micro-coils acutely implanted in V1, and further, that activation thresholds are considerably lower than those from experiments in anesthetized rodents. Results from chronic implantation experiments indicate that microcoil performance remains stable. Temperature measurements indicate that repeated microcoil stimulation can be performed safely.

Conclusions: Our results continue to support the viability of microcoils as an alternative to conventional micro-electrodes. The focal magnetic activation from micro-coils in rodent in vitro and in vivo experiments, built on a theoretical analysis of microcoil magnetic fields, supports the notion that distinct, focal phosphenes can be created that will summate to produce more spatially complex percepts. The generation of phosphenes during a limited number of psychophysical experiments with NHPs provides additional support although further testing is needed to verify that elicited percepts are indeed more focal with magnetic.

Funding: Research supported by the NIH (NINDS and NEI) and by the Dept. of Defense. William M. Wood Foundation, Bank of America Trustee (JSP).

Diego Ghezzi

High-density wide-area cortical visual prosthesis

Ophthalmic and Neural Technologies Laboratory, Department of Ophthalmology, University of Lausanne, Hôpital ophtalmique Jules-Gonin, Fondation Asile des Aveugles, 1002 Lausanne, Switzerland

Visual prostheses provide artificial vision to blind people. Several devices have been developed and tested during the past decades, but the efficacy is still insufficient for everyday life. Retinal prostheses have been implanted in several hundreds of blind patients. Results from these clinical experiences are extremely valuable to the scientific community guiding the design of new visual prostheses, including cortical visual prostheses.

My group focuses on visual prostheses specifically designed to enable safe orientation and navigation. Starting from the results obtained from our work on retinal implants, I will translate this evidence into requirements for cortical visual prostheses. I will especially highlight the important role that the visual field size has during common daily activities. Hence, I will take the lessons learned and translate them into the design of a high-resolution wide-area cortical visual prosthesis.

Our concept is based on a distributed network of highly miniaturized implantable stimulators wirelessly powered and controlled from outside. Preliminary results on the fabrication and validation of this cortical visual prosthesis will be presented

Umut Guclu

Neural Coding and Neuroprosthetics with Deep Learning and Synthetic Reality

Donders Center for Cognition, Radboud Universiteit

In this talk, I will give an overview of our recent work combining neural coding with deep learning to simulate and emulate in vivo neural computation with in silico connectionism for "brain-reading and -writing".

Concretely, we delved into the intricate relationship between macaque visual cortex multi-unit activity and the latent representations from state-of-the-art deep generative models. Our findings accentuated the pivotal role of feature-disentanglement in moulding high-level neural representations tied to visual perception, leading to exceptional spatiotemporal reconstructions of visual experiences.

Addressing the pressing concern of blindness, we also ventured into refining the realm of cortical visual prostheses. We introduced a robust, fully differentiable phosphene simulator that offers biologically plausible visual percepts to aid those with visual impairments. This simulator amalgamates classical and modern clinical outcomes, ensuring realism and practicality in emulating prosthetic vision, thereby setting a new benchmark in simulated prosthetic vision pipelines.

Furthermore, tapping into the potential of deep reinforcement learning, we sought to optimize neuroprosthetic vision by devising task-dependent end-to-end training strategies in virtual domains. Preliminary evaluations in dynamic environments revealed that these adaptive methods surpass conventional feature extractors, hinting at the transformative power of deep reinforcement learning in enhancing neuroprosthetic vision in multifaceted settings. Together, our investigations pave the way for a future where deep learning and synthetic reality coalesce to revolutionize neural coding and neuroprosthetic applications.

Michael Herzog

A psychophysical approach to object rendering in future V4 prostheses

Michael H. Herzog¹, Udo A. Ernst², David Rotermund², Elsa Scialom¹

1 Brain Mind Institute, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland

2 Institute for Theoretical Physics, University of Bremen, Bremen, Germany

Many visually impaired individuals cannot benefit from retinal implants, highlighting the need for the development of cortical prosthetics. Cortical prostheses typically target V1, aiming to implant hundreds or thousands of electrodes, each eliciting the percept of a circular flash of light (phosphene). While V1 visual prostheses initially showed promising results, current implementations still face substantial challenges. As one electrode pulse leads to the percept of a single phosphene, heavy stimulation protocols are needed to deliver high-quality representations of visual scenes. This is not without risk for the blind patients since combined currents from many electrodes could cause brain tissue damage.

To overcome these limitations, we propose an innovative approach: targeting the activation of V4 cells for a future visual prosthesis. We hypothesize that V4 neurons, which potentially encode complex visual features such as curves and corners, could allow patients to discern objects with fewer activated neurons compared to V1-based prosthetic vision. To provide proof-of-principle to this claim, we conducted a psychophysical experiment with 46 healthy participants. We presented 30 objects both as phosphene collections (condition 1) and as corner-like fragments (condition 2) resembling the stimuli favored by V4 neurons. We evaluated the object recognition performance of participants under these two conditions. Our results show that participants required 27% fewer corner-like fragments than phosphenes for object recognition (B=-27.49, SE=3.48, p<0.001, pseudo R-squared=0.14). Detailed analyses showed that 76.67% out of the 30 objects presented were better recognized when expressed with corner-like fragments compared to phosphenes (0.00 < p < 0.48 FDR-corrected, 0.31< r <0.81). We conclude that observers use high-level visual information of local fragments during object recognition, highlighting potential advantages of integrating higher-level object information into cortical prosthetic designs.

Authors are funded by ERA-NET NEURON under grant NEURON-051. UAE and DR are also funded by the “Iris und Hartmut Jürgens Stiftung: Chance auf ein neues Leben” and the “Stiftung Bremer Wertpapierbörse (SBWB)”.

Dirk Jancke

Probing electrical brain stimulation in genetically modified mice

Optical Imaging Group, Institut für Neuroinformatik, Ruhr University Bochum

Our study is part of the EU-funded project “I-See: Improving intracortical visual prostheses using complex coding and spontaneous activation states” (ERA-Net Neuron), which involves a collaborative initiative from different neuroscientific disciplines. New strategies and new approaches for visual intracortical prosthesis shall be developed to increase their efficacy and long-life cycle capacities.

Our part of the project explores options for minimizing electrical microstimulation currents to drive visual perception. Specifically, using wide-field imaging of fluorescent voltage indicators, we explore properties of the coupling between subthreshold microstimulation and internally ongoing (i.e., spontaneous) brain activity in genetically modified mouse models.

We suggest that synergistic interactions between microstimulation and ongoing brain activity can push complex visual information contents across perceptual thresholds. Exploiting the fact that ongoing neuronal activity reflects partly top-down processes and attentional resources may enable to reduce the relatively strong currents as currently used in cortical visual prostheses - particularly, when applying closed-loop stimulation approaches that generate well-timed electrical subthreshold stimulation. In the future, such a predictive stimulus encoder could trigger adaptive microstimulation patterns that resemble basic signatures of natural visual input in blind patients.

Arto V Nurmikko

Wireless Networks of Implanted Microchips for Distributed Cortical Sensing and Stimulation

Jihun Lee¹, Ah-Hyoung Lee¹, Vincent Leung², and Arto Nurmikko^1,3

1 School of Engineering, Brown University, Providence, RI, USA

2 Electrical and Computer Engineering, Baylor University, Waco, TX, USA

3 Carney Institute for Brain Science, Brown University, Providence, RI, USA

Scaling up the number and number density of intracortical microstimulation points, perhaps up to thousands, is being actively pursued by a number of different multielectrode approaches. In this talk we show progress from recent work where autonomous, spatially distributed wireless silicon microchips are being developed to enable space-time patterned intracortical microstimulation. Each chip, no more than 300 µm in linear dimension, is powered at near 1 GHz frequency from an external RF source. A telecommunication co-design enables precise wireless control of the amplitude, pulse duration, and timing of current pulses delivered at each stimulation site across the entire distributed chip population with microsecond latency. The approach is being co-developed in concert with design for chip populations capable of fully wireless multichannel neural recording, with emphasis on spike-type neuronal event detection.

Christopher Pack

Design considerations for an extrastriate visual cortical prosthetic

Department of Neurology and Neurosurgery, Montreal Neurological Institute

Most visual prosthetics attempt to restore vision by stimulating neurons in the retina or the primary visual cortex (V1). These areas contain a high-resolution map of visual space, so that complex images can in principle be reconstructed point-by-point through artificial stimulation. While this approach has shown great promise, I will suggest that an equally fruitful approach could be to target the parts of visual cortex beyond area V1, which are known collectively as extrastriate visual areas. These areas have a tighter connection with perception and action, contain a sparser and more robust representation of visual stimuli, and are particularly adaptable to new behavioral conditions. I will describe these visual functions of extrastriate cortex, as well as preliminary work indicating their causal role in perceptual experience.

Carlos Ponce

Estimating the capacity of neuronal population encoding via image reconstructions

Harvard Medical School, Dept. of Neurobiology

If we want to understand how brains perceive, decide, and act, we must study how neurons work together at the same time. Thanks to advances in imaging, high-density recording probes, and deep artificial networks, we can now study neuronal populations and explore theories of visual coding. However, to truly grasp these codes, we must figure out how individual neurons within these groups relate to the whole population. Can we understand a population code if we learn the function of all its constituent units, or are there unique or emergent properties at the population level?

This talk will delve into this complex relationship in both primate brains and artificial networks, using a geometry manifold framework. We will define the tuning of neurons individually and explore how these responses might combine at the group level to create population representations. Using a technique called 'code inversion' to reconstruct images from these group responses, we will show how highly active neurons (or 'directions') play a key role in processing visual scenes. Our findings may provide new insights into efficient sampling strategies for visual bio-prostheses.

Bogdan Raducanu

Chips to neurons and back: electronic circuits for neural recording and stimulation

Imec, Leuven

For the past 200 years, development has moved from the discovery of the electrical nature of nerves to implants which can supplement a lost body function. Through this presentation we are going to look into the fascinating intersection of electronic circuits and neural interfaces, showcasing their pivotal role in the development of brain-machine interfaces. It explores how sophisticated microchips have revolutionized the field by evolving from simple, generic components to specialized applications, like neural probes, which are supported by a multitude of advanced fabrication technologies and circuit design techniques. This has enabled high-resolution, high throughput neural recording and precise stimulation which can be used for neuroprosthetics, cognitive augmentation, and our understanding of neural processes.

Pieter R. Roelfsema

Visual perception and consciousness and their restoration when the eyes fail

Netherlands Institute for Neuroscience, Amsterdam

Institut de la Vision, Paris

A long-standing dream of scientists is to be able to directly project images from the outside world onto the visual brain, bypassing the eyes. This method could provide a solution for blind and visually impaired patients. It is the only possible solution for patients in whom the connection between eye and brain is lost so that a prosthesis in the eye is not an option.

I will first give an overview of the functioning of the visual cortex, which has low level areas for the analysis of simple visual features and higher areas for the analysis for more complex properties such as object category and face recognition. I will then discuss the mechanisms that determine whether a visual stimulus will reach consciousness or not. It is well established that the electrical stimulation of electrodes in the visual brain leads to artificial percepts called "phosphenes". This method also works in patients who have been blind for decades. The goal of our own research is to bring a prosthesis for the visual brain closer. We implanted 1000 electrodes in the visual cortex to generate complex visual patterns. We demonstrated that this stimulation leads to interpretable images, in the same way that pixels form recognizable patterns on a screen. These new neurotechnological developments take important steps in the direction of prostheses that can restore a rudimentary form of vision.

Andreas Schander

Development of neural probes for chronic recording and electrical stimulation

Andreas Schander & Björn Lüssem

University of Bremen, Fachbereich 1 - Physik / Elektrotechnik

For the development of long-term stable neural prostheses, we need electrical bidirectional interfaces, which allow recording of neural activity and electrical stimulation over a long period of time. This is challenging due to the foreign body response of the brain tissue and the long-term term stability of the used electrode and insulation materials in the harsh saline environment.

In this talk I will present the development results of our institute towards long-term stable neural interfaces for ECoG and intracortical applications. A promising material for these bio-interfaces is the organic semiconducting polymer PEDOT:PSS, which we use not only for the microelectrodes but also in the future for the integration of organic electrochemical transistors on neural probes for improved quality of recordings.

Michael C. Schmid

Influence of visual cortex stimulation on perception: Insights from experiments in non-human primates

University of Fribourg, Switzerland

Newcastle University, UK

Visual cortical prostheses promise to restore basic visual capacities in the most severe forms of blindness. Traditional approaches relying on electric stimulation of primary visual cortex (V1) are increasingly supplemented by optogenetic treatment offering more specific neuronal targeting.

During the first part of the talk, I will present how optogenetic stimulation of macaque V1 triggers neural modulation in local and remote neural networks that are critical to induce a visual percept. Evidence is presented based on behavioral, fMRI, electrophysiological and histological assessment.

In the second part of the talk, I will discuss the limits of V1 focused stimulation approaches and how they might be overcome by additional targeting of higher-level cortical areas. To this end, I will present preliminary results that demonstrate how electrical stimulation of cortical area V4 can support visual event detection.

Martin Schrimpf

Do Topographic ANNs Predict the Behavioral Effects of Neural Interventions in Primate IT Cortex?

EPFL

Particular artificial neural networks capture core object recognition behavior and the neural mechanisms underlying it with increasing precision (see www.Brain-Score.org for overview). These models take images as input, propagate through neural representations that resemble neural recordings at all stages of the primate ventral stream, and produce behavioral choices that resemble human behavioral choices.

We here take this modeling approach to experiments that directly perturb neural activity and measure the resulting changes in visual behaviors. Predicting neural intervention effects is a crucial component to enable computer-to-brain interfaces such as visual prostheses. Specifically, we extend leading topographic models of primate visual processing with perturbation modules that predict the change in neural activity from perturbations such as micro-stimulation, optogenetic, and muscimol suppression.

We then test these models’ behavioral predictions following particular neural perturbations against the findings from experimental literature in a suite of 9 primate IT perturbation benchmarks. Without any fitting to the benchmarks presented here, we find that a model co-trained with a spatial correlation loss predicts all 9 unilateral behavioral results. Models with random topography or topographic unit arrangement after training on the other hand predict less than half the experimental results. Due to a lack of hemispheric processing, none of the models are able to predict bilateral effects. Conversely, the quantitative predictions of even the best model are consistently misaligned with experimental data, over- as well as under-predicting the behavioral effects. Taken together, these findings suggest that topographic models equipped with perturbation modules already capture the qualitative experimental findings from neural interventions - but not yet the quantitative magnitudes.

Fabian Sinz

Exploring the Visual System with Functional Digital Twins and Inception Loops

University of Göttingen, Institute for Computer Science

Deep nonlinear system identification models have set new standards in modelling the responses of large-scale populations of neurons to natural stimuli, yielding models that can accurately predict the response of thousands of neurons to arbitrary stimuli and can account for how behavior modulates responses of visual neuron. This allows us to treat the model as a functional digital twin of the neural population and probe neurons in ways that would not be feasible experimentally. With that, we can derive new hypotheses about the neural populations in silico and consequently verify them in vivo, in a paradigm we call inception loops.

In this talk, I will give an overview of the models, and showcase several examples of how they can be used to derive novel insights for the visual system in mice and monkeys. We believe that the combination of large-scale recordings under natural stimulation and deep data-driven modelling is a paradigm shift in systems neuroscience towards understanding computations in sensory system on complex ecological stimuli.

Anna Wang Roe

A novel interface for cortical columnar neuromodulation with multi-point infrared neural stimulation

Fei-Yan Tian ^1,2, Ying Zhang ^1,2, Jia-Ming Hu ^1,3, Kenneth E. Schriver ^1,3, Anna Wang Roe ^1,2,3,4

1 Interdisciplinary Institute of Neuroscience and Technology.

2 College of Biomedical Engineering and Instrument Science.

3 School of Brain Science and Brain Medicine.

4 National Key Laboratory of Brain and Computer Intelligence, Zhejiang University, Hangzhou, China.

Traditional visual cortical prosthetic designs, largely based on electrical stimulation, have not incorporated known cortical columnar architecture which underlies visual featural encoding (e.g. contour orientation, color, motion, depth information). Here, using pulsed infrared neural stimulation (INS), we have developed an approach with the submillimeter spatial precision needed for interfacing with such cortical columns. Using INS delivered through a linear optic fiber array targeted to orientation domains in cat visual cortex, we monitored, using optical imaging, the effects on responses in contralateral area 18, an area with columns sensitive to motion-induced orientation.

We find that this INS stimulation selectively modulates response to ongoing visual oriented gratings. That is, responses of orientation-matched domains (parallel to array orientation) are enhanced and non-matched domains are reduced. As columns targeted by the array are of mixed orientation, this orientation-selective effect is a higher order, integrated effect. Controls included dynamically applied speeds, directions and patterns of multipoint stimulation, and the fiber array placed at different orientations.

In sum, this interface demonstrates (1) columnar specific effects, (2) visual hierarchical integration, and (3) a means to link across the two visual hemifields. Also important is the ability to achieve functionally selective neuromodulation of ongoing visual stimulation, something pertinent for the low vision population. Combined with our finding that monkeys make saccades to INS induced phosphenes, we suggest this provides groundwork for a conceptually and technologically new generation of BMI.

Daniel Yoshor

Implementing a Visual Cortical Prosthetic: Advances and Challenges

Penn Medicine, Dept. of Neurosurgery

Since the 1960s, the prospect of developing a visual cortical prosthesis for restoration of useful vision to the blind has tantalized scientists, engineers, and clinicians. After two decades of relative inactivity, interest in this approach has been renewed and there are now multiple active efforts to develop next generation visual prostheses and to bring these devices to clinical implementation. We will review the current state of work on cortical visual prosthetics in human patients, with a particular emphasis on our own work, and will highlight advances in the field as well challenges that must be overcome to produce a clinically useful prosthetic device.

Alireza Dehaqani (Poster)

Estimating neuronal population encoding via image reconstruction

Harvard Medical School

Visual cortex neurons are organized to provide information about objects and places in the natural world. While individual neurons show tuning for attributes combining color, form, and texture, overall, information of the natural world is distributed across populations. It remains unclear how and which of these attributes are actually encoded in population response patterns. To investigate the encoded visual information present within population response patterns, we recorded from neurons in macaque V1, V4, and IT using chronically implanted microelectrode arrays. We presented photographs ("target images") to the monkeys and measured the population response patterns of neurons to each image. To quantify the retained information in these response patterns, we conducted closed-loop image synthesis experiments; specifically, we “threw away” the photograph and attempted to recover it using image generators (generative adversarial networks) combined with adaptive search algorithms. The objective was to synthesize new images evoking response patterns that best matched the target-image population response pattern. In parallel, we also used convolutional neural networks (CNNs), which serve as useful models of visual cortex, in a similar experimental setup to estimate the kinds of visual information that can be retained, and the population encoding strategy in the middle layers of a feedforward architecture.

Our findings show that it is feasible to generate images that provoke population response patterns similar to those of the target images, both when using CNNs and the monkey cortex recordings -- there was a measurable increase in the visual similarity between the reconstructed and target images across the synthesis process. Furthermore, experiments involving CNNs revealed that visual information was distributed in the population of units, each of which encodes primitive features. We observed that increasing the population size resulted in better image reconstructions despite a concurrent decrease in the ability to replicate the target population response pattern. However, we also discovered that relying on units that were most activated by the target image resulted in better reconstructions than relying on units that were randomly sampled from the same layer. This suggests that in a multidimensional activity space, strongly activated units provide more information, consistent with a sparse representation. Next, we are investigating how this computational result translates to the ventral stream.

Mansoureh Jalili (Poster)

Illumination patterns for optogenetic applications by means of a transmission hologram formed by two photon polymerization on a single mode fiber

Mansoureh Jalili ¹, Joris Jaruschewski ² , Claas Falldorf ¹ , Ralf B. Bergmann ^{1, 3}

1 BIAS - Bremer Institut für angewandte Strahltechnik, Klagenfurter Str. 5, 28359 Bremen, Germany

2 previously at BIAS, present address: Hermannstr. 16/17, 28201 Bremen, Germany

3 Universität Bremen, Fachbereich Physik/Elektrotechnik, Otto Hahn Allee NW1 and MAPEX – Center for Materials and Processes, 28359 Bremen, Germany

New opportunities in neuroscience and specifically optogenetics can be created using computer-generated holograms (CGHs) as diffractive optical elements across the tip of a single-mode fiber (SMF). The holograms can, for example, be used to influence the behavior of light-sensitive cells by exposing them with different light patterns. Here, we show a CGH fabricated on the facet of a SMF which forms a predefined illumination pattern at a given distance. To maximize the space bandwidth product (SBP) of the CGH, a coreless termination fiber (CTF) spliced to the SMF Is used to illuminate a CGH with a large cross-section. The CGH is formed using two-photon-polymerization (2PP) by direct laser writing. The numerical and experimental procedures will be discussed in detail, and the optical functionality of the system will be demonstrated.

Fatma Karama (Poster)

Linking electrical stimulation to patterns of ongoing activity – Optical imaging in mice with fluorescent indicators

Fatma Karama, Dirk Jancke

Optical Imaging Group, Institut für Neuroinformatik, Ruhr University Bochum, 44780 Bochum, Germany

Our study is part of an EU-funded project (ERA-Net Neuron) which involves a collaborative initiative from different neuroscientific disciplines. New strategies and new approaches for visual intracortical prosthesis are being developed to increase their efficacy and long-life cycle capacities. By targeting higher visual areas and by using subthreshold stimulation protocols, the project explores options for minimizing electrical currents to drive visual perception. Specifically, in our study in mouse primary visual cortex (V1), as well as in higher antero- and lateral visual areas (AL/LM), we explore the physiological properties of the coupling between subthreshold microstimulation and internally ongoing (i.e., spontaneous) brain activity.

Our hypothesis: through synergistic interactions between microstimulation and ongoing brain activity, low current microstimulation can boost complex stimulation patterns of ongoing cortical activity across perceptual thresholds.

Thus, exploiting the potential of ongoing neuronal activity that partly reflects top-down and attentional resources, shall overcome current limitations in cortical visual prostheses that use overly strong currents. In the future, closed-loop low-current stimulation approaches could enable acquisition and processing of data gathered from a visual scene by a camera and generate well-timed electrical subthreshold stimulation patterns. In blind patients, such a predictive stimulus encoder could trigger adaptive microstimulation pattern that resemble basic signatures of natural visual input.

Elsa Scialom (Poster)

Optimizing objects rendering in future V4 prostheses: a psychophysical approach

Elsa Scialom ¹, Udo A. Ernst ², David Rotermund ², Michael H. Herzog ¹

1 Brain Mind Institute, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland

2 Institute for Theoretical Physics, University of Bremen, Bremen, Germany

Blindness is a devastating disease affecting 40 million individuals worldwide. There is often no cure, but retinal and cortical prostheses may offer partial restoration of vision. Most research on cortical prostheses focuses on implants stimulating V1 neurons to generate phosphenes, i.e., percepts of small patches of light. However, phosphenes provide only little visual information, making object recognition challenging for blind patients. For this reason, we aim for a V4 prosthesis which activates higher-level representations of visual scenes, bypassing entire low-level visual processing. Since V4 cells could signal the presence of more complex visual features such as curved segments or corners, patients would need activation of a smaller amount of neurons to recognize objects compared to V1 prosthetic vision. To provide proof of principle evidence for this approach, we conducted a psychophysical study in which 46 sighted participants performed an object recognition task. Thirty objects were both expressed as a collection of phosphenes and as corner-like fragments similar to the stimuli preferred by V4 neurons. Image statistics such as spatial frequencies, fragment’s size, location and number were made comparable. In line with our hypothesis, linear mixed model shows that participants needed on average 27% less corner-like fragments than phosphenes to recognize objects (B=-27.49, SE=3.48, p<0.001, pseudo R-squared=0.14). Object-specific analyses showed that 23 out of the 30 objects (76.67%) needed significantly less corner-like fragments to be recognized compared to phosphenes (multiple Mann-Whitney-U-tests, 0.00 < p < 0.48 FDR-corrected, 0.31< r <0.81). We conclude that high-level visual information of local fragments is used by observers for object recognition. This highlights the need for rendering higher level information of objects in cortical prostheses.

Authors are funded by ERA-NET NEURON under grant NEURON-051. UAE and DR are also funded by the “Iris und Hartmut Jürgens Stiftung: Chance auf ein neues Leben” and the “Stiftung Bremer Wertpapierbörse (SBWB)”.