Your browser doesn't support javascript.
loading
Auto-Spikformer: Spikformer architecture search.
Che, Kaiwei; Zhou, Zhaokun; Niu, Jun; Ma, Zhengyu; Fang, Wei; Chen, Yanqi; Shen, Shuaijie; Yuan, Li; Tian, Yonghong.
Afiliación
  • Che K; School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, Guangdong, China.
  • Zhou Z; Peng Cheng Laboratory, Shenzhen, Guangdong, China.
  • Niu J; School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, Guangdong, China.
  • Ma Z; Peng Cheng Laboratory, Shenzhen, Guangdong, China.
  • Fang W; School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, Guangdong, China.
  • Chen Y; Peng Cheng Laboratory, Shenzhen, Guangdong, China.
  • Shen S; School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, Guangdong, China.
  • Yuan L; Peng Cheng Laboratory, Shenzhen, Guangdong, China.
  • Tian Y; School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, Guangdong, China.
Front Neurosci ; 18: 1372257, 2024.
Article en En | MEDLINE | ID: mdl-39108310
ABSTRACT

Introduction:

The integration of self-attention mechanisms into Spiking Neural Networks (SNNs) has garnered considerable interest in the realm of advanced deep learning, primarily due to their biological properties. Recent advancements in SNN architecture, such as Spikformer, have demonstrated promising outcomes. However, we observe that Spikformer may exhibit excessive energy consumption, potentially attributable to redundant channels and blocks.

Methods:

To mitigate this issue, we propose a one-shot Spiking Transformer Architecture Search method, namely Auto-Spikformer. Auto-Spikformer extends the search space to include both transformer architecture and SNN inner parameters. We train and search the supernet based on weight entanglement, evolutionary search, and the proposed Discrete Spiking Parameters Search (DSPS) methods. Benefiting from these methods, the performance of subnets with weights inherited from the supernet without even retraining is comparable to the original Spikformer. Moreover, we propose a new fitness function aiming to find a Pareto optimal combination balancing energy consumption and accuracy. Results and

discussion:

Our experimental results demonstrate the effectiveness of Auto-Spikformer, which outperforms the original Spikformer and most CNN or ViT models with even fewer parameters and lower energy consumption.
Palabras clave
Texto completo
Añadir a Mi BVS
Imprimir
XML
PubMed Links
Buscar en Google

Texto completo: 1 Colección: 01-internacional Base de datos: MEDLINE Idioma: En Revista: Front Neurosci Año: 2024 Tipo del documento: Article País de afiliación: China Pais de publicación: Suiza
Texto completo
Añadir a Mi BVS
Imprimir
XML
PubMed Links
Buscar en Google

Texto completo: 1 Colección: 01-internacional Base de datos: MEDLINE Idioma: En Revista: Front Neurosci Año: 2024 Tipo del documento: Article País de afiliación: China Pais de publicación: Suiza