Skip to search boxSkip to navigationSkip to main content

The batch primary components transformer and auto-plasticity learning linear units architecture: synthetic image generation case

Research Output:
Chapter in Book/Report/Conference proceeding
Conference contribution
 Peer-review

Abstract

Context tokenizing, which is popular in Large Language and Foundation Models (LLM, FM), leads to their excessive dimensionality inflation. Traditional Transformer models strive to reduce intractable excessive dimensionality at the among-token attention level, while we propose additional between-dimensions attention mechanism for dimensionality reduction. A novel Transformer-based architecture is presented, which aims at the individual dimension attention and, by doing so, performs the implicit relevant primary components' feature selection in artificial neural networks (ANN). As an additional mechanism allowing adaptive plasticity learning in ANN, a neuron-specific Learning Rectified Linear Unit layer is proposed for further feature selection via weight decay. The performance of the presented layers is tested on the encoder-decoder architecture applied for the synthetic image generation task for the benchmark MNIST data set.

Publication Information

Output type

Research Output:
Chapter in Book/Report/Conference proceeding
Conference contribution
 Peer-review

Original language

English

Publication milestones

  • Accepted/In press - 08/11/2023
  • Published - 02/01/2024

Publication status

Published - 02/01/2024

Publisher

Institute of Electrical and Electronics Engineers Inc., United States

ISBN (Print)

9798350329957

ISBN (Electronic)

9798350318906

Publication IDs

  • handle.net: 10547/626147
  • Scopus: 85183472574

Host publication title

Proceedings - 2023 10th International Conference on Social Networks Analysis, Management and Security, SNAMS 2023

Access to documents

Publication metrics

Metrics