Mamba Architecture for LLM/AI Models

What is Mamba?

Mamba is a promising LLM architecture that offers an alternative to the Transformer architecture. Its strengths lie in memory efficiency, scalability, and the ability to handle very long sequences.

Mamba is based on State Space Models (SSM) and Gated Multilayer Perceptron (MLP) mechanisms.

How does it work?

1. Input Projection: The dimensionality of the input sequence is increased.
2. Convolutional Processing: One-dimensional convolution and an activation function are applied.
3. Gating: The input data and the projection results are element-wise multiplied.
4. Repetition: Steps 2-3 can be repeated several times.

Advantages of Mamba Architecture:

• High Performance: Demonstrates excellent results on LAMBADA and PIQA tests, surpassing even models twice its size.
• Memory Efficiency: Utilizes recomputation during backpropagation, saving memory similarly to Flash Attention.
• Scalability: Outperforms Transformer++ on long sequences, especially when increasing computational power and model size.
• Long Context: Can process context up to a million tokens.
• Efficient Text Copying: Excels at text copying tasks.