Function-Space Transformer (FST) Introduced for Adaptive Function Representations
The new Function-Space Transformer (FST) architecture introduces spatially adaptive continuous latent representations for function learning, outperforming prior models on PDE prediction and image classification tasks.
The Function-Space Transformer (FST) is a new neural architecture designed for learning from functions over continuous domains. Unlike traditional models that rely on fixed uniform grids, FST introduces a spatially adaptive, continuous latent representation by placing feature-bearing anchors at locations predicted from the input data. These anchors, refined recursively through function-space interactions, allow the model to adapt its spatial organization for each input.
FST supports both spatially detailed and finite-dimensional output modes, making it flexible for various scientific and visual data tasks. On benchmarks such as PDEBench Burgers and Darcy flow, FST outperforms Perceiver IO, and is competitive with the Fourier Neural Operator. For image classification on ImageNet-1K, FST achieves better accuracy than Vision Transformer (ViT) baselines with fewer parameters. Ablation studies also validate the impact of the adaptive anchor mechanism and recursive refinement strategy.
What Developers Should Do
- Explore the FST approach if working with data defined on continuous domains, especially physical fields, geometric objects, or signals with spatial and/or temporal structure.
- Consider FST as an alternative to grid-based or fixed-latent neural methods for improved adaptivity and efficiency.
- For computer vision tasks, evaluate whether FST's adaptive anchor mechanism delivers better resource utilization and accuracy compared to standard transformers.
- Review the FST paper ablation results to understand feature contributions and design decisions relevant to your domain.
