If I remember correctly Temporal Fusion Transformers tackle the problem of incorporating non sequential data into a transformer.

Custom fusion

Make it part of the sequence? Kinda like special tokens in language models.

Look at papers dealing with multi-modal tasks. e.g. Perceiver/Perceiver IO by DeepMind

You can encode your data into tokens with the same size using something like an MLP. Then feed these tokens into decoder along with encoder tokens. Should probably also add an learnable embedding for different types of data to prevent signal confusion.

Assuming your goal is an autoregressive sequence prediction, I would just modify the start-of-sequence token. For example: Use some reasonable model which takes the non-sequential context and creates a vector. Add that vector to the the learned start-of-sequence token vector. Future time steps will be able to attend to the start-of-sequence token as needed to retrieve the context.

If you're only using a transformer encoder, and not doing the autoregressive thing, I would just add an additional token to the input. I would most likely used a learned position encoding to add to that context vector rather than the normal sequential position embedding. Any time step will be able to attend to this special token and take advantage of the context clue you're providing.

Or, for vectors, just slam it into the start of the sequence directly (use a normalization technique if you need to align dimensionality).

If you feel the need, place some sort of separator token ('###') between the "context features" and the input data.

Use another model to reduce this context to a vector, then append it to each token. This was the process used in Set Transformers (TSPN)

In Conditional GANs, the condition(class label) is prefixed to the latent space random vector, ie. The input vector becomes one-hot-class + rv. You can learn the 'conditions ' such as style by joint training. I believe similar concept can be applied to transformers, judging from the other answers.

There are a few ways to incorporate non-sequential context into a transformer model:
Attention Mechanisms: One way to incorporate non-sequential context is to use attention mechanisms that allow the model to "pay attention" to relevant parts of the input as it processes it. For example, the transformer model uses self-attention mechanisms that allow it to consider the entire input sequence as it processes each element in the sequence.
Context Vectors: Another way to incorporate non-sequential context is to use context vectors, which are fixed-length vectors that represent the context for a given input. These vectors can be concatenated to the input embeddings or used to compute attention weights.
Multi-Head Attention: The transformer model also uses multi-head attention, which allows it to attend to multiple different sources of context simultaneously.
Conditional Transformer: The conditional transformer is a variant of the transformer model that is specifically designed to incorporate non-sequential context by using an additional input modality (e.g. an image or a set of control parameters) to condition the transformer's self-attention mechanisms.
Hierarchical Transformer: The hierarchical transformer is another variant of the transformer model that incorporates non-sequential context by using a hierarchy of transformer blocks, where the lower-level blocks process the input at a finer granularity and the higher-level blocks process the lower-level representations to capture more global context.
Graph Transformer: The graph transformer is a variant of the transformer model that is specifically designed to process graph-structured data, which allows it to incorporate non-sequential context by considering the relationships between nodes in the graph.

Did chatGPT write this

Also have a look at the slot attention mechanism, which does something similar but arguably more elegantly