Viewing a single comment thread. View all comments

mattsverstaps t1_jefvu45 wrote

Reply to comment by Pas7alavista in [D] Turns out, Othello-GPT does have a world model. by Desi___Gigachad

So the extra dimensions are unnecessary? I just realised that there could be some situations in which non orthogonal dimensions are preferable. I can’t exactly think of them. Doesn’t it suggest a pattern in data if a mapping is found that reduces the dimension? Like I picture from linear algebra 101 finding a line that everything is a multiple of so one dimension would do and that line is a ‘pattern’? Sorry I’m high.

2

Pas7alavista t1_jeg5dhh wrote

>so the extra dimensions are unnecessary

Yes one reason for embedding is to get extract relevant features.

Also, any finite dimensional inner product space has an orthonormal basis, and the math is easiest this way so there's not much of a reason to describe a space using non orthogonal dimensions. There is also nothing stopping you from doing so though.

>Doesn't it suggest a pattern in data if a mapping is found that reduces dimension

Yeah generally you wouldn't attempt to use ML methods on data where you think there is no pattern

>Something something Linear algebra

I think you might be thinking about the span and or basis but it's hard for me to interpret your question

2

mattsverstaps t1_jegdsq4 wrote

Yes the span, so if we discover that a set of points is actually all in the span of a line, that line is a kind of fact or pattern about the points. So probably there is an equivalent in higher dimensions. I am seeing there is a problem whereby we introduce our own bias in creating our model.

2

Pas7alavista t1_jegu8de wrote

The span describes the entire space. It's a set of vectors that you can combine using addition and multiplication in order to obtain any other vector in the space. For example a spanning set over the real number plane would be {(1,0), (0,1)}. This particular set is also an orthonormal basis and you can think of each vector as representing two orthogonal dimensions. This is because their dot product is 0.

However, any set of two vectors that are not on the same line will span the real number plane. For example, {(1,1), (0,1)} spans the real number plane, but they are not orthogonal.

Overall though it is always important to be aware of your input space, and the features/dimensions that you use to represent it. You can easily introduce bias or just noise in a number of ways if you aren't thorough. One example would be not normalizing your data.

2