English has this thing called adjective order, which might not technically matter in a lot of cases, but will still sound wrong. Other times, it changes the meaning. In situations like this one, the order doesn’t really change the meaning but it does help the recipient understand the meaning.

In this case, you put the material (brick) before the color (brown). Only, the color describes both the material and the wall, so the color comes first. In a similar fashion, despite (big) referring to the size of the wall, it also gives insight into the size or quantity of the materials used, so it comes before the material. The result, ‘the big brown brick wall,’ is the only way to arrange these adjectives because for the reader, each adjective helps to clarify the next.

A brick wall is a wall of bricks. A brown brick wall is a wall made of brown bricks. A big brown brick wall is a wall made of a lot of brown bricks. Presented like this might help you see why, in this example, the right order helps reading comprehension.