JimmyTheCrossEyedDog

JimmyTheCrossEyedDog t1_j6osqa5 wrote

Good call, shape is the much better term to avoid confusion.

> If we’re considering the dimensions to be the number of datapoints

To clarify - not the number of datapoints, the number of input features. The number of datapoints has nothing to do with the dimensionality (only the shape).

> Deep learning or CNNs are great because of its ability to extract meaningful features from data with shape > 2

This is where I'd disagree (but maybe you have a source that suggests otherwise). Even for time series tabular data, gradient boosted tree models typically outperform NNs.

Overall, shape rarely has anything to do with how a model performs. CNNs are built to take knowledge of the shape of the data into account (restricting kernels to convolutions of spatially close datapoints), but not all NNs do that. If we were using a network with only fully connected layers, for example, then there is no notion of spatial closeness - we might as well have transformed an NxN image into a N^2 x1 vector and your network would be the same.

So, neural networks handling inputs that have spatial (or temporal) relationships well has nothing to do with it being a neural network, but with the assumptions we've baked into the architecture (like convolutional layers).

3

JimmyTheCrossEyedDog t1_j6odc4c wrote

> When talking about dimensions I meant (number of rows, number of features) is 2 dimensions for tabular data...

Right, but my point is that when people say "NNs work well on high dimensional data", that's not what they mean.

> You could consider an image to have width x height x channels features

It does have that many input features, i.e. dimensions, like you've written below.

> but thats not what a CNN does, the cnn extracts meaningful features from the high dimensional space.

Now we're talking about composite or higher level features, which is different from what we've been talking about up to this point. It's true that for tabular data (or old school, pre-NN computer vision) you generally start to construct these yourself whereas with images you can just throw the raw data in and the NN does this more effectively than you ever could, but this is irrelevant to the input dimensionality.

3

JimmyTheCrossEyedDog t1_j6nv3zg wrote

This feels like a mix-up between the colloquial and mathematical definitions of dimension Yes, NN approaches tend to work better on very high-dimensional data, but the dimension here refers to the number of input features. So, for a 416x416x3 image, that's >500k dimensions, far higher than the number of dimensions in almost all tabular datasets.

> image data 4D (extra dimension for batch)

The batch is an arbitrary parceling of data simply due to how NNs are typically trained for computational reasons. If I were to train a NN on tabular data, it'd also be batched, but it doesn't give it a new meaningful dimension (either in the colloquial sense or the sense that matters for ML)

Also, NNs are still the best option for computer vision even on greyscale data, which is spatially 2D but still has a huge number of dimensions.

edit: I'd also argue that high dimensionality isn't the biggest reason NNs work for computer vision, but something more fundamental - see qalis's point bin this thread

17

JimmyTheCrossEyedDog t1_j44zozr wrote

Not sure if you're tagging to warn me about him using my reddit name in his tweet, but I'm not really bothered (although I do appreciate your tag just in case, as I hadn't seen the edit). Did not expect this kind of childish reaction to what I thought was my, your, and several others' constructive (albeit sometimes a bit blunt) criticism, though - yikes.

1

JimmyTheCrossEyedDog t1_j3viyyk wrote

Neat idea, but

> as soon as we find VC

What do you see as the value proposition of made up search engine results? And what makes this different than every other use of LLMs?

I would caution against the idea of "finding VC" as a sure thing when everyone and their brother is throwing together a GPT-based startup right now, most of which will go nowhere.

14

JimmyTheCrossEyedDog t1_j2lzmhs wrote

As someone who knows next to nothing about electronic components, can you provide some example inputs and outputs? Without knowing what the exact problem is, it's hard to determine feasibility.

Off the top of my head, if the symbolic language is quite simple (i.e., every symbol acts more or less independently of each other, so you can just tack the text for what each does one after another), you can essentially do this with optical character recognition or some computer vision approach and just use a simple set of rules to translate each visual detection into text. If the language of how these diagrams work is more complicated than that, though, it may not be so simple.

3

JimmyTheCrossEyedDog t1_j17e3k4 wrote

Great responses so far. One other thing to consider is the purpose of this model. Will it be used to make inferences on out-of-sample data? If so, you should make sure that the form of data you're training on is representative of the form of data you'll have operationally.

In other words, will the out of sample data also have five replicates like you have for your training data? If not, then you should train using all five replicates, not an average. Otherwise, your out-of-sample data will have variance that has been averaged out by your training process in a way you cannot perform on the new data.

If this model isn't for prediction, you have more flexibility.

4

JimmyTheCrossEyedDog t1_izc6ym1 wrote

This was one of the most impressive AI advancements I've seen in recent memory, so congrats and kudos on such great work.

As I see it, one simplifying factor that Diplomacy (like any game) has is the discrete set of potential actions to be taken. When it comes to extending an AI like CICERO, to other sorts of problems, do you see the possibility of such problems having a non-disctetizable action space as a major hurdle, and are there particular difficulties associated with that and potential mitigations for them?

15