Viewing a single comment thread. View all comments

ZestyData t1_jegdmzo wrote

Putting aside the political undertones behind many peoples' desire to publish "the algorithm", this is a phenomenal piece of educational content for ML professionals.

Here we have a world-class complex recommendation & ranking system laid bare for all to read into, and develop upon. This is a veritable gold mine of an an educational resource.

631

Educational-Net303 t1_jeggs0s wrote

Yeah, like Elon or not, the push for open source is always going to be beneficial to the community. Ironic how twitter is more open than ____AI.

308

Erosis t1_jegj2l9 wrote

Twitter is already established as a brand to near saturation and Elon has more money than god. It's the perfect combo for ML philanthropy. Now waiting for that Tesla vision algorithm...

93

FinancialElephant t1_jeh33j9 wrote

Most infrastructure code like computer vision code, device drivers, etc are either not culturally relevant or have little cultural relevance.

I don't think it makes any sense to prioritize them when things like twitter have much more direct cultural impact. It would be great if my network card driver was open source, but does it really matter? Is it worth prioritizing? Will it likely have any cultural relevance? To most people the answer to all these questions is no.

−6

pier4r t1_jegm5a1 wrote

> world-class complex recommendation & ranking system

https://twitter.com/amasad/status/1641879976529248256?s=20

I mean surely it is great but my recommendations weren't exactly stellar in those years.

38

Ulfgardleo t1_jegoe8z wrote

this aprt is not used for recommendations though. this is for analytics and internal testing and ensuring that different groups (+elon) don't get disadvantaged.

30

ZestyData t1_jeh12gm wrote

Idk man as a fairly well seasoned MLE I find their general architecture and scale of their combined models to be fascinating in-and-of itself.

Twitter sucks ass - but this is a beautiful piece of ML Engineering.

8

grumpyp2 t1_jegkfri wrote

Where to start with, it’s such a huge project 😳

24

LetMeGuessYourAlts t1_jegmjkk wrote

Readme.md

Sorry, had to 🤓

68

Internationalizard t1_jegnp8m wrote

I checked the commit history but it has only one commit. So this is a pretty straight forward place to start: https://github.com/twitter/the-algorithm/commit/7f90d0ca342b928b479b512ec51ac2c3821f5922

22

lordofbitterdrinks t1_jegql5s wrote

So how do we know this is the repo used by Twitter and not some stripped down version of it

14

ZestyData t1_jeh198p wrote

This quite obviously isn't the repo used by twitter.

It is a pretty large and well put together documentation epic & consolidation of multiple microservices.

Whether the content is 100% reflective of whats deployed is completely unclear. But its not "fake" that's for sure, its genuinely too many man-years of work to not be in-essence real.

52

f10101 t1_jegs5yt wrote

It will take time, but I'd imagine it should be possible to derive a method of determining this by observation.

Algorithms like this will have fingerprints.

14

MjrK t1_jegtjqj wrote

We don't and likely we won't know.

Unless perhaps someone internal checks and leaks important missing details that later on...

But for now, it does seem robust enough to be reflective of what they have probably been using up to some recent - but that's still just speculation

10

LoaderD t1_jegsuar wrote

> Here we have a world-class complex recommendation

...You know this is twitter's recommender system right? All the tweets I interact with are ML related from very 'left' people like Jeremy Howard.

My recommender system could legit be:

if interested_in_finance_or_ML:
     recommend_alt_right_hate_speech_accounts()
     recommend_crypto_scam_ads()
24

Educational-Net303 t1_jegta0z wrote

Get rid of the if statement and you just recreated Twitter's recommendation algorithm

28

Roger_Cockfoster t1_jegy0u7 wrote

In fairness, it doesn't really matter what you interact with. Twitter is just a sewer of alt-right hate speech for everyone.

−9