A lot of people here are responding to the message but not to the meaning.

It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of when "Just LLM it" will not be the right choice.

"Build an OS" wasn't a common university project because we were all expected to go out and work on Windows, but because understanding the bare-metal firmware for a computer helps you deeply understand how to intuit building for a whole class of problems.

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities.

The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM company, and still then it's a struggle. Those few that do train, they spend most of their budget on compute and have relatively small teams.

Getting experience in this field requires having access to very expensive hardware to begin with. And the skills will be quite hard to convert into any real value for someone, leading to a decent income, unless you have a ton of funding from patient investors, or you have decent contacts in Bay Area networks to get hired at the right place.

With all due respect, paulg is in somewhat of a bubble, this is not congruent with the global situation.

I'm (more than) twice that age, but I've spent time learning this exactly this from videos by Andrej Karparthy and from books by Sebastian Raschka.

I didn't do it because it was useful to me in a practical sense. It's because LLMs are fascinating and I want to know how they work. From that perspective it's been a great experience. I have afirm grasp of the basics. This makes it much easier to understand frontier concepts like compressed latent attention. I can follow the field and understand it.

Not sure I would have got as much out of it at seventeen. I have a lot of background and experience which made it much easier to learn. I wasn't struggling with the linear algebra or with python. I already knew pytorch and neural networks. That helped a lot and I covered these tutorials fast and could skip over large sections. A few evenings and the odd weekend day over a couple of months was enough for me.

For seventeen year olds the tutorials are good enough to make it possible to learn this but it would have taken a lot longer to understand. On the other hand I would have learned a lot more. I think I would have learned a lot of valuable stuff.

However I also think 17 year old me was studying for his A levels and probably this was right choice in terms of maximising future opportunities. I'm not sure I think learning about LLMs instead is sensible. Indeed it might be bad advice. But I can absolutely agree with the sentiment.I think 17 year old me would have wanted to do this too.

While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias).

Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as this post implies?

I agree more with Yann LeCunn's salty reply. Over long run, knowing how autoregressive language models work from scratch will be just one step in having foundational understanding, and they might become dated... like knowing how a CRT monitor work. Something of historical interest and good for learning, but not crucial to being well-rounded.

There are other types of models like diffusion models right now that are showing more efficiency and have a higher ceiling for improvement. Understanding math and fundamentals are more important.

"...and then train..." and those dummies in openai and google wasting billions, while all they need is some 17 year old and few pizzas

I'll use this post as a shameless opportunity to tell more people about a little side project, I made:

http://languagemodelbuilder.com teaches you (in a few hours to days) how to build an LLM from scratch. It's entirely free, without accounts, and without data collection.

I'd say, if you were 17, you'd probably be better off learning how to socialize or at least gracefully approach and relate to people your own age. That's probably the greatest challenge of the near future, sadly.

I am kind of amazed how negative the comments are here, especially on HN.

Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

I appreciate the sentiment and I'm pretty curious how I could train an LLM, even are really basic one from 5yrs ago without nuts hardware.

That said, I a good starting point for a 17yo is reading about perceptrons[0], then the basics of neural networks[1] (ex. 3-layer perceptron) then writing a program to train a 3-layer perceptron and classifying the MNIST dataset[2] - a dataset of characters.

This can anywhere between a day and a week and you will demystify the basics of neural networks and work your way forward with more advanced contemporary concepts.

Fun fact: any multi-layer perceptron neural net can be reduced to a 3-layer perceptron network.

[0] https://en.wikipedia.org/wiki/Perceptron [1] http://geeksforgeeks.org/deep-learning/neural-networks-a-beg... [2] https://www.kaggle.com/datasets/hojjatk/mnist-dataset/data

Horrible advice. This may have been good advice 10 years ago, but not today. There are no positions for people who "kind of understand how toy LLMs work" because so many engineers do these days. Most of the real LLM optimization work is at the edge of research and highly proprietary and not something you could ever do without infra that costs millions.

But of course, 10 years ago this wasn't obvious.

If I were 61, I'd learn to not be so out of touch with the younger generation

As someone who's at a similar age and was interested in learning how to do this, there just aren't enough resources to do so. Most LLM research is in the form of academic papers, and there isn't any 'popular' way to learn these things, and besides that all said research assumes you have a B200 cluster ready to go. If you have weaker hardware (say an 8GB nVidia GPU, which is what I have) you're going to be limited to fine tuning small models or torturing yourself working the GPU for days per iteration trying to run things like https://github.com/karpathy/nanochat, which is hardly an educational experience. Renting cloud GPUs is expensive, and at this age the most I could muster up for experimentation is probably $100 or so, which only gets me 25 or so hours on a B200 which just isn't enough. So why would I bother myself with this if I'm already at a disadvantage because of not having access to the right hardware and when surely there are better ways to spend my time? I concluded the only way to learn and be competitive is by finding work at an AI lab somehow (not happening at 17), or studying ML at the right university.

Putting this in contrast with programming, I learned coding when I was 8, and it was incredibly stimulating to learn because you can quickly iterate and there were thousands of books and YouTube tutorials that dumb everything down and teach you fundamentals. All you needed was a $300 computer, and you can learn nearly anything you want, without being gatekept from this or that because you don't have enough vRAM / an sm_100 GPU.

I think a problem a lot of people are grappling with here is that due to LLMs and AI generally, it’s basically impossible to predict what the future will look like or what jobs will still be around.

I’d probably say something like: do something you enjoy and seems like it might be useful, but accept that the pace of change may mean that whatever you study ends up being irrelevant.

Whatever solution there ends up being to this, it’s not going to be one that an individual 17 year old can implement. We’re past the point where individual good and bad choices matter that much to economic outcomes.

And what resources would you use for that?

(Besides the obvious nano gpt)

Don't really understand why people are saying this is terrible advice. If you're 17, you should be learning everything. At that age, your brain is a sponge and your energy levels are the highest they will ever be.

None of us know what the future of work, education, or AI is going to look like. But your best bet is to become a life-long learner. Be it LLMs, musical instruments, physics, or business.

I get this is basically advice for young founders and entrepreneurs, but i would ignore that request and encourage 17 year olds to spend time trying to find a happy medium between work and life.

Being a super rich and an unhappy workaholic, or a super-impressive engineer who wakes up one day at 45 and realizes they regret wasting half their life (I ran into way too many of these) is a much worse fate than "not being rich from your startup" and working a relatively regular job while feeling fulfilled and happy by more than just work.

Especially in the US, which is uniquely bad at this and encourages people to work themselves to death, mental health and work life balance are much more valuable things for 17 year olds to focus on than finding good startup ideas.

In case you think i'm being a bit dramatic, let's look at the state of 17 year old mental health in the heart of Silicon Valley:

"The City of Palo Alto and the Palo Alto Unified School District approved a funded contract to place 24/7 human security guards and monitors at all four local Caltrain grade crossings, including the Churchill Avenue crossing directly adjacent to Palo Alto High School."

(in case it's not obvious, it's because of suicides by high school students)

The 17 year olds do not need advice on better startups, and this situation will never get better if we focus our advice on how to be better at work instead of how to be better at life. This will require redirecting the conversations.

if i were 17, i'd learn a trade.

Yeah, no way.

I'd move to the middle of nowhere and work multiple jobs on a farm and in construction. Learn how to grow food, and build things. Meet the farmer's daughter, and marry her. Then, buy my own land, grow my own food, and build my own things.

Most 17 year olds I know don’t want anything to do with AI and see the entire industry as an existential threat.

Nothing wrong with learning the theory and understanding the papers. Getting to that point you’ll have to get your fundamentals down. Might be an interesting exercise.

But as a future? I guess we’ll see. I suspect the next financial apocalypse will determine if there is one. Another AI Winter that may outlast all others so far.

If I were 17 again I'd prepare to go for volunteering overseas after high school for 1-2 years (plenty of free options in the EU where you might only need to cover the plane ticket). See the world, you learn a new language, help others and then think about what you want to do.

I built from scratch a sparse text embedding model trained on a 13T token corpus. Not the same as an LLM because there’s no transformer in the mix, but still I learned a shit load of things in order to solve all sorts of problems that emerge when you try to access big datasets and daily update tables with billions of rows. But if it wasn’t for a specific use case that I tried to solve I don’t think that whatever knowledge I gained could be utilized in the market. Sparse models are a very small niche and most people I’ve come across with similar knowledge are in academic circles, not business related ones. So even if LLMs are all the rage these days I doubt the demand for people who know how to build them is that high. Someone who knows how to setup an open weight model and expose an API might be more valuable to a company these days.

I had the same inspiration!

I spent a days reviewing the lecture notes for CS336: Language Modeling from Scratch - and then trained a nanoGPT-esque model in PyTorch.

I'd recommend trying it for those who are curious. Computational bottlenecks become much more intuitive when you've looked at the overall process.

Mr. McGuire: I just want to say one word to you. Just one word.

Benjamin: Yes, sir.

Mr. McGuire: Are you listening?

Benjamin: Yes, I am.

Mr. McGuire: Plastics.

Benjamin: Exactly how do you mean?

Mr. McGuire: There's a great future in plastics. Think about it. Will you think about it?

---

I love this scene because it so perfectly captures what it's like to be young and given advice, however well-meaning, by an older generation living in a world that no longer exists for the young. And it's ambiguous and trite enough to be essentially useless even if the underlying idea isn't terrible.

I'm not sure what 17 year old me would have done with YouTube tutorials for everything under the sun available.

I'm much older and less wise now, but I still afforded myself the opportunity to follow karpathy's tutorials to build a LLM from scratch. Got to play with a few ideas. Seen similar ideas turn up in frontier model work, which is quite gratifying.

There are so many ideas to try.

Currently playing with autoencoders that takes A and B and produce latents A', B', and C'. Reconstruction of A is from A' and C', B is from B' and C'

The idea is if C' can be made to improve both outputs, it must store as much information as it can about what is common to both inputs.

The only realistic approach is to train on a limited data set which is probably less usable than the comibnation of a custom RAG + one of the many available LLMs.

Also many people/kids don't have access to proper "productive" systems anymore, since the whole computing and electronics industry shifted to make "consumer"-devices like smartphones or laptops made for netflix, gaming and spotify.

Breaking the barrier to build a custom system, install linux (or developer tools for Windows, MacOS) is already a complex AND costly task. It was just way simpler in the late 90s and 00s to get something working.

> Notice that what I would not do is try to start a startup. Instead I'd build the foundation of knowledge to base a startup on later.

There's also this little-known concept called learning things for learning's sake and not always trying to capitalize on it.

> Someone asked what I'd do if I were 17. I'd learn how to build LLMs from scratch, and then train ones as powerful as I could with whatever hardware I could get access to.

He's seeing a future for models running on everyday hardware just capable enough to do what the use case requires.

What corpus would one be advised to use for training? Should one limit oneself to legally downloadable documents or should one follow the examples of Altman, Amodei, and Zuckerberg and use LibGen and Anna's Archive?

Oddly, I just remembered I did the nearest possible thing to this when I was 17... back in 1991.

On an Amiga, I took various public domain text documents from cover disks and counted the probability of the next word given the previous word. Then spat out random sequences of words from it and printed them out. It was called "Splurge". Basically a very very simple single layer statistical language model.

Some of the sentences were randomly not bad sentences, which seemed amazing at the time!

That kind of thing (and Core Wars and Tierra etc) did lead me to getting a job at an artificial life startup at the end of the decade. But that was in turn about 10/15 years too early (no GPUs).

There's some lesson from this about timing, but honestly I've gained the most as a person when I did something that was fun, ethical and gained an audience. A tricky combination.

I thought pg was trying to live forever. Has he learned LLMs from scratch?

What about doing abliteration, weight pruning, representation engineering, etc, directly to open LLMs instead?

Building an LLM from scratch has a hard split between a tutorial project you can complete in a weekend (that's useless for actual usage) and then a solid 1km high brick wall if you want to create anything actually useful from scratch.

Modified open models have a very active community around them, without the need to look much further than Hugging Face.

I really don't understand Paul's reasoning here. Does he predict more scarcity on the model-level? That layer seems to be almost a commodity now + training is damn expensive.

If you are really 17, my advice is to identify use-case for AI (ideally relevant for businesses) that work most of the time and find ways to make them work pretty much every time. AI reliability is the scarcity right now.

I get it the sentiment behind the post…but it has some “let them eat cake” vibes though.

The problem is tech moves so fast that by time kid is adult llm are outdated

This might be of use

https://github.com/raiyanyahya/how-to-train-your-gpt

I think learning how to build your own agent harness framework from scratch and really understand each part of it, what are the modern components that a good agent harness are using these days, is more valuable than learn how to build LLMs, but that depends on what you want to do with your career.

I'd learn a trade in all seriousness.

(Edit: And learn how honest business works)

A teenager in China actually did that and got a paper accepted at ICML. The interview podcast is in Chinese, but you can ask AI to summarize it https://www.xiaoyuzhoufm.com/episode/6a8472b95aeb2a5712e8de7...

When I was not 17 at the times of GPT2, I decided to not bother with learning how to build LLMs because it’s too expensive for an individual. This escalated quickly.

telling a 17 year old to get into tech right now is horrible advice, literally telling them to get at the back of a line with a better part of a million more experienced people in it.

Yeah sure, learning the internals of a technology that’s being actively developed will probably teach you things that remain useful for awhile even if your learning outcomes could end up being different than what he’s implying.

Whether that makes a good long-term investment is quite debatable. IMO, LeCun’s take (at the end) is far more forward-looking since it aims to gain more insight into what could come next based on what we know about the current state of the art.

All that being said, the extent to which these people capitalize on our tendency to be blinded by the halo effect is incredible. A constant stream of bite-sized aphorisms…

This is especially at a different level for early startup figures who happened to be in the right place at the right time and usually did little more than digitizing mundane, traditional day-to-day processes. Yet they’re treated as geniuses and prophets, with people hanging on their every word as though everything they say contains some deeper wisdom. PG and the like often strike me as broken clocks and they’re still profiting from having been very early players in the game, who had good instincts for commercialization and capitalism.

————

LeCun’s reply:

> I would try to figure out why LLMs can write my essays but not clean my bedroom. Then I would study topics in college and grad school that could help solve that problem. I'll figure out a set of methods and architectures beyond LLMs that can quickly learn to perform physical tasks as efficiently as humans and animals. That last item is also what I would if I were 30, 40, 50, or 66 years old

If I were 77 I'd do the same

I learned HTML when I was 17 in about 1995 and it's certainly taken me on a pretty fun career path. Less technical than LLMs for sure, but 'figure out where the industry is going and move what you're learning to there' is solid advice.

I'm making an assumption here, but I think paulg is implying that "learning LLMs" today is like the equivalent of "learning computers" in the earlier days. We could even divide civilization into two eras: Before Transformers (BT) and After Transformers (AT).

When I was 17, I made bad cartoons and was in a band. If I were 17, I'd spend more time learning music and design theory. I also learned PHP at this time, but that was low on the list, friendships came first.

If i were 17, I'd try to distinguish who to take advice from, and would definitely learn that VCs have interest to spread a specific agenda in their message. Also, I would get drunk and have as much fun as could, as the misery of working under the treat of being replaced by AI, would simply kill any desire to live past 25.

You can build the program, but to train it is another beast, billions of docs, images, videos, which a mere mortal doesn't have access to

Second in line, build your own agent, that's more in our ballpark, then customize it to your needs, both virtual and physical

https://deeplearningwithpython.io/ is a decent (free) read for anyone looking to follow this advice.

Writing, supervising and training LLMs are now the purview of... even larger LLMs. Optimising CUDA kernels; hand-writing SIMD assembly to speed up data loading; tinkering with your particular brand of DRAM to see if there's anything to gain from optimising for its memory topology and NUMA --- these are now the job of AI.

There is very little reason for humans to get all too engrossed in this type of work now, today, with the hope of being good enough at it to command a high salary in 3-5 years. AI can already do it incredibly well, and they can do it persistently and doggedly 24 hours a day.

I think it's more about the scar tissue (a.k.a. intuition) when building LLMs from scratch than whether it's transferable to job search. Maybe the person will decide they don't like LLMs and not develop that into a career, or maybe they become a researcher in another field because that's a better way to solve problems inherent in LLMs.

This essentially comes down to choosing the search for substance (here in the form of technical depth) over short term gratification and quick wins in life.

I think this kind of mindset should be taught way more in school so that people really appreciate learning a subject deeply.

My kids CS teacher asked me what they should do after AP CS.

Previously they had a class where they'd build apps for other teachers. Like tracking when clubs are, etc. But now that's become easy for teachers to vibe code themselves.

I suggested they shouldn't prereq this class on CS. Heck invite anyone in interested in "building things" and they can get practice at building apps for other people / themselves. Maybe that becomes a gateway TO CS - people who want to learn how things work under the hood.

I suggested post CS class for the CS people should probably be building an LLM or something :)

i was trying to build llms from scratch at 17. failed miserably because i did not know linear algebra. ended up in a different but adjacent field. when chatgtp got big suddenly there were so many people doing llms and i didnt want to compete like that. so much of what was happening was hype, and that really turned me off. am back to building llms from scratch, but like, its a journey teaching myself all the theory on top of my job. i have decent fundamentals, but i need a better grasp of all the advancements in the field in the past 5 years before i would feel comfortable designing anything. baby steps, essentially. am working on better understanding all the layers of an ai while implementing a rag on my local model as an experiment. 17 year olds should learn whatever theyre interested in but need fundamentals in order to do anything advanced.

A 17yo can train a small GPT this weekend. nanoGPT is a few hundred lines. Understanding why it works is the part that takes a decade.

What would you do if you’re 30 years old now?

As a platform engineer being based mainly out of Australia/Hong Kong, opportunities seem to be getting less unless targeting high frequency trading or banking.

It seems like building a startup with the help of some AI tools might be the best bet.

I'm usually a fan of pg, but this post is ignorant of modern AI technologies. Building an LLM from scratch is both a trivial and a useless exercise. There's probably in the range of 5000 github repos doing exactly that. What makes LLMs work is scale, and what makes engineering and training LLMs hard is also scale. And scale is not something you can achieve in your garage.

If the goal is to understand LLMs deeply, one would be better served by either joining one of the big AI companies or doing a PhD. And to be honest, I think this journey should have been started 5 years ago, because right now there's too much competition.

I wonder at the “worlds” Mr Graham envisions, and what is their cardinality. Is this the only 17 yro reimagining, or is there an army of 17yro, of which this LLM curious persona is but one?

So I am 15. Is it worth trying to build my own archive (s-1.site) of strategy ideas? Do I have any real differentiation? Or am I just wasting time? I figure that I can use it as proof that I have some know-how?

Why is it important to train LLMs or even fine-tune them? LLMs have proven their point, costs are crashing, and there are more of them than most companies need.

The real value is to unlock meaningful insights and directions from existing data that is there inside companies.

I live far outside any tech city, so maybe I do not understand. But working with LLMs full-time, building for clients and tons of own experiments, I see no value in building on LLMs.

Ah nostalgia. At 17, I learned how to write programs in BASIC on a mainframe.

I would (and am) going into MLOps. Not just the general infrastructure/systems administration but how to do inference optimization, caching, quantization, memory pinning, vfio passthrough of gpus etc.

Easier said than done - where do you get the B300s from?

Better to start working with harnesses, evals, statistical analysis, etc. - where you don't need the huge hardware for pre-training etc.

This might be a helpful resource: https://laurentiugabriel.github.io/token-town/

If I was 17 again I would sack off work and study and focus on chasing the opposite sex, without the angst I had at the time. I don't know a single person who regrets having had too much sex when they were young. I would not build an llm, too hard, too expensive to run. Learn how easy life is if your morals allow you to grift money of vcs into your own funds and retire.

The core point here is that AI is a massive thing (at the moment) so it's probably a good idea to understand it deeply. Not sure why people are so worked up about it.

I assume in this hypothetical Cornell just gives away degrees to passersby.

If you're 17, and seriously curious about how modern day AI works, you might as well just sit down and look at a couple of courses on linear algebra + calculus, machine learning, deep learning, and more LLM specific deep learning. Those courses will teach you how to go from writing your first perceptron to a MVP language model. But also so much more.

When I was 17, I wrote poetry and learned how to play guitar. I’m really happy that I did. That’s what I would do now, were I 17. I feel slightly bad for people who didn’t.

As an aside, for someone interested and who's an absolute beginner, can someone please recommend good resources on how to build LLMs from scratch? Thank you in advance.

... and it would be totally pointless.

I mean first that is already what plenty of 17yo are actually doing, because that is what they do at school or in parascholar activities. There are already countless of such tutorials where you can do that in an afternoon.

The pointless part though is precisely why Amazon and others are hunting for rare books, all the low hanging fruits have been picked already so just training a bigger model will simply mean burning more energy and money. Sure training a small one for the basic principle is a great pedagogical thing, training another one, medium, then maybe a large one, is also good in term of learning the process and architecture, but one should not expect it to be useful out of that context.

Pure players are precisely doing everything they can to corner the market by making their own scale unreachable by others. Smaller players with access to lesser infrastructure are thus betting on different market, e.g. embedded systems.

17yos should definitely build their (L)LMs from scratch and whatever bigger model they can train for free, or for cheap, but they should not expect that to bring them any riches.

Seek advice from nice people that you actually know instead of rich people on the internet.

Does anyone have a roadmap ?

LLMs are the new compilers.

I don't think you can really call yourself a developer unless you at least have an idea how to build a more complex software project like a compiler, and maybe have built a toy one either at uni or for fun.

It's not clear how long this LLM age of AI will last (to be replaced by something better), but nowadays any developer should at least understand the basics of ANNs, and more than just the "hello world" of a cat vs dog CNN. An LLM/Transformer is maybe the equivalent of a compiler in that regard - something that we all use and is complex enough to present a bit of a challenge. You should at least understand the basics of how an LLM is built, and maybe building a toy LLM will/should become the new Comp. Sci. degree toy compiler replacement.

If you’re 17, you might learn hands on knowledge, like tacit knowledge in areas such as lathes, precision engineering, metrology, and other very niche fields. You can also try climbing and explore arts like music, painting, and drawing. And, of course, spend some time in nature.

LLMs are incredibly boring to me as a technology..Not in terms of what it can do, but how it works.

It’s probably wise to learn how to build one to understand what you’re dealing with. However, if I were 17 I would lean how to apply an LLM to a problem instead of strictly building one.

An LLM isn’t hard to make - the training data is hard to get and prepare.

The big companies stole the data. The average person can’t do that

Paul G is not writing this for a general audience of your run of the mill “engineer” hoping to be employed by someone. He is writing it for future founders. What knowledge / skills you need to develop today to be well positioned to have a startup worthy insight when you are 24.

certainly better than wasting time with harness and agent workflows that will become irrelevant at the next evolution, same thing happened with 'prompt engineering'

Can any1 share any resource to learn that skill. I want something that has been tried by you. I can too search on the internet...

i am a small fan of pg, nevertheless i find this to be an exceptionally good take and it is strange to me to see so much piling on to this one in particular here.

learning about llms is not useful so that you can make llms later, you want to learn about it so that you can work on next generation architectures. llms before long i imagine will be left in the dust by ebm / physics oriented models especially that can have an embodied understanding of the world. but a lot of things you learn about them are transferable by doing something like this

Why are people so negative about this? It feels like a fun project and at 17 the stakes are not really high. Something one could easily do on summer break in a couple of weeks.

Didn't his swiss watch essay say he'd essentially leave the industry because there will only be bloat from now on?

As if you couldn't learn what LLMs are at any Age.

The core technoology is pretty basic, developing a rudimentary understanding for why the individual parts work as well as they do is tricky.

I read a bit about how LLM works, but as a hobbyist it is pretty frustrating that I won’t be building anything useful without throwing a lot of money at it

I think pg answered the question as “what I’d do as a project” and not “what I’d do as a career.” So the critical comments are kind of missing the point, IMO.

I don’t see why learning how LLMs work is a bad project for a 17 year old.

Optimizing your entire career and the next decade+ of your life on LLMs? Yeah, probably not ideal. It’s almost always a bad idea to make long term decisions based on current trendy things.

And since everyone is using this topic to give their ideal advice to 17 year olds, my advice as a mid-30s guy: seriously consider becoming highly skilled at a specific thing, and don’t be scared off by the idea that it’ll take 5-10-15 years to get there.

When you’re 17-25, the timescale of a decade seems infinite. But it’s really not, and a decade spent “exploring and keeping your options open” sometimes just ends up with you being pretty decent but not amazing at a lot of random things.

Sometimes I wish I had just become a carpenter, chef, electrician, etc. – a specific skill set that leads to mastery over time, rather than the endless exciting-new-thing hamster wheel of working in tech.

I'd just build an agent, it's easier than you think and you'd learn a lot about the "magic" of LLMs.

It’s really disheartening to see how many people don’t know shit about LLMs, by reading the comments… ironic given what OP is trying to say

2 years before everyone was doing custom training. What happened to all those today when frontier models itself become more powerful than custom trained ones?

If only we could train LLMs on commodity hardware, right?

LLMs will take away all jobs except yours, LLM maker. Keep at it, you're safe!

What are the best resources to learn how to build LLMs from scratch for 17 year olds?

I have my opinion on this but I'd like to hear the HN opinion, I will just say one thing:

If you are starting with little knowledge, like a 17 year old would, letting an LLM explain it to you is a terrible idea.

Would building an LLM from scratch imply writing code by hand?

Why do intelligent people still use X? Thanks for the xcancel.com link!

If I were 17, I’d learn how to invest and build financial literacy, and plot potential growth of my networth throughout my life, before even thinking about a career. Then smoke a bowl.

Paul has the same problem just about every tech obsessed engineer has (including myself), he thinks everyone else loves computers too. They don't. At all. I''m a self-taught ex-bartender and I can't tell you how many grown adults in the service industry I've tried to get into computer stuff and they had 0 interest.

And younger people I meet don't even own laptops. I had a genz/millennial cusp friend who wrote all her college papers on her iphone.

17 is such a fantastic age to be free and experience the world, you won't get that much of an advantage as these FOMO groomers are selling you into if you start now versus later.

If you are 17, go be yourself, whatever that is, in whatever way you want that to be, but do it so authentically and fully. Be unapologetic about what you love and what motivates you, and pursue that with passion and commitment.

I would not waste my time with yesterday's fad. The next unicorn generator will be something else.

Paul Graham:

>"Someone asked what I'd do if I were 17. I'd learn how to build LLMs from scratch..."

That's funny, Paul Graham, because if I were 17 again,

I'd learn how to program in LISP.

https://www.paulgraham.com/rootsoflisp.html

https://www.paulgraham.com/iflisp.html

https://www.paulgraham.com/hundred.html

(And/or other LISP derived languages... Clojure, Scheme, Racket, TinyScheme, etc.)

I guess "the grass is always greener..." as that old expression, that old "chestnut", goes... :-)

I've come to accept that some people, when faced with impressive technology, simply want to use it. They genuinely have no interest in understanding how it works. Lately, its even become fashionable to shame people for trying to understand ("you still read code? gross, you know AI can do that for you"..)

I will never understand this mentality, to let yourself be so dependent on something you don't understand at all is to live like a child. But it is very common. I doubt too many 17 year olds will bother even trying to understand what an LLM is, let alone build one from scratch.

When I was 17 I was building Windows Phone apps, bad decision on my part.

I think this is weird advice.

Learn how to make language models from scratch, yes. But learn how to use them, in the context of other machine learning tools, on very small hardware.

When the bubble bursts (and I still tend towards thinking it could burst rather than be deflated in a manageable way), the focus will be on uses of AI that are not like the hyperscalars' products.

People will still be interested in useful AI being added to small things — assistive technologies, home security, garden monitoring, their phones and smartwatches, robotics.

Instead of reductive, reactive make-an-anthropic-competitor advice like this, what about advising 17 year olds to focus on broad, integrated, helpful AI — or on going back through eighty years of history to look at AI projects that failed and reassess them?

Curious, what would you do if you are a 40 year old?

This is about as intelligent as say "If I were 17, I'd learn digital electroinics". You would waste your time. Sure, in theory it's useful, in reality it's not that useful.

yeah i loved tech so started learning circuity and soldering. but it was a waste of time i made my living learning how to program web applications.

All the negative comments here are really missing the point. At 17, you should be building your foundation. Kids that can make a custom CPU, or retrofit an old car with an electric motor, or screw around with nuclear energy... these are kids that are doing it just to see if they can. It's not about jobs, it's about curiosity and stretching limits and seeing what you can do, who you can be.

I remember when I caught PG on reddit arguing with some guy who'd said something mean about him. He didn't reveal who he was. But you could tell from his history - his first post was from before reddit opened to the public.

Good times.

I wonder if you can dig that out of the historical reddit database. I'd like to see that again. I love how everything is recorded now.

When you see anthropic job postings that say the job may not be available in a year... it really doesn't inspire much confidence. I say if you're 17, learn to weld, solder, and fit pipe...

That's applicable only if you live in the US.

if i were 17, i'd do these things:

1) LC until mediums

2) Calculus & linear algebra even if i understood nothing i'd just stare at vectors and derivatives.

Really? because I feel like LLMs are already pretty much a commodity, not to say their won't be advances in LLMs but I don't see the models themselves being all that ripe for disruption the way that the web was and such. I'm guessing chip design and manufacturing processes will be more important than models in the future.

It seems this is poor advice in that it’s suggesting young people should focus on the current problem as opposed to future problems. Focus on the current problem can result in making some money but it will result in making the incumbents more money, which is not disruptive. Isn’t the goal of radical software startups to maximize disruption?

If the two current bottlenecks, for this LLM madness that could very well be a bubble, are processing capacity and accuracy (a second processing problem) then what comes next? Isn’t that where young people should be looking or are we just giving up on innovation?

It’s crazy how much survivorship bias gets repackaged as generic advice.

Wait no it’s not, that was always happening.

What’s crazy is that people still believe in it.

I get the sense things have changed a bit since I graduated and there are lot more jobs in AI outside of academia these days, but it's still a very different field from other SWE pursuits, and it's not really accessible to hacker-minded people.

Learning AI isn't like learning HTML in the 90s then expecting to get a job at a tech company building websites. You can't just "learn how to build LLMs" and expect a frontier lab to hire you so I'd argue this is rather bad advise.

Additionally, unlike web development in the 90s you cant really do anything interesting yourself... All of the interesting/useful stuff will require huge amounts of compute and data so there isn't even much point in learning to start your own thing either.

As someone whose built many of NNs from scratch (hand written code, long before the days of LLMs), it's more or less useless knowledge if I wanted to work in a frontier lab or do anything interesting in the field.

I also think anyone thinking about going into a field which is basically a crossover of CompSci and Maths is absolutely insane right now. Even if you think there is a place for CompSci and Maths post LLMs, there's almost no chance anything you learn today will be relevant to the skills required in say 5-10 years.

Paul Graham wants you to work for him, not be him.

YC Combinator guy says that with a time machine he would learn to build the currently trillions-valued or whatever technology. Okay.

I don't think individuals have the resources to build an interesting llm. The l stands for large. You need a dataset too. Llms are only interesting because theyre large

And it's basically a weekend project to put transformers together in a ML library and train it.

The follow up comment,train it to play a game also doesn't make sense? Llms Sony really play games and there are better ml approaches to do that?

Terrible advice. If I were 17, genetics and bio tech at the next frontier, with opportunities to be more than another corporate drone. AI is a lot of bureaucracy and nepotistic who ya know already.

Let’s make the xcancel link the actual link, please!

On first glance, this is good advice and in general I'd give the same for this age group. With age, you'll see more of these scenarios come up and if you have experience and foresight, you can provide direction that may prove fruitful to young people. My son and his friend asked "what should we look into and learn?", about 15 years ago, I told them Python and Java; Python because of versatility and cryptocoins; Java for the long term stability in the job market. I despise Python personally, but I could see its potential and still do; especially for AI. At the end of the day, kids have more time than money and its great experience for them to get their hands dirty and find out what they might be interested in; its a long life.

If I were a billionaire, I'd learn how to give away a lot more of my wealth.

Neural net maths are hardly above scientific high-school level.

Somewhere in rural America is a 17 year old that doesn't even have working plumbing in their house still.

I'm sure they'll get right on powering up their computer from the hamster wheel, Paul.

My heart goes to all the kids out there that didn't get the fair shake let alone fair access to tech that gets these condescending "learn to code/learn to LLM" bootstrappy talks from rich pricks that don't know what life really can be like for a lot of American kids out there.

[flagged]

This is always such a nonsense question-answer thing, asking a person who already succeeded what they would do if they were young.

Even worse when they ask themselves.

Can't this guy enjoy being rich in silence? His takes get worse with every passing year.

Venture capitalist suggests everyone to become his future employee, just as he has been (successfully) doing for his whole life.

I genuinely don't think telling young people to do anything tech-related is good career advice. We don't even know if entry level roles will ever come back. The situation couldn't be worse for these roles. PG thinks that some random teenager will build a startup and get rich from it, or some shit. Like get real, man. Completely out of touch, tech bro who hasn't worked a real tech job for the last like 20 years... Remind me how many startups succeed again, Paul? What about the market dynamics for LLMs and how one might "secure compute"?

If I were 17, but have the money I have now he means.

What the fuck does this guy know about? I'm sure if we went back through similar statements he's said over the years he's said the same thing about various technologies that are no longer relevant. The guy is a talentless hack who larps as a blogger and his only "redeeming" quality is having lots of money.

Owner of Golf Club Company says I should dedicate my life to golf lmfao.

hackers and painters and kids and ROI and startups and capitalism and the destruction of nature and old guys with money talking like they know better in fascist social networks

The amount of people who missed the point here is absurd. He's advocating for learning about how LLMs work. For the sake of learning. Because no one's going to invent the next thing without at least some understanding of the current thing.

I think collectively we should all stop listening to Mr Graham..

He capitalizes on greed and hype but with a soft, sober and thoughtful voice so as to lull you with rationalism and now 20 years of his “disruption” has mostly ruined modern society and a whole generation of techies have been led astray into trying to “change the world” is the world of today (minus the magic technology really any better than 20 years ago?)

- Good for him and his Tech Bros, bad for the rest of society

Twitter mirror: https://nitter.net/paulg/status/2091544343589060625

[dead]

[flagged]

[dead]

[dead]

[dead]

I do not think it is a proper thing to do for 17 y.o., unless they are exceptionally mathematically gifted, as proper understanding of how LLMs are trained requires a good grasp of calculus, understanding modern OS and SDE tools for proper implementation of pipeline etc.

I'd rather simply write another mnist implementation and check if I really like all that AI stuff at first place. Even then, before going into mature-on-the-way-to-dying tech (LLMs) I'd rather focus on fundamentals - good ols linear models, regressions, stat etc.

Another great quote by PG. I've been really enjoying his essays recently - truly a great and curious mind.

If I were 17, I wouldn't be using a social media plattform run by racist neo-fascists...

If I were 17, I'd be going to parties, music festivals, chasing girls, and enjoying my youth.

But sure, make the kids even more depressed by telling them they need to learn how to build an LLM so they can get a job working themselves to death to make Paul and friends rich.

He bases this decision on all of the experience he has amassed, as a 61 year old man in the tech industry. An actual 17 year old, with 17 years of experience, would not think like this, nor should they.

This is the ultimate builder’s mindset. Tinkering with the hardest technical problems—even for frivolous things like games—always yields the highest return on curiosity. Time to go back to the fundamentals.