Reflecting On My Failure To Craft Luck
Hard learnt lessons
My journey into understanding luck started quite recently when I read about 4 types of luck.
Dumb luck
You bought 1 lottery ticket and you won it.
You did nothing to control the luck.
Hustle luck
You take many small projects for consultancy and one project just grows big.
You controlled luck by spreading risks.
Niche luck
You do only one thing like making ML models for agriculture and people want to hire you for it.
You controlled luck by avoiding competition. A big fish in a small pond.
Passion luck
You make ML models to detect white polar bears in white ice which is tough from high altitude using drones. You do this for free out of passion every year.
National geography decides to count the number of polar bears every year to show the impact of global warming on ice caps. Reduction of ice leads to screwing up of ecosystem leading to a reduction in seals which is the primary diet of bears. This endangers bear population.
So now you will be leading this project and will be paid more than those Deepmind engineers.
You got good at 2 very different things - ML models and understanding polar bears. This makes you a personal brand in itself like Joe Rogan.
As we keep making decisions, the majority of which turn out wrong, we often wonder what is the contribution of luck in all this.
Do we really have anything under control?
Now after doing multiple experiments, reading finance and investing in stocks I have learnt a few things.
We cannot predict which decisions will turn out right but at least a few will be.
It’s similar to “If I researched and bought 10 stocks, I cannot predict which one will grow crazily but at least a few will and the investment will be worth it.”
To predict every stock with certainty is to predict every atom in the world because every stock is influenced by
those who hold the stock
those who might want to invest
its competitor stocks and those who hold them or might hold them
its competitor’s competitors….
I mean this is quite messy. Finance tries a lot to simplify with a few parameters. This idea to model extremely complex probabilistic graph with linear multivariate regression is an abuse of Occam’s razor.
Life problems
The problems of life are sort of NP-incomplete. They have incomplete information, are multivariate in nature, have varied constraints.
A lot of time we try to optimise it via some dynamic programming kind of approach while we should be trying a greedy approach. The greedy approach will lead to sub-optimal results but it will give us signals which can eventually lead to the direction we might enjoy for a long time.
The problem of life is not just to find exciting things but to find those which will keep us excited for a long time.
Big ideas explained via curves
Coming across Seth-Godin curve was the turning point in my life. It answers the question of when not to quit. We can understand it in terms of 3 phases.
Phase 1
We start learning something and get good at things on a higher level. We think we have understood everything and achieved mastery.
This happens a lot with data scientists who do a few courses and basic projects.
Phase 2
As we start to get better, we hit the roadblocks of more complex topics. It’s easier to get articles and books for basic ideas but to become achieve mastery we need to read a lot of research papers and try things out to develop our intuitive understanding.
Learning in this phase gets worse and becomes full of insecurities as we realise we do not know a lot. Every step leads to the rabbit hole of more questions. We really feel we are losing it and never be able to achieve the goal. This step can take years.
The good thing to know here is the resistance in learning means others must have faced the same and dropped out. This is a validation that you are learning something which is known to only a few. A lot of people quit in this phase - which leads to reduction in competition.
The obstacle is the way!
Phase 3
As you spend time and get good at tough concepts, you feel more confident. These newly learnt skills bring more opportunities and the snowball effect kicks in.
The snowball effect leads to exponential growth.
We think the world follows the normal distribution with uniform thin tails. This leads to huge predictive mistakes in finance where tail events are considered rarer than the actual. The truth is these tails are fat.
If we learn to understand the opportunities which exist in these fat tails, we learn a trick to surpass the competition - which is same as crafting luck.
For example, I can make easy money if I know when the chart of a stock behaves in a particular way, it triggers a fat tail event. A fat tail event is a less frequent high return event. If you know how to do F&O, you can retire in one shot.
But to find such an event requires effort and belief in the fact that there exists a fat tail. Normal people do not do this. Even fund managers don’t do this. Only the seasoned day traders and hedge fund managers do. This in part also has to do a lot with Principal-Agent problem where people do not take risks when they don’t have much to gain.
It’s easy to learn compounding effect but its quite difficult to practise. We are inherently biased for simplicity and short term thinking. Well, why shouldn’t we when nothing is predictable in the long term. I have rarely been able to do what I planned more than 6 months down the line. My plans keep changing.
The take away from the above chart is that you can reap great benefits if you are ready to deal with uncertainties. I meet a lot of youngsters who feel uncomfortable about investing in stocks. They say that a house investment or bonds will give them up to 10% avg yearly return. Why go into stocks for 2-5% more!
The Δ is quite small in the short run of 5 years but when seen from the retirement perspective it can reduce your retirement age by 5-10 years as you will accumulate the same wealth earlier.
In the chart, we can see share1 performing worse than F.D.(bond) while share2 outperforming all. The truth is we can’t predict this beforehand. A better approach is to invest 80% in MF and 20% in stocks. Some also like index funds.
The same goes for people who stay away from startups. Almost all smart people I know have a bias for working in startups. It’s one thing to like startups while another to work for them. Many talk highly of startups but eventually join large organisations for self-validation and safety. Why bet our life on the idea of some crazy entrepreneur!
Life is a bundle of stocks!
Working at a large firm for years is like investing in bonds. They give safe predictable 10% returns every year in most of the cases.
While working at a young startup gives you the possibility to tap into hyper-growth. We cannot know for sure about the growth of a startup after you join. Crazy things happen all the time.
Different ways for hyper-growth
Join a startup and not leave till the end
Keep changing startups if they don’t go in hypergrowth
Keep switching from startup to large to startup. This can reduce the risk but can hurt your potential
Do your own startup
You can see yourself as a stock whose value keeps changing with time. Your value changes as you work at places and acquire different skills. Some skills also turn out useless in the long run. The higher risks you take, the more upward potential it can* have.
I enjoy Paul Graham essays! He says extremely successful people are those who are smart + work hard + also get lucky. I have smart friends who do shitty jobs and also have hard-working friends who do shitty jobs. Satisfying one condition is just not enough.
To be exceptional, we need to increase our quality of work + efforts simultaneously.
This is the ultimate luck hack!
Do things which create more possibilities for you. Some of those possibilities will turn out luck.
In the above chart for states S1, S2, S3 and S4
A1, A2, A3 AND A4 are the optimal next steps
A1’, A2’, A3’ AND A4’ are the non-optimal next steps
If we don’t take steps which create more possibilities, we will encounter a sub-optimal opportunity like A3’ after taking A2’. Once you get into a bad road, it gets difficult to come out of it as we have lost the optimal path.
This is similar to losing money in investing. Loosing 50% of 100 requires 100% returns to come back to original.
100 -> 50 -> 100
Some opportunities might require you to take an uncomfortable path but leads to a better path in the long run. This is similar to beam search in text generation.
This brings us to the end!
The title of the article is about reflecting on my failure because I do feel I have failed in many ways. I took me 28 years to get these ideas right in my head. All these years I was taking steps which seemed good at the movement only to turn out wrong in the long run.
I hope at least you will be able to look at opportunities much better than me.
Always do things which work out great in the long run.
Let me know what you think in the comments!
Come join Maxpool - A Data Science community to discuss real ML problems!
Ask me anything on ama.pratik.ai
You can try ask.pratik.ai for any study material.
Excellent article. Thanks for writing articles like this regarding career.