Pratik’s Pakodas 🍿

Share this post

A Data Scientist Without A PhD

pakodas.substack.com

A Data Scientist Without A PhD

Tips from top data scientists in the industry

Jun 29, 2020
9
4
Share this post

A Data Scientist Without A PhD

pakodas.substack.com

Photo by Beau Runsten on Unsplash


If you want a great job,

you need something of great value

to offer in return.

— Cal Newport


Some time back I posted a list of “researchers without a PhD” and it got viral. Even people with a PhD liked the post and it made me think there may be some truth in becoming a researcher without a PhD.

Colin(T5 paper author) tweeted this recently and it seemed like nature’s call to me to write this article. There are many interested in serious research if they get the right environment.

Twitter avatar for @colinraffel
Colin Raffel @colinraffel
I often get emails from enthusiastic new researchers from outside the US. They take free ML courses and develop OSS, but can't afford MS programs, can't get into PhDs w/o publications, have trouble publishing w/o mentorship, and can't get visas for an RAship. Any advice for them?
6:48 PM ∙ Jun 19, 2020
586Likes88Retweets

The below set of question-answer is a personal enquiry on becoming a better data scientist.


Let’s begin…

1. What master chess players do to keep improving?

They think holistically about the pieces and exploit their strengths for the end goal. They will gambit if they can.

2. What does this mean for a data scientist?

We need to have a ‘goal’ in mind and need to achieve it under the constraints of time, data, compute and intellectual resources.

3. What am I saying exactly?

We need to learn to accomplish research with the given constraints.

4. Okay, and what should I work on?

“People who make impact, select important problems to work with”.

— Richard Hamming (Turing award winning researcher at Bell Labs)

5. Who gets the opportunity to work on important problems?

We live in a world of rapid advancement. I am interested in making new language models but I cannot do it without having the compute to run experiments. I do not have access to the intellectual capital of people the way FAANG researchers have it.

To be a FAANG researcher, I need a PhD from a top college.

6. Do I surely need a PhD for doing impactful work?

More impactful work is done by PhDs because they have access to the resources that come with joining a research group.

7. I see…How tough is it to get a PhD at top groups?

Twitter avatar for @hardmaru
hardmaru @hardmaru
“To start a PhD in ML, without insider referral, you need to do work equivalent to half of a PhD. Hence, in Apr 2019, I decided to dedicate all my time until Jan 2020 to publish in either NeurIPS or ICLR. If I fail, I would become a JavaScript programmer.” — @andreas_madsen ‼️
Twitter avatar for @andreas_madsen
Andreas Madsen @andreas_madsen
After getting published in ICLR as an Independent Researcher, I have received nearly 100 messages from others who are looking to do the same. So I wrote a blog post on why I decided to do it and my advice to others. https://t.co/vgD4rYE4AP
5:54 AM ∙ Dec 30, 2019
835Likes155Retweets

8. If I luckily get a PhD, how long will it be?

It seems I need to spend around 3-5 years 🤯

9. Will I spend 4±1 years of my life to improve the next 25 years?

I don’t think so.

10. How will I ever get a chance to do impactful work without a PhD?

I really don’t have an answer but I do know some people who made it without a PhD.

Christopher Olah at OpenAI
Known for LSTM blogs and distil.pub

Victor Sanh at Hugging Face
Known for DistilBERT and Movement Pruning

Denny Britz at ex-Google
Known for blogs on CNN, RNN and algo-trading

Melvin Johnson at Google
Known for work on multilingual translation

Niki Parmar at Google Brain
Known as one of the authors of the Transformer paper

Alec Radford at OpenAI
Known for generative modeling such as Jukebox and Image GPT

Jacob Devlin at Google
Known as one of the authors of BERT

Goku Mohandas at ex-Apple and ex-Ciitizen
Known for work in NLP(Healthcare) and now runs madewithml.com

Madison May at Indico
Known for NLP blogs and finetune(Scikit-learn style model finetuning for NLP)

Andreas Madsen (independent researcher)
Known for his story and Neural Arithmetic Units

Stephen Merity at ex-Salesforce
Known for AWD-LSTM and Single Headed Attention RNN

Jeremy Howard (Who doesn’t know him!)
Known for Kaggle, fastai initiative and ULMFiT which fired transfer learning in NLP!

(There will be many other researchers but I only know a few in NLP)

11. Ahh..there seems some hope! How can I stand out as a researcher?

Twitter avatar for @hardmaru
hardmaru @hardmaru
When time is limited, preparing for rigorous coding/puzzle interviews at large firms can take many months (especially for people like me who were not that good technically). The same amount of time might be better spent working on projects that define the essence of who you are.
Twitter avatar for @sebinsua
Seb Insua @sebinsua
@minimaxir @hardmaru It's very true. These kind of interviews require a lot of broad preparation. It's difficult to do this if you have real niche technical interests that you want to focus your time on. No sour grapes though because we don't need to derive our worth from a test.
4:28 AM ∙ Feb 17, 2020
327Likes43Retweets

12. What about conference papers?

Twitter avatar for @Thom_Wolf
Thomas Wolf @Thom_Wolf
The most enjoyable papers usually fall into two buckets: - the summer internship crazy projects => short ambitious open-ended - the slow & carefully crafted 1-year long projects => read like a novel w. highs, lows and teachings None of these are triggered by conference deadlines
8:58 AM ∙ Mar 2, 2020
544Likes54Retweets

13. Should I go wide or deep?

Twitter avatar for @srush_nlp
Sasha Rush @srush_nlp
Some musings on NLP/ML self-study. tldr: * Reading a paper is really hard, give yourself a lot of time and slack. * Pick depth over breadth. If you do depth well, you get breadth for free. * Be actively skeptical of all claims. Read your own drafts with 2x that skepticism.
Twitter avatar for @RisingSayak
Sayak Paul @RisingSayak
Happy to present an interview w/ @srush_nlp of @Cornell & @huggingface. We talk about his current research interests & works, how he entered into ML for language modeling, his open-source gigs, & a ton of other useful stuff. Read the interview here: https://t.co/Mu2cKOZRl0.
1:55 PM ∙ Jun 26, 2020
707Likes121Retweets

14. And what happens after I do some projects and research blogs, if not papers?

Yessss…this question and the answer.


💥 QnA with the researchers 💥

—> Goku Mohandas at ex-Apple and ex-Ciitizen

What are your research interests?

NLP, healthcare, platforms, network effects

What challenges did you face as a researcher without a PhD?

Initially, it was proving myself to those who traditionally hired PhDs from top tier programs who have several publications at top tier conferences. When you don't have this (because you didn't want to do it or couldn't afford to do it), you need to think about how to differentiate yourself and prove that you are fit for the role reserved for those with higher education.

Even if you were hired by proving yourself, your research colleagues will also look for the same proof of value. So you want to really solidify your value add and even better if you can do something the rest of your team cannot.

What are your suggestions for people wanting to research without doing a PhD?

Be different (applies even if you have a PhD). You can demonstrate a focus on a specific area (much like a PhD) but instead of publications at top conferences, you can have a portfolio of end-to-end projects that you've built that demonstrate research and product (which you can still publish on).

Over the years, we've hired many researchers with and without PhD. The best signal for performance has been a portfolio that demonstrates their ability to apply research. Just because someone has X years of formal school doesn't make them a better candidate or researcher.

Don't be afraid to apply your research (applied research is still research). You don't always have to work on devising new architectures or chasing SOTA on specific datasets. Applying and extending research in a focused area is still amazing research and it can have a profound impact on practical applications in your field!


—> Chris Olah at OpenAI

What are your research interests?

NLP, explainability

What challenges did you face as a researcher without a PhD?

Academia challenges
It’s important to be aware that not having a degree can have several negative long-term consequences.

University degrees have a lot of signaling and credentialing value. They cheaply communicate individuals and organizations that you have some baseline skills, at least in theory. In some fields, you can succeed without a degree by demonstrating skill in other ways (publications, open-source projects, portfolios, talks, awards, work history, referrals, etc). Other fields are less accepting.

How can you tell which type of field you're in? One useful test can be to look for examples of people who are successful without degrees in your field. (Author: The reason for writing this post is now validated 😝)

There's also a weird flip side to all these downsides. Once you establish yourself as competent there is this kind of threshold effect where not having a university degree can suddenly start causing people to actually take you more seriously. This kind of counter signalling effect seems to be common when you do non-traditional things.

Family challenges
Unfortunately, even if you feel confident that you would be best served by taking a non-traditional path, many young people face significant social and emotional barriers to doing so, particularly from adults. While some people are lucky enough to have a family that will support them doing something unusual, many are not.

Social challenges
For many people, university is a period of social development. They learn social skills, make long-term friends, and form romantic relationships. For some people, especially readers of this essay, it seems possible this is the biggest benefit of university.

What are your suggestions for people wanting to research without doing a PhD?

If your plan is to learn or work on projects independently, it’s worth thinking especially carefully. This can be great -- I did it for three years and grew a lot as a result -- but it can also very easily fail.

Some important questions to ask yourself are:

  • Do I have things I deeply want to spend a year of my life exploring or working on?

  • Do I have a way to support myself that leaves me time and energy to grow?

  • Can I really work self-directed for months at a time?

  • Do I have examples of me working hard on a personal project or learning without external structure?

  • Do I have or can I learn the skills I need to work on this project independently?

  • Do I have sources of community, peer support or mentorship for what I want to do?

(You can read his very detailed blog on this topic.)


—> Andreas Madsen (independent researcher)

What are your research interests?

Interpretability, or more generally trust in ML

What challenges did you face as a new researcher?

  • Lack of computational resources.

  • Nobody experienced to proofread papers before submission.

  • Keeping up with research at a broader scope on your own.

  • Finances, how do I pay my rent when nobody is sponsoring me.

  • Lack of support network with similar difficulties.

  • Staying motivated when working in isolation.

  • Fear of getting the paper rejected.

What are your suggestions for people wanting to research?

Pair up with a friend who can also think critically about research. Being the first author is super important at the beginning of a research career, so don't do equal contributions. Instead, work on two papers, where each of you is first-author on your own paper and is critical of the other’s work.

Secondly, do smaller scoped open-source solo projects - its small success will keep you motivated and makes the risk of rejection feel smaller.

Thirdly, find a field with not too much activity and which doesn't require large computational resources. GPT3 might be cool but it’s not for independent researchers.


—> Excerpts from An Opinionated Guide to ML Research

Roughly speaking, there are two different ways that you might go about deciding what to work on next.

  1. Idea-driven: Follow some sectors of the literature. As you read a paper showing how to do X, you have an idea of how to do X even better. Then you embark on a project to test your idea.

  2. Goal-driven: Develop a vision of some new AI capabilities you’d like to achieve, and solve problems that bring you closer to that goal.

While I was working on locomotion and starting to get my first results with policy gradient methods, the DeepMind team presented the results using DQN on Atari. After this result, many people jumped on the bandwagon and tried to develop better versions of Q-learning and apply them to the Atari domain.

However, I had already explored Q-learning and concluded that it wasn’t a good approach for the locomotion tasks I was working on, so I continued working on policy gradient methods, which led to TRPO, GAE, and later PPO—now my best-known pieces of work.

Choosing a different problem from the rest of the community can lead you to explore different ideas.

Sometimes, people who are both exceptionally smart and hard-working fail to do great research. In my view, the main reason for this failure is that they work on unimportant problems. When you embark on a research project, you should ask yourself: how large is the potential upside? Will this be a 10% improvement or a 10X improvement?

I often see researchers take on projects that seem sensible but could only possibly yield a small improvement to some metric. During your day-to-day work, you’ll make incremental improvements in performance and in understanding. But these small steps should be moving you towards a larger goal that represents a non-incremental advance.

Besides reading books, PhD thesis and seminal papers, you should also keep track of the less exceptional papers being published in your field.

Reading and skimming the incoming papers with a critical eye helps you notice the trends in your field (perhaps you notice that a lot of papers are using some new technique and getting good results—maybe you should investigate it). It also helps you build up your taste by observing the dependency graph of ideas—which ideas become widely used and open the door to other ideas.


Industrial perspective from a veteran PhD

—> Ajit Rajasekharan - CTO at nference

50+ patents and a successful entrepreneur

What are your research interests?

I don't do research. I work on specific problems in NLP that are relevant to certain application domains of interest to my company.

What challenges do you face at work?

The challenges I face have to do with finding working solutions meeting deployment thresholds of performance and scale, in short time windows.

While a PhD could certainly help in solving hard open-ended problems, the nature of problems in the industry often requires a unique skill that is perhaps best honed only by solving such problems, under time constraints, requiring solutions that can be deployed at scale - it is not enough to just beat a previous benchmark by a few percentage points.

A specific class of problems I work on, driven by practical constraints of the absence of labeled data, is finding unsupervised solutions to problems traditionally solved with supervised models.

A concrete example is unsupervised NER for custom entity types without labeled data. I converged after many failed attempts, with a reasonably working solution (given some constraints) that can be used in production. I doubt having a PhD would have helped me converge on such a solution for multiple simple reasons. The approach and nature of the solution certainly wouldn't merit a PhD to begin with - it’s a simple solution leveraging an existing unsupervised model.

Unsupervised NER remains an active area of academic research to date and sadly has orders of magnitude fewer papers relative to supervised NER.

What are your suggestions for people wanting to research without a PhD?

There are companies, even if not many, that have hard problems that need to be solved in shorter time windows than a traditional PhD time frame. While they may not be very open-ended both in their objective and solution strategies, the problems they solve are not trivial. An example of this is Tesla. Tesla continues to make tangible incremental progress in self-driving deployed to its fleet of cars. Any Tesla owner can testify to this progress, despite all the inevitable glitches of such a rapid pace.

However, there is the other end of broad open-ended problems that can only be solved by longer time frames in a PhD setting, ideally even without the time constraint of having to publish regularly. Examples of this is progress in Deep learning 2.0 as outlined by Prof. Yoshua Bengio, Yann LeCun, and others.

Those of us in the industry, regardless of having a PhD, would have nothing to build on, without progress in such academic research settings. Tesla largely owes its progress in self-driving to CNNs which had its origins and development spanning academia and research labs(Bell labs - interestingly an early embodiment of CNN was first deployed for the practical problem of recognizing bank checks).

I consciously chose not to directly answer the question above. My suggestion for those entering the field, is to choose one of these areas of work - industry or academia exclusively, the transition between them or even straddle them, based upon your strengths, interests, and circumstances.


All in all

Surely it’s going to be difficult for anyone to simulate the environment of top research groups without access to peers and mentors. Albeit, you can always develop taste for finding good problems, make friends of similar interests, and request someone to be your mentor.

If your end goal is to optimize impact on the world, keep building things others need which others haven’t.

My framework

  • Which are the important problems where I can get high improvements?

  • Which of these problems can I work with my constraints?

  • Whom can I ask for guidance and mentoring for the selected problem?

I hope this inquiry helps you become a better researcher with or without a PhD.

Ending this on another of Richard’s quotes 😊

“Beware of finding what you're looking for.

— Richard Hamming


Come join Maxpool - A Data Science community to discuss real ML problems

Ask me anything on ama.pratik.ai

You can try ask.pratik.ai for any study material.

I am also on Medium, Twitter & LinkedIn!!

4
Share this post

A Data Scientist Without A PhD

pakodas.substack.com
4 Comments
Akshay Kalra
Sep 6, 2020

What are your views for MS? Do you think the same discussion applies to that as well?

Expand full comment
ReplyGift a subscriptionCollapse
1 reply by Pratik Bhavsar
Kavita Ganeshan
Aug 3, 2020

very well written article? I resonate with the thought of researching without a Phd. What's in a degree? The learning never stops as long as you are interested. A paper cant prove that. But yes, acceptance is something that I do relate with what Goku mentioned. Its tough, but have to make it through your own niche way.

Expand full comment
ReplyGift a subscriptionCollapse
1 reply by Pratik Bhavsar
2 more comments…
TopNewCommunity

No posts

Ready for more?

© 2023 Pratik
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing