Pratik’s Pakodas 🍿

Share this post
NLP Model Selection v1.1
pakodas.substack.com

NLP Model Selection v1.1

Making model selection easy

Pratik Bhavsar
Sep 22, 2020
Comment
Share

A lot has happened in NLP from the time I released models.pratik.ai.

I received quite positive feedback and have decided to keep it updated.

Here's what got added recently 👻

Models suitable for mobile

  • MobileBERT (5.3x faster than BERT)

  • SqueezeBERT (4.3x faster than BERT)

Translation

  • Helsinki-NLP/opus

Long text (>512)

  • BigBird (Linear compute with sparse attention)

Semantic Search

  • LaBSE (93 language support)

  • LASER (103 language support)

  • Dense Passage Retrieval + Transformer Reader

Summarisation

  • PEGASUS (SOTA on summarisation)

Domain models

  • Replaced BioBERT with ouBioBERT as it supersedes it for medical tasks

Have suggestions? Create an issue @ github.com/bhavsarpratik/nlp-models

BONUS!

Poor Man's BERT

This paper finds that if we remove the last 6 layers of BERT, we still get the same performance. No need for fancy distillation. Both are of the same size 66M. This clearly means we love over-engineering. We over-engineered BERT. Then we over-engineered ways to compress BERT 🤫


Come join Maxpool - A Data Science community to discuss real ML problems!

Ask me anything on ama.pratik.ai 👻

You can try ask.pratik.ai for any study material.

Let me know your suggestions via feedback.pratik.ai 😃

I am also on Medium, Twitter & LinkedIn.

CommentComment
ShareShare

Create your profile

0 subscriptions will be displayed on your profile (edit)

Skip for now

Only paid subscribers can comment on this post

Already a paid subscriber? Sign in

Check your email

For your security, we need to re-authenticate you.

Click the link we sent to , or click here to sign in.

TopNewCommunity

No posts

Ready for more?

© 2022 Pratik
Privacy ∙ Terms ∙ Collection notice
Publish on Substack Get the app
Substack is the home for great writing