The Zipf Mystery

Share
HTML-code
  • Published: 16 September 2015
  • The of and to. A in is I. That it, for you, was with on. As have ... but be they.
    RELATED LINKS AND SOURCES BELOW!
    twitter.com/tweetsauce
    instagram.com/electricpants
    WordCount.org www.wordcount.org/
    How many days have you been alive? www.beatcanvas.com/daysalive.asp
    random letter generator: www.dave-reed.com/Nifty/randSeq.html
    Dictionary of Obscure Sorrows: trclips.com/user/obscuresorrows
    Word frequency resources:
    [lemmatized] en.wikipedia.org/wiki/Most_common_words_in_English
    www.uow.edu.au/~dlee/corpora.htm
    www.wordfrequency.info
    www.anc.org/data/anc-second-release/frequency-data/
    www.titania.bham.ac.uk/docs/
    www.kilgarriff.co.uk/bnc-readme.html#raw
    en.wiktionary.org/wiki/Wiktionary:Frequency_lists
    ucrel.lancs.ac.uk/bncfreq/
    [PDF] www.wordfrequency.info/files/entries.pdf
    [combined Wikipedia and Gutenberg] www.monlp.com/2012/04/16/calculating-word-and-n-gram-statistics-from-a-wikipedia-corpora/
    corpus.byu.edu/coca/files/100k_samples.txt
    corpus.byu.edu/
    corpus.leeds.ac.uk/list.html
    books.google.co.uk/books?id=ja1_AAAAQBAJ&dq=word+frequency+coca&lr=
    www.ling.helsinki.fi/kit/2009s/clt231/NLTK/book/ch01-LanguageProcessingAndPython.html
    Great Zipf's law papers:
    colala.bcs.rochester.edu/papers/piantadosi2014zipfs.pdf
    www.ling.upenn.edu/~ycharles/sign708.pdf
    arxiv.org/pdf/cond-mat/0412004.pdf
    www-personal.umich.edu/~mejn/courses/2006/cmplxsys899/powerlaws.pdf
    Zipf’s law articles and discussions:
    www.theatlantic.com/magazine/archive/2002/04/seeing-around-corners/302471/
    io9.com/the-mysterious-law-that-governs-the-size-of-your-city-1479244159?.Ej9760cOTJCPS_Bq4mjoww.0
    plus.maths.org/content/os/latestnews/may-aug08/food/index
    judson.blogs.nytimes.com/2009/05/19/math-and-the-city/?em
    plus.maths.org/content/mystery-zipf?src=aop
    www.datasciencecentral.com/profiles/blogs/why-zipf-s-law-explains-so-many-big-data-and-physics-phenomenons
    en.wikipedia.org/wiki/Zipf%27s_law
    books.google.co.uk/books?id=f8GrzlnMSm8C&pg=PA62&redir_esc=y#v=onepage&q&f=false
    arxiv.org/pdf/0802.4393v1.pdf
    www.pnas.org/content/108/9/3526.full
    lewisdartnell.com/language_page.htm
    wugology.com/zipfs-law/
    other Zipf’s law PDFs
    ftp.iza.org/dp3928.pdf
    arxiv.org/pdf/1402.2965.pdf
    arxiv.org/pdf/1104.3199.pdf
    www.lel.ed.ac.uk/~jim/zipfjrh.pdf
    www.ncbi.nlm.nih.gov/pmc/articles/PMC2834740/#pone.0009411-Mandelbrot1
    polymer.bu.edu/hes/articles/pgs02a.pdf
    in untranslated language: arxiv.org/pdf/0808.2904.pdf
    pages.stern.nyu.edu/~xgabaix/papers/zipf.pdf
    www.hpl.hp.com/research/idl/papers/ranking/ranking.html
    statweb.stanford.edu/~owen/courses/306a/ZipfAndGutenberg.pdf
    arxiv.org/pdf/1310.0448v3.pdf
    www.kornai.com/Papers/glotto5.pdf
    Zipf’s law slides:
    www.slideshare.net/guest9fc47a/nlp-new-words
    Pareto Principle and related ‘laws’:
    www.squawkpoint.com/2013/03/pareto-principle/
    billyshall.com/blog/post/paretos-principle
    en.wikipedia.org/wiki/Pareto_principle
    Random typing and Zipf:
    www.longtail.com/the_long_tail/2006/09/is_zipfs_law_ju.html
    health 80/20: archive.ahrq.gov/research/findings/factsheets/costs/expriach/expriach1.html
    Principle of least effort:
    en.wikipedia.org/wiki/Principle_of_least_effort
    en.wikipedia.org/wiki/Satisficing
    www.pnas.org/content/100/3/788.full.pdf [PDF]
    csiss.org/classics/content/99
    self organized criticality:
    journal.frontiersin.org/article/10.3389/fnsys.2014.00166/full
    Hapax Legomenon:
    campus.albion.edu/english/2011/02/15/hapax-legomenon/
    www.dailywritingtips.com/is-that-a-hapax-legomenon/
    en.wikipedia.org/wiki/Hapax_legomenon
    [PDF] www.aclweb.org/anthology/J10-4003
    www.wired.com/2012/01/hapax-legomena-and-zipfs-law/
    oed.hertford.ox.ac.uk/main/content/view/402/450/index.html#_ftn1
    oed.hertford.ox.ac.uk/main/content/view/36/166/index.html
    Learning curve: en.wikipedia.org/wiki/Learning_curve
    Forgetting curve:
    www.trainingindustry.com/wiki/entries/forgetting-curve.aspx
    en.wikipedia.org/wiki/Forgetting_curve
    Experience curve effects: en.wikipedia.org/wiki/Experience_curve_effects
    Forgetting
    and zipf's law: act-r.psy.cmu.edu/wordpress/wp-content/uploads/2012/12/37JRA_LS_PS_1991.pdf
    public.psych.iastate.edu/shacarp/Wixted_Carpenter_2007.pdf
    marshalljonesjr.com/youll-remember-less-than-001-of-your-life/
    en.wikipedia.org/wiki/Forgetting
    www.reddit.com/r/Showerthoughts/comments/3gu9qk/it_only_takes_three_generations_for_you_to_be/
    music from:
    trclips.com/user/jakechudnow
    www.audionetwork.com

Comments • 31 968

  • Dimitris M Papadakis
    Dimitris M Papadakis  8 saatler önce

    One hundred per cent of vcauce fans like Michael's videos, not eighty percent of them.

    • Nixoth
      Nixoth  Gün önce

      These videos are always good to watch. I just wish they still uploaded vsauce videos. We need new vsauce vids

      • ALPHA_sh
        ALPHA_sh  Gün önce

        quizzaciously became used alot more than once after this video was uploaded

        • Farmer Dingus
          Farmer Dingus  Gün önce

          Rip Vsauce

          • Jordan A
            Jordan A  2 gün önce

            speaking of "the", here's a song that doesn't use that word at all: trclips.com/video/zZEumf7RowI/video.html

            • VikkiTG
              VikkiTG  2 gün önce

              Wasn’t the videos name “Zipf’s Law”? Is Michael changing video titles?

              • Clip Crew
                Clip Crew  2 gün önce

                **Hapax Legomenon**

                • BOBMAN1980
                  BOBMAN1980  2 gün önce

                  Had to check out your chart of the languages to see what they were, as I highly doubt Ziph's law doesn't apply to Arabic (spoken by over a billion people) or Farsi. The word "the" in Arabic is used far more profusely than in English; and the "a", as well as the verb 'to be' (or "is") basically don't. Here's how that works--if I remember correctly: to saw The Green Car in Arabic, you say "the car the green"; if you omit the "the" before the adjective, you're saying "the car (is) green"; and if you omit the "the" before the noun, you're saying "(a) green car"--car green. ("A Thousand and One Nights" in Arabic is "Alf Leyla wa Leyla"--(A) Thousand Night(s) and (a) Night." In Farsi, however, I have not seen the word "the" at all--basically doesn't exist, except that it's implied. Also the word "a" doesn't exist unless it's as the number 1, as in one thousand is the same as A thousand.

                  • Nicolás Norambuena
                    Nicolás Norambuena  2 gün önce

                    1:39 Chilean?, Our language are considered apart from Spanish? ¿Lo escribí bien?

                    • Nithin Gowda
                      Nithin Gowda  4 gün önce

                      Damn! It fucks with your head real bad!

                      • Socrates Bandeira
                        Socrates Bandeira  4 gün önce

                        Heck! I want Michael dubbed in PT-BR and I want it NOW!!!

                        • octagonocologist
                          octagonocologist  5 gün önce

                          the fallacy, well one of many, here is that there are not 26 1 letter words. there are 26 letters, not all of them are words

                          • VINCENOIRgothFAIRY
                            VINCENOIRgothFAIRY  5 gün önce

                            I think that confirms it, reality is an illusion and we are simulations the zipf mystery is the algorithm everything is programmed by. There's no other explanation

                            • Paul Paturel
                              Paul Paturel  5 gün önce

                              couldnt we also represent it as gaussian ?

                              • jareddd
                                jareddd  5 gün önce

                                Angels: how should we nerf humans & life in the universe? God: Give a 20/80 ratio buff across all stats & boom, roll out the update.

                                • nat x
                                  nat x  5 gün önce

                                  this is proof that we are simulation, not real world. elon musk is right

                                  • Lukas
                                    Lukas  6 gün önce

                                    Wait. If it's the rate we forget information, it could be linked to that

                                    • Xev
                                      Xev  6 gün önce

                                      20% of vsauce's videos have 80% of his views

                                      • Tony Mcchicken
                                        Tony Mcchicken  6 gün önce

                                        It is not that hard really. All thoughts create a frequency that leaks from the head. It gather's in places where like frequencies gather like bubbles of memory. Basically we are in a sea of human thoughts. Frequencies attract it's pair to itself. Particles orbit they're own sounds.

                                        • Charlie Sammon
                                          Charlie Sammon  7 gün önce

                                          O_o

                                          • Andre
                                            Andre  7 gün önce

                                            80% of your complaints will come from 20% of your customers...ok. dumb ass video. dude would be teaching third grade homeroom if it weren't for youtube

                                            • Digi
                                              Digi  9 gün önce

                                              just to prove this wrong im going to make 20 books and the only word will be sauce and it will be 1000 pages long

                                              • Yeeter Boi
                                                Yeeter Boi  10 gün önce

                                                Woah

                                                • IsThisLeftUnity?
                                                  IsThisLeftUnity?  10 gün önce

                                                  I love how Michal singlehandedly revived "Quizatiously"

                                                  • ItsOnlyJoey
                                                    ItsOnlyJoey  10 gün önce

                                                    The

                                                    • ryuzakilawlight
                                                      ryuzakilawlight  10 gün önce

                                                      Dude your videos make me depressed. :D

                                                      • tofu
                                                        tofu  11 gün önce

                                                        Hi, may I take your order? Yes I’ll have the 0:27

                                                        • Jasun Karner
                                                          Jasun Karner  11 gün önce

                                                          A lot of the words at the top of the list are prepositions, articles, and pronouns; words of that type correspond to the ways that we process and connect bits of information, stuff that's usually represented in language by descriptors, which are situationally specific. I would say that the most commonly used words are ones that correspond to the nuts and bolts of human thought processes, and therefore, it's not surprising to see that shared across different individuals and different languages. What is the word "the"? What does it represent? How you answer that question and how you relate it to the fundamentals of cognition itself, may offer an answer to why it's the most commonly used word.

                                                          • redxlillard
                                                            redxlillard  9 gün önce

                                                            @Jasun Karner r/woosh

                                                            • Jasun Karner
                                                              Jasun Karner  9 gün önce

                                                              Maybe not. I'm often wrong.

                                                              • redxlillard
                                                                redxlillard  9 gün önce

                                                                No, really, are you sure

                                                                • Iron Bars Jeb
                                                                  Iron Bars Jeb  12 gün önce

                                                                  Drinking game: read a book and take a shot every time “the” is used

                                                                  • DaBomb Gaming
                                                                    DaBomb Gaming  12 gün önce

                                                                    That’s it. I’m done. Life is a simulation.

                                                                    • Ed Me
                                                                      Ed Me  13 gün önce

                                                                      lol, bussssssssssssssssssted! Time index 1:11 shows a "ranking" that does not match the list at time index 2:36, or the list at time index 16:15. This is the issue with what is called data mining. There are seemigly patterns because of how the data is shifted around, yet the conclusions are without support. 1:11-- the of and to a in is i that it for you was with on as have but be they 2:36-- the of and to in a was is that he for as it 16:15- the be to of and a in that have I it for not on with How "of" is shifted to make the pattern work in the third; or how "be" is near the end in the first, missing in the second, and second in the third; or "a" is fifth in all three. Just kidding, it is fifth in the first and sixth in the other two. I did this to demonstrate about the ease of shifting for data mining to support a conclusion in readers that my conclusion was wrong. lol, bussssssssssssssssssted, again! The concept of the random typing following zipfs claim, and the applying of zipfs claim to word usage distribution does not match. According to zipfs claim "the" in random typing would occur far less than zipfs claim of "the" in word usage distribution. The zipfs claim is actually a psychological effect of human though patterns "fitting" information into a biased pattern. The video even alludes to this when talking about repeating a word again near itself. A fun game to play with this is to get someone to say tin 10X, then to quickly ask them what an aluminum can is made of. There is a high chance they will say "tin." I saw a video about how people were given a series of numbers and then asked to see if they could then come up with a pattern. Once they had their pattern, they were then given a next number in the series that did not follow their pattern. Most were stumped. Some attempted to explain the next number, or to slightly alter their pattern to fit it. Yet the solution was a pattern that was nothing like their pattern. This is an issue when trying to solve problems and to predict a favorable outcome. When the pattern is not correct, it leads to real difficulties down the road. And this is why biases are problematic. Biases lead to allocation of resources towards a desired outcome based upon a biased pattern, sometimes based upon data mining, and then leading to an undesired consequence.

                                                                      • MoAoX7
                                                                        MoAoX7  13 gün önce

                                                                        nice ending

                                                                        • sipntea
                                                                          sipntea  13 gün önce

                                                                          well a graph can show a myriad of things in the same way, it is odd but the 80 20 imbalance is weird i can't explain it

                                                                          • My Art Page
                                                                            My Art Page  14 gün önce

                                                                            5:00 lmfao why did his parents name him that?

                                                                            • The Right is Always Right
                                                                              The Right is Always Right  14 gün önce

                                                                              And some people still doubt the Bible wisdom: "For whoever has, to him more will be given, and he will have abundance" Mathew 13:12

                                                                              • Anne Reilley
                                                                                Anne Reilley  14 gün önce

                                                                                "Talk:quizzaciously A famous TRclipsr humorously coined this word in a video so it's all over the Internet, but real usage is rare or non-existent. The adjective quizzacious itself is a single writer's nonce word."

                                                                                • Webwood Main
                                                                                  Webwood Main  14 gün önce

                                                                                  the be to of and a in that have i it for not on with he as you do at this but his by from they we say her she or an will my one all would there their what so up out if about who get which go me when make can like time no just him know take people into year your good some could them see other than then now look only come its over think also back after use two how our work first well way even new want because any these give day most us

                                                                                  • Tylerty5
                                                                                    Tylerty5  15 gün önce

                                                                                    4:09 ok is it just me that thinks it’s weird that questions and answer are right next to each other?

                                                                                    • Ali-A
                                                                                      Ali-A  15 gün önce

                                                                                      Literally every Vsauce video in a nutshell: Hey Vsauce, Micheal here! The sky exist, or does it? But first let's question your whole Life

                                                                                      • Some guy
                                                                                        Some guy  16 gün önce

                                                                                        I was right when I said every book is just a remix of the dictionary...

                                                                                        • SSshadowguy1223
                                                                                          SSshadowguy1223  18 gün önce

                                                                                          The what?

                                                                                          • ConnorConnor
                                                                                            ConnorConnor  19 gün önce

                                                                                            7:54 I think your Y-axis label has crashed. Please reboot it

                                                                                            • Cade
                                                                                              Cade  19 gün önce

                                                                                              This is one of the best, most interesting vsause videos ever

                                                                                              • Thapelo Mofokeng
                                                                                                Thapelo Mofokeng  19 gün önce

                                                                                                Michael accurately mocking himself at 18:08 is the reason why I still watch this channel

                                                                                                • Hsiu Fung Cheng
                                                                                                  Hsiu Fung Cheng  20 gün önce

                                                                                                  always get 80% lean and 20% fat for juicy burgers

                                                                                                  • biggie cheese
                                                                                                    biggie cheese  20 gün önce

                                                                                                    How is "like" not top used

                                                                                                    • Cheese Nose
                                                                                                      Cheese Nose  20 gün önce

                                                                                                      9:18 D

                                                                                                      • thecaneater
                                                                                                        thecaneater  21 gün önce

                                                                                                        4:45 Oh, shit, the 80/20 rule of Female Hypergamy put on display in nature.

                                                                                                        • thecaneater
                                                                                                          thecaneater  21 gün önce

                                                                                                          So, has anyone done a word count for this video, yet?

                                                                                                          • Creative Genius
                                                                                                            Creative Genius  21 gün önce

                                                                                                            Someday I will watch this video not remembering that I have watched it before.

                                                                                                            • Eli
                                                                                                              Eli  21 gün önce

                                                                                                              Unfortunately, 100% of what I study came out 0% in my exam.

                                                                                                              • Jairo Marquez
                                                                                                                Jairo Marquez  21 gün önce

                                                                                                                In business 80% of your business comes from 20% of your customers.

                                                                                                                • Dec Harris
                                                                                                                  Dec Harris  22 gün önce

                                                                                                                  I decided to check google trends, and quizzaciously was zero from 2004 (earliest time they had data on it) to August 2015. In September 2015 the word at a 100.

                                                                                                                  • XxGamerGirlxX 980
                                                                                                                    XxGamerGirlxX 980  23 gün önce

                                                                                                                    the of and to a in is i that it for you was with on as have but be they

                                                                                                                    • XxGamerGirlxX 980
                                                                                                                      XxGamerGirlxX 980  23 gün önce

                                                                                                                      zipf

                                                                                                                      • Luca Bralia
                                                                                                                        Luca Bralia  23 gün önce

                                                                                                                        the thank the you the for the reading the this the comment

                                                                                                                        • Nifty Kiwi
                                                                                                                          Nifty Kiwi  23 gün önce

                                                                                                                          So can I assume that 20% of my life makes up 80% of my memories?

                                                                                                                          • j g
                                                                                                                            j g  22 gün önce

                                                                                                                            Yess

                                                                                                                            • Nicholas Parker
                                                                                                                              Nicholas Parker  23 gün önce

                                                                                                                              20:03 I'm sad 20:15 smile

                                                                                                                              • David
                                                                                                                                David  24 gün önce

                                                                                                                                hmmmm .... this is sus

                                                                                                                                • Deepyn dixies
                                                                                                                                  Deepyn dixies  25 gün önce

                                                                                                                                  This is the matrix. Reality is predifine. Self awarness is programmed. Try to scap it. see what happens.

                                                                                                                                  • Stefan Filewicz
                                                                                                                                    Stefan Filewicz  25 gün önce

                                                                                                                                    R u sure that the most common used word isn’t “like”

                                                                                                                                    • Ballz Mahony
                                                                                                                                      Ballz Mahony  25 gün önce

                                                                                                                                      Why say lot word when few word do trick?

                                                                                                                                      • Kevin Rylander
                                                                                                                                        Kevin Rylander  25 gün önce

                                                                                                                                        Vsauce is smarter than more than half of native English speakers. He literally uses the word 'literally' correctly.

                                                                                                                                        • Sophia Rose
                                                                                                                                          Sophia Rose  26 gün önce

                                                                                                                                          S A U C E

                                                                                                                                          • Darwin Kalenor
                                                                                                                                            Darwin Kalenor  26 gün önce

                                                                                                                                            4 years later, Google now lists 7,550 results for "quizzaciously." Michael Stevens has incredibly quantifiably changed the path of word use in the English language. I wonder how much more this word will be used in another 4 years.

                                                                                                                                            • JuzzTheFuzz
                                                                                                                                              JuzzTheFuzz  26 gün önce

                                                                                                                                              It’s a brachistachrone!

                                                                                                                                              • Selfie Trash
                                                                                                                                                Selfie Trash  26 gün önce

                                                                                                                                                my grandpas isnt The its WaHT DID YA SAY?!

                                                                                                                                                • nathn
                                                                                                                                                  nathn  26 gün önce

                                                                                                                                                  0:36 looks like an r/ihadastroke XD

                                                                                                                                                  • Shubham Agrawal
                                                                                                                                                    Shubham Agrawal  27 gün önce

                                                                                                                                                    the ratio of likes to dislikes is just mind blowing

                                                                                                                                                    • iamtheiconoclast3
                                                                                                                                                      iamtheiconoclast3  27 gün önce

                                                                                                                                                      "How much does here weigh?" Priceless.

                                                                                                                                                      • BHALT0S
                                                                                                                                                        BHALT0S  28 gün önce

                                                                                                                                                        the greek name for when a word is used just once across a particular set of works etc. is known as Hapax legomenon

                                                                                                                                                        • Werna Dowf
                                                                                                                                                          Werna Dowf  28 gün önce

                                                                                                                                                          Wow this is probably one of the best Vsauce videos I’ve seen

                                                                                                                                                          • Mayist RS
                                                                                                                                                            Mayist RS  28 gün önce

                                                                                                                                                            I'm surprised no one has made a graph showing how many time each word Michael used in this video.

                                                                                                                                                            • Random Kid
                                                                                                                                                              Random Kid  Aylar önce

                                                                                                                                                              I speak Russian, rarely use the "the"... *Oh shi**

                                                                                                                                                              • James Flowers
                                                                                                                                                                James Flowers  Aylar önce

                                                                                                                                                                Simulation theory? The theory that everything is written in time since the Big Bang and free will doesn’t exist?