Nerdiness Quantified: What is a nerd?


Hello. I am a nerd. Depending on the context. Are you?

What is a nerd? What personality types consider themselves nerdy? What demographics characterize those who identify most strongly as nerdy?

This project uses quantitative methods to attempt these qualitative questions.


Nerdiness Assessment

The Nerdy Personality Assessment Scale is a survey freely available online that aims to quantify nerdiness.

The Nerdy Personality Attributes Scale was developed as a project to quantify what “nerdiness” is. Nerd is a common social label in English, although there is no set list of criteria. The NPAS was developed by surveying a very large pool of personality attributes to see which ones correlated with self reported nerd status, and combining them all into a scale. The NPAS can give an estimate of how much a respondent’s personality is similar to the average for those who identify as nerds versus those who do not.

Personality Testing offers an open and anonymized dataset of approximately 1500 responses to their NPAS survey, which include the data desribed below.

Here is the entire NPAS. Feel free to score yourself!

Procedure: The NPAS has 26 questions. In each questions you must rate how much you agree with a given statement on a five point scale:

 1=Disagree <> 5=Agree

  1. I sometimes prefer fictional people to real ones.
  2. I prefer academic success to social success.
  3. My appearance is not as important as my intelligence.
  4. I gravitate towards introspection.
  5. I am interested in science.
  6. I care about super heroes.
  7. I like science fiction.
  8. I spend recreational time researching topics others might find dry or overly rigorous.
  9. I get excited about my ideas and research.
  10. I like to play RPGs. (e.g. D&D)
  11. I collect books.
  12. I am a strange person.
  13. I would rather read a book than go to a party.
  14. I love to read challenging material.
  15. I spend more time at the library than any other public place.
  16. I would describe my smarts as bookish.
  17. I like to read technology news reports.
  18. I am more comfortable interacting online than in person.
  19. I was in advanced classes.
  20. I watch science related shows.
  21. I was a very odd child.
  22. I am more comfortable with my hobbies than I am with other people.
  23. I have started writing a novel.
  24. I can be socially awkward at times.
  25. I enjoy learning more than I need to.
  26. I have played a lot of video games.
Generally speaking, the higher your total, the more you align with statements from people who call themselves nerdy.
Check out the paper for more information on how this scale was developed.
Here’s a heatmap showing the correlation within these 26 questions. The darker the color, the more people answer those questions the same way. Everything has a positive correlation because these 26 questions were selected precisely because they reflect a similar underlying personality. However the degree to which any two questions correlate varies.
heatmap correlations npas.png
It’s hard to parse this if you haven’t seen a heatmap before, so I’ll just point out that Q11 and Q19, the darkest spots not on the diagonal line (and therefore the most correlated), are these two statements:

Q11 I am more comfortable with my hobbies than I am with other people. 

Q19 I have played a lot of video games.

I’ll leave it to the reader to conjecture about what this says about gamer nerds and their social skills. 😉

Data, cont.

Big Five Personality

In addition the NPAS questions, this particular site also administers a ten-item personality test (TIPI) based on the Big Five model. This test yields a value for each of the following traits of the taker (the “big five traits”):
  1. Openness to Experience (O)
  2. Conscientiousness (C)
  3. Extraversion (E)
  4. Agreeableness (A)
  5. Neuroticism (N)

In addition, this test asks for your personal association with the word nerdy, on a 1-7 scale. This is a very useful question for our quantitative approach! I’ve named the class of people who respond with a 6 or 7 as nerd champions.


A demographic form during the test asked for various demographic variables:
  • Age
  • Gender
  • Race
  • Years of Education
  • Urbanness as a Child
  • English Native
  • Handedness
  • Religious Category
  • Sexual Orientation
  • Voted in a natl. election in past year
  • Married
  • Number of you + any siblings growing up
  • Major in school
  • ASD: Have you ever been diagnosed with Autism Spectrum Disorder?
I thought that the ASD demographic question was particularly interesting, so I chose to focus on it as a target variable. It forms a binary class distinction (either you were diagnosed or you weren’t), so it can be used directly as labeled input to train classification algorithms.
A new question emerges…

How does autism contribute to the definition of nerdiness?

I looked into the representation in my data versus the general population and came up with some basic figures: 
Prevalence of ASD 

  • General Population  
    • ~1.5%  “Prevalence in the US is estimated at 1 in 68 births” CDC, 2014
  • My Sample: 
    • 5.5% of 1418 rows / responses (mostly US-based respondents) 

There is nearly 4 times the rate of ASD in this nerd-survey sample, relative to the entire population. I looked at the people in that 5.5% of my data, and charted a histogram of how they answered the “nerdiness-on-a-scale” question:
count of nerdy levels.png
Here’s how to interpret this chart: the blue bars are stacks of all the people who didn’t report a diagnosis of ASD, whereas the green bars are stacks of all the people who did. So the green bars make up that 5.5% of the data, and the blue bars are everybody else.
By looking at the pattern of just the blue bars, we see that in this sample most people answered between “4” and “7”, centered on “6” as the most frequent response. Looking at the pattern of the green bars, we see that people with autism are quite likely to answer with at least a “5” or higher, rising to “7” as the most frequent response. This means autistic people who take the survey strongly consider themselves nerds.
    I wanted to see if there were patterns in the way that people with ASD answered the questions of the survey. A machine learning classifier like Logistic Regression works by learning a coefficient weight for each variable in the data (e.g. for each question response), and uses the weights in a simple formula that gives a number between 0 and 1 (No or Yes predictions). With continuous values between 0 and 1, you can you treat the result as a probability relative to your other algorithmic predictions. 

    Framing the task for Machine Learning:

    A machine learning project can be described in three elements: The task, the experience, and the metric you will evaluate your algorithm performance.

    Task: Classify a survey respondent by whether they have been diagnosed with Autism or not (ASD).

    Experience: NPAS data: a corpus of survey responses where some respondents indicated a prior diagnosis of Autism.

    Performance Metric: Area Under ROC Curve (AUC)

    Data Cleaning and Transforms:

    In order to get a data set as clean as possible with minimal noise for good results, I:

    • Drop/exclude response rows that do not have complete data for any of: NPAS questions, TIPI personality inventory, or basic demographics. (Thankfully, most people answer all questions)
    • Transform categorical variables to “dummy” yes/no binary variables. Algorithm can’t tell difference between named categories, but it can tell the difference between 0 and 1 very well.
    • Calculate Big 5 personality scores based on TIPI responses and keep only resulting score variables (i.e. drop individual questions as they are then redundant and algorithm-confusing)

    Training and Testing Classification Models

    I used scikit-learn to evaluate each of the following model types:

    • Logistic Regression Classifier
    • Random Forest Classifier
    • Gradient Boosted Tree Classifier
    • K-Nearest Neighbors
    • Support Vector Classifier
    The model evaluation pipeline looks something like this:

    1. choose target (e.g. ASD yes/no)
    2. choose features (e.g. answers to NPAS survey)
    3. split the data randomly into separate chunks for training vs. testing each model
    4. cross-validate and calculate resulting ROC curves for each model
    5. generate an AUC score for each model
    Initially I used only the responses the NPAS questions as input. Meaning I excluded other demographic variables when predicting ASD. Here is a chart that shows the ROC Curves and AUC Score for each of the models with typical hyperparameters (settings):
    roc_curve ASD baseline npas only.png
    If you haven’t learned how to read ROC curves, the main takeaway here is actually that the algorithms aren’t great to start and can only predict slightly better than 50/50 chance (the straight diagonal line). The best algorithm is the line that has the most “Area Under the Curve” (AUC) between itself and that diagonal base line. In the results shown, that’s the Gradient Boosted Tree model.
    However, when I add in demographic data, the algorithms perform better, and simple Logistic Regression does very well. Here is an updated graph overlaid on the previous results showing the improvement.

    The improvement means that the machine learning classifier is picking up on the relationship to ASD in demographic questions, and by encoding that relationship as coefficient weights, it can more accurately predict whether a given respondent has ASD or not.

    I wanted to learn more about which features were helping to improve the response, so I used the very cool ML Insights package to take a look at what features the Gradient Boosted Tree was picking up on. The following chart shows the top 5 indicating variables ranked in order of effect: Q4, age, familysize, education level, and Q6. These gave the strongest discriminating signals for ASD.
    The inventory predictors (Q4 and Q6) are positively correlated with ASD diagnosis. Number of siblings is positively correlated. Years of education is negatively correlated.

    On the topic of age, the fact that this demographic question is about the diagnosis of ASD, and not the actual presence of ASD symptoms, means that people who grew up with different psychiatric practices (i.e. when Autism was less often diagnosed) would be less representative in the data.

    Big Ideas

    Personality surveys are a trove of interesting data. They can tell us a lot about ourselves!

    With enough data, we can algorithmically discern components of “nerdiness” by looking closely at how the data varies (machine learning).

    We can approximate the ways in which personality sub-types (e.g. ASD) contribute to our collective conception of a “nerd”.

    Bonus: Code

    You can check out all of the code I used to create this project in this convenient jupyter notebook here on Github.

    Dark Market Regression: Calculating the Price Distribution of Cocaine from Market Listings

    tl;dr There’s a hidden of illegal drugs: I scraped the entire “Cocaine” category, then made a bot that can intelligently price a kilo. Don’t do drugs.


    Project Objective: Use machine learning regression models to predict a continuous numeric variable from any web-scraped data set.
    Selected Subject: Price distributions on the hidden markets of the mysterious dark web! Money, mystery, and machine learning.
    Description: Turns out it is remarkably easy for anyone with internet access to visit dark web marketplaces and browse product listings. In this project I use Python to simulate the behavior of a human browsing these markets, selectively collect and save information from each market page this browsing agent views, and finally use the collected data in aggregate to construct a predictive pricing model.

    (Optional Action Adventure Story Framing)

    After bragging a little too loudly in a seedy Mexican cantina about your magic data science powers of prediction, you have been kidnapped by a forward-thinking drug cartel. They have developed a plan to sell their stock of cocaine on the internet. They demand that you help them develop a pricing model that will give them the most profit. If you do not, your life will be forfeit!
    You, knowing nothing about cocaine or drug markets, immediately panic. Your life flashes before your eyes as the reality of your tragic end sets in. Eventually, the panic subsides and you remember that if you can just browse the market, you might be able to pick up on some patterns and save your life…


    Dark web marketplaces (cryptomarkets) are internet markets that facilitate anonymous buying and selling. Anonymity means that many of these markets trade illegal goods, as it is inherently difficult for law enforcement to intercept information or identify users.
    While black markets have existed as long as regulated commerce itself, dark web markets were born somewhat recently when 4 technologies combined:
    • Anonymous internet browsing (e.g. Tor and the Onion network)
    • Virtual currencies (e.g. Bitcoin)
    • Escrow (conditional money transfer)
    • Vendor feedback systems ( ratings of sellers)
    Total cash flow through dark web markets is hard to estimate, but indicators show it as substantial and rising. The biggest vendors can earn millions of dollars per year.
    The market studied for this project is called Dream Market.
    In order to find a target variable suitable for linear regression, we’ll isolate our study to a single product type and try to learn its pricing scheme. For this analysis I choose to focus specifically on the cocaine sub-market. Cocaine listings consistently:
    • report quantity in terms of the same metric scale (grams), and
    • report quality in terms of numerical percentages (e.g. 90% pure).
    These features give us anchors to evaluate each listing relative to others of its type, and make comparisons relative to a standard unit 1 gram 100% pure.

    Browsing Dream Market reveals a few things:
    • There are about 5,000 product listings in the Cocaine category.
    • Prices trend linearly with quantity, but some vendors sell their cocaine for less than others.
    • Vendors ship from around the world, but most listings are from Europe, North America, and other English speaking regions.
    • Vendors are selective about which countries they are willing to ship to.
    • Many vendors will ship to any address worldwide
    • Some vendors explicitly refuse to deliver to the US, Australia, and other countries that have strict drug laws or border control.
    • Shipping costs are explicitly specified in the listing.
    • Shipping costs seem to correlate according to typical international shipping rates for small packages and letters.
    • Many vendors offer more expensive shipping options that offer more “stealth”, meaning more care is taken to disguise the package from detection, and it is sent via a tracked carrier to ensure it arrives at the intended destination.
    • The main factor that determines price seems to be quantity, but there are some other less obvious factors too.
    While the only raw numerical quantities attached to each listing are BTC Prices and Ratings, there are some important quantities represented as text in the product listing title:
    • how many “grams” the offer is for
    • what “percentage purity” the cocaine is
    These seem like they will be the most important features for estimating how to price a standard unit of cocaine.
    I decide to deploy some tools to capture all the data relating to these patterns we’ve noticed.


    BeautifulSoup automates the process of capturing information from HTML tags based on patterns I specify. For example, to collect the title strings of each cocaine listing, I use BeautifulSoup to search all the HTML of each search results page for 

     tags that have class=productTitle, and save the text contents of any such tag found.

    Selenium WebDriver automates browsing behavior. In this case, its primary function is simply to go to the market listings and periodically click to the next page of search results, so that BeautifulSoup can then scrape the data. I set a sleep timeout in the code so that the function would make http requests at a reasonably slow rate.
    Pandas to tabulate the data with Python, manipulate it, and stage it for analysis.

    Matplotlib and Seaborn, handy Python libraries for charting and visualizing data

    Scikit Learn for regression models and other machine learning methods.

    [Image: Automated Browsing Behavior with Selenium WebDriver]


    I build a dictionary of page objects, which includes:
    • product listing
    • listing title
    • listing price
    • vendor name
    • vendor rating
    • number of ratings
    • ships to / from
    • etc.
    The two most important numeric predictors, product quantity and quality (# of grams, % purity), are embedded in the title string. I use regular expressions to parse these string values from each title string (where present), and transform these values to numerical quantities. For example “24 Grams 92% Pure Cocaine” yields the values grams = 24and quality = 92 in the dataset.
    Vendors use country code strings to specify where they ship from, and where they are willing to ship orders to.
    For example, a vendor in Great Britain may list shipping as “GB – EU, US”, indicating they ship to destinations in the European Union or the United States.
    In order to use this information as part of my feature set, I transform these strings into corresponding “dummy” boolean values. That is, for each data point I create new columns for each possible origin and destination country, containing values of either True or False to indicate whether the vendor has listed the country in the product listing. For example: Ships to US: False
    After each page dictionary is built (i.e. one pass of the code over the website), the data collection function saves the data as a JSON file (e.g. page12.json). This is done so that information is not lost if the connection is interrupted during the collection process, which can take several minutes to hours. Whenever we want to work with collected data, we merge the JSON files together to form a Pandas data frame.


    The cleaned dataset yielded approximately 1,500 product listings for cocaine.
    Here they are if you care to browse yourself!

    Aside on Interesting Findings

    There are a lot of interesting patterns in this data, but I’ll just point out a few relevant to our scenario:
    • Of all countries represented, the highest proportion of listings have their shipping origin in the Netherlands (NL). This doesn’t imply they are also the highest in volume of sales, but they are likely correlated. Based on this data, I would guess that the Netherlands has a thriving cocaine industry. NL vendors also seem to price competitively.
    • As of July 15th, 2017, cocaine costs around $90 USD per gram. (median price per gram):

    • Prices go up substantially for anything shipped to or from Australia:
    * charts generated from data using Seaborn


    In order to synthesize all of the numeric information we are now privy to, I next turn to scikit-learn and its libraries for machine learning models. In particular, I want to evaluate how well models in the linear regression family and decision tree family of models fit my data.

    Model Types Evaluated

           Linear Models
    • Linear Regression w/o regularization
    • LASSO Regression (L1 regularization)
    • Ridge Regression (L2 regularization)
      Decision Tree Models
    • Random Forests
    • Gradient Boosted Trees
    To prepare the data, I separate my target variable (Y = Price) from my predictor features (X = everything else). I drop any variables in X that leak information about price (such as cost per unit). I’m left with the following set of predictor variables:

    X (Predictors)

    • Number of Grams
    • Percentage Quality
    • Rating out of 5.00
    • Count of successful transactions for vendor on Dream Market
    • Escrow offered? [0/1]
    • Shipping Origin ([0/1] for each possible country in the dataset)
    • Shipping Destination ([0/1] for each possible country in the dataset)

    Y (Target)

    • Listed Price
    I split the data into random training and test sets (pandas dataframes) so I can evaluate performance using scikit-learn. Since I can’t fully account for stratification within the groups that I’m not accounting for, I take an average of scores over multiple evaluations.
    Of the linear models, simple linear regression performed the best, with an average cross-validation R^2 “score” of around 0.89, meaning it accounts for about 89% of the actual variance.
    Of the decision tree models, the Gradient Boosted trees approach resulted in the best prediction performance, yielding scores around 0.95. The best learning rate I observed to be 0.05, and the other options were kept at the default setting for the sci-kit learn library.
    The model that resulted from the Gradient Boosted tree method picked up on a feature that revealed that 1-star ratings within the past 1 month were charateristic with vendors selling at lower prices.

    Prediction: Pricing a Kilogram

    (Note: I employ forex_python to convert bitcoin prices to other currencies.)

    I evaluate the prediction according to each of the two models described above, as well as naive baseline:

    1. Naive approach: Take median price of 1 gram and multiply by 1000.
      • Resulting price estimate: ~$90,000
      • Review: Too expensive, no actual listings are anywhere near this high.
    2. Linear Regression Model: Fit a line to all samples and find the value at grams = 1000.
      • Resulting price estimate: ~$40,000
      • Review: Seems reasonable. But a model that account for more variance may give us a better price…
    3. Gradient Boosted Tree Model: Fit a tree and adjust the tree to address errors.
      • Resulting price estimate: ~$50,000 (Best estimate)
      • Review: Closest to actual prices listed for a kilogram. Model accounts for most of the observed variance.


    Darknet markets: large-scale, anonymous trade of goods, especially drugs. Accessible to anyone on the internet.

    You can scrape information from dark net websites to get data about products.

    Aggregating market listings can tell us about the relative value of goods offered, and how that value varies.

    We can use machine learning to model the pricing intuitions of drug sellers.

    (Optional Action Adventure Story Conclusion)

    The drug cartel is impressed with your hacking skills, and they agree to adjust the pricing of their international trade according to your model. Not only do they let you live, but to your dismay, they promote you to lieutenant and place you in charge of accounting! You immediately begin formulating an escape in secret. Surely random forest models can help…

    Book: "The Universe in the Rearview Mirror: How Hidden Symmetries Shape Reality"

    Where did the big bang happen? It happened everywhere.*

    A trip around big ideas in physics and the cosmological rules (re: the symmetries) that govern our universe, The Universe in the Rearview Mirror: How Hidden Symmetries Shape Reality is an easy read covering some less easy theoretical concepts. Relativity (general and special), sub-atomic particle physics, the directionality of time itself, all are among the topics illustrated by author Dave Goldberg. Goldberg, an astrophysicist by trade and frequently asked physicist on, skillfully delivers conceptually dense material with levity, in a familiar format well suited for the general scientifically-minded readership, and with an often tongue-in-cheek style much like I imagine he employs when teaching his undergraduates at Drexel University.

    Like any good survey of a scientific field, The Universe in the Rearview Mirror** is salted liberally with quotes from historically influential figures. One that Goldberg utilizes in his introduction as a succinct justification of the book’s premise comes from Nobel laureate Phil Anderson:

    “It is only slightly overstating the case to say that physics is the study of symmetry.”

    And from there each chapter of the book gives a progressively compelling case for why such a statement, characterizing physics as the study of symmetry, is indeed only slightly overstating the case. Building from the more intuitive forms of symmetry (e.g. the symmetry of a [rearview] mirror; CPT symmetry; Lorentz invariance) up through mind-bending internal symmetries, critical at the most fundamental levels of physics, and ultimately on to how the breaking of certain symmetries is the crucial factor to the universe we see around us, The Universe in the Rearview Mirror orders the daunting complexities of modern theoretical physics into elegant underlying symmetries, allowing the rest of us to make some sense of it, even if only a little bit.

    Of course the symmetries don’t explain everything. But they aren’t supposed to either. In fact much of the book has author Dave Goldberg pointing out just how wrong we often are when it comes to understanding this material intuitively. The theme of symmetries gives us a model with which to gain a better working understanding of the universe. And yet, in the end what we have is still a model and not the universe. Werner Heisenberg said it best:

    “We have to remember that what we observe is not nature herself, but nature exposed to our method of questioning.”

    The spinning disc of Antworld

    *Why did the Big Bang happen everywhere? Because the universe expands like a stretching rubber sheet, not an explosion.
    **While there isn’t a footnote on every page, this book probably does have nearly as many footnotes as it does pages. If you don’t like writing with frequent asides consider yourself warned. References to the bottom of the page aside, the notes themselves are often chuckle-worthy.

    Review: The Secret Life of Pronouns: What Our Words Say About Us

    The Secret Life of Pronouns: What Our Words Say About Us
    The Secret Life of Pronouns: What Our Words Say About Us by James W. Pennebaker

    More interesting than just a look at pronouns grammatically, this is really a psychology book about how we put our words together (and what that can tell us), encompassing the class of “function words” (including pronouns) that make up a substantial part of our speech. Because of the role that function words play in establishing the structure we use to fill in the rest of the words we use (i.e. nouns, verbs, words with primarily semantic content), Pennebaker looks at patterns in function word frequencies and finds strong correlations with interesting real-world classifications: personality types, rhetoric, political speech, gender differences, even income and education gaps. Elucidating these various correlations forms the majority of the book.

    Using function word analysis and modern Natural Language Processing techniques, Pennebaker shows how you can make predictions about the author of an anonymous text, and perform simple culturomics (e.g. gauging national “mood” after the 9/11 disaster) by surveying the text of blog posts across on the internet, all without recourse to more complex semantic information.

    A recommended read for anyone interested in psychology and language, and also for those curious to see what modern technology applied to language analysis can tell us about ourselves.

    View my other reviews on Goodreads

    Review: Connectome: How the Brain’s Wiring Makes Us Who We Are

    Connectome: How the Brain’s Wiring Makes Us Who We Are by Sebastian Seung

    An accessible book to introduce and help explain the exciting theory that the mind is entirely encoded in the particular architecture of your brain. The central theme of “Connectome” is that such a mapping of the connections between neurons provides a far more complete picture of mental activity than other brain models. As Seung explains, mapping a brain’s connectome would enable highly specific examination and treatment of a brain, going so far as to allow correlation of neuronal activity patterns with memory and conscious experience itself.

    The catch is the monumental technical challenge of obtaining and handling so much data, as mapping a connectome, like mapping a DNA genome, is a computationally expensive process. In fact, mapping the connections in a human brain is many, many orders of magnitude more complex given the density of neurons and the intricacy of their connections in brain tissue. Furthermore, technology with the proper specificity to automate the delicate task is still in early stage development. Thus a corollary theme in the book relates to the pace of technological change: the field of connectomics banks on the continuation of exponential growth in computer processing speed (e.g. Moore’s Law) and accompanying technologies. Assuming that technology continues to progress as it has, Seung proposes that connectomes will naturally become the substrate of which we discuss our mental selves and our conscious identity.

    Other notes: The fundamental idea of the connectome is persuasive and fascinating, but perhaps because of such preexisting interests, this book was less in-depth than I was hoping for, and much of the content therein will be familiar to other fans of cognitive science or avid tech enthusiasts. Seung devotes the end of the book to the interesting future possibilities of cyber immortality, but they come with the usual speculation & caveats and don’t yield much of a takeaway message. Seung’s writing style is natural if not as crisp as a science journalist, just occasionally veering too folksy for the science (with a few awkwardly stilted metaphors).

    I was originally introduced to Sebastian Seung’s “Connectome” in his excellent 2010 TED Talk.

    Reviews at Goodreads

    The Modern Denial of Human Pacification: In Review of "The Better Angels of Our Nature"

    In his lauded but controversial best-seller “The Blank Slate: The Modern Denial of Human Nature“, Steven Pinker set out to quash a romanticized nostalgia for the lifestyle of people in pre-state societies: the myth of the “noble savage”. The noble savage, a conception of peaceful human coexistence before civilization, emerged in part as a reaction to the philosopher Thomas Hobbes’ famous suggestion that these savages indeed lived savage lives, described succinctly as “solitary, poor, nasty, brutish, and short”. While Pinker doesn’t write to expressly vindicate Hobbes’ characterization, he does present a strong case for rejecting the overly romantic conception of pre-state peoples as non-violent noble savages, and suggests that we do ourselves a disservice in misrepresenting the history of mankind. Now, in his new book “The Better Angels of Our Nature: Why Violence Has Declined“, Steven Pinker extends this rectification of prevailing but misguided opinion to grand scale, presenting a strong case for our ennobled present; we are living in the most peaceful era humanity has ever known.

    Reading peacefully in the Stanford University Oval

    In the famous 1969 novel “Slaughterhouse Five“, Kurt Vonnegut remarks resignedly to the effect “that writing an anti-war book is like writing an anti-glacier book”, a fatalistic assessment of the human condition, consigned as we are to inevitable war despite a growing wherewithal to protest it. So it goes. Well Pilgrim, Steven Pinker has good news: our novel expressions of knowledge, empathy, and reason really have turned our world away from war. Glaciers won’t budge, but we have Better Angels.

    Pinker blows the reader away (forgive the violent metaphor) with sheer weight of analytical shot. Slowly. At 700 pages of text interspersed with graphs and heaps of reference data, “Better Angels” is thorough-going and methodical because it has to be; contradicting common folk theories (like the noble savage), overriding an often overwhelming sense of unceasing or imminent violence from media coverage (see compassion fatigue), and compensating for a general lack of statistical thinking and probabilistic understanding in the lay public is no easy task. People are right to be skeptical of controversial theories, and knowing this Pinker has patiently lain it all out for us to see for ourselves that violence truly has declined with clear and unambiguously downward direction.

    Image credit: Wall Street Journal

    “Better Angels” is structured around an inventory of six Trends, five Inner Demons with four Better Angels, and five Historical Forces (Pinker can’t help but enumerate). More than half of the book is dedicated to a chronological exploration of the Trends of our history, six paradigm shifts in the human condition: The Pacification Process, The Civilizing Process, The Humanitarian Revolution, The Long Peace, The New Peace, and The Rights Revolutions. The bulk of the remaining half of the text is a fascinating look at psychology and sociology, showcasing a combined total of nine human traits (the Better Angels & Inner Demons) that dictate our behavior depending on their interplay with our environment and circumstance (so intriguing that I will have to return to these explicitly in later writing). The last five items in Pinker’s syllabus, the five Historical Forces, feature in the concluding chapter and encapsulate much of the book’s overall content by reflecting combinations of historical trend and human trait. For this reason I will focus mostly on these five developments for this review.

    The Five Major Historical Forces for Peace:

    • The Leviathan (the state; reigns in internal violence)
    • Gentle Commerce (economic incentives for cooperation)
    • Feminization (empowerment of women; men are the violent sex)
    • The Expanding Circle (empathy; sympathizing with ever wider classes)
    • The Escalator of Reason (rationality; application of empathy)

    Hobbes’ commentary about the always warring nature of savage man (Bellum omnium contra omnes) came in a seminal 1651 work of political theory on social contract, Leviathan. Pinker co-opts Hobbes’ metaphor of the Leviathan throughout “Better Angels” to represent the state, a dispassionate central government with a monopoly on violence and the doling out of punishment. Such an authority, Hobbes argued and Pinker shows, reduces overall violence because it represents the people, the ‘bystander’ seeking to minimize collateral damage in any conflict between two parties under the Leviathan’s umbrella (‘aggressor’ and ‘victim’). The three roles form the corners of a “violence triangle”, and an understanding of the relationships between the roles leads to an understanding of conflict, its causes, and our collective capacity to keep it in check. Knowing how to utilize the monopoly of force to control citizenry doesn’t preclude a brutal autocracy from being an ultimately pacifying Leviathan for its people (any ruler can see that keeping citizens in line yields a stronger state, not to mention more tax revenue), though democracies have proven quite capable of enforcing the rule of law without resort to violent suppression as primary means. Thankfully, modern Leviathans are increasingly democratic, a trend that has continued to proliferate (e.g., the Arab Spring of this past year).

    The Leviathan, referenced throughout the book as critical to the historical trends of the Pacification and Civilizing Processes, is a governmental construct that has proven to promote peace through the simple game theoretic logic of internecine deterrence. That is, minimizing conflict within the Leviathan for the greater good of the Leviathan. Pinker frames this effective deterrence in a relabeled Prisoner’s Dilemma called the Pacifist’s Dilemma, showing that when the Leviathan imposes penalties on conflict then the rational choice for both parties (Players A and B) shifts to pacifism (cooperation) rather than aggression (defection, which would be the go-to choice in an otherwise dangerous world, hence the dilemma).

    An example of the Prisoner’s Dilemma game model. Prefer a comic?

    [An aside on the Prisoner’s Dilemma: the always enlightening radio program Radiolab aired in their “Good Show” on the roots of altruism a fascinating segment on the Prisoner’s Dilemma and human behavior, explaining the game theory behind the thought experiment by retelling a real-life World War trench scenario. They talk to mathematician Steven Strogatz and political scientist Robert Axelrod about the tit-for-tat strategies that emerge in a prolonged (repeating) dilemma. Steven Pinker touches on these Iterated Prisoner’s Dilemmas explicitly in his section on revenge and he shows that the game mechanics can be seen extended in all group interaction, and is thus critical to an understanding the nature of conflict.]

    Among the many rational ideals incubated by the Enlightenment that have contributed to the ongoing decline of violence, the recognition that both sides in a seemingly zero-sum game (like the Pacifist’s Dilemma) can actually increase their mutual benefit by exchange (positive-sum) is among the most pacifying. Pinker, adopting Norbert Elias’ descriptor “Gentle Commerce”, describes this idea as often under-emphasized (“I suspect that among researchers, gentle commerce is just not a sexy idea”) but nonetheless a powerful incentive for peace, citing it in his conclusion among the most meaningful historical developments alongside the Leviathan. When the economic benefits of trade increase the value of maintaining a pacifist relationship, war becomes less attractive in comparison. In the terms of the Pacifist’s Dilemma, it increases the reward value of mutual cooperation for each player beyond what they might win with aggression. Exchange of specialized goods or culture never acts as a guarantor of peace alone, but rather gently encourages it like an undercurrent beneath a turbulent surface.

    Feminization is the natural result of a world that has increasingly advocated equal rights for women and allowed them greater influence in the decision making process that shapes societal values, policies and norms. It is perhaps the most obvious historical force for peace; men are far more inclined to be violent than women. A little evolutionary biology can give us indications as to why the genders are not equal in their propensities. Males of the species need to compete for reproductive access to females, who have incentives for a stable family environment to raise their children (with fathers who aren’t getting killed all the time). When men are in charge without input from women, their psychology of dominance contests carries over into cultures of honor where there is strong pressure to back up claims to status with violence, from the level of mano a mano duels and family blood feuds up through politics and interstate war. Women have good reasons for abjuring violence and war, psychology likewise rooted in the evolution of our species, and Pinker offers an abundance of evidence to suggest a strong link between the empowerment of women and the decline of violence.

    My favorite metaphor in this study of violence is one I’ve often appealed to: The Expanding Circle. Another of Pinker’s developmental standouts, The Expanding Circle (adapted from ethical philosopher Peter Singer’s coinage) is a growing category comprising those who elicit our sympathy. Pinker explains how perspective-taking, something we do whenever we read a novel, inflates our circle of sympathetic concern. When Pinker compared the rates of literacy and long-distance idea vectors (“books”) to trends in cruel treatment and homicides (related caveat: this book is chock full of grisly torture descriptions), he found that increases in literacy corresponded to the advent and perpetuation of the Humanitarian Revolution, fueling The Republic of Letters in the 17th and 18th centuries (a spiritual ancestor of our interconnected Global Village of the present day). The chronology of factors is in the right direction for a causal claim: technological advances in publishing, mass production of books, expansion of literacy, and the popularity of novels all preceded the major humanitarian reforms of the 18th century. The suddenly liberated flow of ideas, along with technologies that gave the people carrying those ideas newfound mobility (especially to move into and between cities, crucibles for progress), fostered cosmopolitan worldliness, more rapid evolution of ideas with the rise of urbanization, and critically, peace and empathy. Looking to more recent history (with the help of wonderful culture-surveying tools like the Google NGram Viewer) we can thank developments in the Expanding Circle for what Pinker calls the Rights Revolutions of the late 20th century: Civil Rights, Women’s Rights, Children’s Rights, Homosexual Rights, and Animal Rights. The extension of rights followed awareness of not just the range of human experience, but other sentient beings as well (in stark contrast to the utterly routine cruelty to animals during most of our coexistence). Modern sensibilities uphold the individual right to autonomy as an implicit social norm for minorities downtrodden in all but the last few decades of human society. Part of what fascinates me most about the Expanding Circle is just how rapidly it expanded in the 20th century. The idea that communication and technology work together as primary drivers of the mechanism that inflates our circle of sympathetic understanding is too enticing to ignore. Pinker expounds:

    “If I were to put my money on the single most important exogenous cause of the Rights Revolutions, it would be the technologies that made ideas and people increasingly mobile. The decades of the Rights Revolutions were the decades of the electronics revolutions: television, transistor radios, cable, satellite, long-distance telephones, photocopiers, fax machines, the Internet, cell phones, text messaging, Web video. They were the decades of the interstate highway, high-speed rail, and the jet airplane. They were the decades of the unprecedented growth in higher education and in the endless frontier of scientific research. Less well known is that they were also the decades of an explosion in book publishing. From 1960 to 2000, the annual number of books published in the United States increased almost fivefold.” – Steven Pinker

    Closely related to The Expanding Circle and perhaps Pinker’s favorite metaphor if only for the Better Angel of our nature at its core: The Escalator of Reason. The Escalator takes us step-by-step higher toward the ideal of pure objectivity, Nagel’s view from nowhere, a “superrational vantage point” that through the power of reasoning allows us to consider our own interests and the interests of another as equivalent in the grand scheme of things. Our cumulative gains in reducing violence may have been impossible without faculties of reason to help us determine how and why we should encourage our Better Angels and subdue our Inner Demons. While reading I found myself thinking of reason as the ‘Better Archangel’ of our nature, playing a vital role in every step of moral progress we’ve made since its rise to ethical power in the hands of Enlightenment philosophers.

    “Once a society has a degree of civilization in place, it is reason that offers the greatest hope for further reducing violence. … Reason is up to these demands because it is an open-ended combinatorial system, an engine for generating an unlimited number of new ideas. Once it is programmed with a basic self-interest and an ability to communicate with others, its own logic will impel it, in the fullness of time, to respect the interests of ever-increasing numbers of others.” – Steven Pinker

    Returning to the terms of the Pacifist’s Dilemma, the Escalator of Reason and the Expanding Circle change the game by blurring the lines between Player A and Player B. As one player increasingly empathizes with one player and vice versa, and reason abstracts the suffering as equally unjust for either side, the values in each outcome are summed together and the result is identical rewards or punishments for both players. This togetherness makes all but the mutual cooperation outcome a net loss, in essence merging the players into one and eliminating the dilemma.

    If humanity has been developing its way toward peace, why are some of the most memorable slaughters in history from just the last century? Chroniclers of war often call the early to mid 1900s the Hemoclysm to describe the deluge of bloodletting in World Wars and genocidal episodes like Maoist China and Khmer Rouge Cambodia. In our revulsion to these massive immoralities it is important to remember that despite the vast number of people killed, the magnitudes of these atrocities are not exceptions that belie humanity as more violent than ever, but rather reflections of humanity as more populous than ever. (In 1800 there were one billion people on the planet, in 1900 that had almost doubled to just under two billion, and by 2000 the world population skyrocketed to over six billion.) In light of this striking variability in absolute numbers we need to instead consider the proportion of people affected if we are to look at long-term historical trends.

    World Population

    With proper perspective we see that modern societies experience only a tiny fraction of the violence that even the most peaceful of pre-state societies endured. Even factoring in egregious 20th century atrocities and World Wars does little to raise the overall percentages of violent death by comparison. Atrocitologist / necrometrician Matthew White’s list of (Possibly) The Twenty (or so) Worst Things People Have Done to Each Other, reproduced in the section of the book addressing the common objection that the 20th century has been the worst yet, helps put our vivid cultural memories of the most recent atrocities into historical perspective. Of particular relevance is the adjusted ranking of the top twenty that Pinker provides, showing the surprising finding that the two most brutal 20th century episodes (WWII casualties and famine under Mao Zedong), while leading with the greatest absolute death tolls of the twenty (or so) listed, are only 9th and 11th respectively when ranked in terms of relative population killed. Meaning there are at least eight recorded events in our history before the 20th century that killed a greater percentage of the world’s population. Living in pre-state societies, Pinker documents that you would have had anywhere from an average of 15 percent up to a 60 percent chance that your cause of death would have been that another man had killed you. Even living during the Second World War, your odds of dying violently during the worst of the 20th century (roughly the first half of it) are nowhere near, at around 2 percent. Today in the 21st century, western countries tally a rate of violent death at less than .01 percent, counting homicides as well as war deaths.

    Since the end of World War II in 1945 man has been living in the “Long Peace”, a dramatic near-cessation of major conflict, particularly among the dominant world powers (something new for them). In the past 20 years even other forms of extreme violence have declined, a recent period Pinker dubs the “New Peace”, marked by a decrease in three forms of violence that many inaccurately assume to be increasing: civil & guerrilla wars, genocide, and terrorism.

    An area histogram. The combined height of each stack represents the total of battle deaths for that year. Image credit: WSJ

    There are important psychological reasons that we overestimate rates of violence that are actually in marked decline. One is the common assumption that violence in human society is metaphorically hydraulic: the amount of violence in the world is constant and only pushed around or pent up during peace before inevitably bursting into new conflict. Pinker makes it clear that this mental model is not supported by the facts, but the ready supply of hyperbolic news makes it hard to notice the overall trend without a step back to look at the whole analytically.

    Other big reasons for our skewed exaggeration of present problems compared to past, cognitive tendencies I heard cited recently by Freakonomist Steven Levitt among many other social scientists, have to do with the magnitude of events and the ease with which we can remember them. People pay more attention to and ascribe disproportionately greater probability to the biggest, most intense events, which because of exponential population growth means that the 20th century had literally billions more potential victims. And as we saw earlier, the conflicts of the past hundred years take the top absolute tolls but are in line with relative tolls of destruction, the metric relevant to our comparison with the rest of history.

    We live in a big world (that’s big on media), so even the progressively tinier fraction of violence within it presents us with an absolute amount of violence too much for any one mind to handle. You could spend every waking hour consuming attention-grabbing news of civil strife, true crime shows, and terror ad infinitum. And some people do; you don’t have to try hard to be exposed to media coverage of violence. How could anyone presented with such an onslaught of evidence for our capacity to commit violence think that we are becoming more peaceful? The human mind makes intuitive predictions based on the accessibility of examples in memory, which will naturally favor the vivid and recent examples. Without counter-balanced coverage (and the media much prefer the distressing to the mundane), we assign probabilities for the likelihood of future violence that are distorted by our shortsighted perspective of past violence. The result for most people is a gut reaction to reject any claim of human pacification (Pinker notes in the preface that immediate responses to his book’s premise are typically “skepticism, incredulity, and sometimes anger”, even from friends and family).

    Image credit

    It’s hard to find real bones of contention to pick with “Better Angels”. Steven Pinker maintains an equitable, analytical tone and sticks to credibly supported theses. But if I had to pick one to quibble over it would be in a challenge to the idea that 20th century disasters were coincidental bad luck, caused by the particularly bad apples Hitler, Stalin, and Mao. Granted, their killer combinations of charisma, ambition, and psychopathy persuaded millions to follow their incredibly destructive lead. And if only a critical decision point in history had gone the other way: Hitler killed in his 1930 auto crash, or Stalin assassinated, or Mao deciding that Marxism was kind of silly in the first place, maybe the world would have had been spared a few megadeaths in the 20th century ledger. Pinker is hardly the first or only scholar to ponder these counter-factuals, summarizing the position in a recent interview as: “Many historians have argued as follows: No Hitler, no Holocaust; no Stalin, no Purge; no Mao, no Great Leap Forward and Cultural Revolution.” But the problem as I see it is our limited ability to continue pondering these counter-factual worlds beyond their theorized fork in time. I trust the historians know what they’re talking about when they claim that the preemptive removal of these authoritarian leaders would have averted their ability to wreak their respective atrocities, but I’m not sure I trust any man to know with certitude what it would have been like even 5 or 10 years into such an alternate history. Like the defiantly unpredictable patterns of the weather driven by chaos theory, the further we extrapolate from a set of initial conditions the less we can predict about a dynamic system with confidence. Re-imagining the incredible systemic complexity that would be a world history without Hitler, Stalin, or Mao entails acknowledging the unforeseen possibility of alternatives equally if not more catastrophic. Pinker himself would be quick to point out that any bumps in the road to non-violence, Mao or no Mao, are only bumps in a clear downward trend. Why then, besides in wistful contemplation, entertain myopic fantasies of a smoother trend in the predictive isolation of Hitler-less 1940’s Germany or Leap-less 1960’s China? I’m not sure Steve gains anything for his main thesis in these counter-factual considerations.

    Minor quibbles aside, “The Better Angels of Our Nature” assiduously justifies its subtitular contention: violence really has declined, and now it’s not so hard to see why. Steven Pinker has assembled vast quantities of data to support his position, sourced in turn by the assemblies of other preeminent scholars in ethnography, anthropology, and the history of man. Add to this a trove of lab-tested social psychology, game theory, and the areas of Pinker’s own expertise in cognitive psychology. The resulting dissertation, structured with the incredible skill and forethought that define Steven Pinker’s books, sums these component analyses into the rational juggernaut needed to upend the conventional wisdom it is up against. Though consistently dispassionate in tone and bearing throughout, the title of this book betrays its emotional impact: optimism for humanity.

    Review: The Information: A History, A Theory, A Flood

    The Information: A History, a Theory, a Flood by James Gleick

    The Information: A History, a Theory, a FloodA thorough exploration of information theory, how communication functions at its most fundamental. Language, mathematics, cryptography, memory, computing, the history of telecommunication, the history of intuitive human information theory before and after it was formalized.

    Most intriguing is the third of Gleick’s informational themes: the Flood, our modern immersion in quantifiable information. The book ends with allusions to Borges’ “The Library of Babel”, a short story that seems ever more apt and readily appreciable as time goes on.

    Freeman John Dyson has written a substantive synopsis and review entitled “How We Know”, here:…

    View other book reviews

    Review: Physics of the Future: How Science Will Change Daily Life by 2100

    Physics of the Future: How Science Will Shape Human Destiny and Our Daily Lives by the Year 2100 by Michio Kaku
    I’m not sure about anybody’s ability to predict a century into the future (especially if you give credence to the idea of accelerating returns in technology), but I was willing to give this book a shot after hearing Michio Kaku in interviews. In particular he piqued my curiosity with the claim that all the ideas in the book are grounded on currently existing prototypes or established scientific theory.

    Physics of the Future: How Science Will Shape Human Destiny and Our Daily Lives by the Year 2100Now after having read it, I think Michio is only giving a survey of some select topics, and the only ones that I think he handled well were the ones most closely linked to physics (e.g. space travel, nanotechnology & quantum behavior, global power generation). The other fields he dives into, particularly his analysis and extrapolation of consumer technologies, were disappointingly off target or lacking in proper depth.

    The book is occasionally so superficial in its treatment of some prototyped technologies that it reads somewhat like painfully outdated sci-fi from Michio’s childhood in the 50’s. The book is written to be highly accessible, but he does uninformed readers a disservice by giving equal weight to illogical and ‘improbable but not impossible’ possibilities.

    My biggest problem with his predictions are that they center on only a set of technologies that Kaku has experience with, extrapolated all the way out to 2100 without much consideration of how all the unmentioned possibilities will change his visions for the future.

    As an example, Michio doesn’t do the best job keeping our present circumstances and his far future predictions from mixing anachronistically: e.g. the frequently repeated “…when we carry around our own genomes on a CD-ROM” for a “2030-2070” range prediction. I worry that Michio Kaku is just paraphrasing some of the ideas out there without really thinking about them any more critically, like a mediocre science journalist or sci-fi writer. Again this could be an artifact of his intentionally writing this book to be broadly accessible, but I don’t think he found the right balance.

    The best parts of the book are in the second half, particularly his chapters on The Future of: Energy, Space Travel, Wealth, and Humanity (respectively) and I did enjoy most of this material despite a scattering of the same problems mentioned above. Sadly, I think Michio Kaku completely botched his concluding chapter, “A Day in the Life in 2100”, and I think the preceding Future of Humanity chapter would have been a much stronger ending. The “Day in the Life” conclusion is silly speculative fiction and the best (worst?) example in the entire book of his anachronistic and muddled sci-fi visions.

    Michio Kaku is great when talking about physics and large scale trends closely linked to humanity’s knowledge of physics, but judging from this book alone he doesn’t put together upcoming technologies into realistic or compelling future scenarios very well at all, ending up with an incomplete picture somewhere in the uncomfortable border between imaginative thinking and unwarranted speculation.

    Other reviews

    Review: Beyond Boundaries

    Beyond BoundariesBeyond Boundaries by Miguel Nicolelis
    A neuroscience memoir of thought-provoking work, experimental brain interfaces and thought control tests told through the lens of Nicolelis’ own academic history and Brazilian based life story.

    The book offers specific and compelling evidence for not only controlling robotic systems remotely, but also for how our brain is naturally built to incorporate external apparatus and sense data directly into the body map and further into the sense of self, for brain connected robotics that restore the ability to walk to the paralyzed, for thought-based personal interaction, and even for direct brain to brain connections that create literal brain networks and a higher order of complexity.

    Very inspiring concrete experiments to shake some of these formerly sci-fi concepts loose from their intermediate fiction. Indeed the specifics of the experimental methods are sharp enough to be double-edged, disengaging from the overall visionary narrative to bring the reader back down into the due diligence of science and Nicolelis’ experience as researcher and academic, which, while important to establish the validity of the book’s premise, are less accessible than the grand ideas described in the preceding paragraph. Still, Nicolelis does it right by interspersing anecdotes of Brazilian football matches or personal history to keep the book moving.


    Review: The Moral Landscape: How Science Can Determine Human Values

    The Moral Landscape: How Science Can Determine Human ValuesThe Moral Landscape: How Science Can Determine Human Values by Sam Harris

    Harris says a lot of things that need to be said. Doesn’t mean I always like the way he says it.

    The Moral Landscape is a look at morality as a landscape, with peaks and valleys of “well-being”, essentially ‘goodness’, happiness, or fulfilled life. The conceptual premise is that humanity should use science to navigate the landscape, with the goal of moving people to higher moral ground through objective consideration. Through this metaphor Harris advocates a Science of Morality, where values can and should be determined strictly via scientific endeavor, rather than via purportedly causally distinct domains such as religious belief or abstracted philosophy. Harris describes this moral question as an illusory fact-value distinction that can be reduced to fact only, and as such determined scientifically. This is in opposition to David Hume’s well-known “is-ought” distinction, and it seems there’s been plenty of controversy over this type of philosophical supposition ever since. I’m inclined to side with Harris overall, but I can still logically at least entertain some of these philosophical challenges, making it difficult for me to endorse Harris wholeheartedly from the outset.

    I find Harris’ vitriolic rejection of competing ideologies off-putting, but I can understand why he would mix invectives into his logical argument if trying to garner support for an inchoate field of inquiry against sometimes even more zealous opponents. The parts of the book I found most interesting were where Harris takes on the gentler logical tone and analyzes contemporary academics in Positive Psychology, a branch of psychology focused on the concept of happiness, primarily through the lens of clinical studies, statistical analysis, and neuro-imaging (i.e. scientifically). I’ve enjoyed a number of the prominent books in this field recently and I was initially surprised when Harris directly contradicts the authors at times throughout the book (e.g. Jonathan Haidt in particular), making a point of exposing problems with certain sociological conclusions. I did not find these to be refutations of these psychologists’ findings, but rather a compelling challenge to rethink conclusions that may be unwarranted.

    In total I find it hard to argue against the proposition that science will influence mental states and understanding given the society we are living in today. Modern use of non-invasive technology (e.g. transcranial magnetic stimulation of spirituality) and manipulation of biochemistry provide convincing examples of how physical action is causally linked to belief and states of being, and therefore a science of morality could be used to determine effective action, whatever that may be. That is to say, while a science of morality may be continually revising the ends, hypotheses about what constitutes goodness, as the “is” we discover physically redefines the “ought” we perceive philosophically, we can be sure it will refine the means to further moral understanding. As Harris so forcefully asserts, we do ourselves disservice by objecting to scientific inquiry. Perhaps “ought” is merely the best possible “is” we can envision, and moral science a powerful tool to widen our metaphysical gaze.

    (Additional note: Much of the material is a general condemnation of religiosity, and “New Atheism” seems to be the term for Harris’ particular brand of it. To me New Atheism doesn’t really seem to be a different kind of atheism so much as a category to address modern outspoken religious critics like Harris, Dawkins, Hitchens, etc.)…

    View book list on Goodreads