ChatGPT: The Robot, the Myth, the Legend

2022 has been called the “year of AI” by many media outlets (Forbes, Washington Post, many others) In particular, ChatGPT has sparked both the imagination and fears of the public. I personally find that many misunderstand what ChatGPT actually is and instead get caught up in what it seems to be. So why am I writing a blog piece aiming to do my part to help clear up these misconceptions? ChatGPT is a powerful landmark AI application that garners both undue fear and exaggerated claims – I hope that you, dear reader, would be in a better place to navigate between these two extremes after reading this article.

What is ChatGPT?

ChatGPT (Chat Generative Pre-trained Transformer) is a large language model (LLM) develped by OpenAI for conversation modeling. What the heck is a large language model?

First – what’s a language model? A language model is a model that is shown many examples (“trained”) of how to continue a sentence. (What is a model you ask? I will return to the word “model” in a bit.) For example, if I typed:

At the stroke

a language model could continue it as:

At the stroke of midnight [70%]

At the stroke of a pen [30%]

Where the percentages are the frequency those sequences of words occur. Your google searches, smart phones, and email applications already incorporate language models to do this. This is because the language model has been shown enough text (remember – Apple, Google, etc have so much data on you) — it has learned what words are likely to follow others. But what if the sentence previous to “At the stroke” contained the name “Brian Gutekunst” (the General Manager of the Green Bay Packers for non-sports fans out there) , or “Cinderella”? How does that change how a language model generates text? If the previous sentence contains the name “Brian Gutekunst”, the continuation is more likely to be:

At the stroke of a pen, Brian Gutekunst made Aaron Rodgers the highest paid Quaterback in the NFL

However, if the previous sentence contains “Cinderella”, the continuation would more likely be something like:

At the stroke of midnight, Cinderella’s carriage turns into a pumpkin

Language models that takes into account the context like the examples I gave above are called “Large Language Models” – LLMs. So – ChatGPT is a LLM that has been trained for conversation modeling – how to continue a conversation. I should mention that ChatGPT is more specific/narrow in scope than OpenAI’s GPT-3 (Generative Pre-trained Transformers – 3rd generation) which was released in June 2020. If you are interested in the differences betweeen ChatGPT and GPT3, feel free to reach out to me. For the purposes of this blog, GPT-3 “can do more things” than ChatGPT. (Also GPT-4 is coming later this year)

So, basically, ChatGPT is a LLM “fine tuned” from GPT-3 to produce text in a dialogue system (the “Chat” in ChatGPT) that meets human standards. They did this by scraping internet conversations and bringing users (specifically, OpenAI hired a bunch of hourly contractors) into the loop to incorporate context on which possible word continuations would be more convincing to a human.

ChatGPT Can’t Reason

ChatGPT sounds pretty cool, and in many cases, it is! But it also has many, let’s say, weaknesses that I will get to later. I’ve personally noticed that much of the fear and exaggerated claims around ChatGPT is rooted in misinterpreting the impressive chat modeling as demonstrating some sort of human-like reasoning. I suppose two reasons contributing to the misunderstanding is that A) the dialogue interactions can be convincing and B) ChatGPT, like most other LLMs, are built with algorithms called neural networks that are loosely inspired by connections between neurons in the brain.

ChatGPT can not reason. It can not do math. It can not solve logic puzzles. It can not invent new knowledge. It’s definitely not sentient. Ask it standardized math questions for high school students – specifically AMC10/AMC12 – and watch it fail horribly. My favorite example is this hilarious example from the AI luminary Andrew Ng, where ChatGPT thinks an Abacus is faster than a GPU. (Which is equivalent to saying an Abacus is faster than a PlayStation 5) The conversation itself is complete nonsense.

ChatGPT isn’t close to demonstrating human-level reasoning, and I am personally skeptical that the AI field is even close. Both OpenAI and DeepMind want to build machines with human-like reasoning capabilities through research in what has been coined “Artificial General Intelligence” – or AGI. But there is no universal agreement on how AGI should be defined or how reasoning should even be quantified at the level of a machine. The community is split on whether AGI even should be a goal of the field, which you can read more about in this excellent discussion at MIT Technology Review. How can a scientific community make progress if the community can’t agree on its foundational premises and most pressing problems? It will take some time to cut through the current level of confusion around AGI. At the other skeptic extreme, I’ve heard the claim that ChatCPT simply memorized every interaction on the internet. This is definitely not the case and is mathematically impossible – from information theory a LLM memorizing all the text on the Internet would take more space than it would take to store the entirety of the Internet in compressed form.

ChatGPT Approximates, Which is Very Useful

So if ChatGPT is not reasoning, nor is it memorizing, what is it doing, exactly? It’s approximating. Specifically, what make neural networks so powerful is that they are fantastic function approximators. I suppose “buy our awesome function approximator” isn’t a great marketing strategy, and it’s perhaps hard to capture the imagination of the media and the general public that way. But like the force gives a Jedi its power, so does the Universal Approximation Theorem (which is quite deep and profound) gives neural networks their power. “Oh no – not a math lecture” you may be saying, dear reader, but stay with me I promise I’ll be brief and cute animal pictures are coming. By the way – for the purposes of this blog you can consider “function approximations” and “models” (Remember Large Language Models, LLMs, from earlier? Of course you do) as synonymous.

Much of science, engineering, economics, and finance have functional relationships that can be explicitly written down. For example, the market price of a house usually exhibits a linear relationship with the size of the house:

You can easily fit a straight line with the above example. But say you want to write a function to automatically recognize pictures of cats and dogs?

How do you even begin to write a function (or – if you prefer – build a model) to do this? With neural networks, you don’t have to, it will approximate the function for you if shown enough examples, ie, it is “trained”. A bit of oversimplification, but there are two broad model classes: Discriminative and Generative:

In our cats and dogs example, a discriminative neural network model will approximate a function that separates cats from dogs. For a brilliant illustration and satire, Mike Judge nails it in this short and hilarious HBO’s Silicon Valley clip. A generative neural network model will attempt to learn the underlying probability function of cats and dogs separately. This is where the “G” in “ChatGPT” comes from – Chat Generative Pre-trained Transformers.

So what exactly is ChatGPT approximating? It’s approximating conversations from both the internet and transcriptions from OpenAI contractors, emulating the style and word choices of these conversations. ChatGPT guesses what words and phrases should come next in a conversation, and assigns a probability for which word /phrase should come next based on the conversations it has previously seen . We don’t know for sure what data was used exactly in training, but it is some combination of guided dialogue transcriptions they sourced from their contractors and data gathered from the internet called the “common crawl” that is filtered for internet conversations from sources such as twitter, reddit, stack overflow, and stack exchange. This is the “P” in ChatGPT, Chat Generative Pre-trained Transformers

So what is ChatGPT useful for? Generally, neural network models are useful for automation: making difficult or mundane tasks much easier/faster. (As an “elder millennial”, I remember the days of physical maps and Mapquest. The seamless integration of models in Google Maps has automated navigation, for example) As ChatGPT is currently the most sophisticated conversation modeler, it’s immediately useful for automating many use cases revolving around Questioning and Answering (Q&A). However, since conversation modeling is quite broad, it can also be used for many things outside of Q&A. Some early applications include:

1. Customer Service and Triage. Customer service is one of the bread and butter applications of Q&A. There are already many chat bots in the wild for automated customer service, and they will likely get more sophisticated and ubiquitous as LLMs such as ChatGPT mature.
2. Content Creation, such as outlining and form letters. ChatGPT has seen many outlines during training. So, ask ChatGPT to create a structure of an article, or a book, or a thesis, or a proposal, or a letter of intent, and it will do quite well. This also works for the complete content of things like form letters.
3. Code snippets. This seems surprising, but remember that since ChatGPT approximates conversations from its training set, it has seen many lines of code on Stackoverflow, which also links to open source programming documentation. Ask ChatGPT to write code that corresponds to the type of code that exists in code documentation and it will do quite well at reproducing it. The actual code may not do what you want it to do (indeed, Stackoverflow banned generated code for this reason), so you should test/edit the generated code in small pieces. However, this will change programming education, possibly lowering the bar to entry. Which leads me to….
4. Education. As ChatGPT is very good at drafting content (see 2. and 3.) the applications to education is obvious. However, this is currently quite controversial.
5. Domain-specific assistants. It’s beyond the current scope of ChatGPT, but they have a lot of future promise as domain-specific assistants.
6. Language Assistants. ChatGPT isn’t there yet, but the components are there to make Babel Fish a reality.
7. Generating Better Product Copy. Related to 2.; I do see ChatGPT being used for generating product descriptions that make products easier to sell.

ChatGPT is still new so there is likely to be other innovative applications.

Is ChatGPT a glorified BS artist? Weaknesses, pitfalls and Future

The limitations and pitfalls of ChatGPT are many. I’ll start here:

1. The model does much better in areas well represented as digital text (computer code, politics, science) than in areas that are not. The more sentences there are of a topic in the training set, the better ChatGPT is able to generate responses. (The more aligned the conversation is with the “P” in ChatGPT, the better it does) Remember – ChatGPT is a very sophisticated function approximator, and it can only approximate responses based on the data it has seen.
2. ChatGPT will reproduce misinformation from any of its input sources — it is not an intelligent system that tries to balance or weight different perspectives.
3. Because the choice of words captures tone and reveals biases, ChatGPT will tend to reproduce the tone and bias of the articles in its input corpus. Confident, scientific, or racist, it will reproduce anything.
4. Speaking of bias – as ChatGPT is a function approximator, (sorry for repeating myself), the bias-variance tradeoff is unavoidable.
5. The sources of individual fragments are lost. This is not information retrieval. (And because this is not information retrieval, declarations regarding the end of Google are premature)
6. Generative models (the G in ChatGPT) are notoriously hard to verify.

What does one do to verify truth of responses from ChatGPT? How do you handle its bias? Since ChatGPT can not do information retrieval, how can you verify its sources? These are serious questions. What do you call a system that sounds confident about what it says regardless of whether what it’s saying is true? Harry Frankfurt wrote an entire book cautioning against such a person – the Bullshitter. Some folks react to this fact with a hint of nihilism – Wired Magazine argues that of course ChatGPT is a BS artist because the internet is inherently BS. Blind trust in ChatGPT can cause a lot of confusion and harm, such as perpetuating harmful stereotypes and biases, increased political polarization, lead business leaders to make costly ill-informed decisions, and scam artists trying to rip off the unsuspecting public, which is already happening.

LLMs such as ChatGPT are here to stay. What does that mean moving forward? ChatGPT is clearly disruptive, with many useful applications, but I want to leave three parting thoughts about ChatGPT moving forward. My first comment is a sobering reality check: We will inhabit an AI future that offers most significant returns to investors. Microsoft just recently invested 10 billion dollars, taking a 49% stake in OpenAI. (For a company that started as a non profit, how times have changed) My second comment is that I hope more leaders will speak out on the ethical concerns of LLMs specifically and AI in general moving forward. I am encouraged that Demis Hassabis of DeepMind spoke to Time Magazine yesterday urging caution. I anticipate that the messaging may be mixed moving forward, but I hope more leaders follow in Demis’s footsteps. My third and final comment is that I advocate what Wired Magazine calls “AI supply chain transparency” to build public trust. It is not a panacea, but it’s a place to start.

Post Script: A Cheers to Google.

To quote Isaac Newton – “If I have seen further [than others], it is by standing on the shoulders of giants.” The “T” in ChatGPT stands for Transformers, which was invented by Google Research in their landmark 2017 paper, Attention Is All You Need. It is a landmark AI paper, and one of the most influential AI discoveries in the history of the field. Large Langauge Models are possible in large part due to their work. So, cheers!

Top Ten Films of 2018

In my opinion, 2018 has been a weak year for films, and easily the weakest year this decade. It appears like the stark prediction that Steven Spielberg and George Lucas made over five years ago is coming to pass.  Their theme park analogy is particularly relevant with the increased power Disney has in the film industry, particularly with their purchase of the 20th Century Fox movie studio. The other major studios are predominantly concentrating only on a handful of tentpole films, and the division between television and film for low budget films is blurring. Indeed, Netflix and Amazon have picked up the art house film torch. Netflix has a lot riding on Roma and have more prestige films in production including Martin Scorsese’s next film, The Irishman. I personally worry about the tech giants having this much power in the film industry, as I fear that their presence would widen the gap between the film haves and have nots. Sure, respected filmmakers such as Alfonso Cuaron and Martin Scorsese will be fine, but how would the next Barry Jenkins be discovered? Film is about people, not about algorithms.

Still, there are great films released even in a down year such as 2018. Here’s my personal top ten list. Two caveats when reading through my list: I personally favor visually-oriented films, and I of course haven’t seen everything released in 2018.

1: Roma – Alfonso Cuaron

I was fortunate enough to see Roma a few months prior to its wide release at the Philadelphia Film Festival, where it cemented its status as the best film of 2018. After back-to-back commercial and critical sci-fi/fantasy successes including Harry Potter and the Prisoners of Azkaban, Children of Men, and Gravity, Alfonso Cuaron delivered something unexpected with his latest film, Roma. Roma is an intimate neo-realist film made with the eye-popping spectacle of a modern sweeping epic, complete with the director’s trademarks of elaborate tracking shots, meticulous choreography, and detailed world building. Neo-realist films (made popular by Italian filmmakers such as Fellini and Rossellini) are characterized by being more concerned with depicting the day-to-day life experience of their characters in a particular time and place rather than a traditional plot-driven three act narrative. The particular place is the Colonia Roma neighborhood in Mexico City, and the particular time is the early 1970s, both of which are brought to life with vivid black and white cinematography and the director’s technical virtuosity. As Cuaron is primarily concerned with depicting the film’s protagonist Cleo’s daily life experience as a nanny in Mexico City, (based on his own nanny from childhood), the film itself requires patience that is rewarded with a powerful ending. Beyond commenting about class in Latin culture and finding grace in every day life, Alfonso Cuaron emphasizes the interconnectedness of human experience and that all actions, no matter how small or large, have an effect on others.

2: If Beale Street Could Talk – Barry Jenkins

There’s no sophomore slump for Barry Jenkins, whose follow up to Moonlight is a heartfelt love story set on a backdrop of racial injustice in 1970s Harlem. Barry Jenkins has a gift for using color, lighting, and music to set the mood of his films and convey the inner state of his characters. His cinematic technique has an Asian sensibility; indeed Barry Jenkins himself has stated that he’s quite influenced by the great Hong Kong filmmaker, Wong Kar-Wai.  (He also effectively borrows a bit from Martin Scorsese for Beale Street) Barry Jenkins fundamentally understands that the strength of cinema lies in portraying human experience rather than narrative arcs, and Beale Street conveys its social justice message through the experience its excellent characters falling in love and the obstacles they face as the audience shares in Tish and Fonny’s tenderness and hardship.

3. Annihilation – Alex Garland

Annihilation continues a yearly tradition of mature and smart science fiction that started with Gravity in 2013.  Blending Solaris, Apocalypse Now, and Alien, Alex Garland deftly manages disparate tonal shifts to make a film that is equal parts contemplative and horrifying. Like a few films on my list this year, the narrative struggles toward the end. However, Alex Garland uses inventive and distinctive visuals to make a film that’s a metaphor for grief, impermanence, and the fragility of life.

4. Widows – Steve McQueen

“Heat for the me too era” is probably the best way to describe Widows. Both heist films share a film noir sensibility and feature complex characters whose actions are driven by broken family dynamics. Michael Mann in his 1995 classic used the tropes of the heist genre and his characters to comment on the relativity of good and evil. Steve McQueen goes further in Widows, by suggesting that the inherent corruption in American political power structures forces the characters in his film to make morally ambiguous choices. Steve McQueen makes his case with visual storytelling, best exemplified with the technically exhilarating but startling dialog scene filmed on a car dashboard as it drives through a gentrifying neighborhood in Chicago.

5. Mission Impossible: Fallout – Christopher McQuarrie

I personally had more fun with Mission Impossible: Fallout than any other film in 2018. An exercise in pure cinematic spectacle, the formula established by Christopher McQuarrie and Tom Cruise is to provide spy movie thrills in the form of high adrenaline action sequences embedded in a cat-and-mouse caper storyline. (Which, upon close reflection, is admittedly completely ridiculous) The action sequences in Mission Impossible: Fallout are among the technically most accomplished in the action genre, which use the geography of its international locations in ingenuous ways. I couldn’t help but get a retro vibe when watching Mission Impossible: Fallout, with its reliance on intricate stunt work (famously performed by Tom Cruise himself) and insistence of capturing as many special effects in camera as possible. In an era where tent pole films are computer generated affairs, Mission Impossible: Fallout increasingly feels like filmmaking from another time and place.

6. First Reformed – Paul Schrader

Religious themes are probably the most challenging ideas to address in film, and Paul Schrader’s main character in First Reformed is stuck in the Dark Night of the Soul. Played brilliantly by Ethan Hawke, (who much deserves an Oscar nomination) Ernst Toller seeks consolation in the church he leads and in the writings of Christian Contemplatives in order to help him move beyond a personal tragedy, but consolation is nowhere to be found. First Reformed asks difficult existential questions about religious life. Is over-reliance on a religious institution an impediment to spiritual growth? Can personal faith be more than a cultural or political tribal allegiance? Is the greatest temptation on the spiritual journey not carnal pleasures, but in finding solutions to suffering that are unable to bring lasting happiness? Can true consolation only be found within, by helping others, or a combination of the two? Paul Schrader treats these questions seriously and explores them earnestly. An almost masterpiece, First Reformed falls apart during its lackluster, grindhouse ending that is overly inspired by Paul Schrader’s own screenplay from Taxi Driver. The ending simply doesn’t work and clouds the central themes of the film, however, few films in 2018 has stayed with me like First Reformed. 

7. A Quiet Place – John Krasinski

 

 

A Quiet Place is the most tense and suspenseful movie experience of 2018, a genuinely terrifying horror movie brought to life with expert sound design, flawless acting, and smart editing. Taking a page from Ridley Scott’s Alien, John Krasinksi emphasizes atmosphere and tone rather than gore. There is a core of humanism beyond the scares and the horror movie tropes that asks what defines a family and what is needed to preserve it.  The film’s well-drawn and empathetic characters each offer something unique to the family’s survival, including the smart and brave protagonist, Regan, who also happens to be deaf. Although her disability seemingly puts her at a disadvantage with the monsters of the film, John Krasinski cleverly subverts this assumption at every turn.

8. The Ballad of Buster Scruggs – The Coen Brothers

 

The Ballad of Buster Scruggs is a Twilight Zone-style anthology set in the Old West made with a Coen Brothers sensibility. The Ballad of Buster Scruggs consists of six short stories that manage a delicate tonal balancing act between hard-edged realism and exaggerated tall-tales, with the narratives blending the brutal with the absurd. The best Western since Tarantino’s Hateful Eight, the Coen Brothers turn Western tropes on their head. The first two short stories recall the Coen brothers screwball comedy roots, but the anthology soon enters the darkest territory the Coen brothers have covered since No Country For Old Men. Indeed, death is a central theme pervading all the anthology short stories.

9. A Star is Born – Bradley Cooper

A remake of a movie with the same name with Barbara Streisand and Kris Kristofferson, A Star is Born is part fairy tale, part Hollywood melodrama. Excellent acting and intimate direction elevate the film above its melodramatic tropes, as both Bradley Cooper (channeling both Eddie Vedder and Jeff Bridges) and Lady Gaga have great on-screen chemistry. In some ways, A Star is Born is the anti-Roma. Whereas Roma uses the cinematic language of epic spectacle to portray every day life, A Star is Born relies heavily on closeups, lending the film an intimacy and putting the emphasis on the actors. (Even the songs sung on stage are shot mostly in closeup)

10. The Favourite – Yorgos Lanthimos

The Favourite is an unpredictable blending of Downton Abbey and Veep. The screenplay is viciously funny, and the entire conceit of the film wouldn’t work without the excellent acting of Olivia Coleman, Rachel Weisz, and Emma Stone. The film is at its best when it enters Mad Men territory, reflecting on how a person of power who seemingly has everything could be so lonely and unhappy. The film stumbles during a lengthy setup and an anti-climactic ending, nevertheless The Favourite is the freshest costume drama to be released in a long time.

Honorable Mentions

• Burning – Lee Chang-Dong: I’m not as in love with Burning as the art-house film community is, but it’s nevertheless a thought-provoking slow-burn thriller about class warfare in Korea.
• Black Panther – Ryan Coogler: My personal distaste for Marvel Studios is the only thing keeping Black Panther from my top ten of the year, but it’s the best Marvel film to date.
• BlackKlansman – Spike Lee: Tonally off, but still provocative and Spike Lee’s best film in years.
• Isle of Dogs – Wes Anderson: Weaker characters than his other films, but still colorful, joyful, humanistic, and thematically very Wes Anderson.

Top Five Takeaways from NeurIPS 2018

Although I’ve known about NIPS since my physics days, I never could have imagined the sheer massive scale of this world-famous conference on Artificial Intelligence. Renamed to “NeurIPS” in 2018, the 32 year old AI conference had over 1,000 accepted papers and nearly 8,000 registrants in attendance in Montreal for one day of tutorials, three days of invited talks, parallel sessions, and poster sessions and two full days of specialized workshops. Despite the scale and significant tech industry presence, NeurIPS is first and foremost an academic science conference.

The sheer variety and volume of topics at NeurIPS makes it impossible to see everything at the conference. Because of this, other attendees might come away with quite different takeaways than I have. I should also mention some personal disappointment in not seeing ethical issues in the field (such as diversity, systemic bias, and privacy) being more broadly addressed at the conference. With that in mind, however, here are my top five NeurIPS (formerly NIPS) takeaways:

1. Subtle Architectural Advancements

After a few years of major architectural papers and announcements, such as Generative Adversarial Networks and Capsule Networks, 2018 was a quiet year in comparison.  My own personal view is that the major architectural advance in 2018 belongs to Neural Ordinary Differential Equations (already being coined ‘ODENets’ by some) by Ricky Chen et al. at  the University of Toronto. https://arxiv.org/pdf/1806.07366.pdf. The paper also shared top honors in the NeurIPS organizing committee’s best paper category.

‘ODENets’ aren’t new. You can find a good introduction at https://srome.github.io/Using-Ordinary-Differential-Equations-To-Design-State-of-the-Art-Residual-Style-Layers/. What is new from the paper from Chen et al. is their thorough investigation of ODENets properties, including tests on image recognition and continuous time models. Both will likely be relevant for deep learning on videos, and continuous time models would likely be helpful for challenging time series modeling tasks such as connected devices with a high sampling rate. Possibly the most useful results of the paper, however, is in showing how well ODENets perform in density estimation tasks.

2. Improved fundamental understanding of existing approaches

To me, ‘Sanity Checks’ was the dominant theme to NeurIPS 2018. Quite a few papers focused on improving fundamental understanding of neural network architectures. A good ‘sanity check’ example comes from Pierre Baldi from UC-Irvine. He defined Neuronal Capacity using arguments from information theory (https://papers.nips.cc/paper/7999-on-neuronal-capacity) to show that deeper neural networks have “less neuronal capacity” than shallow networks, meaning that shallower networks can approximate more functions. However, the functions deep neural networks can learn are smoother and better behaved, thereby acting as another form of regularization that he termed ’structural regularization’

Another good ‘sanity check’ paper is from Shibiani Santurkar et al. “How Does Batch Normalization Help Optimization”. They convincingly demonstrate (http://papers.nips.cc/paper/7515-how-does-batch-normalization-help-optimization) that Batch Norm makes gradients more stable, smooths the loss function landscape, and thereby making neural network training faster and more reliable. If BatchNorm can be applied to your use case, definitely use it

Other authors pushed popular algorithms forward. U-Net is a popular architecture for image segmentation, and a team at DeepMind defined a ‘probabilistic U-Net’ (https://arxiv.org/abs/1806.05034) for applications such as medical imaging where image segmentation can strongly depend on the underlying pathology. Graph Neural Networks are a perpetually popular topic, and Stanford presented an innovative pooling technique (https://arxiv.org/abs/1806.08804) to summarize (i.e ‘pool’) local information in a graph.

There are many other examples in addition to the papers above. It is encouraging to see researchers both in universities and in the private sector spending significant time to understand how neural networks function.

3. Popular algorithms were shown to have substantial limitations

The other side to the ‘sanity checks’ theme is that many popular algorithmic approaches were shown to have substantial limitations. Q-learning is a popular algorithm to learn policies (which action to take) in Reinforcement Learning, however Google Brain showed that Q-learning can lead to some severe biases known as ‘Delusional Bias’ https://arxiv.org/abs/1806.08804. In image recognition, several popular techniques exist to construct saliency maps from neural network gradients to rank which input pixels are most important in feature construction and classification. One of the most important NeurIPS results again comes from Google Brain (http://papers.nips.cc/paper/8160-sanity-checks-for-saliency-maps), who developed a rigorous saliency map testing framework. Unfortunately, several popular approaches failed the tests in this framework, calling into question many previously published research results in the image recognition domain.

4. Improved collaboration between researchers/engineers and stakeholders are key to drive AI forward

“ Those [Reinforcement Learning] simulations may help make results reproducible, which is good, but they miss a lot of the complexity of the natural world, which may make the work less meaningful, and less rigorous” – Joelle Pineau

This was encouragingly a widely held sentiment at NeurIPS 2018. “From personal experience, 90% of our failed projects was because of poor communication and/or collaboration, not lack of technical expertise” remarked a managing Machine Learning Engineer at Doc.ai. Barbara Engelhardt in her talk in the NeurIPS ML4Health workshop observed that trust from physicians in Machine Learning is currently fragile, and machine learning practitioners need to deeply collaborate with medical professionals for artificial intelligence to truly have an impact on health care.

 

5. AI in Drug Development is still Young

Although there were nearly 8,000 participants in NeurIPS 2018, the representation from the Pharma industry was relatively small. (See for example https://medium.com/@longevity/how-serious-is-big-pharma-about-ai-in-drug-discovery-f4cfe23cfe85) Additionally, many professionals in the tech industry lump the challenges of Machine Learning in health care and drug development together with one, broad brush stroke of “health care”, even though the problems are often quite different.

Nevertheless, it was great to attend an ‘Artificial Intelligence in Drug Development’ lunch with researchers from the industry applying the latest in advancements in AI/ML to drug development, including representatives from GSK, Pfizer, and AI startups InSitro, phonemic.ai, and benevolent.ai. Among discussing the usual challenges with applying these approaches to complex, multidimensional data, there was universal agreement that the field is in desperate need of standardized benchmark datasets to compare algorithmic approaches. (This is direct analogy with common benchmark datasets in the vision recognition domain, MNIST and CIFAR) There was also a desire to have a dedicated ‘AI in Drug Development’ workshop in a future NeurIPS conference.

Top Five Takeaways from the Strata Data Conference

This was only my second Strata conference, but it was interesting to see an emphasis on Data Science and Machine Learning in a conference that was historically had more of a bent to big data platforms and software engineering. This is not a criticism; the conference was a fantastic mix of talks both on the platform side and the data science side. I was impressed with the quality of talks, including such industry giants such as Jeff Dean from Google Brain. I was also impressed with the conference organizers and the program committee in providing a breadth of topics and a generally well-run conference. Here are my top five takeaways from the March 2018 Strata in San Jose:

1.Machine Learning vendors and developers are focusing on tools that make model deployment easier

Machine Learning developers for the past few years have focused on developing tools that make it easier for data scientists to build machine learning models. This has included support for a wide variety of models and architectures, as well as providing APIs that make working with these models tractable. Frameworks such as TensorFlow open sourced by Google has accelerated the use of neural networks to solve practical business problems. In addition, new data science platforms have appeared to help accelerate data science work, including machine learning model development from companies such as Anaconda, Cloudera, and Domino. Features vary, but these platforms generally combine a computational notebook development environment with self service data management tools that all work from a web browser.

Representatives from these Machine Learning frameworks and data science platforms at Strata have predominantly focused their talks on how to practically deploy models into production. For example, Tensorflow’s fantastic talk by Google’s Rajat Monga mainly focused on their feature road map that makes using and deploying TensorFlow models easier, including full interoperability with the Keras API. Microsoft gave a talk on practical considerations when deploying finicky Recurrent Neural Network models that rely on high velocity streaming data. Companies in the field gave similar presentations. Although the specific details vary, a recommended production deployment workflow has emerged, which consists of the following steps:

• Model Creation

     This is where you train your model, test your model, and export your finalized model.

• Model Containerization with Docker

     In the big data world, making sure that the right software and package dependencies are installed in your cluster is a DevOps nightmare. Model containerization simply means you pack all the dependencies necessary with the model in a ‘container’ before you ‘ship it’ to production. Docker is the most popular container engine that makes this process convenient.

• Model Hosting in a production cluster

Once you’re ready to ‘ship’ your model, you have to make sure your production cluster can ‘receive’ the model, run the model, and provide remote access to the model. Different machine learning products accomplish this differently – Tensorflow has a service called “TensorFlow serving’ for this puprose, for example.

• Production deployment with Kubernetes

     Once you have your model ‘packaged’, and the production cluster is ready to ‘receive’ your model, you have to ‘ship it.’ This is where Kubernetes comes in. Kubernetes is the FedEx/UPS of the deployment process – it ‘orchestrates’ docker containers and makes sure they find the appropriate place in the production environment.

The big tech players are not the only ones jumping on this workflow – the Data Science platform companies are incorporating new features to make this workflow easier. Kubernetes deployment is one of the main new features in version 5 of the Anaconda Data Science platform, for example.

2. Hadoop is on life support

The above model deployment workflow should somewhat startle you. Using Kubernetes for container orchestration implies that YARN – the Hadoop resource management and job scheduling tool – becomes significantly less important for data science workflows, to the point that it’s worth asking the question of whether YARN specifically or Hadoop generally is still relevant. The increased popularity of cloud object storage frameworks (such as Amazon S3) is also rendering Hadoop obsolete. Common Big Data stacks increasingly use a Spark cluster in conjunction with Amazon S3, bypassing Hadoop all together.  It is abundantly clear the community is moving away from Hadoop. Even Cloudera is aware of this, and to their credit their platform roadmap is deemphasizing the Hadoop ecosystem.

Although Hadoop as a distributed file system and YARN for resource management is on life support, I’m not fully ready to call Hadoop dead. For one, options for on premises object storage solutions aren’t fully mature, and standards haven’t been fully set. There are still enterprise applications where the the Hadoop Distributed File System (HDFS) still works well, and the database solutions built on HDFS are well-understood and work quite well for certain domains.

3.  Data Science is maturing as a field and data-driven companies are able to offer valuable lessons learned

Netflix, Gap, Google, Slack, Cloudera, and Blizzard Entertainment are among the companies who have mature data science organizations that offered invaluable lessons learned from building real life data science products. Michelle Casbon from Google offered a great ’10 lessons learned from using Kubernetes for streaming NLP applications’ talk. Google yet again has provided thought leadership on a disruptive new tool, Kubernetes in this case, that is only beginning to be adopted by the community. Slack offered insights on enriching search results with Machine learning. Ted Melaska from Blizzard offered lessons learned from successful data science projects from the managerial and team building perspective. We are entering an era where data science is a mature discipline being used by data-driven companies to make business decisions. They offered valuable insight of approaches to try, pitfalls to avoid, and how to successfully transitioning a company to become a data driven organization. It’s a testament to how much Strata (formerly ‘HadoopWorld’) has changed as a conference.

4. Niche Machine Learning applications such as AutoML are becoming mainstream.

I was surprised to see what used to be considered niche areas in Machine Learning research are becoming mainstream features in data products and overall data strategy in certain organizations. AutoML used to be a niche research area of automating machine learning model selection and hyper parameter optimization, being evangelized by such Machine Learning researchers such as Randy Olson formerly at the University of Pennsylvania. DataRobot was one of the first out the gate to productize AutoML, and now AutoML everywhere. Google published a blog piece back in November (https://research.googleblog.com/2017/11/automl-for-large-scale-image.html) regarding AutoML, and it’s now a Google Cloud product as well as a feature being integrated within TensorFlow. H20.ai partnered with Nvidia for their version of AutoML, called ‘Driverless AI’, that runs on Nvidia GPUs.

I was also surprised to see other niche approaches becoming more mainstream, such as Active Learning and other semi-supervised Machine Learning approaches. Paco Nathan from O’reilly media gave a fantastic talk regarding actual business applications of semi-supervised learning and using domain expertise to fill in label gaps.

 

5. Ethics in big data and data science are more broadly addressed.

It’s refreshing to see ethics in data science and big data being more broadly addressed than a single talk at Strata. Natalie Evans Harris, former senior data policy advisor to Obama’s Chief Technology Officer, offered case studies on using open source cloud data integration services focused on social programs, including a use case involving a homeless intervention and prevention program in Indiana. Her take home message regarding ethics in data science is powerful, where she quoted congresswoman Barbara Jordan to challenge and wake up the data science and big data community: “There is no executive order; there is no law that can require the American people to form a national community. This we must do as indivudals, there is no President of the United States who can veto that decision. We must define the common good and begin again to shape a common future.”

More broadly, talks brought up fascinating ethical concerns. Ryan Boyd from Neo4j did a graph analysis of Russian trolls on Twitter and open sourced his work. What is the ethical responsibility of the tech community as a whole to combat fake news? What is the ethical responsibility of the US Government to continue this work for our defense? To what degree do the social media companies share this responsibility?

Seth Stephens-Davidowitz from the New York Times gave a talk called “Everybody Lies’ based on his book of the same name. In his talk, he touched upon several points of how big data and the Internet can reveal important facts about ourselves. Of particular interest to me was that Google search histories can actually be more accurate than Gallup surveys in certain questions, such as suicide prevention and quantifying racism. This brings up an interesting ethical dilemma. We are all concerned about data privacy, and where do we draw the line to make decisions based on data that was not originally collected to be used in a formal survey?

 

Top Five Takeaways from the ‘AI Pharma Innovation in Drug Discovery’ Conference

‘We don’t know how to analyze data, so let’s make a dashboard’  – Leonardo Rodrigues, BERG

The pharmaceutical industry faces serious challenges. It takes 15 years and 2 billion dollars on average to develop a new medicine, and failures are costly. If the development time of new drugs could be reduced with modern data-driven approaches, one can reduce the cost of drug development and help patients faster. Machine learning has revolutionized the modern digital economy through industries ranging from e-Commerce to the promise of self-driving cars. Machine learning is not new to the Pharma industry, but its application to drug discovery has been limited. Every step of the drug development pipeline could benefit from novel Machine learning approaches. It’s been shown that genetic evidence nearly doubles the probability of success for a new drug. It takes on average 5,000 molecular assays to find a lead drug candidate, so ‘lead optimization’ by reducing the number of experimental assays could enable early drug discovery to be more agile, save cost, and focus more time on finding novel targets. Finally, drug candidates that make it to clinical trials often don’t make it to market due to efficacy failures, and these clinical trial failures are extremely costly.

It’s with the above context that ‘Artificial Intelligence in Pharma’ conferences are starting to appear all over the place, including the 2018 ‘AI Pharma Innovation in Drug Discovery’ conference in San Francisco that I attended. ‘Artificial Intelligence’ is a term that I believe is being overused now, and everyone defines ‘AI’ differently. ‘AI’ in this conference is predominately used as a synonym for Deep Learning.  Here are my top five takeaways from the conference:

1) It’s all about the data

‘If the data is not easily accessible, all AI initiatives are destined to fail’ remarked Leonardo Rodrigues of BERG on the opening day of the conference. Modern deep learning approaches are data-hungry, and Pharma isn’t great historically at making data across the discovery pipeline easily accessible to R&D researchers, encouraging data silos and analysis results (in the form of excel spreadsheets and powerpoints!) that disappear into the ether of mapped network drives.  It was strangely comforting to see that these problems are still endemic across Big Pharma, and there’s universal agreement that data needs to be widely accessible and reliable for Deep Learning initiatives to be successful. Hugo Cuelemans of Janssen commented during his Virtual Drug Screening talk that although Janssen has found Deep Learning to be extremely powerful for predicting compound activities, the actual approach and specific neural network architecture was less important than having as much historical, clean assay data as possible.  It’s a difficult problem, since the large variety of pharmaceutical data is significantly more complex than other industries, such as data from e-commerce sites.

2) The Pharma Machine Learning startup field is getting crowded

The amount of new startups focused on using machine learning and deep learning to solve specific drug discovery problems is staggering. If you didn’t know better, one would think AI drug discovery is completely a solved problem! Imaging provides the lowest hanging fruit in applying deep learning technology; as such many of the early Pharma Deep Learning initiatives are specific to imaging applications. This was reflected in the startup world for a while (Arterys, for example) as well. Today, there are a plethora of new startups applying deep learning methods to a wide variety of applications including immunotherapy modeling (Kadmon), pharmacokinetics (Numerate), biological network mapping (e-Therapeutics), among other applications.

3) Pharma is beginning to appreciate the value of Transfer learning

One of the pleasant surprises from the conference is that there is a general understanding of the value and power of Transfer Learning. Promising approaches in machine learning publications are normally trained and validated with public data sets, and therefore some transfer learning approach would be necessary to test such approaches on proprietary pharmaceutical data. Janssen went even further, suggesting that Pharma companies could form consortiums to share deep learning models, but not proprietary enterprise data. This could be made possible through transfer learning, and might be a great strategy to help accelerate the adoption of machine learning in drug discovery.

4) There’s anxiety that Deep Learning is over-hyped and simply a fad

Despite the general excitement surrounding Deep Learning at the conference, there is still a lot of anxiety and skepticism surrounding the ability of the field to make a significant impact in drug discovery. This is due to several reasons. One reason deals with recent high-profile failures in the field, such as IBM Watson Health’s partnership with MD Anderson, the DeepMind collaboration with the NHS in the UK, and a few negative recent publications such as this medium article (https://medium.com/the-ai-lab/artificial-intelligence-in-drug-discovery-is-overhyped-examples-from-astrazeneca-harvard-315d69a7f863) that highlights examples from AstraZeneca, Harvard, and InSilico Medicine. The other reason is that many veterans in the industry are skeptical of Deep Learning because these same veterans have seen Big Pharma adopt cutting edge technologies such as high throughput screening and massively parallel genome sequencing early, yet the costly and lengthy drug development pipelines have only worsened. The anxiety is understandable, but representatives from companies such as BERG, Genentech, and Roche mentioned that Deep Learning is here to stay and Pharma has to modernize due to the ‘disruption’ the field has caused in other industries such as self driving vehicles.

5) No one addressed deep learning model deployment

The conference addressed a wide array of topics which were broadly classified by the conference organizers into the following categories:

• The fundamentals of setting up a successful machine learning approach in drug discovery
• Matching AI & Machine Learning to Biopharma’s challenges
• Enhancing drug discovery productivity with AI & Machine Learning
• Practical applications of AI & Machine Learning in drug discovery
• What is the future of AI & Machine Learning in drug discovery

 

None of the talks addressed the practical considerations of how to deploy deep learning models to production. I saw many approaches and initiatives  but I didn’t see any success stories; meaning I didn’t see any examples of how deep learning insights were used to make specific decisions in drug discovery. I did not witness any ideas of how to turn deep learning models to data ‘products’ that help scientists and managers to make recommendations. Although it is the early days of machine learning adoption in Pharma, there are valuable lessons learned from other industries in building machine learning models, (one of my personal favorites is from a Netflix alum https://medium.com/@xamat/10-more-lessons-learned-from-building-real-life-machine-learning-systems-part-ii-93fe7008fa9) many of which are only sensible and obvious with experience. I am emphasizing this point because I firmly believe practical deployment considerations are a bottleneck for the success of Deep Learning initiatives in the Pharma industry. (Would Amazon be Amazon if their product recommendations were flyers they mailed once a month rather than easy to use and consume recommendations from your personalized home page?)

 

My Top Ten Films of 2017

To call 2017 a challenging year is of course an understatement. The films of 2017 often thematically reflect these real-world social issues and anxieties. Consequently, my best films list is somewhat darker in tone than an average year. 2017 was also a challenging year for the movie industry as a whole with the revelations unveiled by #MeToo and financial instability. Historically, more people go to the movies in troubled times, but that is not the case today and the film industry is in real financial trouble. (Some good writing about this can be found at Variety, Vanity Fair, and ReelViews) In troubled times, it’s miraculous that great films can still be made and released.

1. Dunkirk (Christopher Nolan)

Dunkirk is a survival story told through the fog of war and experienced through a state of perpetual disorientation and chaos. Christopher Nolan achieves this with limited dialog and relying on unparalleled craft and technique in camera work, editing, and sound design. (Nearly everything is captured in camera, including mounting IMAX cameras on WW2-era airplanes) Nolan has experimented with expressing time differently in cinema with mixed success, but finally nails it here. Time doesn’t exist the way it does in a conventional film, where you safely watch a narrative unfold. Instead, the intention is to convey the experience of the British military during the Dunkirk evacuation. You come through the other side with a very different take of heroism, echoed by the final trailer: “Hope is a weapon. Survival is victory.”

2. The Florida Project (Sean Baker)

Twice as many Americans live in poverty today compared to 20 years ago, and Sean Baker’s The Florida Project shines a light on American poverty through a unique lens of a ‘slice of life’ story of a struggling single mother living in the shadow of Walt Disney World.  Possibly the most empathetic film of 2017, Sean Baker avoids ‘misery porn’ territory primarily because of how he’s able to work with amateur actors to give incredible performances. Willem Dafoe – the only professional actor in the cast – provides his best performance in years. The performances are enhanced by rich cinematography captured on 35mm film that seamlessly switches between child and adult perspectives, offering both childlike wonder and hard realism. The film ultimately succeeds not because it offers preachy moralizing or sentimental solutions, but because the characters are given human dignity.

3. Blade Runner 2049 (Denis Villeneuve)

This long-delayed follow up to the classic science fiction film could have gone oh-so-wrong, but this movie is oh-so-right. Wrapping up old questions from the original film while taking the story in new thematic directions, Blade Runner: 2049 explores many of the same philosophical ideas of other great sci fi-movies: defining human identity, the nature of memory, and what it means to be self aware.  Like the original, this is all wrapped in the frame work of a detective film-noir story and the film is brought vividly to life with the best cinematography of the year by the incomparable Roger Deakins.  Denis Villeneuve already made a great science fiction film with last year’s Arrival; I can’t wait to see what he does with Dune.

4. Three Billboards outside Ebbing, Missouri (Martin McDonagh)

“Acting out of Anger will not yield the results you think”, is the central message of Three Billboards. Ostensibly set up as a revenge thriller, the movie constantly defies expectations – I was not able to get a comfortable read on what the incredibly complicated characters would do or where the story was going. The movie walks a thin line between tragedy and outright comedy (akin to a few Coen Brothers movies) that it somehow pulls off without melodramatic sentiment. Martin McDonagh’s background in stage plays is evident here as this is a performance and dialog-driven movie. Francis McDormand is rightfully the front runner for best actress.

5. The Shape of Water (Guillermo Del Toro)

‘Magical Realism’ is a well-established genre in Latin American literature that has resulted in many fantastic novelists and a Nobel Prize in Literature. Magical Realism is less established in film, where the nature of the medium makes it difficult to pull off. No one arguably pulls it off better than Guillermo Del Toro. The Shape of Water works as a romance, a fantasy, and even a monster movie. Guillermo Del Toro’s cinephilia is present in every scene, enhanced by meticulous set design, a fantastic musical score, and dynamic cinematography. (The camera nearly always moves in the Shape of Water, including quiet dialog scenes)  Like all great magical realist stories, the fantastical elements always come with a huge dose of allegory. Del Toro champions marginalized outsiders whether it be due to disability, race, or nationality and questions traditional American power hierarchies. It’s Guillermo Del Toro’s best film since his Spanish Civil War allegory, Pan’s Labyrinth.

6. Loveless / Nelyubov (Andrey Zyaginstev)

What often differentiates a great film from a good one is how long it sticks with you. Few films have remained with me the past year as Loveless, Andrey Zyagnistev’s follow up to 2014’s Leviathan. A portrait of a divorcing couple whose only son runs away, Andrey explores two primary themes: 1) you aren’t guaranteed to learn the lesson you’re supposed to from a crisis and 2) It is more difficult to find loving families if the state does not value empathy and compassion. It is this second idea that got Andrey Zyyaginstev in trouble with the Russian government – Russian criticism is common in his work, and the Russian Ministry of Culture so disapproved of Leviathan that they refused to provide funding for Loveless, hoping to silence him. Andrey Zyaginstev miraculously got this movie made through international financing from France and Belgium. (Ironically, the film was ultimately so well-received internationally that Russia ultimately submitted Loveless for Oscar consideration to save face.) Stark, brooding, challenging, but provocative, Loveless makes an impact. Zyaginstev is a very precise filmmaker –  every camera move, lighting setup, and edit are deliberately used to advance the story or offer insight into the characters. Loveless cements Andrey Zyaginstev as Russia’s best living filmmaker.

7. War for the Planet of the Apes (Matt Reeves)

Forget Disney Star Wars – The re-imagined Planet of the Apes franchise is the best sci-fi/fantasy trilogy of the 2010s. Everything you’d want from a summer tent pole movie is here, rousing action, amazing special effects, and sheer spectacle, but the film has a depth not usually present in a movie like this. Matt Reeves continues the biblical allegory set up from the previous film, while bringing in a variety of cinematic influences including westerns (particularly The Searchers), The Great Escape, and Apocalypse Now. Andy Serkis is completely convincing as Caesar – Modern day special effects are so advanced that they can be reliably used to make a ‘digital costume’ for an actor.

8. Get Out (Jordan Peele)

Even without Jordan Peele’s social commentary, Get Out is still one of the most original and well-crafted horror movie in years. It’s an edge-of-your-seat horror/thriller, and Jordan Peele achieves this in part because Get Out stands tall with Dunkirk and Baby Driver as the films of 2017 with the best editing and sound design. Underneath the effective genre exercise, however, is biting satiric commentary on systemic racism and the myth of a post-racial America.  Despite being his first feature film, Jordan Peele made Get Out with the confidence and skill of a veteran filmmaker.

9. Tie: Lady Bird (Greta Gerwig) / The Meyerovitz Stories (Noah Baumbach)

 

I admit that it’s a cheat to have two films tied, but both Lady Bird and The Meyerovitz Stories (A Netflix original) share a cinematic voice, both stylistically and thematically.  (It’s probably not a coincidence since both filmmakers are domestic partners) Lady Bird and The Meyerovitz Stories both have a lot in common with Woody Allen’s older films  – namely dialog-driven movies which rely on long, extended takes from a wide master shot, limited but deliberate use of music, and stories that try to balance comedy with drama. Thematically, both movies explore similar territory: Family members who ask each other ‘Yeah I suppose I love you, but why can’t you be different?’ and the narratives flesh out the consequences of this attachment on the characters.

10. Baby Driver (Edgar Wright):

If you like car chases, see this movie immediately. Next-level camera work and some of the most inventive editing in 2017 result in flawless chase sequences and a frenetic pace that persists throughout the movie. The film itself is a fun mash-up of different genres, such as heist movies, action thrillers, and musicals. (Guillermo Del Toro hilariously calls Baby Driver “An American in Paris on wheels and crack smoke”) Although the narrative suffers a bit near the end of the movie, Baby Driver is still one of the most fun, funny, and exhilarating movie experiences in 2017.

Honorable Mentions:

 

Coco (Pixar): Coco just missed my top 10. It’s one of the most imaginative and joyous movies of 2017. The film is not only respectful to Mexican culture, but it’s seamlessly integrated into the stories and characters with an incredible and authentic attention to cultural detail.

I, Tonya (Craig Gillespie): If Goodfellas and Fargo had a baby, it would be I, Tonya. If that sounds appealing, seek this movie out. Margot Robbie is very good.

Phantom Thread (Paul Thomas Anderson): A movie about fashion in 1950s London might sound like a snooze fest, but this is Paul Thomas Anderson.  Beyond the incredible craft, the movie surprises with unconventional plotting, mysterious lead characters, and an uneasy mood that feels like a twisted version of a PBS Masterpiece special.

Mudbound (Dee Rees): It’s an honest portrait of post-WW2 South. A Netflix original

 

Unsupervised Deep Learning of Videos – thoughts on NIPS 2017 Advancements

NIPS 2017 has come and gone. What was historically a sleepy academic neural networks conference is now all the rage – Tech industry heavy weights from Apple demoing their self driving car program, to NVIDIA announcing the Titan V, are using NIPS as an opportunity for press announcements and recruitment. Although in my opinion the original point of NIPS is starting to get lost with industry involvement, there were plenty of great keynotes and advancements.

One of the more impressive results at NIPS is in the area of Deep Learning of Videos with generative networks. Despite the rapid and impressive development of Deep Learning in imaging the past couple of years, videos remain a challenge. In particular, machine learning is traditionally not great at telling the difference between what moves and what doesn’t move. The industry lingo for this problem is Pose Recognition. Why is this important? We are talking about developing ‘AI’ here, and humans can instantly recognize and distinguish things that move from things that don’t.

NYU scientists presented a new model at NIPS named DrNet that specifically aims to address this problem. In addition, they address this problem with generative networks, which allows them to address this problem, but also create machine-generated videos, which is really cool for a lot of reasons.

High Level Summary / TL;DR version

DrNET aims to disentangle the parts of a video that stay the same from the parts that change. In addition to recognizing these different parts of videos, these features can be used to generate new videos in which an object’s location can change over time. The approach uses two parallel networks, one that learns the content of the video, for example what a person looks like, or the background of the video that doesn’t change, while the other learns the temporal variations/dynamic parts of the content, for example when a person walks around. In order to make this work, the secret sauce is to properly penalize the network that is trying to learn the dynamic part (‘learn the pose’ as it’s called), so it wouldn’t capture too much information from the static content.

The results are great. They achieve a ~93% accuracy in disentangling dynamic from static content in a synthetic test video set. Applying this method to the first few frames of a video, they are able to not only accurately predict actions ~100 frames into the future of the video, but generate video frames hundreds of frames into the future.

These results are an important milestone, for several reasons:

• This approach can perform ‘pose identification’, or discriminate dynamic vs static content of videos, accurately.
• It can be used to generate videos, which is useful for many applications. (Archival video retrieval, text-to-video generation, even drug discovery)
• If you combine this with ‘Object Segmentation’, or algorithms that recognize different objects within a video, (This is actually an easier problem to solve) then Deep Learning with videos will enter a stage of unprecedented power and maturity.

For those interested in “getting into the weeds”, read on!

Neural Network Architecture

The neural net architecture for DrNet is given above. (Taken from their paper). The various components of the architecture are:

• The Pose & Content Encoders

The DrNet authors construct two separate encoders, a pose encoder and a content encoder, designed specifically to learn features of static and dynamic content in video frames, respectively. The abstracted features are concatenated.

• The Frame Decoder

The encoded pose and content features are reconstructed into a specific video frame by a Frame decoder. In some ways, the overall approach is an abstraction of an autoencder approach, as the overall network architecture seeks to reconstruct the video frames it encodes. However, the overall neural network consists of multiple neural network architectures, which individually borrow many ideas from Generational Adversarial Networks. (GANs)

• The Scene Discriminator

One of the secrets to the success of GANs is the concept of a discriminator network, which is a neural network trained to discriminate real images from images generated from random noise. The authors here borrow this idea to create a Scene Discriminator neural network that takes pairs of abstracted pose features from the pose encoder and discriminates whether these features come from the same video scene or not. This is critical to extract coherent dynamic features within a video scene, and allows the network to generalize to more than one scene. Scene changes are historically a huge problem and stumbling block to deep learning with videos.

• The Adversarial Loss

The real innovation with DrNET is their derived adversarial loss. Videos are dynamic, and a video can have multiple scenes. Scene changes in videos have been a huge stumbling block to deep learning approaches, and the adversarial  loss term attempts to overcome this problem. The idea is that dynamic objects within a scene change their state, but are still the same object. With different scenes, objects can change. Think of a movie you’re watching – within a shot, actors can walk around, speak, or do any other action, but they are still the same actors. The setting/background of a shot doesn’t change. Once a different scene starts, (or if you have a camera edit) the scenery can change, and different actors can be in the scene or the shot.

The authors introduce an adversarial loss between the scene discriminator network and the pose encoder, described above. The objective function of the discriminator estimates if  a pair of dynamic features come from the same scene, and the adversarial loss calculates the cross-entropy between the discriminator output and the pose encoder output:

(There are actually two cross entropy terms, depending on whether the second pose feature is being tested is from the same scene, or a different scene.) Inspired by GANs, the adversarial loss encourages the pose encoder to create abstract features that fool the discriminator, which helps the encoder identify dynamic features both within a scene and when a scene changes.

The overall objective function for the combined neural network is:

Where the reconstruction loss is the standard per pixel mean squared error of the reconstructed (decoded) frame and the actual frame and the similarity loss is a regularization loss that captures the mean squared error between frames of the content encoder. The similarity loss encourages the content encoder to capture static features in a video.

Predicting Future Video Frames

The output of the trained DrNET is a set of static features, and a set of dynamic features within a video. You can take the dynamic features, and use them to perform inference of future video frames, and generate synthetic video frames if you wish. The authors provide an example of inference with a standard configuration Long Short Term Memory Model (LSTM):

So how does DrNET perform? DrNET is able to discriminate static and dynamic content of synthetic videos with a 93.3% accuracy, which is astounding. In addition, the figure below shows the inception score, which is a metric measuring the quality of predicted video frames:

Finally, here are some generated videos with DrNET:

Implementation in Torch

DrNET has a github page, check it out! DrNET is implemented in Torch. Some things to note:

• It looks like DrNET is implemented in native lua rather than PyTorch!
• The public data sets they’ve used were synthetic videos created from MNIST, NORB, and SUNCG as well as real videos of human actions from KTH
• In nearly all cases, the pose and content encoders, as well as the decoder were Deep Convolutional GANs (DCGANs)
• The KTH video data set uses different architectures. The Pose encoder is ResNet, whereas the content encoder and the decoder uses the Oxford VGG 16 layer convolutional neural nets.
• Consult the DrNET paper for hyperparameters.
• Torch is probably a more natural deep learning framework for video deep learning than TensorFlow. Both Torch and TensorFlow define neural networks with Directed Acyclic Graphs, but TensorFlow graphs are defined statically, and Torch networks are defined dynamically, and can naturally handle dynamic input data from sample to sample. With the heavy Google investment in TensorFlow, however, this can and likely will change.

All in all, DrNET is a really cool milestone in Deep Learning with Videos. I’m excited to see more video advancements during the coming year.

 

Top Ten Films of 2016

Anyone who knows me realizes that I’m a big film geek. Every year at this time, I compile a list of my ten best movies of the year.  I normally keep this list private, but this year I figured, what the heck – let me throw it out there. How do I define a good movie, and how do I include it in the list? To me a good movie answers this question most clearly: “Why is the movie a movie instead of a play, a novel, a musical, a magazine, or something else?”  Generally, to me, these are movies that use sight and sound most effectively to tell a story with interesting characters.

A few caveats before the list. I, of course, haven’t seen everything in 2016. So there are gaps. Also, every film fan is biased, and I am no exception. What is my personal bias? I have a few. I like science fiction, and there are two sci-fi entries in the list this year. I also think Martin Scorsese is one of the best American filmmakers of all time. No other living American filmmaker has been making challenging and essential cinema for over 40(!) years. His latest, Silence, takes my top spot. (A slight cheat, since I didn’t technically see it until 2017)

 

Without further ado, here is my list of my top ten films of 2016.

 

1. Silence

Director: Martin Scorsese 

Adapted from a novel of the same name from Shusaku Endo, Silence is a deliberately-paced, contemplative film about two Portuguese Jesuit priests in 17th century Japan searching for their mentor, who renounced his Catholic faith in order to save the lives of Christians being persecuted there. Martin Scorsese, typically known for emotional intensity rather than contemplation, channels his inner Akira Kurosawa in how he establishes location, frames shots, paces his scenes, and sparse use of music. Silence is difficult and complex. The movie asks if missionary work can be contaminated by ego, and how this can affect choices made when faced with challenging dilemmas that have no clear solutions. None of the characters in Silence are free from moral burden, and offers no firm conclusions to the ambitious questions it raises. Ultimately, and perhaps quite timely, Silence asks difficult questions about the value of a spiritual life in a world that has no use for it.

 

 

2. Moonlight

Director: Barry Jenkins

Barry Jenkins’ coming of age tale of an African American man, Chiron, discovering his sexuality in the Liberty City neighborhood of Miami could have descended into either Hallmark melodrama or misery porn in the hands of a lesser filmmaker. Instead, tight writing, haunting music, sympathetic performances, and cinematography that perfectly captures Liberty City come together in a great coming of age film.  The screenplay by Barry Jenkins and Tarell McCraney splits the story into three stages of Chiron’s life that connect together seamlessly. What I find particularly clever about the screenplay is that although Moonlight is an intimate film, it liberally applies Joseph Campbell’s narrative template of The Hero’s Journey, which is a template normally used for epics. A twist on the last stage of the Journey has Chiron challenged by Kevin (Chiron’s potential love interest) after Chiron returns to Miami from an extended stay in Atlanta. “Who is you?” – challenges Kevin – not buying Chiron’s transformation in Atlanta.  Perhaps self acceptance is the key to his journey.

 

3. Arrival

Director: Denis Villeneuve

What begins as a solid, but conventional science fiction procedural film about humans making first contact with extra-terrestrial life morphs into a reflection on the nature of free will and how perception can be shaped by language. Heady themes to be sure, and the film only works because it’s anchored by a well-written protagonist that is brought to life by Amy Adams’ terrific performance that gives the movie balance between heart and intellect. A must see for fans of serious science fiction.  Denis Villeneuve (Prisoners, Sicario) proves himself to be a master of the procedural, and holds the reins of the upcoming Blade Runner sequel. Blade Runner 2047 can’t come soon enough.

 

4. The Handmaiden

Director: Park Chan-Wook

This genre-bending pulpy thriller from Park Chan-Wook (Oldboy) starts with a simple enough premise – a Korean woman gets hired to be a handmaiden to a Japanese heiress in 1930s Korea – but not everything is as it seems. Borrowing from diverse genres of film noir, historical costume drama, thriller, and “forbidden romance” melodrama, Park Chan-Wook crafts a visually rich, well-told, and unpredictable story. A word of warning – the film does get sexually explicit and goes to some twisted places – but The Handmaiden is the most visually inventive film of 2016 that puts diverse genres into a blender and plays with audience expectations. Distributed by the newly formed Amazon Studios, The Handmaiden will be widely available on Amazon Prime Video.

 

5. Manchester by the Sea

Director: Kenneth Lonergan

“To what degree does closure from tragedy come from external events or from within?” is the central question explored in Manchester by the Sea. Visually restrained with minimal music and deliberate pacing, the film rests on the shoulders of the actors to bring to life Kenneth Lonergan’s complex screenplay, whose characters handle grief in different ways. Casey Affleck is great, where his character is balancing the grief of his brother’s untimely death with his own tragic history, while trying to look after his nephew.  It’s a difficult film to watch, but it’s a great film that will stay in your mind.

6. Green Room

Director: Jeremey Saulnier 

Take the classic setup of an Alfred Hitchcock movie, where an ordinary character gets caught up in an extraordinary situation, and crank the resulting tension to eleven in this terrifying white knuckle thriller about a struggling touring punk rock band who unknowingly performs a show at a neo-nazi venue in rural Oregon. Featuring a brilliant casting choice against type of Patrick Stewart as the neo-nazi leader, this indie thriller features snappy dialogue, tight editing, and breathless pacing. (A word of warning – it’s extremely violent)

 

7 . La La Land

Director: Damien Chazelle

La La land borrows heavily from classic musicals, in particular Singin in the rain and umbrellas of Cherbourg, but it’s very much its own film about the struggles of making it as an artist and the possible relationship compromises you’re faced with in pursuing your dream. A catchy soundtrack combined with dynamic camera work give this bittersweet (mostly sweet) film a lot of energy.

 

8. Rogue One: A Star Wars Story

Director: Gareth Edwards

I’m in the minority who was dissapointed by the Force Awakens; I found it to be a somewhat shallow remake of a new hope that treaded familiar ground. Rogue One has its flaws, namely too many underdeveloped characters. However, Rogue One is a better movie than the force awakens precisely because it explores fresh ideas while still feeling like a Star Wars movie and enriching the saga. The best film-as-spectacle experience of 2016 – Rogue One delivers everything one expects from a Star Wars movie: a lot of action, eye-popping special effects, and a beautifully realized sci-fi world, but also adds a gritty side to the Star Wars universe by having its characters make sacrifices in fighting for a just cause.

9. Hail, Ceasar!

Directors: Ethan and Joel Coen

The Coen Brothers’ screwball comedy about a movie star (George Clooney) kidnapped by Communists in 1950s era Hollywood is an entertaining send up of the movie business. Leisurely paced and slight at first glance, there is more beneath the surface as the Coen Brothers both celebrate the artistry of Hollywood, while questioning the commercialism of the movie industry and exploring many of the same themes present in their more “serious” films.

 

10. Sully

Director: Clint Eastwood 

Clint Eastwood’s workmanlike portrayal of the real life emergency landing of US Airways flight 1549 in New York’s Hudson River works in part because of Tom Hanks’ great acting. A classy, satisfying, and uplifting movie about duty and being good at your job. Sully avoids being a straight Oscar Bait biopic by reflecting on the fear and self-doubt of its protagonist struggling with being labeled as a hero.

 

Transitioning from Academic Physics to Data Science – Lessons Learned

Although I am a Data Scientist, most of my career has been spent as a physicist in academia. It’s what brought me to Philadelphia; a city I now call home after deciding to stay after my postdoc. Saying goodbye to the academic track is a deeply personal decision for many physicists, and deciding what “life after physics” looks like can be a scary and maddeningly confusing road.

A transition to a Data Science career made sense for me personally. An “Experimental Neutrino Astrophysicist” by training (say that 3 times fast!), I spent a lot of time processing, analyzing, and modeling data from large, collaborative experiments. My typical work in the private sector really isn’t that different from physics in a lot of ways. Many of the tools and techniques are the same.  The main differences are that I work on solving practical business problems, rather than solving interesting, but abstract physics problems, and project times scales are significantly shorter in industry.  My transition to the private sector wasn’t easy, however; I had to learn a lot of lessons by trial-and-error, and sometimes the hard way with some pretty strong feedback along the journey.

I can’t say whether a career in Data Science is for you. (Keep in mind, though, that you will be spending a lot of time programming and analyzing data in some way. If that sounds unappealing, you should probably look elsewhere for a career path.) I can say, however, that physicists are uniquely qualified to be Data Scientists and can have a hugely positive impact in the private sector. I won’t define what Data Science “is”, (I find the term a bit silly – how can you have science without data?!) but you can find plenty of good resources out there through Google. What I like to do is to share some lessons I’ve learned from the transition – I sincerely hope this could be useful to anyone looking to make the jump. This is written from the perspective of my experience as a physicist, but some of this might apply to other fields as well.

Lesson 1: Put effort into the job search and show that you “want it”

Put effort into your resume

When I first started looking for jobs, I had the attitude of “Cool – All I need to do is put my academic CV into a Microsoft Word Resume template, then I’ll be golden.” That was completely the wrong attitude. Resumes and CVs serve vastly different purposes, and it helps to get the perspective of a friend who’s in the business world in some way. Some thoughts to keep in mind:

• Your resume should be like a “movie trailer” – a preview of a specific story that would want a potential employer to follow up and ask more questions.
• The hardest part of writing a resume for me was not only putting my experience in language a non-expert can understand. I also needed to put in what I like to call a “so what” clause, why did I do what I did and what was the impact?
• I firmly believe that resumes should be the maximum length of one page for academics looking to transition
• I think Personal Statements are generally a good idea. I found this article from the Guardian helpful.

Don’t say “The academic thing didn’t work out, so I’m looking for options”

Even though if it’s the truth that a tenure track/permanent academic position not working out is the primary motivation for your private sector job search, you have to be careful of how you convey this message. There is a very real danger that this can be taken as needless complaining and can really hurt your chances of getting hired at a company. In addition, many employers want to feel like you want the job and are excited in some way about the opportunity.

Show some understanding of the company and their “domain”

It might seem like a simple thing, but if you demonstrate some interest in what a company does, a bit of understanding in the specific area/industry that the company works in, and ask direct questions for clarification, it shows a lot of focus and maturity. You don’t have to convey expertise; a little goes a long way in showing that you’re serious and prepared. In a lot of ways, this is just a sublesson of “be prepared” – and can really help you stand out.

A personal portfolio of relevant example work (for example on GitHub, a kaggle competition) shows that you’re serious

To use a cliche, if I only knew then what I know now. Once you land a job in industry, you’ll be very busy, and your work will likely be covered under some sort of non-disclosure agreeement such that you won’t be able to share your awesome/innovative machine learning and analysis projects publically. Coming out of academia is a perfect time to get a personal portfolio of work in order. Most academic work can be publicly shared, and spending time into putting public GitHub page in order is a great showcase of your skills and problem-solving thought process.

Network. It’s important.

I was constantly told when I started my job search how important networking is. I didn’t initially “get it”. I kind of figured that a potential company would see my work, get a sense of how I can solve problems, and that should be enough to land a job. But there are a lot of shortcomings to this attitude, and networking is about more than your personal job search, although healthy networking will help you certainly land a job that’s right for you. Some thoughts:

• Your peers on the job market right now are your friends, not your competition. Reach out and join forces with them. Let’s face it. The physics world can be a bubble that can reinforce some bad habits if you aren’t careful. And physics is fiercely competitive, in terms of landing grant funding and the purple unicorn of a permanent job. There is plenty of room for everybody in Data Science, and every physicist has a lot to offer. Join ranks with your peers looking to make the transition. You can help your peers fill in gaps in skills and the transition process they might have missed, and they can help you do the same.
• Stay in touch with your friends/colleagues who leave academia. Your friends can help you get a sense of the things they’re working on, the industry they’re focused in, and provide guidance on applying for jobs, skill gaps to fill, and the transition in general.
• Meetups can be useful – many companies use meetups for recruiting. I’m on the organizing committee of the DataPhilly meetup, and a lot of practicing data scientists attend meetups. Companies can and do recruit there. Meetups tend to be focused either on information sharing, such that you’ll learn about different activities going on at companies with a local presence, or hackathons. We do both at DataPhilly.
• Consider joining a Professional Networking Group – There are a lot of great professional networking groups out there. Beyond your job search, you can connect to a community and make some good friends. It’s also a not-obvious, but fantastic opportunity to do science outreach!
• Professional Social networking (like LinkedIn) has worked for a lot of people – LinkedIn is not magic, but it can work. I’ve personally landed both of my jobs in the private sector by being very active on LinkedIn. An old, but good Forbes Article discusses how recruiters use LinkedIn more actively than traditional job boards. Being active on LinkedIn goes beyond your personal job search too – a lot of useful professional development articles get posted there, which you can pay forward to your network. Skill endorsements can be very helpful, which not only help you on your search, but you can also pay forward to your network as well.

Lesson 2: Be Confident

If you take away only two points from this blog post, remember: 1) As a physicist, you are uniquely qualified to be a data scientist and can have a huge impact and 2) Companies only care about two things: Are you smart? Can you independently solve a business problem quickly? 

A typical job ad in data science reads something like this:

Come join Initech! You will join our rapidly growing team to develop and implement predictive models that would accurately forecast printer failures.

Skills and Qualifications:

Advanced degree (MS or PhD) in mathematics, statistics, computer science, or related field

At first glance, you might think to yourself – “Wow – they are looking for mathematicians and computer scientists! I can’t compete with these guys, this is going to be a hard sell.” Having been on the other side for some time now,  this is generally not the case. Most companies aren’t looking for a specific background, they just want smart people that work independently and who can solve business problems quickly. So hiring managers put down a list of fields where they feel smart people come from.  Your task, dear physicist, is therefore to convince an employer that you can work independenly and solve business problems quickly.

So what’s the secret sauce physicists have that mathematicians and computer scientists don’t? Experimental physicists have experience with real data, where as most mathematicians and computer scientists do not. Don’t underestimate how valuable experience with real data is. I have personally seen great computer scientists who have dealt with subtle machine learning algorithms in academia, on clean idealized data sets, struggle a bit when dealing with the messy world of real data. If you’re a theoretical or computational physicist, don’t fret! Theoretical and computational physicists tie physical models and computational techniques to observables, (unless you’re a String Theorist ;-P ) which is of incredible value.

This is not to denegrate computer scientists and mathematicians, of course. This is just to say, as a physicist, you have unique experience that mathematicians and computer scientists don’t, and DO make potential employers aware of this. The other fields that have a focus on data and observables in this way are statisticans, bioinformaticians, and some engineers. In general, though, there are fewer statisticians and engineers applying to data science positions for a variety of reasons, and bioinformaticians tend to focus on the life sciences/health care area.  In addition, as a physicist you’ve had to teach yourself a wide variety of hard (programming, electronics, statistics, you name it) and soft skills (recruiting, paper writing, conference presentations, convening working groups, making progress on your thesis while only talking with your graduate advisor once a year) that are valuable in industry.

Lesson 3: Fill in gaps in skills and vocabulary

Physicists make great data scientists. There will inevitably be, however, gaps in skills and nomenclature that arise since there are software technologies and vocabulary which are more common in industry than in academia. To give an example, in particle physics, we often say we “fit a distribution to the data”, where as in industry they say “we perform a regression”.  There are certain software technologies that are specific to physics (ROOT, cough, cough) that aren’t directly used in the private sector, and there are common technologies in the private sector that are not typically found in the physics world. Some thoughts:

• Choose an analysis language: Python or R Sorry MATLAB/LABView/IDL/ROOT fans. You really need to have an intermediate knowledge of one of these two languages. The good news is that you only really need to know one. (As an aside – although there are other analysis packages out there such as SAS,  you really can get away with just Python or R when on the private sector market for the first time)
• Familiarize with prevelant machine learning algorithms. Machine learning can be daunting. There are countless approaches being tried all the time, and countless of libraries and packages out there. However, certain machine learning algorithms are extremely prevalent in the field, and being familiar with the common ones is very helpful. This list is not a bad start. (I don’t consider logistic regression “machine learning” in any way – but it’s a very common technique used in the private sector.) Unless you’re applying to a very machine learning-specific position, you likely can get away with these.
•  Review probability and statistics. Especially Bayesian statistics. Bayesian statistics are much more important in the private sector.
• LEARN BASIC SQL. The caps are not a typo. Basic SQL knowledge is a must. Is your dream job at Facebook? Learn advanced SQL.
• Review/Familiarize with basic computer science algorithms and data structures. How much computer sciencey stuff you are expected to know depends on the position, but defintely be familiar with basic data structures, and fundamental algorithms regarding operations such as sorting and search. You also need to have at least a fundamental knowledge of Big-O complexity.
•  Learn Git/GitHub. It’s effectively the standard version control system used in industry
• Learn basic data science vocabulary. The vocabulary can be different in the data science world. Blogs can be a good resource to familiarize yourself:
  • 21 questions every data scientist should know.
  • KD nuggets
  • Kaggle Blog
  • Open data science

Formal training can be useful to help close some of these gaps. Online training from Coursera, while not completely rigorous, is helpful. The Data Science fellowship programs Insight and The Data Incubator are geared toward closing these gaps I’ve talked about for PhDs and postdocs, and have a great track record of placing their fellows in great data science jobs. Masters degree programs in data science are starting to become more common, and might be worth looking in to if you have a BS/MS. As a PhD student or postdoc, though, I am not so sure these Masters degree programs are necessarily worth the money.

Lesson 4: Draw parallels between Academic and Industry Work

As I mentioned back in Lesson 2, your goal is to convince a company that you are intelligent, can work independently, and can solve a business problem quickly. An important step in convincing an employer that you can do this is you have to connect your academic work with the position in some way. Some specific thoughts:

• Connect your thesis, skills and academic work to the job description and domain. Especially emphasize your hands-on analysis work with real data, and/or specific computational/theoretical modeling use cases.
• Be concise and practice explaining your work to a non-physicist. I get it, it can be challenging explaining physics work to muggles. But if you spent any time doing science outreach, (guilt trip alert) you should be well-prepared here. Keep in mind you’d have the thread the needle between being
  • too technical (“We rejected the null hypothesis of neutrinos not oscillating with a p-value of…”)
  • too self-deprecating (“oh, it’s too obscure/boring to get into details…”)

Lesson 5: What kind of Company are you Looking For?

Ask yourself: What kind of company do I want to work for? What does an employer need to offer in order to empower me be the most useful contributor and collaborator, while  not harming my quality of life? When I first started my data science job search, I didn’t answer these questions clearly. I didn’t have a clear idea of exactly what kind of company I was looking for, other than I wanted to build analytical models for a company that had a decent core of ethics. When I started searching for my second data science position, I developed a list of priorties that were important for  me to have in a workplace, not only for personal quality of life but to help me be a better collaborator. Some considerations to keep in mind:

• Corporate Culture.  Is diversity valued in the workplace? How collaborative / individualistic of a company is it? Do such things as working in a modern, collaborative open office space format, complete with free lunch and Pac Man arcades matter to you? Is remote/virtual working a priority? How competitive is the general culture of the company? Do you want to help shape a coprorate culture, such as joining a start up/small company? Is there a clear path of advancement in the company?
• Benefits and Lifestyle. Is having more vacation important? Good health care? 401K benefits? Do you strictly want to work 9-5? Do you only care about having a good salary?
• Cutting-edge Technology Do you want to get your hands dirty with the latest advancements in machine learning? Big Data technologies and databases? Optimizing models and technologies for scale?
• Domain Expertise Is working in a specific industry, such as e-Commerce, healthcare, etc. important to you?

Lesson 6: Beware of Pitfalls

I won’t sugar coat it – just because there is a lot of opportunity in data science specifically and the technology sector in general, doesn’t mean there aren’t serious pitfalls you have to be careful of. Despite the oft-spoken message that organizations want to “change the world and make it a better place” through technology, organizations are still businesses looking to make money. This isn’t necessarily a bad thing, and many companies do a good job of innovating, having a positive societal impact, and making money in the process. The flip side of the same coin, however, is there are plenty of companies that don’t. Technology pervades our lives, and how we use technology both professionally and on a daily basis have real effects on others and society.  A book I highly recommend that reflects on these issues is called Weapons of Math Destruction.  Things to keep in mind as you go on the job market:

• The technology sector has a huge diversity disparity.   I’ve seen it, it needs to be fixed yesterday, and unfortunately very few companies make meaningful changes to address it. Engadget has a good article of how lack of representation of women and minorities in technology can strongly interact with entrenched company culture. IBM itself ranks 20th on DiversityInc’s  list, but IBM can still do better.  And the Engadget article doesn’t go far enough. Be vigiliant, and I encourage all job seekers to explicity discuss race, religion, socioeconomic status, gender, sexuality, and/or disability with your potential employer and how diversity informs the company’s values and culture.
• Be sensitive to Ethics. Companies need to make money, but ethics are important. A company can be ethical, help others, make the world better, and make money. It’s very possible. But not all do. If something smells fishy, pay attention because it probably is. I remember I interviewed with a prominment credit card company down I-95 in Wilmington DE, and they flat out told me how they made money was to specifically target low-income minorities with high interest rate credit cards.
• Be somewhat careful with headhunters Not all headhunters are bad. Some really  help people find full time jobs at good companies that are a good fit. Other headhunters, though, unfortunately can be like snake oil salespeople. And once you start getting out on the job market by submitting your resume and making a LinkedIn profile, you better believe you’ll be contacted by one. The conversation goes something to the effect of – “I have this GREAT opportunity for a BRILLIANT candidate like yourself to solve interesting Big Data problems for a Fortune 500 company!”  These conversations ring the following warning bells:
  • The job description is vague, and usually  just consists of technology catch phrases
  • No clear indication who the company is
  • These jobs are usually underpaying contractor positions with poor benefits that last 6-12 months.
  • Headhunters can make grand promises of how these jobs can be contract-to-hire. They can’t promise a full-time position, so don’t fall for it,
• Beware of false advertising – Data science is a catch phrase, and companies can often relabel other kinds of jobs that have nothing to do with data science as “data science” jobs to attract more applicatons.
• Beware of “Data Science Boggle”- One thing to keep in mind about Data Science is that it’s a new field. Many companies get excited about what data science can do for them. “Data-driven decision making” is a new catchphrase in the business world. A company might want to start a data science division. They get so excited, they open up a job add and fill it with every data science buzzword that they can Google.  Although generally well-intentioned, a company often doesn’t have a clear vision as to what business problems they want to solve with data science. This can be a challenging place to be, especially if this would be your first Data Science position. I’ve probably spent at least half of my work in consulting clarifying data science goals for organizations – this is an extremely common problem, even in large companies.
• Be suspicious of the interview process. The data science interview process varies per company, but usually goes something like a general “get to know you” interview, a technical interview, and an in-person interview.  The technical interview can be Industry’s version of the Qualifier. (Physicists will know what I’m talking about) My view is that the technical  interview should be “just technical enough”, not too technical/academic, not too general. 21 questions every data scientist should know is the “goldilocks” level, in my opinion. It has also become common to have some sort of take-home data science assignment to supplement or even replace the technical interview. This can be a good thing, but be suspicious of the time investment for this assignment if it’s longer than 4 hours or so.

Lesson 7: Don’t worry about the “perfect job”

My final lesson, dear physicist, is that it’s ok if  your first private sector job doesn’t check off all your boxes to start or doesn’t end up being what you thought it was. Take a deep breath. Jobs are impermanent, projects are singificantly shorter in the private sector, and turn over in technology can be quite high. Once you get used to being in a corporate enviornment (which will happen quickly) and get a better sense of what you want and how you can be most useful, it’s perfectly fine to move on to another job if things don’t exactly line up in your current one. It happens all the time – data science is a young field, and you’ll have plenty of options. It happened to me. My first industry job was at a large consulting firm, and after a litle while it made the most sense for me to move on to IBM, where I work now.

We physicists are so conditioned to playing the long game. Our PhD programs in the US can take 6+ years. Our projects seemingly last forever, with many ups and downs. And the longer we stay in physics, the less options we have. You take the postdoc you can get, and you’re on the hook for three years or more. Rinse and repeat, because one postdoc usually will not be enough to land a “permanent” academic job. If you are lucky enough to find one, you’ll have one or a couple of options. Once you take one of those options – “tenure” is the name of the game, and you aim to stay there for good. If we make the transition to the private sector, we are so conditioned to see the world from this experience. There is nothing wrong with this – just rest assured that things are signficantly  more elastic and dynamic outside the ivory tower.

The AI Pundit Part 2: IBM Watson’s Editorial on the Presidential Debates

Introduction

On Part 1 of this blog post, I analyzed Hillary Clinton’s and Donald Trump’s acceptance speeches at the National Conventions with IBM Watson’s Tone Analyzer service. For Part 2 of this post, I move on to the Presidential deabtes. Like part 1, I take the transcripts of the debates and pass them to IBM Watson. (This means only the language is scored – the “Hillary Shimmy” is sadly left out of the analysis)

The First Presidential Debate

The above two bar graphs show IBM Watson’s Tone Analyzer’s emotion and language scores , respectively, for Hillary Clinton and Donald Trump from the first presidential debate. Pervailing opinion shows that Hillary Clinton handidly won the first debate, but at first glance, IBM Watson scores the language of the two candidates quite similarly. In particular, the emotion scores are dominated by anger and fear, which is similar to the kind of language that dominated the acceptance speeches of both candidates. What specific statements are behind these scores?

Donald Trump’s anger is primarily driven by Hillary Clinton, while Hillary Clinton’s anger is primarily driven by policy.

The Washington Post declared Donald Trump a big loser in the first debate, in part to the Post’s observation that “Trump seemed to resort to his worst instincts — interrupting Clinton and shouting “not true” while she spoke, with very little backup for those claims. Not a good night.”  IBM Watson would agree with this observation, in the sense that Donald Trump’s anger score is driven largely by attacks on Hillary Clinton and his interruptions. Here’s an example exchange that the Tone Analyzer scores high on anger:

 TRUMP: “Why are you just thinking about these solutions right now? For 30 years, you’ve been doing it, and now you’re just starting to think of solutions.”

CLINTON: “Well, actually…”

TRUMP: “I will bring — excuse me. I will bring back jobs. You can’t bring back jobs.”

Hillary Clinton’s angry language, by contrast, is driven mostly by policy and only indirectly by Trump:

“ We had the worst financial crisis, the Great Recession, the worst since the 1930s. That was in large part because of tax policies that slashed taxes on the wealthy, failed to invest in the middle class, took their eyes off of Wall Street, and created a perfect storm” – Hillary Clinton

Hillary doesn’t directly reference Donald Trump often in the first debate, and therefore attacks on Donald Trump have only a minor impact on her anger score, in stark contrast to her acceptance speech at the Democratic National  Convention.

Hillary Clinton scores higher on Analytical language than Donald Trump, while being percieved as more tentative.

The Atlantic declared Hillary Clinton the clear winner of the first Presidential debate, in large part because “Clinton delivered a series of detailed answers on subjects ranging from race to the Middle East to tax policy.” Indeed, IBM Watson scores Hillary Clinton significantly higher than Donald Trump on “Analytical language”. Statements such as:

“Independent experts have looked at what I’ve proposed and looked at what Donald’s proposed, and basically they’ve said this, that if his tax plan, which would blow up the debt by over $5 trillion and would in some instances disadvantage middle-class families compared to the wealthy, were to go into effect, we would lose 3.5 million jobs and maybe have another recession.” – Hillary Clinton

drive Hillary’s higher Analytical score, and IBM Watson concludes that Hillary Clinton used a more analytical language style than Donald Trump. Many media outlets interpret the precise and detailed nature Hillary’s answers as more substantive.

IBM Watson scores Hillary Clinton as somewhat more tentative than Donald Trump. This is driven by statements such as

“I hope that I will be able to earn your vote on November 8th.” – Hillary Clinton

“I think we come at it from somewhat different perspectives.” – Hillary Clinton

The other major reason the Alantic gave for Hillary winning the first Presidential debate is that “Clinton keeps her cool”, and many of her responses that the Atlantic describes as “if not always inspiring, it was professional and careful” are being scored by IBM Watson as “tentative.”

Hillary Clinton is afraid of the wrong economic policy choice, and Donald Trump is afraid of foreign threats stealing from America

Although the IBM Watson Tone Analyzer scores national security statements high in “fear”, Hillary Clinton’s  statements on economic policy is driving her fear score:

“So we’re now on the precipice of having a potentially much better economy, but the last thing we need to do is to go back to the policies that failed us in the first place” – Hillary Clinton

The same theme of policy, particularly economic policy, is behind Hillary Clinton’s high anger and fear scores. Depending on how Hillary’s specific phrasing, a particular statement could score high on anger, fear, or even both at the same time, as is the case in the first Presidential debate.

“The strongest moments of the evening for Trump came right out of the box, as he assailed Clinton for her support for free-trade deals that have driven American jobs overseas and left lower- and middle-class Americans hurting” – The National Review commented – and IBM Watson clearly picked up on the isolationist economic message that Donald Trump expressed early in the debate.  These statements scored high on the fear scale:

“We have to renegotiate our trade deals, and we have to stop these countries from stealing our companies and our jobs.” – Donald Trump

The Second Presidential Debate

I have to confess – I didn’t really see any of the second presidential debate. I was busy watching the Packers hang on against the New York Giants. From what I’ve read, though, the media generally described the second Presidential debate as taking a dark turn. The Washington Post described the second presidential debate  as “taking the low road as attacks and slurs dominate”, while the New York Post declared the second presidential debate as the “nastiest presidential debate in modern history”. From the limited amount I saw of the debate, it seemed like pretty nasty stuff. The debate transcript itself reads like some nasty rough draft of a VEEP episode that the writers cut from the show. I had IBM Watson analyze the transcript without actually having the context of seeing the debate. How did it score?

It seems like IBM Watson would agree with the general media take, in the sense that the “Anger” score far and away dominates the other emotion scores. As before, however, you need to dig in to what’s driving these scores to get some context:

Both Hillary Clinton and Donald Trump expressed anger over each other and in responding to personal attacks. Donald Trump additionally expressed anger at the moderators, Bill Clinton, and ISIS.

The New York Times observed that Donald Trump was “tense at first, and then increasingly angry as he grew more comfortable on the attack”.  IBM Watson picked up Hillary Clinton as the clear subject of many Donald Trump’s angry attacks. However, Donald Trump also scored high in anger when responding to attacks from Hillary Clinton, as well as responding to the moderators,  (Anderson Cooper and Martha Raditz), attacking Bill Clinton, and ISIS in general:

“That was locker room talk…. And certainly I’m not proud of it, but that was something that happened. If you look at Bill Clinton, far worse, mine are words, his was action. This is what he has done to women. There’s never been anybody in the history of politics in this nation that’s been so abusive to women, so you can say any way you want to say it, but Bill Clinton was abusive to women” – Donald Trump

” I will knock the hell out of ISIS. We’re going to defeat ISIS. ISIS happened a number of years ago in a vacuum left because of bad judgment” – Donald Trump

“And I will tell you that when Hillary brings up a point like that and talks about words that I said 11 years ago, I think it’s disgraceful and I think she should be ashamed of herself”  – Donald Trump

“I’d like to know, Anderson, why aren’t you bringing up the emails? I’d like to know. Why aren’t you bringing up the emails?”  – Donald Trump

I find it remkarable that not only does IBM Watson’s Tone Analyzer scores Donald Trump’s first quote above high in anger, but also the New York Times uses the same quote as evidence for Trump’s harsh attacks. He appears to be angry about… well… everything. I was surprised to read of how much Trump went after Bill Clinton, especially as a deflection tactic when confronted about the Access Hollywood tape.

The same New York Times article mentions that Hillary Clinton “broadened her indictment of Mr. Trump beyond the 2005 recording” and that “Mrs. Clinton found herself on the defensive more than in the first debate”.  Here, again, I find that Watson agrees with the New York Times conclusion, as it scores her general attacks on Donald Trump and her responses to personal accusations high on “anger”:

“with prior Republican nominees, for president, I disagreed with them. Politics, policies, principles. But I never questioned their fitness to serve. Donald Trump is different. I said starting back in June, that he was not fit to be president and commander in chief.” – Hillary Clinton

“After a year long investigation, there is no evidence that anyone hacked the server I was using and no evidence that anyone can point to at all anyone who says otherwise has no basis.” – Hillary Clinton

This all seems to fit from the transcript I read, but we at IBM might need to build in some sort of “Ridculuousness” metric into Watson: “that was something I said about Abraham Lincoln after having seen the wonderful Steven Spielberg movie called “Lincoln.” It was a master class watching President Lincoln get the Congress to approve the 13th Amendment.”  Really, Hillary? I should bogart this move and blame some of my faux pas on  Martin Scorsese movies…

“Disasters” disugst Donald Trump

IBM Watson thinks that disasters generally disgust Donald Trump, as he used the word thirteen times during the second debate. His declarative statements on disasters register on the Tone Analyzer’s “disgust” scale”:

 “NAFTA signed by her husband is perhaps the greatest disaster trade deal in the history of the world” – Donald Trump

As the “orange-haired boy who cried wolf”, there seem to be enough factual distortions around his “disasters” that it’s hard to take is frequent disaster statements seriously.

Hillary Clinton again scores higher on Analytical language than Donald Trump, while again being percieved as more tentative.

It’s interesting to note that, identical to the first debate, Hillary Clinton is scored as using more analytical language than Donald Trump, while also being percieved as tentative:

“But if [anything] else, the 170 million of of us who got insurance through our employees, got big benefits. Number one, insurance companies can’t deny you coverage because of a preexisting condition. Number two, no lifetime limits, which is a big deal in you have serious health problems. Number three, women can’t be charged more than men for health insurance, which is the way it used to be.” – Hillary Clinton

Whether this means that Hillary Clinton is more intelligent than Donald Trump or not, she certaintly speaks more coherently and forms longer sentences.