I read and Consumed™️ a number of things the first half of 2023. Let me detail some of them:
- Braiding Sweetgrass by Robin Wall Kimmerer helped me reconcile disillusionment and frustration I had with academic science, and to help me appreciate different philosophies of thought.
- The Magicians by Lev Grossman is a grounded examination of the fantasy genre within a well-fleshed fantasy world. If "Houlden Caulfield seeks Narnia but goes to Hogwarts" sounds interesting, then check this out.
- This is How You Lose the Time War by Amal El-Mohtar and Max Gladstone is a short story about star-crossed lovers from two advanced, warring, timeline-hopping scifi empires. I didn't like it that much, but it was 200 pages and fun to read.
- The Before Me podcast series is a well-constructed narrative built around interviews with the narrators mother, who arrived to America fleeing the Khmer Rouge in Cambodia. It's worth a listen.
How "Braiding Sweetgrass" helped me come to terms with limitations of science as a philosophy and practice
TLDR: Science^TM is good, but spending time as a scientist showed me the cracks. disillusionment is hard to reconcile. If you can relate, I highly recommend reading Braiding Sweetgrass.
During my years as a PhD student, I found myself increasingly frustrated with academia. I felt that researchers career incentives were not aligned with the pursuit of knowledge, and at least within machine learning, I felt that trends and assumptions go unchallenged (to the detriment of the field.)
I think of science as the best-effort cultural practice of knowledgemaking. I'd say science refers to, all at once,
- The act of research, or observing the world to learn new knowledge,
- A protocol for articulating that knowledge to others,
- A cultural practice for maintaining that knowledge between generations,
- and a catch-all term for the knowledge that was gained through science.
This is why doing research, writing journal articles, public outreach, and teaching classes are all jobs of A Scientist.
I understood that the Academic Science to which I belonged follows a single (flawed) tradition of knowledge ("the scientific method") which can be traced back only a few thousand years. But I also knew that humanity worldwide had developed other ways of gaining and communicating knowledge, in the form of songs and stories, and much later, writing. That is to say, Academic Science was not the only science.
The only other major lineages of Science that I knew of (and took seriously) were Indigenous practices. I vaguely knew that a huge amount of our medicinal and agricultural knowledge could be traced to Indigenous people. More recently, Indigenous knowledge was able to supercede common scientific understanding, such as the practice of controlled burns.
I expressed this all to a friend (who was also a PhD student in machine learning) and she recommended Robin Wall Kimmerer's 2013 "Braiding Sweetgrass: Indigenous Wisdom, Scientific Knowledge, and the Teachings of Plants". Part autobiography, part philosophy, part narrative, this book explores academic and indigenous sciences, humanities, and politics.
An aside-- On science and learning
I was a machine learning researcher. I spent a lot of time thinking about how we write algorithms which identify correlations and build useful models. We would call this "learning".
The job of figuring out how to usefully model the world is also the job of science! That is to say, machine learning can be thought of as a scientific method. In this sense, the field of machine learning is a scientific field about a scientific method.
There are some good meta-papers analyzing the field of machine learning through the selfsame lens of machine learning. One interesting problem is that of overfitting due to test-set re-use (see "A Meta-Analysis of Overfitting in Machine Learning" by Roelofs, Fridovich-Keil et. al, 2019). Thinking of science as machine learning lets us frame our field as overfitting, and so lets us better identify and articulate problems in our field.
More generally, I think machine learning should be thought of as a humanities. After all, there's no way to be a machine learning researcher without constantly taking deep epistemological shrugs. No free lunch? Solomonoff complexity? ¯\_(ツ)_/¯
Other books, short books, and podcasts
I listened to an audiobook of The Magicians by Lev Grossman at the recommendation of a friend. This book series is excellent. Often described as "Holden Caulfield goes to Hogwarts", it's a grounded and critical take on the fantasy genre. It subverts without being overly cynical and it grounds the story without taking the fun out. It's almost like science-fiction in some parts, where the author uncompromisingly works to make things consistent.
As much as The Magicians is critical of fantasy, it is a fantastic fantasy series on its own. The author is a grand storyteller, especially at setting scenes. Paragraphs are just descriptive enough to make the world feel tactile and real. Just be prepared to feel disdain for the main character. I ended up listening to the other two audiobooks in the series. I can wholeheartedly recommend the first one at the least.
~~I was also gifted a short book, "The Little Book of Satanism" by La Carmina, by a friend. It details the history of Satan as a mythological and literary figure, recent "Satanist" movements, as well as the recent history of "The Satanic Temple".
~~I did appreciate learning more about "The Satanic Temple", and how it is a distinct, different religion than the LaVeyan Church of Satan. In short, the TST is more lib-left than the CoS's lib-right, but both movements were born from contrarianness. I'm a big believer that to be cringe is to be free, so I can appreciate Satanist's inherent disregard for any worry about the label "cringe".
~~I am writing this in 2023, a time where the American fascist movement is also entirely Christonationalist, so I see a lot of value in TST's activism and existence. The ironic thing is that TST's tenets line up pretty identically with the virtues Jesus Christ supposedly espoused. (And, given Jesus did fulfill the role and act of "satan" in the original Hebrew sense, and given Jesus was killed for opposing the state and existing religion, perhaps we could say Jesus was one of the first Satanists?)
I've also read the collection of short stories "Squirrel Seeking Chipmunk" by David Sedaris. I appreciate how numerous and short they are, and they're generally high quality, but none were memorable enough to make this a "must read". It makes me appreciate how accessible a printing press is, because I imagine this would have been a best-seller if it were written in the 1400s.
I also enjoyed "This Is How You Lose the Time War" by Amal El-Mohtar and Max Gladstone. As a book, I found it hard to follow and hard to appreciate, but enjoyable to read in short bursts. I'm usually a big fan of narratives which get creative with time, such as Homestuck or Tenet. I think it's difficult to craft a good narrative which proposes an interesting time mechanism (say, with two characters experiencing time in different directions). When crafted well, the literary ironies we learned in school can lap together like waves rolling over waves, as characters learn (or unlearn) key facts. The audience's view of the world is the one true timeline.
The science fiction in TIHYLTTW was simply too obtuse for me. The two main characters, Red and Blue, are soldiers in two different opposing factions, each with their own high-fantasy scifi world, each alone too large to fit into the occasional hint and glimpse. The end result are too many scattered pieces of information to allow me to piece together a coherent picture.
I also listened to several podcasts. I recommend the "Before Me" podcast series. The narrator interviews her mother, a Cambodian-American refugee who arrived in the United States fleeing the Khmer Rouge in Cambodia. It's hard not to be constantly aware that the living memory of historical events are dwindling, dying, and disappearing with each passing year. This podcast is an effort to preserve some of that memory.
Other good papers I read or skimmed
When I read papers as a PhD, I was actively interested in considering how I would improve or involve this paper. Without the motivation of imminent implementation, I've found myself absorbing little from most papers.
These were ones that I read with enough interest to take something away.
- "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜" by Bender, Gebru et al (2021) is a paper worth reading, but not worth surmising here.
- "Training language models to follow instructions with human feedback" by OpenAI, 2022 is a primer on RLHF.
- Aside: My very first completed learning project was a senior design NLP project where my team and I converted BDD descriptions to executable test-cases. In a last-ditch effort to fine tune the model the night before demo day, I made a simple REPL that let us train the model by hand, epoch-by-epoch. We took shifts training the data. It made the demo a bit better!
- "Efficient Transformers: A Survey" by Yi Tay et al, 2022-- this is a paper I skimmed when I was interested in implementing a transformer model for a Homestuck bot. I ended up not creating the bot.
- I also read a bit on diffusion models, as I had not read about them at all during my time as a PhD student, so I took a quick review.
- "Diffusion Models in Vision: A Survey" by Croitoru, Hondru, Ionescu, Shah 2023. I skimmed this when I wanted to create the aforementioned Homestuck bot, but then relented and decided to read the original:
- "High-Resolution Image Synthesis with Latent Diffusion Models" by Rombach, Blattmann, et. al 2022. This was another one that I only skimmed, coming to realize that I didn't care about that Homestuck shitpost bot so much.