#73: Merlin for sound-based bird identification, CCAI's big climate grant, and finger spelling with AI

Hey everyone, welcome to Dynamically Typed #73, a classic links issue. Today for productized AI I’ve got Merlin, an app for identifying birds by their calls; and an Oxygen Digital panel I joined on the future of AI-assisted coding. For ML research I found Stanford’s new Mistral language model training framework. For climate AI, I’m excited to share a thesis on datacenters and the energy grid that I got to co-supervise; also, Climate Change AI is launching a big new research grant. Finally, for cool stuff there’s fingerspelling.xyz, a web experience to help you learn to spell in American Sign Language.

In other news, this week I’m at an offsite with Dexter. We’re spending the week in a chalet in France, working most days and going for hikes and climbing on other days. Have I mentioned that we’re hiring?

Productized Artificial Intelligence 🔌

• 🦜 Merlin, an app by the Cornell Lab of Ornithology, identifies birds based on their songs and calls. The app’s Sound ID feature currently supports 450+ birds in the US and Canada. It works by visualizing an audio recording of a bird’s song or call as a spectrogram — where the x-axis is time, the y-axis is frequency, and each point’s brightness represents decibels, so it’s essentially a monochromatic image — and then classifies it using computer vision. Because the vision model runs on-device, Merlin also works without a cellular connection. Beyond Sound ID, the app also has a Photo ID feature that directly classifies photos of birds, and one that guesses which bird you saw based on three simple questions (how big it was, what its main colors were, and what it was doing) — that last one is probably just some clever filtering though, not an AI model. Links: App Store, Google Play.
• 💬 I joined Oxygen Digital for their AI Series panel on AI-assisted coding (YouTube link). To our own surprise, we filled the whole 90-minute slot — it was a lot of fun! We discussed GitHub Copilot and OpenAI Codex, and lots more about the future of professional software engineering as tools like this become a part of every IDE. As I also wrote in Towards talking to computers with Codex, I’m most excited about how these code generation AI models will unlock the power of working with APIs to people who don’t know how to write code.

More productized AI: stories (22), links (71)

Machine Learning Research 🎛

• ⚡️ Stanford HAI’s new Center for Research on Foundation Models (“foundation models” is their name for large self-supervised models like GPT-3 and CLIP) has open-sourced Mistral, a “framework for transparent and accessible large-scale language model training.” It’s on GitHub at stanford-crfm/mistral.

More ML research: stories (14), links (81)

Artificial Intelligence for the Climate Crisis 🌍

• 🔌 Earlier this year, I wrote about the climate opportunity of gargantuan AI models: like many other types of workloads, the offline training of AI models could be scheduled dynamically based on electricity market signals of over- or under-supply. This way, these power-hungry datacenters can provide demand-side response for balancing the power grid — an increasingly important problem with the growth of renewables — so that we’ll be less dependent on supply-side balancing from coal and gas plants. I’m excited to share that over the past few months, I co-supervised Hongyu He’s BSc thesis project at the VU Amsterdam’s @Large Research group on exactly this topic. In his 158-page (!) thesis, Hongyu extended OpenDC, a datacenter simulator I helped develop during my own BSc, to incorporate electricity price signals from Dexter’s asset optimization product into the virtual datacenter’s workload scheduler. He then simulated different ways for datacenters to participate directly in power markets, and found that these could be profitable (and therefore helpful in balancing the grid). Hongyu’s full thesis is on arXiv: How Can Datacenters Join the Smart Grid to Address the Climate Crisis? This simulation work is an important step for convincing stakeholders to pilot and deploy this in the real world; I hope to have more to share on that in the future.
• 💰 Climate Change AI is launching an Innovation Grants program that “will fund year-long research projects at the intersection of climate change and machine learning for up to USD 150K per project, for a total of USD 1.8M.” Their areas of interest include AI approaches to: mitigation; adaptation; climate science; low-carbon technology research and development; behavioral and social science related to climate; and AI governance in the context of climate change. The submission deadline is October 15th.

More climate AI: stories (6), links (25)

Cool Things ✨

Yours truly learning to spell “Love” on Fingerspelling

• 🖐 Fingerspelling.xyz is a web experience that helps you learn to spell in American Sign Language. It uses an on-device hand tracking model to both visualize the position of your fingers and judge whether you’re making the correct sign, and then walks you through spelling different words. The site is super well-polished: it’s fast and it even highlights which of your fingers are in the right and wrong places in real time. Definitely the must-click link from today’s DT. (Only works in Chrome, Edge or Firefox; not Safari.)

More cool things: stories (5), links (25)

Thanks for reading! As usual, you can let me know what you thought of today’s issue using the buttons below or by replying to this email. If you’re new here, check out the Dynamically Typed archives or subscribe below to get a new issues in your inbox every second Sunday.

If you enjoyed this issue of Dynamically Typed, why not forward it to a friend? It’s by far the best thing you can do to help me grow this newsletter. 🏞