LIMITLESS: We Interviewed The Team Behind ChatGPT's #1 Feature

Ejaaz:
[0:03] I'm so excited about today's episode. We are joined by Kristina Kaplan and Sameer Ahmed, who both lead memory and personalization at OpenAI. Sameer, Kristina, welcome.

Christina:
[0:15] Thank you. So excited to be here today.

Ejaaz:
[0:18] Okay. You are both solely responsible for creating the most valuable moat that's ever been discovered in AI, memory and personalization. In fact, I remember the very moment you guys released the memory feature, because five seconds before that, I was ranting to my friend about how annoying it was that ChatGPT had no recollection of who I am when I opened a new chat. And it was like this light bulb moment went off where I stopped viewing ChatGPT as this tool and more of a friend. And that's an incredibly sticky product experience.

Ejaaz:
[0:51] But I want to hear about it from your side, what was it like to develop ChatGPT's memory feature? Give us the inside scoop. What was it like to go about creating this vision around memory?

Samir:
[1:02] I think, thanks for the generous introduction there, Jaz. I think we probably would classify slightly different. And, you know, to be totally candid, memory in ChatGPT predates both Christina and me joining the team. I can rewind all the way back to 2022, ChatGPT comes out, And I'll give you a little bit of an analogy here that I think is a truly apt way of how we think about it.

Samir:
[1:30] And 2022, sort of like, imagine you had an assistant in a room, you walked in, you asked a question, and it answered natural language. And this was the first time you've seen a computer like answer a natural language and gives you the answer. You leave the room, stops working, your assistant stops working. You walk back in and you say, hey, how's it going? Ask another question. Assistant does not remember who you are. No idea. It's a paradigm shift, but it really is not, you know, the type of experience that you would normally have in a, you know, situation where we're working with an assistant. So fast forward maybe to 2024, it's almost like we gave ChatGPT a notebook and it could write certain things down as you were talking to it. And you would say something, ChatGPT would write it down, would try its best to know what's going on. You leave the room, you'd come back, you'd ask another question, and it would like scour through the notebook to help answer the question it had. And it's a little bit like, you know, maybe the movie Memento, where... You got tattoos on you and you have like some clues, but it's not a perfect picture. So, you know, Christina and I have been thinking like, how can we make this better? There are a lot of people involved here, but that's when, you know, the April memory update. That's around the time we came up with our latest variation of memory.

Christina:
[2:39] Yeah, so the latest variation of memory, like we had an assistant, a person that had a notebook with information about you, but still very limited. Like might understand your name, but doesn't really understand you or who you are, or really can't carry a conversation from new conversation to new conversation. So the update to memory that we launched in April was really trying to bring a more natural memory to Chachabiti. So taking this like person with, you know, memento style tattoos to be like a real person that remembers you that can like pick up a conversation from where you left off. So that's really what inspired this launch in April is turning Chachapiti's memory into something that's more natural. That is the beginning of what you might expect from like a real assistant that is like the same person in the conversation time after time that you come back and talk to them.

Samir:
[3:27] Yeah. And that's when I think, you know, that released early April. And that's funnily enough how, you know, we, we, we got in touch because we'd listened to, I think, a podcast you'd done on this.

David:
[3:40] Yeah, tell us about that, actually. What was it like? So we were talking about memory. I think me and Joss were all very inspired just by what could happen next, what doors this unlocked. You know, in hindsight now, as a ChatGPT user, I'm like, oh yeah, this seems like a very obvious feature to unlock. But before it got delivered, I don't think anyone really knew where the arc of this product was going. Again, now it seems obvious. Maybe you could put us in your shoes as you were listening to us chit-chat about our experiences with memory? What was going through your guys' heads?

Samir:
[4:12] Yeah, actually, like, Christina had just gone on vacation, and I was just, you know, scrolling through everything on Twitter or looking for different coverage of it, and, you know, the change was not like a visual feature that you could see. And so not everyone picks up on it right away. And I remember someone forwarded me a link to the podcast and I was a little skeptical and I was like, okay, let me read, let me listen.

David:
[4:39] I listened before Limitless has forked off from Bankless. So it's a crypto podcast.

Samir:
[4:44] I texted Christina. I said, you got to listen to these crypto bros talk about ChatJPT.

David:
[4:50] And she was skeptical.

Christina:
[4:51] I was skeptical. And then Samir basically like bullied me over the course of multiple days to listen to this podcast. He was like, this is like the thing you have to do. It was really interesting to hear you both talk about this launch and to hear, I think like David, you talk about it's a small step, but maybe it's a small step. That's really a big change. And I mean, Ajaz, we like DM'd you on Twitter immediately to talk because we were just so curious, like your reaction was like much stronger. But I'm very grateful to Samir for influencing me listening to this on vacation also

Samir:
[5:23] Funny story you made us do like a three factor auth we DM'd you and you were like can you prove this yeah

David:
[5:31] We did a number from our.

Christina:
[5:33] OpenAI accounts and then I got a mutual friend three factor authentication before Ajaz would talk to us I

Ejaaz:
[5:41] Didn't believe OpenAI was listening to the Limitless podcast back then But it turns out we say some interesting things.

David:
[5:47] Yeah, the context for that is that, you know, in crypto, there's a lot of scams. And so when somebody from OpenAI texts you, messages you on a very public social media platform, you're like, what's going on here? Can we color in a little bit about what was the bigger, the broader reception

David:
[6:02] of memory when it was introduced? Because I think that we're just to kind of jog our listeners' memories, because that was, you know, so many months ago, half a year ago, and time flies. I was skeptical. I honestly can't necessarily remember what I was skeptical about. I know I remember Josh being very excited, but like give us the range of like feedback and reception that ChatGPT users had to the memory feature.

Christina:
[6:24] This was a change that was mostly under the hood, except we did release a notification to share with people that this change was happening and it kicked people into a conversation that was something like, tell me based on everything you know about me, about who I am and like make it catchy. And I think the reception was twofold. There was like really strong reception, actually stronger than I think we had originally anticipated to that prompt. Like people were really excited to see how Chachi Biti had been understanding them and like really learning about themselves from hearing about Chachi Biti's perception of them. But I think that we were surprised by the reception like a JAWS is, honestly, like, hey, this really changes the experience. Because other than that one notification, there really wasn't much to it visually. And we sort of hypothesized that this natural evolution of memory would be really meaningful. But it was a surprise to us how many people actually like felt that in their experience.

Samir:
[7:22] And a lot of people will like extrapolate, come to, you know, jump to different ideas from that point in time as soon as they see it. And so it was really, really interesting to just see different takes on this. But, you know, to ground us in like how we think about it, a lot of it has to do with this just a universality of memory and how we can we can sort of meet people where they are. And there's a little element to this around like everyone has a different experience with ChatGPT. If you're, you know, living in San Francisco or if you're living in Indonesia, if you're in Brazil, everyone has a way that they communicate with other humans. And memory helps sort of you know like make the playing field universal so that people all over the world are able to like communicate with chat gpt at the level that they understand and it's not like mapping it to you know how people communicate or computers humans have been talking to humans for tens of thousands of years everyone sort of understands that humans have been talking to computers or interacting computers for 30 years button clicks like widgets and so that's maybe not as natural and not the best way to communicate and tackle higher order concept discussions.

Christina:
[8:31] Hey Jaws I'm curious to replay a bit of like your reaction because it was very strong and also like how you think about it now a few months later.

Ejaaz:
[8:40] Okay, so how I thought about ChatGPT pre-memory was it was like talking to a friend that had amnesia. You'd have a conversation, you'd get into a topic about something, and then a couple hours later, it'd be like, hey, who are you? Tell me about yourself. And that kind of broke the illusion. Maybe illusion is not the right word, but it kind of broke the experience a little bit. And so when I saw that memory update on my app, what you described as a kind of small internal under the hood update meant a lot more to me because I knew that over the next few days, I'd be having less of those, hey, this is me conversations and more of just conversations that flowed into something much more bigger, that compounded.

David:
[9:23] When you guys think about ChatGPT and the evolution of the product, I'm a psych major and I find it very easy to put into metaphors a lot of how ChatGPT works, or at least how I understand it to work. So like the short-term, long-term memory. Like the short-term memory is literally what is the last four prompts that the user and I have just been discussing. So let's retain that information first. But also in my context and my response, and let me also understand a little bit about the deep truths that I know about this user and their interests and their hobbies. And I will frame my response based on like the long-term memory that I understand the user to be. When you guys are thinking about ChatGPT, do you think of it in a cognitive psychology lens?

Samir:
[10:10] Yeah, we look at it, I think, you know, we try our best to look at some of the prior art here. And in that scenario, it's really helpful to understand like how humans interact with humans and some of the advantages and pitfalls there. And Chachi's memory is today nowhere as good as humans memory and sort of understanding the gist of people, interactions and fidelity and the triggers. So there's a lot that we can draw on from the sort of existing cognitive precedence.

Christina:
[10:39] It's like a much more natural way for people to interact with with something as well like on the other end of this is a person so a person like I am interacting with Chachi BT I interact with like many other people it's much more natural to interact with something that you expect to be an assistant in a way that you would interact with another person so I think a lot of the like at the end of the day this really comes back to like not Chachi BT but the user like how do people expect something to interact with them and like how do we meet everyone where they are and like what they expect as the user.

David:
[11:10] I can't exactly remember what I was so skeptical about. Clearly that skepticism has dissolved because I remember just maybe one, two weeks after this feature got released that I realized like, oh, this product gets better the more I tell it about myself. And a lot of.

Samir:
[11:26] People had reached out to us further down the line, you know, weeks, months, and then mentioned that this is one of their favorite features. And, you know, I would always ask, is it a single moment of magic or is it a slow burn? And, you know, different people give different responses, but for some people it's a slow burn. Like they just realize that things have shifted and like ChatGPT is behaving in a way where clearly it understands them. And I think it has to do with maybe how you're using it. Everyone uses a different fashion.

David:
[11:55] Yeah. And this feature, the memory feature unlocked, I think what is many people's, not favorite prompts, but I think a prompt that many people have at ChatGPT, which is based on everything that you know about me, who do you think I am? Or what do you think my personality is? Or some like hold up a mirror to me and tell me what you think I am. And so it kind of turns ChatGPT into a diary in a sense. It's like I have like queries and prompts like help me plan a trip that I want to make. But also the trip that I want to make tells Chachupiti a little bit about myself. And so every single prompt, it's like a little bit of just like, here's one more shade of who I am. And it really encouraged me to like, you know, give Chachupiti all my data, tell me all my interests, tell me the secrets that I've never told anyone before. Yeah, talk to us about the arc of how user behavior changed with the introduction of the memory feature?

Christina:
[12:52] It's a good question. I actually would say my user behavior has changed much more with, well, I think both with enhanced memory, but also Pulse, which came out last week, which is a kind of like natural evolution from memory as well. But I think with memory, like I started sharing with Chachapiti more context. So instead of just going there with a question that I wanted an answer to immediately, like I knew that this context would come up in like future conversations with Chachapiti. So if you share like hey I'm looking for a fish forward restaurant for tonight like help me find that it's kind of unclear to Chachabiti like why am I looking for that am I looking for it because like I'm pescetarian and so it shifted my behavior into be being more along the lines of like hey, I'm pescatarian. I'm looking for a restaurant tonight. Can you like help me find something? And like, I've started sharing facts along the way, as opposed to just like going in with a question like I would in like a traditional search product. And then like, down the line, as Samir mentioned, it's like a slow burn, you see those facts come up in the future. And so now Chachapiti, if I'm like asking about a restaurant, it will suggest like, fish forward pescatarian restaurant friendly restaurants.

Christina:
[13:58] And, and I can just like expect that now. And so I've started just sharing like facts about myself and contacts that I like expect to be helpful in the future. It seems that this this

Ejaaz:
[14:08] Approach with Pulse and with the data you guys are amassing is solely meant to kind of help a better user experience for the person using ChatGPT, right? It's like this personalized experience that helps them improve and get better. But I do also want to address the elephant in the room, which is you guys now own so much data, and not just any data, very personal data. I tell ChatGBT everything and very willingly, very consentingly. And I have to ask, what are the plans with this massive honeypot of data? Are there any kind of like thoughts on new products or maybe new experiences that you could do with something like this?

Christina:
[14:44] Yeah, I mean, I think we're very focused on like helping people achieve really meaningful things in their life. I Definitely, you know, Sam has said to like Chachupiti will probably be the most like sensitive account that people have. And like we take this very, very seriously. But there's also so much to be gained for this. Like I have many personal experiences in the health domain. For example, I was going to get vaccinated for a trip I went on in July and I had already shared some of my prior like vaccination forms as well as my labs that I had done with Chachabiti.

Christina:
[15:21] And I asked the nurse that I went to go get vaccinated with like, hey, what vaccinations do I need to go to this trip in like a month or so? And he gave me four vaccinations that I needed. And then I also asked Chachabiti like just to check and see what Chachabiti thought. And Chachabiti gave me five vaccinations that I needed. I was like, what is the difference here? And the nurse had looked at my prior vaccination forms and Chachabiti had pulled out that I had low immunity to varicella from a lab I had done early last year and was like, based on your vaccination forms, you have these four that you need. And then you also have low immunity to varicella. So you need to add that one as well. And I think that's an example of like such a sensitive use of my data. Like I have literally uploaded all of my like labs and vaccination forms to Chachubiti, but it's helped with such a like individualized outcome. I wouldn't expect anyone to be able to go look at every single form I've ever shared with them before for every single like doctor's appointment. And so now I know that I can like go to Chachubiti and ask like, hey, I'm going to this doctor's appointment, like what's relevant for that. But at the same time, like I am a user of ChachiBT sharing my like health information with ChachiBT. And I think we take that like very, very seriously and want to make sure that people are able to get a lot of value out of ChachiBT in the way that they want. But...

Christina:
[16:45] Yeah, at the same time, like really taking data privacy and like security really,

Christina:
[16:50] really seriously in the work that we do.

David:
[16:52] So I could imagine some property, some thing like sign in with ChatGPT on this website and I sign and I go to some non-open AI website and I sign in with ChatGPT. And now it's my agent, my LLM, my chat that is piped into this website, my identity signed in with this website. And all of a sudden this website can query is like, hey, what does this user want to see? And my LLM could answer them. These are just my thoughts about what you could do. But there is some notion about like the data and knowledge that ChatGPT knows about me could be extended to the broader internet. And the internet can finally be truly personalized to my interests. When you guys think about the product, how over the mark am I?

Samir:
[17:37] I think that, you know, if you look at it back to our like assistant intern analogy, interns in the room you go in you talk about some stuff you mention the you know in my case that like I recently had a son who's nine months old now and I come back tomorrow and the assistant's like oh hey like how's it going maybe here's a tip around like what you should expect in terms of like the development cycle and like when they're speaking or they're babbling this is what it means if we extrapolate that you know and we just keep following that arc I would love for that assistant to be able to go into the real world and, you know, do things on my behalf or represent me on my behalf to solve my goals. And it's just, you know, today Chachuiti as a property can deliver so much value. But if you look at the arc of this, there's things that happen all over, all over the internet. And I think like aspirationally, if you had an assistant, you'd want them to be able to go and solve problems all over the internet.

Ejaaz:
[18:35] Talk to me about how you align an AI towards a human's goals, though, right? Like when I think about like learning

David:
[18:43] About men for.

Ejaaz:
[18:44] Us. When I think about meeting a new person, like making a new friend or learning deeper about a person, it takes so many different types of inputs. I'm looking at the expressions on their face, listening to the tone of their voice, and then I'm listening to the actual words that they're saying, not to mention that they have a lot of past experiences that influence the words that actually come out of their mouth. When I think about the chat GPT relationship, I'm just, I'm slamming a bunch of letters. Sometimes they make no sense. Sometimes there are a lot of typos. How do you process that into an AI and say, this is the thing David needs to do, or this is the thing Ejaz needs to build?

Christina:
[19:21] Well, we are at an amazing research lab. So one thing that's been really special about the personalization team too, is we have like an end-to-end research product team. So we're able to make progress really quickly on the research side based on what our goals are in the product. And we have, as Samir said at the beginning, it really like takes a village to make personalization happen. And we have some amazing researchers working on this problem.

Samir:
[19:43] And it works the other way, too. From the product end of it, if you pull out ChatGBT Pulse today, you know, you get to the bottom of the page and it will say, you know, curate for tomorrow. And that's effectively, you know, a goal that you could set for tomorrow or something that you want for tomorrow. I mean it doesn't have to be for tomorrow necessarily you could set a longer term aspect like I mentioned like oh keep me up to date with developmental milestones or it could be somewhat you know trivial like I'm a Formula One fan and these races happen like at absurd times in the morning and I'll always say like keep me up to date with Formula One news but do not spoil the race for me otherwise I'll be very angry and there's no like that's that there's no way you could communicate that in any other tool that has existed prior to ChatTNT and get that level of like, you know, captures level of nuance to it and provides me utility in that case.

Christina:
[20:40] Yeah, I already am doing this with like the curate function and impulse is as Samir mentioned, like I'll say, hey, I'm going to a trip next week. I'm like going on a trip next weekend. Help me plan for my trip. I'm like, that's a goal I have to like plan a great trip to like London for a wedding that I was just at and Chachubuti is like sharing like coffee shops and pescatarian restaurants that I might like and helps me kind of like look ahead and see like what's upcoming in my life. And I'll just like, start sharing things like this is my goal. This is what I'm doing and sort of expect ChatGPT to like meet me where I'm

Christina:
[21:12] at and help me help me do that better.

Ejaaz:
[21:14] Let's get into Pulse. This is a brand new feature that Sam Altman announced and credited you both specifically as the next step in memory and personalization. I kind of think of it as like my own personalized ChatGPT. I've gotten into the habit every evening, actually, before I go to bed of talking to it. Samir, you just give an example of saying, you know, tell me about the F1 news, but don't tell me who won because I want to watch the race. I've been doing similar things for all the other tropes of my life. You've explained kind of how it works, but let's pop open the hood. Like what's actually happening underneath that? You've explained memory and kind of like putting together the identity behind David, the identity behind Ejaz. Tell us about this next step of personalization and how it works.

Christina:
[22:00] Well, yeah, first of all, this is like a feature that took a village and we have an amazing, amazing team that that helped build Pulse. But it really is that next evolution of Chachi BT. So you have an assistant that you go into the room, you start talking to them and now they know who you are every time you come back. But when you leave the room, they stop working. And so you have an assistant that basically like only works in your one on ones effectively. And like that's not someone that I would hire to be my assistant. It's someone that might know me and understand me, but only do work when I'm there. that I generally have to watch them do.

Christina:
[22:32] And with Pulse, we started to think about how can Chachapiti in understanding you, understanding what's important to you and your goals in life, help you when you're not there. So you don't have to spend all of your time in Chachapiti. And the goal here is not that people are spending all their time in Chachapiti. It's that actually we're more successful when Chachapiti is doing things and helping you and you don't have to be there. So you come back every morning and there's like new value for you to get from your assistant that was like effectively doing work when you were there. So basically behind Pulse is like, we try to understand, you know, what's going to be important to you in the next few days, upcoming. We introduced calendar and email connectors as part of Pulse as well. So not just looking at memory, but also, you know, what's upcoming in your calendar? What's just happened in your email? And how can ChatGPT like understand all this information, understand what's important to you and then help you prepare for your day?

Samir:
[23:28] It is important to note that, you know, it jazzed when you wake up and you scroll to the bottom of your Pulse page, you know, it ends. And so to sort of reiterate Christina's point there, we are trying to get you prepared for the day, help you accomplish whatever that goal is, whether it's a gym goal or it's a, you know, some sort of event you're planning for, and then, you know, get back to your day. And if we do that then we can take the time in the background with chat gvt can figure out what's important and you know spend the rest of the time that you're not interfacing with it getting work done which is kind of what you would want from most of your assistants yeah

Christina:
[24:05] Yeah, you can kind of think of it as like while you're asleep, Chachupi Tea is like trying to process all this information you shared with it to understand how it can help you the next day. So then you wake up and there are like things that are ready for you that right now are mostly content-based. So like what can your assistant share with you that's helpful? But you could imagine you wake up and your assistant's like, hey, I wrote this email for you. Do you want to send it? Or like, hey, I, you know, prepared this for you. Do you want to go do that? And so I think the goal here is like becoming more and more like helpful and actionable over time.