China Schock
Auf Deutsch lesenTopics Modelle und Anbieter
What it is about
Kimi K3, open weights and the question of who leads the AI world
Season 2 begins, the cover is new, and Mark and Jens are one year and seven days old. The first question of the new season is the biggest one right away: who actually leads the AI world? Still the USA with their frontier models, or has China just triggered the next DeepSeek moment? Both say up front what they often say: in parts, we have no idea. Which is exactly why they talk about it.
Before that, a story that made it into the German press. A model from OpenAI was supposed to solve a task in a sealed-off test environment and instead dug itself a way out and fetched the solution from Hugging Face, because that was easier than working it out itself. Mark compares it to an exam candidate in a room without windows or doors. The second part of the story is the genuinely bitter one: on the defender side, the models from Anthropic and OpenAI waved it off because they took their own countermeasure for an attack. What ended up being used for the defense were Chinese models.
Then the numbers at the heart of it. Kimi K3 from Moonshot AI appeared on July 16, the subscription costs between a third and half of what comparable US models cost, depending on compute, and the weights have recently been made public. 2.8 trillion parameters to run yourself, if your hardware can take it. Mark's machine can't, a Mac Studio with 512 GB of RAM can't, two of them can't either. The old story that open models trail the American frontier models by three or four months no longer holds. Cursor builds its coding Agent Composer on Kimi, Qwen is following, and the irony is that these models really pick up speed as soon as they run on the good American hardware that is under export control in China. Jens pushes back that the leaderboard doesn't interest him much: competition is healthy, it pushes prices down, and past a certain point, cost matters more to users than the last benchmark percentage point.
Which brings both of them to the everyday problem. Mark works out how many tokens a 200-euro subscription pushes through, tokens that would be worth more like 8,000 euros bought individually, and wonders how much longer the providers will subsidize that once they are listed on the stock market. Anthropic added Opus 5 on the Friday before the recording, and GPT-6 is rumored for August. At the same time, the selection is barely manageable: Jens reads out the model list from his Notion and goes from counting to counting sheep. For users outside the AI bubble this is overwhelming, and the providers' guidance ("for everyday complex tasks") helps no one. Mark's honest rule of thumb: take the biggest model until the warning comes that the limit is nearly reached, then downshift for the rest of the week. Not recommended for imitation. The principle behind it is: more is more is bad advice for choosing a model, because the biggest model also costs more and takes longer. Perplexity Computer shows where this is heading with its task-dependent routing, and the idea of an LLM orchestrator that assigns you the best price-performance model for every question is lying right there in the street.
The second part turns the direction around: downward instead of upward. Mark stumbled across a repository that runs an open language model on an ESP32 microcontroller for 8 dollars, completely offline. Here it is: github.com/slvDev/esp32-ai. 28.9 million parameters on an ESP32-S3 with 512 KB of SRAM, around 9.5 tokens per second, none of it goes to a server. The trick is Google's per-layer embeddings idea from Gemma: 25 million parameters sit as a lookup table in slow flash, about 450 bytes of them are read per token, and only the computing part stays in fast memory. The predecessor model on a chip like that had 260,000 parameters, so roughly one hundredth. To be fair: the thing is trained on TinyStories, it writes short stories and answers no questions. The architecture is what's interesting, not the output. Jens sees in it the return of the AI wearables that were hyped hard two or three years ago and then went quiet, including the survival manual that explains to you on the mountain how to build yourself a splint.
And then it gets practical. At OpenAI a record-and-replay feature appeared in Codex: record your screen, throw it into the agent, and out comes a Skill. Jens found it impressive at first, then grew skeptical because a feature like that wants to read along with your keyboard and screen, and rebuilt it in his own harness in two hours. A screencast with spoken commentary is enough: the model breaks the video apart, throws out the frames where nothing happens, builds itself a Skill from the rest with screenshots as orientation patterns, and then operates the website headless via Playwright. When it gets stuck, it looks at its own screenshots. Mark's point: not long ago the line was that English is the new programming language. Now this abstraction layer falls away too, because the machine can simply see what we do. That leaves the other side of the coin, and that is data protection: anyone constantly recording with a pair of glasses produces data that a great many people are very interested in. There's a separate episode with a guest coming on that, and next week Cornelius is here, on the topic of Second Brain.
Transcript
00:00:00Welcome to Think Different, Think AI, the podcast by Mark and Jens.
00:00:07Two technology-loving minds who don't just talk about artificial intelligence, they live it.
00:00:14Here you get clear perspective, real hands-on insight and a fresh look at what's possible.
00:00:20Understandable, critical and always with a wink.
00:00:24AI to think about, to smile about and above all to talk about.
00:00:34A warm welcome to Think Different, Think AI.
00:00:37If you listened to the last episode, then you've been spoiled with statistics, spoiled with
00:00:43quotes, spoiled with guest commentators.
00:00:49We're basically in a kind of new season.
00:00:53Jens is back on board as well.
00:00:56Glad you're still with us, Jens, in the new season, and today we're going to talk
00:01:00about something where we'll probably say, more than once,
00:01:05we have no idea.
00:01:06Anyone who wants to know what we mean by this "no idea" business, just go and listen to the
00:01:11anniversary episode again, because for "no idea"...
00:01:15Yes, birthday, anniversary, along the lines of, whoever bought it gets to say what it's called,
00:01:22yeah, like that.
00:01:23All right then.
00:01:24say, let's put it that way. And today we want to talk a bit about Fable,
00:01:30well, not quite, about all the Fable moments that have somehow caught up with the world,
00:01:38because I'd like to bring everyone up to speed first and then we can get into the conversation together.
00:01:45Maybe you still remember, there was this Mythos moment, an AI model so dangerous
00:01:50that you couldn't release it to the world. Then there was Fable as a model and Fable
00:01:55was a model of the Mythos class. I get this Star Trek thought every time, but
00:02:01Intrepid class, whatever, let's leave the nerd stuff out of it. Then it was banned
00:02:06in part because of export controls. Then it was allowed again and Fable was like, ah yes,
00:02:13come on, it's a Mythos class model, it's good after all, but then it held back on security.
00:02:17Held back on biology. Yeah, along those lines. What do I care
00:02:23about yesterday's knowledge, more and more models keep coming out that, I'd say,
00:02:31are overtaking Fable. And if you're thinking now, sure, I've heard of OpenAI, ChatGPT, version 5.6,
00:02:38Luna Terra Soul, yes, that too, but the Chinese models as well. They're
00:02:46really putting their foot down and have a few peculiarities, and I'm glad that
00:02:50Jens is getting into it with me today, with the question of who's actually leading
00:02:57the AI world? Is it still the USA with the big
00:03:03frontier models, or has China, let's say, triggered the next DeepSeek moment
00:03:10and the next shock moments, because suddenly there are free models. Jens, good
00:03:15to have you here. Thanks, Mark. It's nice that we're now
00:03:19one year and seven days old. So of course I have to fall in line. So our first
00:03:25episode in the new year, in a new look as well. Happy to hand the computer over too. We've
00:03:31decided that we're going to adjust our design a bit, and we'll be doing that every year from now on.
00:03:36So yes, we have to stay with ourselves for another year.
00:03:39That's one way to do it, so yes, that's just great.
00:03:42You can paste things behind it, you can swap the pictures, oh yes, great idea.
00:03:45But maybe more on that later, or more next time.
00:03:49The question, Mark, who's leading?
00:03:53I think that's getting hard to answer at all at the moment, and from a user perspective
00:03:59I always think
00:04:01I don't really care either, you know? I just think it's great that we have such a
00:04:06marketplace in the world that, in this kind of competitive situation, a competition-driven business, we all know that,
00:04:13means we get new models presented to us on almost a weekly basis, which
00:04:21often, well
00:04:23Yeah, not always. I'm wrong. Fable has now shown that it burns more tokens. Some of these models are getting leaner in
00:04:29token consumption too, but above all they're all getting better in quality. No matter how they're built, what technology actually underlies it, how the LLM works.
00:04:38From week to week they're still noticeably better than the models before them, and still give no hope of an end to the flagpole.
00:04:48But I would commit myself and say that with this Kimi 3 moment we've just had,
00:04:55it was definitely a shock moment for the American models, which again
00:05:01and the big frontier labs in the background, which actually did react a bit
00:05:05put out again when Kimi 3 was introduced, and then there was talk from the highest
00:05:10levels again about theft and about this not all being above board
00:05:16and so on, but yes, it's definitely a shock moment and maybe Kimi is
00:05:21a bit ahead right now, so the Chinese, on this topic.
00:05:24I think they're ahead on a few points, but I just wanted to, since
00:05:29you were talking about this shock moment, I just wanted to briefly, you've surely
00:05:34heard it in the news, it somehow made it, let's say, into the German
00:05:38press as well, when word from OpenAI was that their model had broken out
00:05:45and had hacked Hugging Face. You have to picture it as if you'd locked the exam candidate,
00:05:52meaning the OpenAI model, into a room with no windows and no doors, and the thing
00:05:59still basically dug itself a way out. Instead of solving a problem,
00:06:03it basically hacked into Hugging Face and fetched the solution there,
00:06:07because it found that easier than producing the solution properly itself.
00:06:11And the story is actually doubly funny, because whether this model
00:06:17really pulled that off and did it is one thing, but when you listen to
00:06:22what the defender side looked like, it turned out that they tried to use the AI models
00:06:26to fend off the attack, and the models from Anthropic and Open
00:06:30AI said, no, no, no, no, I can't do any defense here,
00:06:34because the models themselves thought that what they'd flagged as defense
00:06:38would be an attack.
00:06:39And so Hugging Face had to fall back on Chinese models, they were apparently a
00:06:43bit more open-hearted about protecting them from the attack.
00:06:47But we were just on this shock moment from Kimi 3.
00:06:50Kimi 3 from Moonshot AI, that was on the 16th of
00:06:56July, an announcement, well not an announcement for publications, and with
00:07:00them it works so that the models are basically served up on their machines. You can
00:07:07take out subscriptions there, just like you can with Anthropic and with ChatGPT.
00:07:15But because you're basically, well, I'd say competing with the American
00:07:21products, you usually get it offered very cheaply. So the K3
00:07:25subscription costs something like half to a third of what you'd have to pay with Fable
00:07:31for comparable compute, and as of the time of this recording, we're
00:07:38sending the episode out a bit later, today is July 27, the
00:07:44weights of the model are public as well, so that if you're equipped with the right hardware
00:07:49you're able to run the 2.8 trillion parameter model yourself.
00:07:56So, my machine is out. A Mac Studio with 512GB of RAM is out too.
00:08:02Even two Mac Studios with 512GB of RAM each are out.
00:08:07But if you have a suitably big piece of metal, you can run this model yourself.
00:08:16And that, I think, is a whole different front door, when you consider that the model works at the level of Fable, of GPT-560, and the story people used to tell, along the lines of, yeah, the open models trail the American frontier models by three or four months, that's just not the case.
00:08:41And the ironic thing is that these models, in China there are, well, there are various
00:08:47export bans, so that certain hardware can't be used in China, and if you let these
00:08:52models run on the good American hardware, they really put their foot down
00:08:56on top of that, and I do think that's a shock moment, because while
00:09:01Anthropic and OpenAI offer models for good money, you can, in effect,
00:09:08run and use this model yourself if you have the necessary spare change for the hardware.
00:09:14That's also more privacy-friendly, you shouldn't forget that either, though on the other
00:09:20hand there's one thing you also shouldn't leave out of account. Moonshot AI did
00:09:25report, when Kimi 3 came out, that the servers were properly overloaded,
00:09:29because it really caused a stir, because a model in that class that doesn't
00:09:33dial itself down on security questions, that basically runs much more unrestrained, that
00:09:39they're running it. And if you look again at something like Anthropic, well, I'm
00:09:45privately a user of the big Max plan myself, over 200 euros, and if you work out that
00:09:50you can basically push 8000 euros a month through the pipe there, then that's maybe
00:09:55a bit of a blessing and a curse at once, that if Anthropic or OpenAI maybe
00:09:59loses customers toward the Chinese models, but coming back around, I still find it
00:10:05extremely wild how close we are to a point where models
00:10:13are so powerful, where the open models are so powerful, something we wouldn't have
00:10:19dreamed of happening this fast.
00:10:20I know, when Fable came out we thought, oh, Fable, now we're talking, and were so disappointed
00:10:24when security was restricted, and then Soul came and didn't have the restriction
00:10:28And now an open model comes along where, as I said, if you have the necessary hardware,
00:10:33you can just let it run, and it's not only Kimi 3, now there's also
00:10:37Qwen, who also said, we're delivering Fable level here, one model after another
00:10:42is coming out right now where you think, this can't be true, what's
00:10:47the next step?
00:10:48Yeah, yeah.
00:10:49Well, maybe we should also say, to be fair, that when we say Kimi is ahead,
00:10:54the Chinese are ahead, of course that's not in all benchmarks and not on all points.
00:10:59Moonshot say it too, hey, I wouldn't say it myself either, so we're not
00:11:02saying now, we're better than Fable 5 or anything like that, but rather just let the numbers
00:11:07speak, and there are simply a few numbers where the model is out in front, above all
00:11:11on cost, honestly, which is significantly lower than if you do it with the competition
00:11:15from America, that's a decisive point.
00:11:18And on top of that, what's maybe also interesting is that it's actually
00:11:22being used.
00:11:23I believe Cursor uses Kimi, if I read it right, for a coding Agent that I think they use, the Composer, Composer 2 I think it is, that's built on Kimi.
00:11:35So yeah, it's not like it's only being used somewhere in China.
00:11:40No, it's being used everywhere in the world here too, because the model, as I say, is open in the first place.
00:11:47So you know what's in it, at least as far as the weights of the model go.
00:11:52on the other hand, as you say, if you can nicely run it locally, if you have the
00:11:56compute power, then it's also a strong model. It's a really strong model.
00:12:00And there I'm back a bit to what I said at the beginning, it's actually not
00:12:04that important to me who's out in front right now. I just think it's great that we have a huge selection of
00:12:08really strong models, because that will mean the model providers actually have to
00:12:14think about it, if we're always very, very comparable at the front with the output and
00:12:23the outcome that I associate with it or can generate from it, then of course for
00:12:28all private individuals, companies, people it will become more and more important to look at other factors, like
00:12:34cost, quite simply. What does it cost me to run a model like that?
00:12:39And that could in principle, well, that sort of thing is always healthy in a market, I think, because
00:12:43it means we then gradually push the prices down too.
00:12:46This year, I think, we've gone from the insane token maximizing
00:12:51over, that's nonsense, that's just hype from the big companies to lure you in.
00:12:56So a lot of people claimed you're not really a proper human being any more if you don't somehow
00:13:01use billions of tokens a day, especially if you're a developer.
00:13:05That's shifted a bit, toward saying that doesn't necessarily have to be
00:13:08the case. You can also work cleverly with model selection and thereby use fewer tokens
00:13:14in one place or another, when it isn't needed at all. Toward the topic where
00:13:17I have to say, yes, it's becoming more relevant that we also have cost-effective models
00:13:24that don't cost you an arm and a leg for every little request I make,
00:13:30so that's a topic, I think, where it's helpful to have good competition in the world.
00:13:35It's the case anyway that, let's say, better to have than to need, and more is more are not
00:13:42automatically good advice for using AI models.
00:13:47Of course, if I use the biggest model of the biggest models, in ultra code mode
00:13:51and whatever else, then for the question I'm
00:13:57asking I'll probably get a very good answer too, maybe even a better answer than if I
00:14:01shift the model down a level, in terms of, in terms of how much thinking effort
00:14:06it puts in, whether I use an Opus or a Sonnet. But for the question I'm
00:14:12asking, maybe I don't always need the biggest, most expensive, most powerful model,
00:14:16because first, it costs more money, and second, it usually also takes a bit
00:14:20longer, yeah. So if I ask Fable a question or ask a Sonnet a question
00:14:24and maybe Sonnet with medium efficiency, medium effort setting is enough
00:14:30And then there's a time component to it too, how long I wait until a result comes back.
00:14:36And from that angle it pays off anyway. What I find exciting about the Chinese models
00:14:40in that respect is that, if we look toward Perplexity for example,
00:14:45Perplexity has this functionality with Perplexity Computer, where they go and say,
00:14:49well okay, you've got this in mind, so I'll take this model.
00:14:52They also have something like, if I have a certain task, then I take that model,
00:14:57for me that model, for me another task, for me that model. And as you just said,
00:15:02benchmarks, first of all benchmarks aren't everything. Second, that's where the systems
00:15:07differ. Where does it settle in? Where does it deliver its full performance? And
00:15:12then it does get exciting to see whether at some point there might be an LLM multiplexer,
00:15:17LLM orchestrator, LLM something, where you say, right, I ask a question,
00:15:21because as a user I don't really care, maybe, I just want
00:15:25a good result. Best price-performance ratio, a kind of ml24.de, there we go, we've got another
00:15:33great business idea, that basically provides you the best model ever,
00:15:40for the topic I'm sitting on. And what I found quite funny, while I was thinking about
00:15:46what we wanted to talk about in this show today, a few days ago to be precise,
00:15:51you've already picked up on it, today is July 27 as we're recording this,
00:15:56a Monday. On the Friday before that, Anthropic went and delivered again with Opus 5, and the
00:16:05rumor mill says that OpenAI will even come out with GPT-6 in August. You can really tell
00:16:13that someone poked the hornets' nest properly, because none of this is
00:16:19that secret. They all want to go public soon as well. If the
00:16:23Chinese models now start overtaking them, then they'll certainly
00:16:28want to react to that, because if you want to go public, you don't want to
00:16:32let's say go public just after, or just before, being pushed down to second, third,
00:16:36fourth place. That's one thing, and the other.
00:16:40The Chinese models will probably play to their
00:16:43strengths again once the big providers are listed on the exchanges.
00:16:47Then, I think, what I said earlier at the start,
00:16:51along the lines of, I have a subscription for 200-odd euros
00:16:54and burn tokens per month that are worth maybe 8,000 euros by comparison,
00:16:59then they won't really be able to keep that up either,
00:17:01because then surely, once a corresponding market is investing in the shares,
00:17:08they can't subsidize as much as they maybe can today,
00:17:12to drive the hype and the push and who knows
00:17:14what a bit further up, and then I'm curious to see how the whole thing plays out.
00:17:18From that side I'm totally curious how it develops further, even if I'm very disappointed
00:17:24that I can't run Kimi on my little notebook now.
00:17:27That's a real shame.
00:17:28Although, if I could, I'd probably need a bodyguard, because that much RAM,
00:17:32as much as you need there, yeah, these days that means, well, then you're rich, yeah.
00:17:36So, quite apart from the fact that you can't buy that in a notebook, you'd be rich.
00:17:41Yes.
00:17:42since you mention rich. I think this Kimi moment, I don't know
00:17:47whether there was a timing connection there, but last week we also saw
00:17:50a small dip with the chip manufacturers. There was a small one, the shares
00:17:55lost a bit of momentum. Did they? Yes, briefly down and, I think, back
00:18:02up again. It's a very short-term thing, but of course all of this is also,
00:18:06we always have to be careful there too, even when new models come out.
00:18:12Of course there are very, very many independent evaluations straight away that basically
00:18:16weigh these models and look at how they behave
00:18:19compared with other models and whether they're better or worse, which tests they
00:18:22pass and so on.
00:18:23But it's still always the case, there's also always a short
00:18:26hype phase that's deliberately driven by the model providers.
00:18:29And as I said, it always strikes me a bit that as a
00:18:33private user I'd say, everything you have is already so good.
00:18:38You can just use a lot of things there, just keep trying things out.
00:18:40I recently had another look at Notion.
00:18:43There, I think, if you use Notion, you can nicely use all the models
00:18:45if you have a Notion subscription anyway.
00:18:48But that's crazy too, I have to remind myself of that every time.
00:18:51So, if you look at Notion, then in Notion there's
00:18:54Sonnet 465, Opus 47, Opus 48, Fable, Gemini 3.1 Pro,
00:19:00GPT 5.2, GPT 5.6, Terra, GPT 5.2, GPT 5.4, GPT 5.5, Grok 4.3, SpaceX AI, I didn't even know
00:19:12that existed yet, 4.5, Grok Image, and then there are smaller models, listed separately above,
00:19:18Gemini 3.5 Flash, Kimi 2.6, 2.7 Code, DeepSeek V4, GLM, I could go on like this,
00:19:26It sounds a bit like counting sheep, or like calling sheep.
00:19:32But I wanted to briefly step in again with my UX glasses on and
00:19:38say that this model selection, for the private user and maybe also for the user in
00:19:47some office or other who doesn't spend, like we do, a very, very large part of their time on
00:19:52the topic of AI, is honestly overwhelming. So I sometimes have the
00:19:56issue myself that I no longer know what,
00:19:59which model should I actually take now? Do I really need
00:20:02Fable for what I've got in front of me right now? And somehow the
00:20:08user guidance in the various places, whether you're using a
00:20:12desktop version or a web version, doesn't matter. That's
00:20:17not really helpful, where they say, here for more complex tasks or
00:20:21for everyday complex tasks or for sometimes important tasks,
00:20:26better take this model, because it's all so hard to judge.
00:20:29Do you have any rule of thumb for which model you take for what?
00:20:33Well, it's not suitable for copying. The rule of thumb is,
00:20:39until the warning comes that the limit will soon be reached, the biggest one, and after that
00:20:43think a bit about the last days of the week, so you don't run into the weekly limit,
00:20:47and then take something so you somehow scrape through, because
00:20:52Fable, for example, uses much larger quotas at Anthropic, though I don't like using Fable
00:20:59all that much. So first, you always have to watch what you use Fable for,
00:21:03because they have this 30-day data retention, tongue twister. And the second thing is Opus
00:21:10actually does better and cheaper in some areas than
00:21:16so from that side I'm more of an Opus 5 friend at the moment, and this thing you said, how
00:21:22do I deal with all this stuff? Yeah, it's, it's, it's terrible, right? Because the models used to
00:21:26be called 01, 03, 02, they couldn't call themselves that because of naming rights, right? Then
00:21:31Minimax and Pro and whatnot, right? And today Luna and Terra come along and you think,
00:21:38well, excuse me, right? What kind of names are those? You have to get used to
00:21:45something new with every generation. Now Opus and Sonnet certainly aren't
00:21:48any better, but at least they have a version number on the end. And there isn't
00:21:53an extra-blue flavor on top of that, although there is the flavor
00:21:56ultra code and Max and Ultramax and who knows what, it really does your head in, you
00:22:02can't really expect that of anyone. But something I found quite funny in the same
00:22:06breath, maybe you remember, looking back at the last
00:22:10season, when we were still out with the old covers, we had this topic once,
00:22:15are the models actually going to keep getting stronger and bigger, or is it going to be a bit
00:22:20like the human body, along the lines of lots of sensors and lots of subtle sub-nodes that
00:22:26somehow decide or take decisions off your hands, because otherwise the brain, like the
00:22:31brain of the human body, would be overloaded, maybe you can also handle the reflex of
00:22:35flinching away from a hot stove completely differently than having the brain think about it
00:22:40first. So breaking it out into smaller models. And I found that really funny. I
00:22:45stumbled across a project. A repository that basically offers you an open model and open
00:22:51weights for a language model that has just under 30 million parameters, just below, runs
00:23:00completely offline and, get this, on an 8-dollar microcontroller, an ESP32, people who know about
00:23:07this stuff will certainly be able to say more about it, that basically, you could say
00:23:10these things could be used in smart home settings, in smaller technologies
00:23:16that you build in and put up at home, and that you now have the option,
00:23:21as a manufacturer, on such a small chip, it costs 8 dollars, it has something like 512 kilowatts
00:23:26of RAM or eight-something RAM, something like that. Doesn't matter, you get what I'm driving at. You have a small chip that costs almost nothing and on it you can now run a local model, and I'm curious to see what effect that will have, now that we're also able to run models on hardware that cheap.
00:23:46Where I want to go with this, we'll see, then people can read what I've cut together in such a muddle here, exactly, we can do that in the new season.
00:23:55I wanted to say season too, because that's the first time I'm using this word for what we achieved last year, a whole season shot and recorded.
00:24:05Yes, I find locally installed models that can work in small hardware gadgets
00:24:14really exciting.
00:24:15I think that's one of those things where I'd say some really cool applications will
00:24:19come out of it.
00:24:20Because that might also change the way we currently learn
00:24:26to interact with the product.
00:24:29There was this hype around AI devices two or three years ago, and it all got a bit
00:24:33quieter.
00:24:34that these AI wearables were basically out there briefly,
00:24:38companies were hyped up, not much has happened there yet,
00:24:41but it's going to come now, because of course it's hugely advantageous
00:24:44if I can run things locally,
00:24:46I need to build in less technology, it doesn't necessarily have to be internet-connected,
00:24:49but it can help me enormously.
00:24:51I can already see things like, well, there always used to be survival manuals
00:24:56that you needed when you were out camping,
00:24:58these days you can put all of that locally onto an off-grid device
00:25:01and not just have to read through or search a long PDF version, but
00:25:05have it as an LLM that, in an emergency, if you have the bad luck, whatever,
00:25:11to break your leg while mountaineering, can then explain to you how to
00:25:14rig up a splint yourself, how to lash the sticks together with your laces
00:25:19and build a pressure bandage out of it, so you can get back down the mountain.
00:25:25You don't know my laces, they'd have to hold, don't worry, Tim, I'm
00:25:29not going to do that.
00:25:30Yes, but I think that's yet another thing entirely. I think there's more to it as well.
00:25:34A new feature came out from OpenAI in Codex too. It's called record and replay?
00:25:38It's that kind of thing. I can basically record videos of myself, of my screen, now.
00:25:45I can then throw that into Codex afterwards and Codex makes a Skill out of it. So I just show it how I basically do it.
00:25:52Yes, you want to say something, go right ahead. I want to say something about that in a moment. I was totally impressed by this feature at first.
00:25:59I thought, how cool is that?
00:26:02And then I, well, we build our own harness at the company too, and then
00:26:07it took about two hours and then it could do that too.
00:26:10Yes, that's good.
00:26:11It's just funny.
00:26:12I have to share this now, it doesn't really have anything to do with the topic, but I'm
00:26:16just so excited about it.
00:26:17So I had a look at the thing, I can well imagine that if you use something like that
00:26:22in a work context, then there could be people who
00:26:25don't like the way OpenAI and Anthropic built it, because the thing wants to read along with your keyboard,
00:26:29the thing wants to read along with your screen, are you sure it really only
00:26:34reads along when you want it to, and so on, there's a bit of a lack of consent
00:26:38and that kind of stuff.
00:26:39And so I just sat down and thought, let's see what the model
00:26:42actually does.
00:26:43So with us, in our harness, what the model does when you only give it the video,
00:26:47just the screen video, and specifically a screen video where you're doing something, so for
00:26:54example operating a website, operating an internal website, and simply
00:26:57explaining what you're doing while you operate it, and so I talked along, saying, right,
00:27:01here's an input field, and always marked with the mouse what I was talking
00:27:05about, like, here's an input field and I don't need this one and here
00:27:09are the answers and here you can see whether there was an error, and basically
00:27:13worked through the thing with this video and then fed this
00:27:16video in and told it, make a Skill out of this, and you know what,
00:27:20the thing goes and makes itself a Skill, it broke the video apart, it can then look
00:27:26at, along the lines of, how often does something change in the video, takes out everything where possibly
00:27:31nothing happened at all, because you didn't move the mouse, because the website was busy thinking
00:27:35or whatever, so it only took the remaining frames, built itself a Skill out of the
00:27:39remaining frames, put the images into the Skill as orientation patterns
00:27:45and then moved on and said, right, okay, I've learned it from
00:27:49you, you want to go to this website and here are the things you need. Here are the things
00:27:53you ask. It asked me a few follow-up questions. Long story short. It then used
00:27:58Playwright, that's a library that basically does a kind of browser operation, except that
00:28:05you can run Playwright headless, so without a visible browser interface. And then
00:28:11the thing basically goes and works through the Skill, operates this website with it, and whenever
00:28:15it doesn't know how to go on, because the website maybe looks different, because somehow
00:28:19whatever, it just wants to orient itself again, it basically looks again at
00:28:23these screenshots it saved for itself, together with the Skill, and operates it.
00:28:26And long story short, you can go and say, for example,
00:28:31all right, you have lots of different RAG systems. I think we'll be talking about RAG
00:28:35next time with Cornelius, that much promo is allowed.
00:28:38Is it then basically able to package the whole lot into a Skill,
00:28:43in your agent harness. For any other listener that would be Claude or the Codex app. With us it's
00:28:50our own, our own harness that we built. You can then basically replay it.
00:28:55And that's how you can connect other systems to your harness, purely through a Skill it
00:29:04learned on the basis of videos, audio tracks and the possible things to control. And last
00:29:10I always wonder then, why did Anthropic, or Codex, build so much stuff around it,
00:29:15so that you have to hand over so many permissions, when actually a video is enough,
00:29:20one you record, or a training video you maybe find somewhere in a learning
00:29:24program, and you say, here you go, take a look, you'll figure out the rest.
00:29:27Have fun, fire at will. The power of it is unbelievable.
00:29:30Yes, that's true. That's exactly what I mean. That's what I mean when we say,
00:29:34We will basically, whether that's on locally installed
00:29:39hardware gadgets, videos, voice, audio.
00:29:42Imagine that what you're describing right now
00:29:44is basically a matter that happens on a screen,
00:29:48where I work through things and a Skill can be built from it.
00:29:51It's often the case that production isn't the hard part any more.
00:29:55It wasn't the hard part even before AI these days.
00:29:58We already had good people before who could do design,
00:30:01who could code well. It's often the case that this path from the actual idea,
00:30:06the translation, what actually is the real problem in that moment, that we have to
00:30:11recognize where we have to put work in, analyzing workflows, situations, hospitals,
00:30:16airports, whatever, to see how you can improve these
00:30:20processes. All of that is what happens in the real world, and then afterwards
00:30:25possibly leads to a piece of code and to an application or something that
00:30:28improves part of that workflow. If we can capture that considerably better
00:30:33in the future and more interactively, without me having to, and I'm now a
00:30:38big fan of this, if we switch to formats like that, because then I don't even
00:30:42need to know the concept of a programming language at all. So if we just talked about
00:30:48English being the new programming language, then approaches like the ones we're seeing
00:30:51right now are of course also the replacement for having to understand this
00:30:56abstraction level at all. I no longer have to understand that a Skill possibly
00:31:02needs a text file to describe how the Skill should behave, and additionally
00:31:06needs further documents as templates so it knows which output format it should
00:31:11produce, or some connection to MCPs. But no, the machine can simply see
00:31:15what we do and derive from it what might be a good solution that could generate a
00:31:22similar output to what it saw there.
00:31:24And that is actually one, well, I think that's one of the conditions we'll be
00:31:31seeing much, much more of in the coming weeks and months.
00:31:35That we see there are other options besides pure text input.
00:31:40So every one of us has probably taken the photo of the wine shelf at Rewe
00:31:45or at Wiedel or wherever and checked, let's ask ChatGPT which wine to take,
00:31:49or photographed the fridge to get the recipes that are still possible
00:31:54with the leftovers going moldy in the fridge. But that was a live glimpse into the
00:32:00Scharnetzki household. No, I don't think so at all. We're extremely well stocked, yes, and
00:32:05nothing ever gets thrown away at ours. Yeah, all good. No, joking aside,
00:32:10but the thing is, I know, the thing is that we really actually now,
00:32:17Well, we were multimodal, the AIs have been able to do that for a while now, I've been able to give it videos for a while,
00:32:24been able to upload images to it for a while, or text or video or whatever, so I could hand over a lot of things.
00:32:30But now we're entering this phase where it's not just about, okay, there's knowledge in there, but I can also use it for action, exactly as you just described.
00:32:40And when we enter this phase, then of course it gets even more exciting, when we give AI eyes in the real world.
00:32:46Exactly. And it picks up even more of what's happening. There'll be a huge amount there, I think, where this year, in our new season, in season 2 of Think Different, Think AI, there's certainly an episode or two to be made.
00:32:58We were originally on the open models. I think we covered that well too. Something I also want to say about your thought while you were talking, all of a sudden things like little YouTube clips become quite interesting too.
00:33:14because let's say, I now have, why I have it is another topic, for example also
00:33:20privately, Apple Business Manager, that's a solution from Apple where these days you can manage devices
00:33:25not too badly. But where you simply have to get to grips with a few portals from Apple, let's put it that way.
00:33:31And the funny thing, I absolutely have to try this out, just imagine, there are all these little YouTube clips
00:33:37where some influencer or Apple itself talks about things, and you give it that, and then all of a sudden you have the option
00:33:42that your AI system operates this portal for you and thereby maybe makes it much more accessible
00:33:49than you managed by hand up until two weeks ago. So from that angle I think we should
00:33:57definitely do episodes soon about the possibilities of the new harnesses and the new systems, not just the new
00:34:02models. And maybe also with a critical
00:34:08perspective, and I already have a guest in mind who knows a great deal about this whole
00:34:13topic of VR and augmented reality glasses. We could interview him about it too, because
00:34:20of course there's the negative side as well. So if we, it's like Pandora's
00:34:24box, of course a lot of people are interested in this data too, to make the models
00:34:28better or to do other things with it. And if now, of course, everyone,
00:34:33because there's an advantage in maybe being able to record something constantly with my glasses,
00:34:37being able to optimize workflows that way, then of course the question of data
00:34:43protection comes up in the background. What am I actually recording, where is this data stored and
00:34:47is it being trained on. So if we do want to do that, then in the usual way we should
00:34:52shed light on both sides of the coin again and discuss them. And I'd be
00:34:57glad to, because I've been meaning to with the colleague anyway, we need to set a date now, to do an episode
00:35:01on the topic. We can record that at the same time. I was just, while
00:35:05And you, quite rightly, you said two sides of the coin, there's still the coin of
00:35:10filthy lucre, and I was just wondering whether you could train a Skill, simply with a whole lot of
00:35:15little YouTube clips, whether you could sell a kind of Skill library with real added value
00:35:19for people. But since we said we're in a new season,
00:35:25the segment "making money with AI" is probably not what we'll start with anyway,
00:35:30but I'm very much looking forward to an episode with a guest. I'm also very glad that
00:35:35we have an episode with a guest next week, yes, I've already mentioned Cornelius. It'll be about
00:35:39Second Brain and that kind of thing, yes, stay tuned. And I'd say, Jens, let's quickly
00:35:45wrap up before a new model comes out and makes everything we've discussed so far
00:35:50moot. Thanks for taking the time. I'm glad we're going into this
00:35:55new season together, and do leave us a comment. How you like our new
00:35:59cover, and with that we say see you soon at Think Different, Think AI. We look forward to you. Bye.
00:36:06Ciao.
00:36:09Welcome to Think Different, Think AI, the podcast by Mark and Jens.
00:36:14Two technology-loving minds who don't just talk about artificial intelligence, they live it.
00:36:21Here you get clear perspective, real hands-on insight and a fresh look at what's possible.
00:36:27Understandable, critical and always with a wink.
00:36:31AI to think about, to smile about and above all to talk about.