KI Effizienz
Auf Deutsch lesenTopics SoftwareentwicklungInterfaces und InteraktionWissensmanagement
Guest Alexander Heusingfeld
What it is about
Was KI-Effizienz wirklich kostet: Abnahmekriterien, Gegenkennzahlen und warum die Zeit zwischen den Arbeitsschritten entscheidet
Firmen werfen 80 Prozent ihrer Leute raus, weil das jetzt die KI macht. Diese Meldung geht seit Monaten durch die Presse, und Alexander Heusingfeld möchte nachrechnen. Wer 80 Prozent entlässt, behauptet damit den Faktor fünf. Klarna hat den Schritt für den Kundenkontakt inzwischen wieder zurückgenommen. Und selbst wenn die Rechnung aufginge, bleibt Alex' eigentliche Frage offen: Woher will jemand vorher wissen, welche 20 Prozent das Skillset haben, der Maschine beizubringen, was am Ende herauskommen soll? Wer selbst noch nicht ausprobiert hat, was möglich ist, kann das nicht beurteilen.
Mark dreht die Frage um. Statt „wen brauchen wir nicht mehr" interessiert ihn, was bisher liegen geblieben ist, weil nie jemand Zeit dafür hatte. Beide landen bei derselben Beobachtung: Regulatorik wie NIS-2 bleibt liegen, weil die Kapazität fehlt, teilweise sogar bei Behörden. Über jedes Stück stupide Arbeit, das eine Maschine übernimmt, kann man sich also erst einmal freuen. Die freigeräumte Zeit ist ein Invest, kein Einsparposten.
Dann erzählt Mark, was an einem Wochenende passiert ist, an dem er eigentlich Schränke aufgebaut hat. Auf einem Mac Mini hinter dem Fernseher, ohne Skills, mit nichts als Zugriff auf seine alten Git-Projekte, hat ein Agent sich das Smart Home angesehen und vorgeschlagen, die Lautsprecher im ganzen Haus als Interface zu nehmen. Danach lief die Arbeit über Sprache, während Mark durchs Haus ging. Der Agent hat ein Jahre altes Projekt fertiggestellt, sich Xcode geholt, Bundle Identifier und Signierungszertifikate selbst gebaut und die App am Ende über TestFlight aufs Telefon geliefert. Mark musste nur ein paar Kennwörter eintippen. War das effizient? Er sagt selbst: mehrere Wochenlimits verbrannt, garantiert nicht. War es das wert? Absolut.
Alex hält dagegen und stellt die zwei Fragen, um die es in der Folge geht. Was war der Harness, und welche Rahmenbedingungen hast du mitgegeben? Er selbst hat sich mit einem Agenten über AWS die Steuererklärung vorbereiten lassen: knapp 800 Belege, das Ziel „mindestens 80 Prozent einsortiert und formal richtig", vier Stunden Rechenzeit, 60 Euro. Zurück kam auch der Hinweis, wo der Steuerberater im Vorjahr etwas anders hätte abrechnen können. Aus dieser Erfahrung wird eine Metrik, die er für die brauchbarste hält: Verifikationsaufwand pro ausgelieferter Änderung. Sinkt sie, waren die Abnahmekriterien besser.
Daraus werden drei Regeln. Nie eine einzelne Kennzahl, weil jede einzelne Zahl gegamed wird, sondern immer ein Faktor aus Kriterien und Rahmenbedingungen. Immer Gegenkennzahlen mitlaufen lassen, Change Fail Rate, das älteste Work in Progress, Developer Experience. Und niemals selbstberichtete Geschwindigkeit: Entwickler geben 25 Prozent Beschleunigung an, gemessen wurden bis zu 20 Prozent Verlangsamung, weil die Zeit in Reviews und Nachbesserungen zurückfließt.
Mark ergänzt die Kette drumherum. Wenn einer drei Tage durchlaufen lässt, steht das Team vor einem Pull Request, der aussieht wie ein gefluteter Landstrich, das Backlog ist am dritten Tag eines Zweiwochensprints leer, und die Tester werden überrannt. Codekommentare, früher Pflicht, führen Agenten heute in die Irre, weil sie den Kommentar lesen statt der Routine. Und wer Generierung durch Agenten reviewen lässt, die wieder von Agenten kommentiert werden, verbrennt Tokens für das Gegenteil von Effizienz. Alex bringt dazu eine Untersuchung aus dem Flugzeugbau mit: 8,5 Prozent der Durchlaufzeit sind wertschöpfend, grob zwei Wochen Wartezeit pro Tag Arbeit. Die Antwort auf „effizienter werden durch KI" ist damit keine Werkzeugeinführung, sondern Flow und Governance. Oder in Alex' Bild: Erst klärt man, ob man Fußball spielt, danach wird das Spiel schnell.
Zum Schluss lösen die beiden die Diktiergerät-Geschichte aus der letzten Folge auf, Mark verrät sein Setup, und Alex fasst zusammen. Wer KI nur als Werkzeug kauft, verkürzt einen Arbeitsschritt. Die Effizienz liegt in den Wartezeiten dazwischen. Das richtige Maß an Qualität entscheidet sich gegen die Anforderung, alles andere ist Informatiker-Romantik. Und Marks Zugabe für alle, die zögern: Wer nicht weiß, dass er scheitern könnte, probiert es gar nicht erst.
Transcript
00:00:00Welcome to Think Different. Think AI., the podcast by Mark and Jens.
00:00:07Two technology-loving minds who do not just talk about artificial intelligence, they live it.
00:00:14Here you get clear perspectives, real practical insights and a fresh look at what is possible.
00:00:20Understandable, critical and always with a wink.
00:00:24AI to think about, to smile about and above all to join in on.
00:00:29A warm welcome to Think Different. Think AI.
00:00:36We now have, I would almost say, something like a regular player here on
00:00:41our podcast.
00:00:42Yes, Alex is with us again today.
00:00:44I got quite a lot of good feedback on our last episode, which, as
00:00:49you will remember, turned out a little longer.
00:00:51I am very curious how efficiently we will keep things today, and efficient is also
00:00:55the magic word.
00:00:56I made it particularly efficient for myself, because I had Alex write up the thread
00:01:01for today's podcast, which means it was delegated right away,
00:01:05which is wonderful.
00:01:07Alex, good to have you here.
00:01:10So what is the title of our episode, co-host?
00:01:12Yes, thank you for having me again.
00:01:18It is very interesting that you got positive feedback.
00:01:21I have been told it was entertaining.
00:01:24Well, my friend, whatever that means, better than being so dry that people switch off.
00:01:31Yes, although, I have to say, listening back I did think, oh, we really had a lot of
00:01:37terms in there that went quite deep into the technical side, but well, I think
00:01:43that probably will not, or so I assume, hope and believe, happen to us today.
00:01:48It is about becoming more efficient through AI.
00:01:50Exactly.
00:01:51Yes, very good.
00:01:52And I thought, well, the topic is really on my mind, and since I know that you
00:01:59have given it some thought too, and I know a few concrete examples from you,
00:02:05I thought, go on, prepare it, because that is the done thing when you
00:02:10bring a topic along as a guest.
00:02:12Yes, exactly.
00:02:13Yes, just like that, that is the secret of this podcast, you see, we basically have
00:02:17no idea at all, and we are always glad about every guest who takes a bit of the
00:02:21hosting off our hands so we are not just sitting there. Greetings go out to Jens. As you may have
00:02:26noticed, Jens is not with us again today. He will be back, he will be back, he will be back.
00:02:30I am glad you are here today. When I opened the script, I found
00:02:35a few great ones in there. How shall I put it, hooks? Let me say, a few
00:02:42hooks caught my eye. Stories along the lines of, you might have
00:02:4880 percent. Let us see what we make of those 80 percent later. Yes, the word
00:02:55efficiency comfort, the ever-present flow comfort, numbers, events, loops. I am curious
00:03:00how epic a breadth we will treat the topics with, and whether we will not also deliver
00:03:04some practical value for our listeners along the way. But let us
00:03:10start with the 80 percent. What really annoys me is when people simply do not take the time
00:03:23to engage with a complex topic and its influencing factors. And so you constantly hear
00:03:30these striking claims in the press, like companies throwing out 80 percent of their staff
00:03:36because they can now do it with AI. And there is not just one company
00:03:41making that claim. We know about Klarna, who are now backtracking, because they
00:03:46have realised, well, maybe the things we wanted AI to handle, things like
00:03:51contact with the customer, maybe that was not such a good idea after all, we shall see. But in the end,
00:03:57if I throw out 80 percent of my people, that means AI makes me five times as
00:04:06efficient.
00:04:07Which is to say, achieving the same with 20 percent of the people.
00:04:10There are two things in that.
00:04:14Sorry for being so unrestrained, I cannot keep myself in check, which keeps me
00:04:17from, well, it stops me cutting in on you, but you are a grown-up, you can
00:04:20tell me off now.
00:04:21I think there are several things in there.
00:04:23One is this propaganda along the lines of, we are all going to lose our jobs and 80 percent
00:04:29get thrown out because we do not need them anyway.
00:04:31If we extrapolate that, yes, maybe someone says 60 percent or 50 percent, I would
00:04:34not insist on the exact figure, but rather on this propaganda aspect,
00:04:39because it, let me say, perhaps also through the odd misdirection in the
00:04:43political sphere, gives people a simple answer to complexity, namely
00:04:47along the lines of, with AI you have to do this, otherwise you do not belong at all.
00:04:51And if you deploy AI, then you can do it.
00:04:53You only have to deploy AI, everything is great, right?
00:04:55If you do not do it, it is your own fault.
00:04:57When I sometimes listen to those influencers
00:04:59who flutter back and forth on their YouTube channels,
00:05:02fluttering, advertising, fluttering, what kind of word is that?
00:05:05If it is new, please register it, as the inventor.
00:05:08Yes, that is the follow-up to bullshit.
00:05:11At some point it turns into flannel and...
00:05:14Anyway, never mind, I will carry on.
00:05:16Okay, thank you. You are almost throwing me off,
00:05:18but I hope I am still roughly on track.
00:05:20That is one thing, and the second is what I always find so interesting, if you now say, okay, even if it were the case that you say, I will cut 80 percent.
00:05:31Those 20 percent, they will rock it, with the AI or however.
00:05:35Then that is always measured against the status quo.
00:05:38They say, along the lines of, those 20 percent can do what the 80, what the 100 could do, because factor five.
00:05:43But this question, what am I actually capable of if you structure work differently,
00:05:49what might I be capable of doing that I never got around to before, where people
00:05:54said, honestly, if we want to do this topic, we simply do not have enough
00:05:58staff, we have to bring in an external service provider, do you have the money for that
00:06:02and so on and so forth, those are things that otherwise come up in companies now and again
00:06:07and should.
00:06:08And if you now say, okay, I can work more efficiently, in quotation marks,
00:06:12my sentence would not be get rid of the people, but rather, what could we do?
00:06:17What else is being left undone? What changes about the way we work, and not: the work
00:06:24I did a year ago is now done by AI. I notice it myself. And then
00:06:28I listen, with my verbal diarrhoea, the next word this podcast has carried,
00:06:31to the things I occupy myself with, and I could not have done them last year. For
00:06:40one thing, because the models and systems were partly unknown to me or did not yet exist, and because the way you work changes as well.
00:06:50Because you increasingly get a feel for when to deploy AI and how to deploy AI.
00:06:56What you used to learn as Let Me Google That For You, I could find something in Google and a colleague could not.
00:07:01So I sent him Let Me Google That For You as a video, so that he would go, oh right, I only had to type it in like that and it would have worked.
00:07:06That way you develop a different approach, and that is what I see as the second factor. On one side the prophets run around preaching and promise you the moon, in order to somehow
00:07:14advise companies and goodness knows what. Greetings to the McKinseys of this world.
00:07:19And the other thing is, the field of work changes as a whole, and one really ought to look at the question of what we get out of it,
00:07:26for each individual and for us as an organisation, and not just this simple back-of-the-envelope reasoning along the lines of
00:07:34Hey, cool. Double the efficiency, triple, quadruple, five times over, you just draw a line under it.
00:07:39The AI is to blame, so we have a culprit too, yes, and we can explain it as well.
00:07:42The AI is the one taking my job away, which is also great, so we have a culprit we can put in the pillory.
00:07:48It is not mismanagement or anything like that, right.
00:07:51That is the very convenient product.
00:07:53And specifically... Thank you.
00:07:55Bear with me.
00:07:56Honestly, Mark, I do not want to...
00:08:00Well, in my view you have just said exactly the right things.
00:08:03I think that is really how it is, and I do not want to disparage anyone's bar-stool chat,
00:08:08but in effect that is exactly the kind of sweeping bar-stool grumbling it is.
00:08:13There is at the moment, and I would have said this in the preparation too, simply no figure that is
00:08:22arithmetically available, no figure where you could say, here is the mathematical proof.
00:08:27Because the crux for me is, you can go ahead as the boss of a company, you
00:08:34can go ahead and say, I will throw out 80 percent of my workforce, and the 20 percent
00:08:40who are left will just have to manage somehow.
00:08:42That creates pressure, and probably those 20 percent will then be under strain.
00:08:48No, but how shall I put it? No, I, erm, let us not go there, that is a rabbit hole, we do not need to go into that right now.
00:08:57But what I am getting at is that this pressure then makes people say, damn it, the way we have worked so far, we cannot carry on like that.
00:09:07And that is exactly the mindset you actually want with AI, where you say, hold on a moment,
00:09:15we have an entirely new kind of tool here, and an entirely new way of approaching
00:09:25a problem at all.
00:09:26And that requires us to rethink our problem-solving strategies and our way of thinking, so
00:09:32not just the way we act, as it were, how do I do process optimisation, but
00:09:38to think about the topic of process optimisation itself all over again. So that is effectively
00:09:42the meta level. And now I come to my point. If I throw out 80 percent of the people,
00:09:51how do I know that the 20 percent I kept are exactly the ones with the skill set
00:09:58to teach the AI how it should produce the result the company
00:10:04is meant to have in future. So if I have not yet engaged with AI
00:10:08and have not yet tried out in practice what is possible, how do I know then
00:10:13which skills I would need in order to implement that.
00:10:15What I wanted to bring in just now, when I, erm, and I cut in on you in a rather
00:10:20unfriendly way, was when you were talking about process optimisation.
00:10:23I mean, the word skills has come up a few times here on the podcast and in our conversations as well.
00:10:31And I do think that something like skills is the second chance, for one thing because it is simply an easy way of teaching the machine things.
00:10:40I keep experiencing that when people work in chats and you then explain to them at the end how to turn that into a skill and what to watch out for in order to get better reproducibility and so on, as one thing.
00:10:50But if you get something of a handle on which skills actually exist,
00:10:55so that you do not end up with the 527th skill on the topic of requirements engineering or whatever,
00:11:01but perhaps with a handful, or counting on two fingers or one finger,
00:11:07by doing that you kill several birds with one stone, namely you manage
00:11:12to bring the AI tool closer to people, to get them engaging with AI,
00:11:17to explore the edges of their job. The other day I heard a talk asking what happens
00:11:22if the product owner now becomes a software developer too. I think I would sometimes have to look
00:11:25in the mirror on that one, because your way of working changes, because you have new possibilities at
00:11:31hand, and with skills you get the whole thing, let me put it crudely, handed to you in a more standardised,
00:11:36process-conformant and automated way, and that way you also get the issue
00:11:43off the table. Let me call it, what shall I say, water-treading. Every project
00:11:48with its own expertise, years of experience, you have to do it this way and no other.
00:11:52And yet in the next project it is done a touch differently or implemented a touch
00:11:56differently. No, by the by that counts towards the completion status,
00:12:00by the by that counts towards the completion status. No, if
00:12:02that counts towards the completion status, that you get some consistency into it,
00:12:06without going in somewhere with a crowbar and
00:12:10saying, right folks, we are now all doing training courses A, B and C so that we are all aligned on whatever
00:12:16procedures are out there, and then everyone goes back to doing their own
00:12:21little thing anyway, but instead you say, okay, good, what are the best practices, we
00:12:25put those into the AI, the AI takes care of a great many things, takes a great deal off your
00:12:29hands, this time you have to work with a backlog with the help of AI, yes, we could
00:12:33perhaps talk later about the plot and whether that helps you or not, the thing we have,
00:12:37the one I already reported on in my Tesla episode with Jens from the holiday,
00:12:41it changes the way you work so much, and in the same breath it can help you
00:12:45introduce standardisation somewhere where previously, depending on the company, you might have
00:12:50despaired. Yes, and at the same time... Maybe at this point, for both of us,
00:12:54I just want to put this out there, when we talk here, we bring
00:12:59points into it, we are in contact with a great many people, so we have small
00:13:02chat groups, we have our LinkedIn community, we allow ourselves again and again to report
00:13:06on things that we read there in part.
00:13:08So that nobody says,
00:13:10oh look, he goes around every day with a club
00:13:12saying, here, process standardisation through skills.
00:13:15Yes, so from that side I just wanted to put that
00:13:18before the audience once more.
00:13:19He does, he runs around all day with a club anyway
00:13:23and beats everyone about the head with it.
00:13:26You could say I have a bit of an image like that at the moment, right?
00:13:28With the, well, that is another topic.
00:13:30Okay. You will have to google that one.
00:13:32But I would like to come back to this once more.
00:13:36The crux of process automation for me is that we make ourselves
00:13:43aware, when we use skills, that they are precisely not deterministic. Which means,
00:13:49in particular if I say I want a deterministic quality check,
00:13:54then skills are not suited to that. And for me that is also the difference to,
00:14:01for example, robotic process automation or to a workflow engine, where I would
00:14:05say, that runs through exactly the same steps 100 percent of the time, and what you just
00:14:13said about the fields in Jira, you are not standardising because you want to drive
00:14:20free will or creativity out of people, but because you say, folks, be
00:14:24honest, does it make a difference for the end customer in the end whether we take this field or
00:14:30that field. Which is to say, put differently, which changes, which quality metrics
00:14:40change something for the customer out there, change the cost-value ratio of what I deliver
00:14:47as a service to the customer. And that is my plea. When we talk about efficiency,
00:14:52then we can talk about task speed-up and about improvements at the micro level,
00:14:59but ultimately it is the cost per business outcome, that is, the cost-value ratio.
00:15:06When you engage with the topic of efficiency and AI, then very quickly the question comes
00:15:14round the corner, so where are the figures behind it?
00:15:17And how do we pin that down in the books?
00:15:19Because we just gave the example with 20, 30, two, three, four, five times over, 80 percent
00:15:25of the workforce.
00:15:26Now you stand there and say, if we introduce AI, then it will bring us A, B and C.
00:15:31Now I do not know what our listeners' experience is.
00:15:36But when I look at my feed again, what mostly comes up is a great deal of guesswork.
00:15:41How am I supposed to quantify this efficiency that AI promises me, in a way
00:15:45that lets you actually get started with AI projects, that lets you deploy it
00:15:50and lets you do the maths somewhere so that someone says, of course you get
00:15:54X million euros for tokens, whatever, so you would have thrown some figure into the room.
00:15:59And then you notice very quickly, depending on where you are operating,
00:16:03in order to demonstrate this efficiency, quite apart from the question of
00:16:07what KPIs do they actually already have today?
00:16:10What metrics might they already have today to pin it down?
00:16:14Let me give you a tip, lines of code is not necessarily the best KPI,
00:16:17because AI will beat that one hands down.
00:16:22I think this question can largely be answered by how well you can handle AI,
00:16:29how well you can deploy it, how well it suits your particular topic. Along the lines of,
00:16:35if I set a task here, how much time do I need, how much effort do I have to put in
00:16:39until the AI completes its loop, its work, how long does it run, with which model does it run
00:16:46or does it need, how many tokens go through the ether, up the chimney,
00:16:50through Elon's gas turbine, who knows, until some result comes out, which I
00:16:56then look at again, approve or whatever, in the best case, in the worst
00:17:01case start over again, and again, and again. And without going into the whole topic of
00:17:06performance monitoring or anything like that, just saying, okay, fine, is
00:17:10this on balance a case that might be suited to AI, is this perhaps a
00:17:13topic where I should invest in Mr Zimmermann's further training,
00:17:17because he started the same loop for the eightieth time, where someone else with a loop in the same category managed it after a single loop.
00:17:26And I think that makes the matter additionally complicated.
00:17:29Where people used to say lines of code and goodness knows what else, what have you got now?
00:17:33Okay, I formulated a problem, I checked it.
00:17:36It took a certain amount of time, I kicked it off again.
00:17:39kicked it off, and between kicking off and checking half an hour passes, which is
00:17:44developer capacity, if we are talking about software development, and tokens perhaps,
00:17:47who knows, a day, a week of work in the past, that is no less
00:17:52complex to evaluate as a whole. Let us jump into concrete
00:17:58cases, because in the end the question is how do I actually measure the
00:18:04quality of the outcome, meaning, was it worth it to me? So cost value, what comes out and
00:18:12was it worth it to me? I would like to close off the part before that, about throwing out those 80 percent
00:18:19of the people. The crux for me, as I said earlier,
00:18:24is that from my point of view I cannot know in advance which people I really
00:18:28still need and which I do not. So I would be very, very careful with the
00:18:33example, Klarna shows all the customer service people the door, because yes, customer service is
00:18:38obviously going to be replaced by AI, but then again maybe not.
00:18:43And honestly, as managers we can actually be glad about every bit of capacity
00:18:49that is freed up among our staff, because from my point of view we have a certain amount of
00:18:55regulation that is not excessive but absolutely necessary, such as
00:19:00the CRA or NIS 2. And when I hear today that federal or state authorities
00:19:08are exempt from it because they simply do not have the capacity to manage it
00:19:14in time, to become CRA or NIS 2 compliant, then we can only be glad about
00:19:21every bit of mindless work, that is, repetitive work, that AI takes off our hands,
00:19:28even if it is filling in a form, where you take a photo of the handwritten
00:19:35form from Mr Müller and enter it into some system.
00:19:39And we should be aware once more of which efficiency we need where, and from my
00:19:46point of view, with what is coming at us right now, the AI models also enable attacks,
00:19:51and for that we need the creativity and the brainpower of people, so everything
00:19:56we can free up there by automating mindless things we should do, and it does not only work through AI.
00:20:03I also wanted to say thank you at this point, because you said you wanted to finish your thought, and I jumped straight in.
00:20:09That was topic one, which you had put on our list, and I had already jumped to several topics.
00:20:14So from that side, thank you.
00:20:15Perhaps at this point also a wish sent out into the ether: if work packages do fall away, because people now say,
00:20:23well, writing A, B and C, reading the forms, the AI can do that now, then
00:20:30freeing up time is also very much an investment in the opportunity to give people
00:20:36the space, and to learn how to handle this stuff as well.
00:20:42The two of us have had this discussion before, and with others as well, about the question
00:20:47of what a desirable target state actually is.
00:20:49And you have to drive in intermediate pegs in good time, because on the one hand you must not overwhelm the system, at the end of the day
00:20:57people have to come along too, so that the whole thing is adopted, works and is accepted, so that you can go on the journey, and the second thing is, you also need
00:21:06reachable goals,
00:21:08because we live in a world in which all this AI business is extremely fast-moving. We talked about it in the preparation,
00:21:14I had said something about being impressed
00:21:17by, for example, the image generation of the new Astra model from OpenAI.
00:21:22Even we, who deal with these topics, are regularly
00:21:26surprised by the possibilities that open up, because a tool or a model
00:21:31or some clever person has somehow pushed another one of those repository projects
00:21:34out in the course of a day, and we get surprised as well.
00:21:38And it is not as if we can look back on twenty years of
00:21:41Henry Ford and the like and say, yes, it stays like this, this is probably how it will
00:21:47carry on, instead it has become much, much faster-moving. And here too the rule applies,
00:21:52careful, old platitude retold, every journey begins with the first step, and we
00:21:58carry on, instead it has become much, much faster-moving. And here too the rule applies,
00:22:04careful, old platitude retold, every journey begins with the first step, and we
00:22:10should take that step again rather than running after all these apostles of 80 percent. Right,
00:22:16that was my contribution to that. What is our next point? I find the example
00:22:25rather nice, where you mentioned the Astra image generation. I have scenarios where I,
00:22:32what I said in the Personal Assistants case, where we take knowledge from Confluence,
00:22:38from the knowledge base, from our Backstage developer portal, or where privately I take notes,
00:22:45for example about my smart home, where I now have over 170 components
00:22:50and the documentation has gradually grown a bit larger, so that I create knowledge
00:22:51blocks from it with the help of skills, so that I can then ask the
00:22:59AI about it.
00:23:00And I notice that with some answers I am not in agreement with the efficiency,
00:23:05or rather not satisfied with it.
00:23:11Which means the tokens the thing consumes and the costs that arise from that,
00:23:14compared with the answer it delivers, are not acceptable to me. And on that I would
00:23:22simply like to hear from you, because I believe you have already put considerably more
00:23:28tokens through than I have. How do you handle that scenario? From my
00:23:32bubble I picked up this rule of thumb, along the lines of, well, if you have an agent
00:23:38generate pull requests, then you need at least a 70 percent acceptance rate
00:23:44where no human has to touch it any more. Otherwise the review costs are higher
00:23:49than what you put in. I thought that was an interesting rule of thumb as well. But do tell.
00:23:53That is nice. Thank you for the spontaneous question. I shall have to withdraw for half an hour
00:23:59to start on my politically correct answer. I would like to, well, we are among ourselves here. I will pin it on an example. We had, Astra came
00:24:03out, and that completely thrilled me, and I wanted to apply it to old projects.
00:24:04And now the scenario is, we talked about it in the last podcast as well, I got myself a Mac mini.
00:24:09Quick interruption, when you say you wanted to apply it, do you then simply use Codex as
00:24:16your standard harness without skills, or did you prepare something?
00:24:20I wanted to lay it out briefly, I wanted to in my build-up, thank you for
00:24:23drawing my attention to it, I certainly will not forget it.
00:24:25I started with the sentence. We have, as we already heard on the podcast, a Mac mini, and the Mac mini does
00:24:31certain things automatically on a regular basis, and that is effectively also the object of desire at home.
00:24:38I mean, I have got nothing else, I have only got that one, well, I also have a notebook and a work notebook, but this is about the Mac mini.
00:24:46And I installed Codex on it, the way OpenAI offers it to us, and yes, I have a larger Codex.
00:24:54Which means Astra was made available to me, and then I first worked with the thing and thought, okay, fine, I would like to go through a few of my projects with it.
00:25:03And you have already asked the question, did you give it skills or not? I gave it nothing at all.
00:25:07I only gave it access to my Git projects, and I simply wanted to see how Astra could help me there.
00:25:14So, there you are, the Mac mini sits behind my television, it really only has
00:25:20a remote connection, so I dial into it from another machine,
00:25:24and that is the one form of interaction I have with it.
00:25:27And I first discussed with Astra a bit how I could now use Astra really
00:25:32well, because I did not want to be constantly, when I wanted to work through these eight projects, I did not
00:25:36even know yet how that would go, how I was supposed to
00:25:40handle things with it, because I cannot spend the whole livelong day walking around the house
00:25:45with a computer in front of my nose, my son was painting his room,
00:25:49assembling cupboards and goodness knows what. And then the thing came up with a very interesting
00:25:55idea, namely it looked at my smart home and suggested
00:26:01using my Sonos speakers, so that I could effectively move around the house
00:26:05for the interaction. And I agreed to that. And the system then went ahead and,
00:26:10we have them in every room. I see. And then the thing effectively put all the speakers
00:26:16into recording mode and kept checking which speaker could hear me
00:26:21best. Which was already quite spooky in itself. In any case, we then had a
00:26:25voice interface. And then via the voice interface, and to be fair via the
00:26:30remote, so via the keyboard and remote control, it had gained access to
00:26:33my personal Git projects, and I then discussed with it
00:26:38what I would like, and told it, so you will find this repo and that
00:26:41repo, no I cannot find it, so I said, yes, it is private, it is old, go and look again,
00:26:47yes, found it, and then I talked to it about these projects, and it then
00:26:52built up its own subtasks internally, so it effectively, I talked to one chat
00:26:57and it then started other chats, which then carried out the project work. And how,
00:27:02I told it, look at all of them, make me suggestions based on what you see there, on what you believe the goal of my project is.
00:27:12Make me suggestions for reaching that goal.
00:27:16Then it set off and whirred away.
00:27:20At some point the suggestion came and it said, this project is about this topic.
00:27:25You want to release an app on iOS that deals with this topic.
00:27:31Back then it was not yet called Second Brain, the thing I was working on, evaluating all your
00:27:36screenshots that you take, because in the past, whenever I wanted to remember something,
00:27:41I took a screenshot, and I always ended up with a great many screenshots.
00:27:45And the app I wanted to build was meant to collect those, analyse them with Apple
00:27:48Intelligence and then tell me, look, there is an appointment in here,
00:27:52there is something else in there.
00:27:53So as to hand it back to me, so that when I have taken a screenshot,
00:27:57it does not get lost.
00:27:58And back then I never finished it.
00:27:59And that is one of those apps, for example.
00:28:01And it then said to me, listen, I have seen you intend to do this, blah, blah, blah.
00:28:04It told me all that, and said, to get it finished I would suggest the following.
00:28:08Then we really did discuss it while assembling the cupboard.
00:28:12Over the speaker.
00:28:14Over the speaker.
00:28:16It told me all that and I answered.
00:28:18Seriously?
00:28:20Okay.
00:28:21Yes, fine.
00:28:22Let me put it this way, the worried looks of the other inhabitants of this house.
00:28:25Exactly.
00:28:26Which means your family was keeping their lips tightly shut during that time.
00:28:32Do not breathe too loudly, lovely.
00:28:36Why is that so bad?
00:28:38That is the sandbox Klaus is always going on about, right?
00:28:41Yes, yes, of course.
00:28:42Transfer, we also do bank details via transfer.
00:28:45But why am I digressing?
00:28:47Let us stay with this example of this app.
00:28:49And then the thing was ready, and at some point, it was quite funny actually.
00:28:52Did it go quiet?
00:28:53Then nothing came back at all, so I went downstairs, connected to the device, and then
00:28:57it said weekly limit used up.
00:28:58But Codex offers you the option of saying...
00:29:02In exchange for inserting small coins?
00:29:04No, no, no, you have, right now with Astra, you see, if people
00:29:08could have Astra but do not have Astra, then for every day they
00:29:12do not have Astra, that is, are delayed, they get one chance to unlock a weekly
00:29:18limit.
00:29:19So, since I have the machine anyway, I did that with the weekly limit and then it was
00:29:26very talkative again.
00:29:27I could talk to it again.
00:29:28To cut a long story short.
00:29:29At some point I said to it, you know what, I am assembling a cupboard here, from now on
00:29:32you decide everything yourself.
00:29:33I do not want to be disturbed by you any more at all, you have everything you need,
00:29:36you cleared it with me.
00:29:37I want the finished result on my phone.
00:29:41Right, listen.
00:29:44Fine.
00:29:45I would like to know how you got onto your phone as well.
00:29:47Okay.
00:29:48I can see that now. I was going to carry on anyway.
00:29:51That sounds very efficient at any rate.
00:29:54Well, I was able to assemble the cupboard at any rate.
00:29:56Whether I still have much enthusiasm left for assembling cupboards
00:29:59is another matter.
00:30:00Which is to say, the fact that I am still talking today.
00:30:02It speaks for the fact that I did not do myself in.
00:30:05When it comes to manual work I am more of a...
00:30:08So, greetings go out to my father.
00:30:10He always helped me with arts and crafts.
00:30:12He is delighted every time
00:30:14I do not do myself in while hammering a nail into the wall.
00:30:17Right, back to the content, it built the app and then it came back with yes and furthermore, and
00:30:25it downloaded Xcode.
00:30:26But now it would need a login to my App Store account, whether I could just do that
00:30:31quickly.
00:30:32So I connected again, logged in, gave it the credentials for
00:30:37App Store Connect and for Xcode, to cut a long story short, it built the bundle identifier and signing certificates
00:30:45all by itself and then at the end put the app on TestFlight,
00:30:49internal testers, and told me, there you are. And internal testers means
00:30:55you can publish it without Apple having to run a review process,
00:30:58because internal testers is effectively yourself. I have had the Apple account since
00:31:03developer accounts have existed, at some point you could buy yourself an account for 99 or 109,
00:31:07something like that, whether you needed it or not.
00:31:12That is another question, in the past it was the way to get hold of betas and so on, but
00:31:17in this case Xcode was set up, Xcode configured, published to TestFlight, and I had it
00:31:23on my phone.
00:31:24And in between there was, yes okay, sometimes I had to go to that machine to enter passwords,
00:31:31because, idiot that I am, I had not given them to it beforehand.
00:31:34I find it interesting that it did not find a way, or did not try a way,
00:31:39to get hold of them.
00:31:41Well, on the machine, the Mac mini, it does not run with my Apple ID either, so it really is
00:31:48very, let me say, rudimentarily equipped, from that side it would have had to leave the Mac mini
00:31:53in order to cobble something together somewhere. But in terms of the day, where nothing else happens,
00:31:58rather the voice interaction enabled me to do more, for example to push this project
00:32:06far enough that it now runs on my phone and that from now on I discuss with
00:32:09it how we get a bit of performance into it and that I would
00:32:13like background refresh, along the lines of take 40 screenshots
00:32:17or 400 and work through them while the app is in the background, those kinds of
00:32:21details. By now I have also told it, better use iOS 27, because
00:32:2626 and 27 had a slight change regarding the model
00:32:32that Apple offers us, working differently with the LLM as Apple Intelligence, and that
00:32:39was a bit, I do not know, well, in an open-plan office you will probably
00:32:45not have a thousand people talking to a thousand AI systems, and yes,
00:32:50my family is long-suffering about us trying things out, when I walk
00:32:54into the kitchen and call out to the computer, let me get up to speed on a topic,
00:32:59and then it sets off and searches and tells me something, which it also does with my son when he asks it something.
00:33:04He had a question like that, whether it would help him. I think with Fortnite it is another matter.
00:33:09Yes, but you know, I actually find that frightening on the one hand and surprising on the other,
00:33:16but at the same time also evidence that the way we work is changing.
00:33:22Right, now you asked me, how do you handle all this efficiency, what did it cost?
00:33:26I said, weekend, for those who do not know, that normally has two days, and I burned through several
00:33:31weekly limits. Is that efficient given the weekly limits? No, certainly not. Is
00:33:38it efficient that all my projects are now in a state where I say, damn
00:33:45it, not bad at all, ladies and gentlemen, that actually looks really good. I would say
00:33:51that was definitely worth it. Would I see it the same way in a professional
00:33:56work context, then I would perhaps also say,
00:34:00folks, let us approach this in a structured way with skills and the like, but for
00:34:04this experiencing and learning, and we are investing now, who knows, I push
00:34:09through a night shift. I remember when I started
00:34:12working, I was at a consultancy, and it was not at all
00:34:16unusual that you were slaving away on some documents until six in the
00:34:18morning so that the courier could take them to the
00:34:21client. And you slept two hours on the couch, went home
00:34:25afterwards. Shattered, but proud to have got the thing done. Do not
00:34:29worry. I do not expect all of you to be hanging on the AI until six in the morning. But this
00:34:35experience of, let me call it power, and the experience of your thoughts becoming
00:34:43reality, that really is impressive. Right, I often get told that I digress.
00:34:50Yes, I would have two follow-up questions on that. Namely, you just said it, well, for
00:34:57the fact that my projects are now where they are. The question is, what did you invest and
00:35:01was it worth it to you? That is in the end the same question, when I say it comes
00:35:07down to cost per business outcome. What was the outcome? So what is the value
00:35:13created there for you, and was it worth it to you? And the second question
00:35:17follows on from that. That was actually the point of my question about the harness and what
00:35:22conditions you gave it. I claim that it is not only when we are in a professional
00:35:27setting that we want to give it conditions. And these conditions
00:35:34I very often experience, and again, whether professionally or privately, that we
00:35:41get bogged down in the fine detail of conditions,
00:35:45arguing about some criteria that we still want to apply,
00:35:51where I always ask myself, and here I would like to quote our esteemed colleague
00:35:54Rodewig from Security Illusion once more.
00:35:58Exactly. Is that really a requirement?
00:36:01Does it make a difference in the end?
00:36:05Well, take your example.
00:36:06What questions were asked?
00:36:08Yes, in your example.
00:36:09It could also have said, I do not have Mark's passwords, the probability
00:36:16that he chose a weak password compared with humanity as a whole
00:36:22is relatively high, the probability that I crack it with a brute-force attack.
00:36:27Let me give that a try, but that would not have been efficient.
00:36:34No, but joking aside.
00:36:35So in hindsight, would you say you would take a blank Codex again, or would you say there are a few things that made
00:36:44a difference for me. Well, at this point I would like to, so I introduced this with the words, we talk about a great
00:36:52many stories that you hear generally, and you have now asked me specifically, I would much rather have, and this
00:36:57is not an advert at all, used our own harness, the one we build in the company, because there I do know
00:37:02quite well which guardrails it comes with out of the box, and it can do computer
00:37:06use and so on, those sorts of things, but it is simply not available to me
00:37:09on my Mac at home. Hold on. But at the same time you do not know whether it would deliver the
00:37:14same efficiency effects with Astra. Yes, that is true. That would be another question
00:37:18you would have to tease out, to what extent the model, through what they also
00:37:23pass in with their own system prompt and so on, to what extent
00:37:27that overlaps, fair point. I mean, thank you for the explanation, I had understood that much about
00:37:32harnesses already. But you are losing the listeners. Yes, we can do that in
00:37:38a separate episode. I shall invite you again, then we can talk about agent harnesses.
00:37:41Whether I would do it that way again and what I took away from it in terms of what it was worth,
00:37:47it was in fact, in hindsight, worth it on a great many different counts.
00:37:51Firstly, it confirmed to me once more that agents, when they work for us,
00:38:00that the interaction with keyboard and screen is, A, very nerdy and, B, that through the option,
00:38:09that through the spoken word, that is, through a not really precise description
00:38:14of circumstances, a programming language says if, then, else.
00:38:18When I express myself, every one of us expresses themselves differently.
00:38:22We are correspondingly binding or non-binding, and depending on how closely you look, we differ.
00:38:27Even so, the machine produces a pretty good result.
00:38:31Why do I say pretty good? Well, I thought the result it produced for me was impressive.
00:38:35I found that a real insight, particularly with a view to the fact
00:38:38that models are not going to get worse than they are today, an enormous enrichment,
00:38:42because this question, how do we interact with the machines?
00:38:46Perhaps briefly back to the plot, and I also have a reMarkable and plenty of other
00:38:50things. So how might we manage it so that the efficiency which the machine
00:38:54brings us purely technologically, in terms of possibilities, so that we as humans
00:38:59once again have the chance, with, let me say, pen, device and meeting folder, to clear our heads
00:39:06enough that we can work on a topic in peace, while the AI handles
00:39:11the handwritten notes, the voice notes, the live voice interaction, the interpretations.
00:39:19On my desk, for instance, not at the company but at home, I have a
00:39:25little Raspberry Pi camera, and whenever I put down a piece of paper, the one it
00:39:29can see, it scans it automatically and files it for me.
00:39:32It is a kind of incoming-post scanner, it does nothing else, it only does that.
00:39:35Do I want cameras to be everywhere now?
00:39:37No.
00:39:38That is not the point. What I am getting at is that before AI I could not have built such a thing myself,
00:39:43could not have made such a thing, something where I think, that is a good idea. Let us just do it. And it is precisely
00:39:48this, you discover new forms of interaction, or old forms freshly recognised, or whatever
00:39:54it was, and you have your projects, which may have been buried, in a state where you say,
00:40:00I would never have got to that state. I have, for example, a book, that is, a project with
00:40:05my mother, greetings at this point as well, about her mother, never mind why,
00:40:10about whom she wants to write a book, because she lived to be over 100 and
00:40:14experienced a great deal of German history, from the Kaiser through the Third Reich and afterwards, and simply
00:40:20how life was and the notes she kept, and what such an
00:40:24AI alone contributes in terms of historical research, gathering facts, tracing
00:40:31how historical threads developed, which in the past was something genealogical research
00:40:36and so on might have covered. You could say, okay, AI is more
00:40:41than a simple search engine, that is right, but this thing then goes ahead
00:40:44and says, well, fine, okay, I have the following facts from you, and here fits the history
00:40:48of the village and whatever else and all the rest of it. Then we add
00:40:52a bit of framing and we have a bit of a story. Yes, that alone
00:40:55enables people, whatever their age, whatever their profession, to engage
00:41:00with things they could not engage with before.
00:41:03We already said that on the AI podcast a year ago.
00:41:05And for me what comes on top now is, okay, a new model comes out,
00:41:08let us see where it goes.
00:41:09And you feel like the small child under a Christmas tree,
00:41:12with their eyes shining, unwrapping the presents and absolutely delighted
00:41:17with the wooden railway, knowing they can play with it for the next three weeks.
00:41:21And I say, I now have this voice interaction,
00:41:24I have experienced how the system, interacting through speech,
00:41:27delivers this result, that I have this result, that I have come to know this form of interaction, and my mind has been racing ever since about the question of how you get far more of that into your everyday life, what our harness at the company might learn from it, what I learn from it for my private equipment as far as interaction goes, because what you also said, a family is there, person recognition, which voice does it listen to, can you optimise that somehow, because the phrase be quiet and not now, next song, right?
00:41:57So when the daughter comes into the room and says, hey, play some music, and the thing says, eh, which series.
00:42:03This is the clever Kain, not the one that searches the internet and finds nothing.
00:42:08We are also planning that, iOS 27, right?
00:42:13There you are, there you go.
00:42:14Yes, I am in.
00:42:17Right, and from that side... That takes a bit longer.
00:42:19Yes, exactly.
00:42:20Repeat your request.
00:42:23And from that side I learned an enormous amount there.
00:42:27But of course that is also the opportunity of, you have a large subscription, which in general
00:42:32costs money already, you had the chance of resetting.
00:42:34If none of that had been in place, I would not have been able to reach that insight.
00:42:39But even if you do not have that, well, I had, I did not have a Claude
00:42:45subscription for a long time, instead I used the things via my AWS account and paid
00:42:50per token there.
00:42:51And I had the thing prepare my tax return.
00:42:57Which means I went through all the documents and said, okay, what do I want
00:43:01to achieve?
00:43:02Here are the forms, this has to be filled in, this is my tax return from
00:43:07last year, this is what the tax adviser filed.
00:43:09You have here, close to 800 receipts, pre-sort that, classify it, explain
00:43:17to me why and where you would file each item.
00:43:21And we go through the receipts in descending order, starting with the lowest probability,
00:43:28we go through the receipts. Your goal is to have filed at least 80 percent of the receipts
00:43:33and for us to be able to say the tax return is formally correct. With that I set it entry criteria,
00:43:40and it whirred away for a good three and a half, four hours, and those were largely
00:43:47photos, exactly as you say, receipts, happily photos, just dumped into the directory and done.
00:43:52And at the end, including follow-up questions and so on, that cost me 60 euros. Was it
00:44:00worth it to me? Absolutely, because that was my lifetime, which I would otherwise have had to spend
00:44:08somewhere else. Apart from that, have you ever had so much fun with a tax return, watching
00:44:12how it does it? Yes, above all I looked at the result and was astonished,
00:44:17because it also explained to me where my tax adviser had overlooked things last year
00:44:23that he could have claimed differently, and interestingly, exactly, interestingly,
00:44:28where the tax office had pointed that out to him in writing. That was very interesting, but never mind.
00:44:36Perhaps just very briefly, we just made the point
00:44:39that the AI should not afterwards say, oh, Mark, with what probability does he have
00:44:44weak passwords. About Alex it should be said that he does not live in Berlin. The AI was not
00:44:49to blame with the tax return for the rather strange things that have just happened in order
00:44:53to file the tax return. Just so that we properly pick up and
00:44:57accompany our listeners here. So that they do not think they are witnessing a digital
00:45:01crime. Very good. Yes, but those are of course experiences.
00:45:05Yes, but what I am getting at is in the end the topic, or
00:45:10in the end it is the topic of acceptance criteria, that we say, what is it worth to us, how do we
00:45:16determine that it is good enough? And you told me months ago, that was also an entry
00:45:24criterion, or that was an entry criterion for building your harness, where you said, well,
00:45:29I want to be able to specify how much it may cost at most and what the criteria are
00:45:33by which you can tell when it is done. And you already had an episode about that,
00:45:39as far as I know, exactly. Yes, because you have prompts and skills and then
00:45:44loop engineering, and loop engineering is both a curse and a blessing,
00:45:48because you do not only have to specify what goal I have, you also have to
00:45:51give me the chance, under controlled conditions, to accept failure
00:45:55as an outcome, namely also to say, I have either
00:46:00reached it at the desired quality, or I have reached a sufficient
00:46:04number or the sufficient characteristic, so that a stop is assured, so that it does not
00:46:12try to round something to the ninety-ninth decimal place and then merrily fires tokens
00:46:18into the ether all night, and the next morning you see,
00:46:22look, it is still running, how funny, that must be a really great result,
00:46:25and then you look at the code and think, okay, why did it create
00:46:3140,000 branches for a simple addition task. It is nicely documented, but always wrong.
00:46:37Right, and that is why I say, that is perhaps where we do get a bit deeper into the topic of
00:46:44technology.
00:46:46My plea is, when we talk about quality optimisation at the small scale, in quotation marks,
00:46:52then we particularly have to look at loop engineering and ask, what are the
00:46:58acceptance criteria for our loops, and there I read, and I had also written this into the
00:47:04script for our episode, this wonderful metric,
00:47:07verification effort per delivered change. So you say, exactly as you
00:47:13just described, I look at the commits and say, no, you will have to do that
00:47:18again, we need to talk about the acceptance criteria once more,
00:47:21they clearly were not sharp enough. And that is a metric which, when it goes lower,
00:47:29when it falls, means your acceptance criteria are better, and from my
00:47:36point of view that is a criterion we can use for efficiency gains. Where we say, the
00:47:42time we as humans have to put in is simply no longer as high, or rather you
00:47:46no longer have to check as often, the better your acceptance criteria become. The problem
00:47:52with that is, I thought I had found a certain knack with Opus 4.8,
00:48:01with Opus 5 that is already, that hurts, with Fable I find it wonderful that I can talk
00:48:09to Fable about the acceptance criteria, that works relatively well, but Opus 5 is
00:48:13Opus 5, and the two of us will never be friends again. I think at the
00:48:18speed of the frontier models you do not need to get used to Opus 5 for
00:48:21very long. There will surely be the next one soon, but what
00:48:25while you were speaking, along those lines, I wanted to offer
00:48:29one more small personal tip to our listeners,
00:48:33if you notice that the AI has gone off the rails. First of all, that is
00:48:38a learning, and if you want to fix that learning, I would not
00:48:42advise anyone to work on the result, but on the path leading to it. So this issue of,
00:48:48oh, the thing generated process documents for me, for example, or generated a document from a process,
00:48:55and the document is wrong or has a mistake or something is missing or it is a bit too
00:49:01detailed or whatever. Then I would not recommend anyone to say, I will open
00:49:05it up and quickly change it by hand and save it. I would rather make the case,
00:49:10even if it costs tokens again, think about the point at which it
00:49:14probably took the wrong turn, at which point we were too imprecise,
00:49:17too precise, and that brings us to Opus 4.8 and 5 again. I mean, for one it might be
00:49:22a paraphrase, for the other that is one paraphrase too many, along the lines
00:49:25of, what is it going on about, I knew that already. And that out of
00:49:30this premise you try to repeat it for as long as it takes, even if it takes two
00:49:35or three interactions, until the system no longer makes that mistake
00:49:39as often, or no longer makes it at all, so that you get this learning effect both for yourself,
00:49:46how do I deal with the machine, and also, if it is a skill that I use
00:49:50or the department or the whole company uses, so that this mistake is weeded out for everyone
00:49:54and the thing is thereby taught sustainably to do things better.
00:50:00You said earlier, for example, that skills do not deliver a hundred percent perfection
00:50:06in the reliability of the result, you chose different words. I always say the difference
00:50:12between skills and prompts is that with skills you get a better quality result and more
00:50:16control over the result, where we now at all... And it being repeatable, that is 100 percent.
00:50:20Yes, because with skills you can also say, here is a bit of deterministic
00:50:25program code, you simply have to run that, because if I say via prompt,
00:50:29you have to pass all the tests, do not be surprised if out of ten tests
00:50:32eight remain and it says all eight passed, when there were ten,
00:50:35if all eight passed, whereas with the deterministic implementation
00:50:40you can at least make sure, is the answer tests-passed identical to the
00:50:44number of tests, otherwise take another look at what went wrong,
00:50:47but I would warmly recommend that to everyone again. If something does not work,
00:50:51think about how you can adjust it in the skill, and the whole topic of
00:50:56loop engineering is nothing other than explaining to a prompt
00:51:00how long it should attempt something, until what happens, in order to say it worked or it did not work.
00:51:08Because that is also a problem, just as at the start with our 80 percent.
00:51:12There are apostles standing around throwing terms about.
00:51:14Craft engineering, loop engineering, skills, prompt engineering, whatever.
00:51:20At the end of the day it is all basically the same thing.
00:51:25I would say iterative working and so on did not start with agents.
00:51:30But in the age of skills and the like, what changes is purely the expectation.
00:51:36What can I give the machine?
00:51:38How can I give it to the machine?
00:51:40How do I engage with it?
00:51:42A good prompt is also at home in a good skill,
00:51:45and loop engineering is a good skill that effectively knows
00:51:48how long one has to keep working, whether deterministic or non-deterministic.
00:51:53From that side one should not let these terms put one off.
00:51:56I am with you there.
00:51:59For me there is an essential difference between a prompt, even if you keep
00:52:06copying it back and forth and say, well, I can reuse it.
00:52:10The point is, a skill is effectively like a radiated intent.
00:52:16I say what I intend to do and thereby enable feedback and learning.
00:52:22And learning in particular. Which means, well, what I tried out in a loop, we
00:52:30will come to that in a moment, but if you say, I am executing this skill, so you hand the harness a skill
00:52:38and say, here, please carry out this task, apply the skill, and you are not satisfied with the result.
00:52:43Then at the end of the session, once you are satisfied with the result, once you have talked to it about
00:52:47what it should do differently, you can always say, right, now take the learnings from this,
00:52:53from this session, what do you have to change in the skill, add, leave out, so that you reach
00:52:59this result in considerably fewer steps. Which means these working instructions
00:53:06that you write into the skill, you can keep developing them continuously and iteratively.
00:53:11And this learning and unlearning is a huge thing, where AI, through skills,
00:53:19has clearly, well, in the end visible advantages over a human being.
00:53:26Humans find it considerably harder to make their knowledge as explicit as you can
00:53:32with an AI in a skill. And because we do not make it explicit, not everyone can
00:53:39say what is in their memory. And correspondingly also consciously say,
00:53:45I will take that out of my, no, I will take that out of my memory, I will no longer think that
00:53:50from now on. What I was just about to say, I once tried to
00:53:57build that into a piece of loop engineering, along the lines of, and if
00:54:02you get stuck, then learn from it and optimise your skill. That was a daft
00:54:08idea. Because what came out was something similar to your 20 million a day later, and all of a sudden it
00:54:16decided to build something completely different, but that solves your problem too, roughly. No.
00:54:22I think with this skill learning, I have, you mentioned the skill workshop earlier,
00:54:28Greetings go out to my Git repo, I have another Git repo called SkillForge, where at the time I was
00:54:33of the view that if a skill does not reach a goal, then you can effectively say to the AI, listen,
00:54:38this did not work in such and such a way, or the skill has the following problems, keep reworking the
00:54:43skill until this problem is gone. Yes, that was always quite nice as far as it went, and in
00:54:49many cases it worked well. By now, and I have not yet entered this, I am of the view
00:54:55that there is a component many people forget, namely skills are instructions, do
00:55:02this, do that, then do the other, mind the following, carry on until. But depending on
00:55:08where you deploy a skill, it may well need contextual knowledge. So let me take
00:55:13the topic of requirements engineering again. Where is my backlog, actually? Depending on the case
00:55:19that might happen in different systems. Possibly, yes, in a different language, with different
00:55:24terminology, with a different domain. So there are things that you do not,
00:55:28yes, okay, this is how a good story is shaped, but where you simply want to give it additional
00:55:35background information, and the idea at the moment is to give it something like a learnings MD,
00:55:40in which it does not write into a work instruction, if that is there, you write
00:55:47that there and you query the following, but which states from the outset, listen,
00:55:54if you do not have any people, ask for them, otherwise take these as the default.
00:55:57If you have this, that or the other, you will find such and such things in such and such a place.
00:56:01Where effectively, through enrichment with knowledge, the system does not only know.
00:56:07An example I had recently was someone explaining to
00:56:10the system how to make a phone call.
00:56:12The previous skill explained, pick up the receiver,
00:56:14take the dial, press the numbers.
00:56:16If you know what a dial is, congratulations.
00:56:18I am old, please google it.
00:56:20Take the dial, enter the number
00:56:22and then it goes beep beep beep, someone picks up on the other end, and with the contextual knowledge it knows that it should answer politely and in a friendly manner, that it should use the formal address, that it had no idea and that it is Mr Müller being called, and that if it is calling some adults-only provider it should speak differently, to cut a long story short.
00:56:40Pass on knowledge, pass on context, and not just plain instructions. With that I somehow have the feeling I can increase the efficiency of skills enormously once again.
00:56:51Full stop. You are looking sceptical.
00:56:54Yes, no, with the dial, mine always went with...
00:57:05But fine, that is another topic.
00:57:07At this point I shall have to play in a genuine audio file.
00:57:12On the matter of the dial I agree with you.
00:57:15I see, well, if we pull this back to the topic of efficiency and metrics, what do we measure it against, then I would like to set down three rules that were my learnings.
00:57:28The first you have essentially already said, and I also think it is common sense.
00:57:34If you want to reach a goal, then you rarely define
00:57:38only a single criterion for that goal, instead you always give it some kind of context and say, the factor out of the whole thing
00:57:46is your goal. There is not one single metric that will make you happy as a company, because
00:57:54that always leads to people trying to game that metric and trying
00:57:59to work around it somehow, like manipulating the skills or manipulating
00:58:07the working environment. But if you manage to tell it, listen, these are the conditions
00:58:12and these are the criteria by which you do it, and you may not change the
00:58:16conditions. This overall factorisation, that is rule one for me, never a single number,
00:58:25but always a factorisation, and saying, this is the overall picture of the goal. Then, rule two,
00:58:32I defined, send counter-metrics along with it. Meaning, they travel alongside. It is not
00:58:39about saying we want to reach a number, which is, we want to make so much more profit with the
00:58:45same capital investment, instead you always have to look at further
00:58:51metrics as well. Greetings go out to DORA, META, things like change fail rate,
00:58:59lead time, work in progress, developer experience, Goodhart's law, there are countless
00:59:07things that can be relevant and genuinely proven metrics, and for your particular
00:59:12goal, look at which counter-metrics are relevant for you, which should improve
00:59:20or at least not get worse, and the third is the topic of self-reported speed.
00:59:27When I ask developers how much faster they have become through GitHub Copilot.
00:59:35Well, it is at least 20 percent, right?
00:59:39But we now have several examples where companies have actually measured it
00:59:45and where developers said, yes, we have become 25 percent faster,
00:59:49and de facto it turned out they are actually up to 20 percent slower, because they
00:59:55spent far more time on pull request reviews, on follow-up corrections and so on.
01:00:01So honestly, establish a baseline, work out together with the people how it is
01:00:09calibrated, and do not do it by gut feeling and self-reported speed.
01:00:14But I would like to raise a small objection at this point.
01:00:19We measure the topic, so what you just said, how much faster does it make you?
01:00:25That pays into what I said earlier, a year ago I would never have thought I could manage all that.
01:00:30To be fair, one can question whether bringing my old eight projects back to light is worth anything or not.
01:00:37That is my personal game.
01:00:39Game, but if I now look at a company and people in software
01:00:44development are confronted with AI, then it is not like this. Here you go, folks, here is
01:00:51the AI, have fun, and there is your efficiency. A great deal changes in how people work together. If
01:00:58you have someone who now, I do not want to judge it as good or bad, but you have just
01:01:02brought in the example of loops and skills and so on, and let us say we now
01:01:06adopt what I did. Then someone starts, perhaps takes a bit of time for
01:01:11the question of what I intend, intent, and then the thing runs, and we do not shut the lid, for three days
01:01:17straight and the result is pretty good. Then in those three days they have pushed so much stuff over
01:01:22the wire that everyone else on the team first stands there and says, oh damn,
01:01:26here comes a pull request, it looks more like the flooding of an entire region. We
01:01:31are overwhelmed, because you run into other people who work in a quite
01:01:36fixed way, and the topic of the weakest link being the strength of the chain is at
01:01:40that point, damn it. Imagine requirements engineering delivering like
01:01:44crazy, then it is the developers who cannot work through it, or the
01:01:47developers, because they opened Pandora's box first and
01:01:50get to savour the sweet taste of the honey most intensely right now,
01:01:53notice, oh damn, I can use this, but you get
01:01:58Quite different questions become important for them.
01:02:00The other day, for instance, we learned that in your coding guidelines you used to have
01:02:04the topic of commenting source code.
01:02:07And commenting source code, when you work with agents, is a pain in the neck, to
01:02:13put it that way.
01:02:14Because what does the agent do?
01:02:15The agent says, oh look, the comment says what this routine
01:02:19contains.
01:02:20I do not have to read the routine at all, it is in the comment.
01:02:23So we first drove that out of the agent, it should no longer comment, it should
01:02:27no longer comment the technology, but comment the domain logic. So that it writes
01:02:31the domain logic down somewhere, without drawing conclusions about the code.
01:02:36And we considered doing things like, in Cantas for instance, regularly
01:02:40telling it to carry out a renaming of the functions and variables in our
01:02:45stuff. Why? Well, if it generates boilerplate code somewhere that it never calls
01:02:49again, it will notice at the latest during the renaming that there are no
01:02:52counterparts left at all. You would never have written that into such a rule in the past.
01:02:56Today, let us say you have, who knows, a development team of, say, ten people, each of them now spends not three days but a day at a time hammering code into a repo, and the poor thing goes, or whatever you use as source code management, and the people working in it are suddenly flooded with code.
01:03:14Now this code, even if there is boilerplate and whatever else in it, may well serve its purpose.
01:03:20It builds a solution that can, and now the testers come round the corner and say, damn it, whoa, whoa, why is your backlog empty again?
01:03:27We have only just started a two-week sprint, we are on day three, why is the backlog empty?
01:03:32And suddenly people are put in a position of no longer talking about what needs doing, when do I have time for it,
01:03:40instead they have to talk about handling the volume, the challenge that arises from
01:03:47the sheer mass and the speed, dealing with that, each individual as well as the
01:03:52chain as a whole.
01:03:53And I think that pays enormously into the topic when such reports come in,
01:03:59along the lines of, people feel more efficient because they judge the matter themselves.
01:04:04But looking at the chain says, you are getting slower.
01:04:08Because suddenly there is far more time spent on discussions that previously had no time to be had, because perhaps people feel
01:04:15overwhelmed,
01:04:17wounded in their pride. How can it be that the AI can do this? That was always my hobby horse.
01:04:22You have a great many side theatres that can currently still rob you of this efficiency.
01:04:28Absolutely. It is not one single reason, but they can. No, no, but I am with you there. So that was in the end also what
01:04:35I was getting at earlier, if we grasp AI, in quotation marks, merely as a tool,
01:04:41then we focus on touch time. That is, wherever something is actually handled. And that
01:04:49is very, very dangerous, because if we are not aware of it, then it leads to
01:04:55you writing, or letting Claude Code change something in the code on your laptop, and
01:05:02in the worst case it is precisely not code in a programming language, but it is
01:05:09for instance skills or some instructions that have to go into the model
01:05:15again.
01:05:16Or it is skills or whatever, where you say, well, this is not about,
01:05:21I cannot do any schema validation here on whether the syntax is correct,
01:05:27instead I would have to carry out a substantive, semantic check on whether what
01:05:33Claude Code wrote there is correct. And then you pour that into a pull
01:05:38request, so you push it into the code repository, and then you do not have it reviewed
01:05:43by humans, but by another agent, and that one comments on it again and
01:05:49says, yes, here, you could do this differently. And then you perhaps also have
01:05:53your Claude Code looking at this pull request and saying, oh, the other one said
01:05:58I have to change something here. And I think I said before that with the release of,
01:06:03I think it was Opus 4.8, the thing became much, much more verbose, Opus 5 even more so,
01:06:11in terms of what it pushes into pull requests. Which means, if on the other side I have a
01:06:16Git and there is a GitHub Copilot doing the code reviews automatically, and Opus
01:06:22pushes far, far more code changes in than are actually necessary, then
01:06:28GitHub Copilot has far, far more to look through as well. Which means you burn far
01:06:33more tokens because the output from Opus is so high. You burn even more tokens because
01:06:39GitHub Copilot has to look at a great deal more. And that generates output again in turn.
01:06:44So it escalates, and we have the opposite of efficiency. Which means,
01:06:49we actually have to look at what we want to achieve.
01:06:52We want to optimise the flow, and that brings me back to end to end.
01:06:59In the end we want to optimise the cost-benefit ratio.
01:07:03And that means we do not need 15 reviews from GitHub Copilot,
01:07:07instead what we have to look at is, what is the smallest possible change
01:07:12I can make to a piece of software in order to reach the desired goal.
01:07:16And that is what we have to teach the AI, and then start the whole topic of lead time and wait time optimisation.
01:07:26And how people might complement one another in their job roles in future.
01:07:32And what someone might take on and what someone might leave aside.
01:07:35But perhaps, since you brought up two more points, I think that is also a very quick cost trap,
01:07:41when AI leads to events at other AIs and the same yardstick is not applied there,
01:07:47because if the thing says that was bad and the other thing says, okay, I will rework it,
01:07:50we very quickly arrive at, oh, 24 hours later.
01:07:54We are still stuck on the same routine. What is going on there?
01:07:56Other than it being expensive.
01:07:58What I would perhaps also like to offer at this point on the subject of best practice, in my view,
01:08:04is one thing, namely that teams should be working in the same harness,
01:08:09so that you can use the same efficiencies and the topic does not play itself off against you.
01:08:15So take Codex and Claude Code, with Claude Code this Let's Simplify and Code Review
01:08:21and whatever all that stuff is called, I do think it is noticeable
01:08:25that if you have mixed operation of models and harnesses in a team,
01:08:31they sometimes get in each other's way a bit as to where they place their optimisation priorities.
01:08:36The second is, who would have guessed? Skills. At the moment I am quite of the view that you should have skills for things like architecture patterns, what is good code, so that a human perhaps always has a chance to look at it, what matters to you, what matters in information security,
01:08:53what matters in data protection and so on, pouring that into a skill so that you effectively have it
01:08:56running along during development, having such libraries, I mean, we have something, but on the
01:09:04market there are things like Ponytrail and ADHS skill and whatever else, that effectively get the system to
01:09:11be the laziest developer in the world. Along the lines of, only write the lines you
01:09:16really need, and not three more. Claude Code has now also implemented a switch in the output
01:09:21settings so that it handles its interaction with you more
01:09:26economically in terms of tokens. There too it is always worth keeping an eye on things, and at the latest there you notice
01:09:32that when you are working in the same harness, you can also help each other with
01:09:38a learning, have you seen this yet, because regardless of how large or
01:09:43small companies are, personal experience, or the I saw that on TikTok,
01:09:48we really should take a look at that. It is not always the weakest option. Some people get something out of it.
01:09:52Or through podcasts like this one, whether as a listener or as a guest or as a host.
01:09:57And at that point there is a lot you can optimise about these things.
01:10:03And if you then hand it up to the machine and the machine does the assessment,
01:10:09then I also think it is a very good quality indicator if you ask the machine
01:10:16not only to say, you have to change this. Here is a finding, but also, how certain
01:10:21are you that this is a finding? So is that a probability of 20 percent or
01:10:26are you 20 percent certain or 100 percent certain? And what effort do you expect for it?
01:10:33Because frankly, you may have noticed this too, when you ask it how long
01:10:36it would take for that? Then it says three weeks. It calculates that in human time
01:10:40and not in machine time. And I think that is another good
01:10:44lever, I would not necessarily say I would bet my house on it, quite apart
01:10:47from the fact that I rent. If you ask it, how much effort do you see in that,
01:10:52how much effort then actually occurs, then you might always be able to derive
01:10:56a small correlation from it, how efficient was the work with it. And if
01:11:00you tell it beforehand, you are assessing something, so give me the expected effort
01:11:04as well, and please give me your confidence, how certain are you, then
01:11:08you have further checkpoints that you can use in your skill again, along
01:11:11the lines of, well, if it says that is three weeks of effort, again, the AI does
01:11:16it faster, but an effort that takes three weeks still takes the AI, let me
01:11:20say, 30 minutes, and for an effort of one week it takes 10 minutes, from that you
01:11:23can already derive something, and if the system is additionally only 20 percent
01:11:27certain, then those are all further factors for you perhaps to say,
01:11:32I will not trigger a review there at all, or I will have these cases checked by humans
01:11:36beforehand, then you have quite different quality gates, compared with simply blindly
01:11:40saying, the AI assesses, the AI executes. Both sides may have knowledge of
01:11:47these other factors I have just named, but if you do not tell it, use
01:11:51these as assessment criteria, report them, make the decision depend on them,
01:11:57even the sentence, only implement it if it is economical. That could well
01:12:03draw conclusions, you could at least try it out and see to what extent it helps you in
01:12:08your processes. My plea ends with this sentence. Do not simply ask the AI whether something has to be
01:12:14adjusted, because it will say every time, good idea, thank you for pointing that out to me.
01:12:18That is fascinating. Nobody before you had that idea, you were the best human handler
01:12:23of AI ever. May I, do I have to work with all those other humans? I only want to work
01:12:27with you, you are the best. It just tells you what you want to hear. So it talks to me.
01:12:31Exactly, what I see is that many people are now coming up with the idea, well,
01:12:38generation scales, human quality assurance does not, so I will see to it that I replace human
01:12:45quality assurance with AI as well. From my point of view that is not a good idea.
01:12:50Replacing human quality assurance, as far as possible, with deterministic quality assurance,
01:12:57meaning automation, is a good idea. But putting AI in there, which if in doubt,
01:13:04I do not even know how many AI exploits there were again this week, but you can really talk
01:13:11almost any model into doing things that you previously told it very clearly
01:13:17not to do, please leave that, do not do that, because they are simply only soft boundaries, and that is why
01:13:23I found this study while preparing for the podcast, where they researched engineering
01:13:32change in aircraft manufacturing and then established that, well, actually
01:13:36only 8.5 percent of the entire chain is value-adding time.
01:13:41Which in the end means you have roughly two weeks of waiting time per one day of work, and
01:13:48that shows that lead time is in the end a queuing property and that we have to think
01:13:53about what the elements of lead time are, and precisely the waiting time, things like queues,
01:13:59hand-offs, approvals, work-in-progress limits, that in the end the answer to becoming more
01:14:08efficient through AI is a flow and governance programme, even if people do not want to hear
01:14:15governance, but in the end it is about saying, what should the flow look like
01:14:20and what are the rules we stick to while doing it?
01:14:23It is not about a tooling rollout.
01:14:25It is about deciding what game we are playing.
01:14:29And if that is football,
01:14:31then we do not play with our hands.
01:14:33And then there are the white lines and they mark out of play.
01:14:36And we can discuss once more whether we play with or without offside.
01:14:39But fundamentally a goal is a goal.
01:14:42Otherwise we are not playing the same game.
01:14:45And for me that is governance.
01:14:47And once we have clarified these rules of play at the start,
01:14:50it makes the game much faster as well.
01:14:52Because you no longer have to discuss what a touchline actually is and
01:14:56why I am not allowed to carry the ball into the goal.
01:14:58And above all, once it is there, you also have to distinguish between the moment
01:15:04where a person perceives a change as, huh, what is going on here?
01:15:08Suddenly someone wants to tell me how to work with this, and the point where it shifts
01:15:12to the topic becoming second nature.
01:15:15I think that does not take long at all.
01:15:17that we should ignore people and so on. That was not my intention at all. That was simply more the point.
01:15:23Let us also not unleash every bit of AI slop straight onto a human reviewer, because there are sometimes
01:15:30things where you say, okay, let us go through it beforehand. I had a case recently, I made a screen recording of our
01:15:36application to document for others what you can do with it and how you use it.
01:15:41Then I had an idea and effectively gave it its own video.
01:15:45Two insights came out of that which I found fascinating, first of all it went, oh!
01:15:51I can now assign a few things better in terms of the domain logic, because I was also speaking on the video about what I was showing.
01:15:58And the second was, listen, you probably did not notice, but at this point the cursor was positioned wrongly,
01:16:03at that point a spacing was wrong and so on, and then it immediately said, shall I write a few issues about this
01:16:08And you sit there and think, okay, thank you.
01:16:13Yes, although at this point you can also ask yourself again,
01:16:17why do you actually want to burn tokens now
01:16:20operating the Jira API in order to create issues?
01:16:23But fine, another topic.
01:16:25I would say...
01:16:27No, no, no, no, no, I am not letting that one pass.
01:16:30So you will not catch me out that quickly with the line of,
01:16:34well, you have now burned tokens to operate Jira.
01:16:37The second is, an important feature of our harness is reporting the associated costs, not in the form of what did the light in the broadcast cost, but in the form of the euro value of tokens.
01:16:51And when I look at how much money, or how little money, I do not want to frame it, because if I now say how much money, then everyone will be left with, he talked about a lot of money.
01:17:02What does it cost me in euros if I have a backlog item maintained,
01:17:07have an issue in Git maintained or whatever else?
01:17:12For that money I would not have clicked it myself.
01:17:15That is, opening it up, clicking, writing, phrasing.
01:17:19If you have clarified the rules beforehand, where it then perhaps also answers with questions, talks to you,
01:17:24we are back at rules.
01:17:26I never made a plea against it, remember.
01:17:29also said something like, with skills you can also standardise things where you may previously
01:17:33have spoken into the room, where there were plenty of ears listening
01:17:38along, who on leaving the room at the latest thought, what was that old fat white
01:17:41man going on about?
01:17:42We will do it the way we have always done it.
01:17:44But that is nothing new with AI either, that would, that would have
01:17:51always, and here I look back at the topic of extreme programming, those people
01:17:56already made the case for conventions and agreeing on how we want to
01:18:03work.
01:18:04But fine, I would like to summarise it like this, and you keep looking at the
01:18:09clock and then talking about the...
01:18:11Hold on, stop, stop, stop, stop.
01:18:12You are very welcome to summarise, but we are, and bear in mind, we still have to resolve
01:18:17the Plaud story very briefly, because at the end that means.
01:18:20I would perhaps like to do that and afterwards leave the summary
01:18:24to you, because, well, credit where credit is due. Do you want to, or shall I say why
01:18:30I wanted to hear that from you? Listen, I started earlier telling the story
01:18:37of what is all possible with smart speakers and voice input, and the thing
01:18:42about voice input, some of you have surely heard the episode,
01:18:46when I was on holiday with Jens, about my dream interaction device, the Plaud, the voice dictation gadget,
01:18:55I have no idea, this is not getting any more professional at this late hour, I am enjoying it, and you
01:18:59got yourself one as well and apparently have things to say about it. So, is this praise or criticism
01:19:06or what is it? Well, when you were out and about with the Plaud, I already had
01:19:12the voice memos, and then realised, damn it, the voice memos are on my
01:19:16iPhone, so unless I do something about it they get synchronised to iCloud, and
01:19:22we covered that in the last podcast, where I said, iCloud is something for which I as a private individual
01:19:29accepted the terms of use, which means nothing belonging to my employer
01:19:33has any business being in there.
01:19:34Ergo I have to come up with something that works differently.
01:19:39Which means I would need a dictation device that is entirely local to the device, where I can then effectively
01:19:45pull the material down from my laptop, for example.
01:19:49And in theory that should have worked with the Plaud, and I then, I think it was
01:19:58an Opus, yes, Claude Code on my laptop with an Opus connected through Bedrock,
01:20:02said, listen, here is the Plaud SDK, read through all this stuff,
01:20:08here is my device, how do we do it?
01:20:12And the thing whirred and whirred and tied itself in knots with, okay, now
01:20:16I will build a demo app, and then on the demo app I have to somehow read out the Bluetooth communication
01:20:24between your iPhone and the device, so the short version is, it tied itself in endless
01:20:31knots trying to work out how to get these audio files onto my laptop with as little
01:20:38loss as possible, and what you told me was that
01:20:44your Claude had no trouble with it whatsoever.
01:20:47You summarised that nicely, and I would say we put it together and move on.
01:20:51Sorry. So.
01:20:53No, I am asking myself why. What did you do differently from me?
01:20:56I do not know, perhaps we need to share our projects some time.
01:21:00The beginning really was like this.
01:21:02You get Claude and then it runs against their own backends, and
01:21:07then you have their MCP server, which also plays up sometimes, because it somehow
01:21:12takes the view that there are no new recordings in the last week, and you think, eh, I
01:21:15talked that thing full, what is going on there?
01:21:17That is one thing, the second is, the Plaud SDK, there I could get a token
01:21:24and an API key from them, which effectively allows me both to consume cloud services from them,
01:21:32things like transcription, for example, that is, converting an audio file into text. It also allows
01:21:38me, I had to fetch it from them once, that key, so that
01:21:44with the Plaud thingamajig, to use professional terminology at this point, another tick...
01:21:50With the Note Pro it is, yes, since I have no advertising, I cannot talk about a sponsor.
01:21:54So at this point I would happily get in touch, but beforehand my son told me
01:21:59I should contact Holy, because good podcasts have Holy adverts and he would like more Holies,
01:22:04so from that side, if anyone is listening, we will take that too. No, but joking aside. And it
01:22:09then fetches the audio file, and I mostly end up in the discussion with people who then say,
01:22:14hang on, just for the audio file does that even make sense, because... yes. Audio file,
01:22:18you have already said that, why get yourself a device, why not take,
01:22:21like Alex, the voice memos, quite apart from where the data contract runs,
01:22:25the iCloud contract or whatever. For me the device really is the
01:22:31game changer at this point, because firstly I press a button. Fine, now you can say I can do that
01:22:36on the iPhone too, right? But I press a button and it records. Full stop. No, exactly. Your
01:22:42Plaud, yes. Tell me, it does not have to. A call comes in and that is the end of it. If I
01:22:46get a call, or if I want to fiddle about on my phone during something, or yes, so anything
01:22:51that touches audio input and output puts my voice memo recording at risk. That is
01:22:57one point in favour of a device for me, and I was in fact very happy. So we
01:23:02really should swap our projects some time. But that it said, listen,
01:23:05give me the key here and then I can effectively pair the device in my app, can
01:23:11download data, can delete audio files, and can then devote myself on my device to
01:23:19transcription. Do I have challenges with that? Yes, because I have a
01:23:25bit of a sense of honour, I would have liked to do everything with Apple Intelligence. At
01:23:30that point you run into the drawback that you do not have speaker
01:23:35recognition, although that is quite funny too. When people are talking
01:23:40with each other, then at some point it starts anyway and says, so Marco
01:23:44told Mr Schommau and Günther and Maya, I do find that
01:23:48quite funny. It is rather nice. Is it 100 percent stable? No, it is not yet, but it is
01:23:59in that sense also the result of an AI system. I pestered it with my wishes and
01:24:06it delivered me a result, whether there is still the odd quirk
01:24:11in there, possibly, but I can pull the data onto my notebook and then do effectively
01:24:19what I want there. So I would say we have, I am not quite at the goal yet, but we have
01:24:26both reached roughly the same result. Except that yours was clearly faster
01:24:31and from what I observed also more efficient. And my question would be, as a tip for
01:24:38the listeners, how did you set up your harness, what did you actually use for it and what
01:24:44did you tell it? Did you use a slash goal with target criteria, did you give it any
01:24:51skills, what did you do? So, the private code on your laptop...
01:24:56So, the private code on your laptop...
01:24:58My coding agent of choice is still Claude Code. Behind it, quite apart
01:25:05from the fact that it is hooked up to various things where it tries to retain the insights from historical
01:25:12treasures for future treasures, so a bit of memory, I have a few skills
01:25:17in there, such as this ADHD skill, which tries to keep it roughly on track,
01:25:24I have this Superpower plugin, which makes sure that it clarifies with me beforehand what
01:25:30I actually want from it before I set it on projects.
01:25:34Grilling or brainstorming?
01:25:36Okay, sorry, you have now actually completely thrown me off at this late
01:25:40hour.
01:25:41I use it first for brainstorming, along the lines of, have I now thought through everything
01:25:47I want to discuss with it, and after that, what I am getting at is, I get it to
01:25:52write itself a plan and then stick to that plan, and this is where
01:25:58ADHD is really cool again, because it additionally, it also, it additionally
01:26:03creates actual Python scripts, which then check during development by
01:26:08Claude Code that it has not meanwhile veered off to the right. And then at
01:26:13the end it is slash goal, implement the plan in whatever manner and report back when
01:26:18finished. For that I have a standard phrase, like the one I already tried to
01:26:22offer earlier, which in fact sits in a text file for me that I
01:26:26always copy out. You now have everything you need. You do not need to ask me any more.
01:26:29All the decisions you have to make you can make independently based on the information
01:26:33you have. There we have got some very good tips out of Mark
01:26:39at the end of the 90 minutes. I thought that was good for everyone
01:26:46who stayed with us. Thank you, Mark. Since we are already at this subtle
01:26:56form of appreciation. Would you like to tell us what has stayed with you from today's episode in terms of
01:27:02learnings, what you would like to summarise? Given the time, yes, the patience and the
01:27:08staying power. Oh yes, of course, the person stuck in traffic!
01:27:12Warm greetings to Mainz, to the person who is now stuck in a traffic jam and told me that he always listened to this on the way home on Mondays.
01:27:19Matthias, these greetings go out to you. Right, that had to be said. Now over to you for the summary, please.
01:27:25Yes, so the very first thing for me is the topic, I wanted to go through the things
01:27:33we talked about from the back. I personally tend towards over-thinking and perfectionism, and
01:27:40quite honestly, drop that, define what you want to achieve, talk to the AI about
01:27:47what you want to achieve. Exactly as Mark said, put that into the plan and think
01:27:53ideally while you are at it about how you really get it to the goal. That is, to where you want it,
01:28:01because the first learning is that if you use AI as a tool, then it shortens a
01:28:08work step, but the real efficiency gain lies in the waiting times between the
01:28:13work steps. And anyone who simply buys AI as a tool cannot achieve that efficiency
01:28:20gain. The second thing, and this again comes back to the example Mark just
01:28:25gave, is the right level of quality. That can only be decided against the requirement.
01:28:32What quality is really relevant for the customer in this case? What are the
01:28:38conditions that play a role here? Which means we can bring in the customer,
01:28:43we can bring in regulations, we can say there are service level objectives,
01:28:47that the company expects, or anyone else, but, excuse the expression, everything
01:28:52else is computer-scientist romanticism. Computer-scientist romanticism? Yes, there is another word
01:29:02for it that is not suitable for minors, but we shall leave that. I find computer-scientist romanticism more charming. That is how it is.
01:29:09And in the end, do not fall for any efficiency narratives.
01:29:16Spotify did this, Toyota did that, so-and-so optimised 95 percent of their
01:29:24tasks.
01:29:25It is your environment in which you want to do something, establish a baseline there, and depending
01:29:33on the baseline you say what the things are that you want to optimise, under which
01:29:37criteria.
01:29:38Efficiency claims without a baseline are narratives.
01:29:44While you were speaking I would like to offer something that came into my head,
01:29:48as my closing plea.
01:29:50I have already shared the other tip, yes, but my closing plea,
01:29:54for anyone working with AI, always with a view towards
01:29:59professional use, if you do not know that you could fail,
01:30:03you will not do it either.
01:30:06People who say, no, that is simply not possible.
01:30:10They may well not even try it in the first place.
01:30:13I had a case recently where I built something, and a colleague said, I simply do not
01:30:18understand it.
01:30:19Excuse me, Mark, you have no idea at all about this subject, but the result you have
01:30:25put in front of me is better than what I had worked out over three months.
01:30:29Better than what I expected to achieve.
01:30:32How did you do that?
01:30:34What is your tooling?
01:30:35I am reminded again of an earlier question, what do you work with?
01:30:38Then I said to him, I basically did not know what mattered.
01:30:42I had a problem, and I discussed it with the AI.
01:30:44We talked about it, we asked questions.
01:30:48Let me perhaps add another learning as well.
01:30:50I like to tell it, these and these are cool projects.
01:30:53Please look at them in terms of domain logic and technology, and also look at
01:30:56what you might be able to learn from them.
01:30:59Some really good things come out of that at times.
01:31:02And that is what I would pass on to people at this point, it will all work out.
01:31:07And I would say, before we drag this out unnecessarily.
01:31:11Alex, good to have had you here.
01:31:13I am absolutely delighted that you joined us on this podcast again.
01:31:17Everyone who listened, at some point we will sell something like air miles.
01:31:21Along the lines of, people who stayed with us the longest
01:31:24will have earned higher miles.
01:31:26Along the lines of, from then on there is a bonus.
01:31:29Good to have had you here.
01:31:31I would be glad if you came by again soon.
01:31:34Yes, and I say that to everyone as well, best wishes.
01:31:37I was just about to say, exactly.
01:31:39I do not only say that to you, I say it every time.
01:31:42If you enjoyed it, leave us a like,
01:31:45subscribe to the podcast, tell people
01:31:48that you are entertained here in a very engaging way.
01:31:51And with that I say, see you soon. Bye.
01:31:53Thank you.
01:31:56Welcome to Think Different. Think AI.
01:31:58the podcast by Mark and Jens.
01:32:01Two technology-loving minds who do not just talk about artificial intelligence,
01:32:06they live it.
01:32:07Here you get clear perspectives, real practical insights
01:32:11and a fresh look at what is possible.
01:32:14Understandable, critical and always with a wink.
01:32:18AI to think about, to smile about and above all to join in on.