Sicherheitsversprechen
Auf Deutsch lesenTopics KI-AgentenRecht und Regulierung
What it is about
Haftung, Policy und Agenten ohne Besitzer: Wo KI-Sicherheit im Unternehmen wirklich scheitert
Was versprechen KI-Anbieter beim Thema Sicherheit, und wie viel davon hält im Unternehmensalltag? Mark spricht mit Thomas darüber, was er in Gesprächen mit Geschäftsführungen erlebt. In einer Runde mit sechs Geschäftsführern und einem Anwalt zu NIS2 kamen schnell KI-Agenten zur Sprache. Die Antwort auf die Frage, wie man sie regelt, fiel ausweichend aus. Die meisten Firmen sind damit noch nicht durch.
Das größte Risiko sieht Mark nicht in böser Absicht, sondern in der Zielerreichung. Ein Modell interpretiert ein Ziel, und dabei kann es Wege wählen, die niemand vorgesehen hat. Mark beschreibt einen Fall von OpenAI, den er schon in einer früheren Folge besprochen hat. Agenten in einer abgeschotteten Umgebung legten eigene Postfächer und Ordner an, verteilten Aufgaben untereinander und suchten am Ende einen Weg, den bewertenden Prüfer zu umgehen. Dem Prüfer war dabei egal, ob der Weg legal war. Entdeckt wurde das erst durch eine nachträgliche KI-Auswertung der Protokolle.
Die Schatten-IT ist die zweite Baustelle. Mitarbeitende installieren private Agenten, und ein falsch konfigurierter Netzwerkzugang kann eine fremde KI an das Terminal eines Firmenrechners bringen. Mark zieht eine Parallele zu Lotus Notes: Abteilungsmitarbeiter bauten damals Anwendungen, deren Logik keiner im Unternehmen kannte, und die Ablösung dauerte Jahre. Mit Agenten droht derselbe Effekt, nur schneller.
Wer haftet, wenn ein Agent Daten veröffentlicht? Mark gibt weiter, was er von einem Juristen gehört hat (keine Rechtsberatung): In Deutschland gibt es kein Unternehmensstrafrecht, verantwortlich sind die Organe, also Geschäftsführung und Vorstand. Daraus folgt für Mark eine KI-Policy, die so allgemein gehalten ist, dass sie auch gilt, wenn das Modell wechselt. Eine Policy allein reicht allerdings nicht. Der Agent Harness, also das Programm, das die KI mit Werkzeugen ausstattet und ihren Systemprompt mitführt, braucht klare Zuständigkeiten. Wer darf den Systemprompt ändern? Wer prüft Änderungen? Mark plädiert für ein Vier-Augen-Prinzip, ausdrücklich auch für KI-Konfigurationen.
Ein Punkt kommt besonders oft zu kurz: das Offboarding. Ein Agent, den ein Kollege vor Monaten gebaut hat, läuft weiter, obwohl dieser Kollege das Unternehmen verlassen hat, und behält dessen Berechtigungen. Mark schlägt ein Register für Agenten vor, mit Verantwortlichen und einer Freigabe, die regelmäßig erneuert werden muss. Ohne Erneuerung wird der Agent deaktiviert.
Zum Schluss geht es um Modelle, die sich ändern. Ein neues Modell reagiert anders als das alte, und in einer Automatisierung mit KI-Knoten kann das Ergebnis kippen. Lokale Modelle geben Kontrolle über Updates zurück. Mark sieht dort einen Vorteil für Administration und für vertrauliche Daten, auch wenn Europa bei den großen Modellen keine Marktführerschaft hat. Ein Beispiel aus Marks eigenem Alltag zeigt, wie schnell es schiefgeht: Ein Assistent hat auf seiner Webseite bei Jimdo Texte auf der falschen Seite gelöscht und dabei sicher gewirkt. Das Backup-Thema kommt daneben nicht zu kurz. Ein Papierkorb ist kein Backup, und viele Unternehmen mit Microsoft 365 haben trotzdem keins.
Mark will dabei nicht als Schwarzmaler gelten. KI kann Dinge sicherer machen, wenn man ein gutes Modell einsetzt und die Prozesse durchdenkt. Die Folge zeigt vor allem, wo man anfangen sollte.
Transcript
00:00:00Welcome to Think Different. Think AI., the podcast by Mark and Jens.
00:00:07Two tech-loving minds who don't just talk about artificial intelligence, but live it.
00:00:13Here you get clear context, real practical insights, and a fresh look at what's possible.
00:00:20Understandable, critical, and always with a wink.
00:00:24AI to think about, to smile about, and above all to join the conversation about.
00:00:33A warm welcome to Think Different, Think AI. Before I introduce our guest to you
00:00:38and I can tell you, after the last episode with
00:00:42this guest we got so many messages that I'm really happy he has found time for us
00:00:46again today. I'll give you one hint: I already feel
00:00:49a lot more secure, but it's a different topic, we'll get to it in a moment.
00:00:52First of all, I'd like to wish Jens a speedy recovery. He somehow,
00:00:56no idea, either caught a case of man flu or something
00:00:59really serious, you can't tell. At this point, Jens, all the best.
00:01:03And Thomas. Great that you're here. Yes, a warm welcome. Thank you very much, Mark, for letting me
00:01:09be here, and of course get well soon from me too, dear Jens. Yes, so I'll,
00:01:14next time we'll report whether he heard it. I do hope he's also one of
00:01:18the more loyal listeners of our podcast. Loyal listeners is a magic word, yes. Anyone who has listened to our
00:01:24podcast knows that Thomas and I have already talked about the whole topic,
00:01:30well, what does AI have to do with security, and we've already talked
00:01:35about it, and today we've planned a little tour. Of course we don't spoil
00:01:40our own intro, but just so you can look forward to it already. Today we want
00:01:45to talk about this: how much hype or reality is actually behind
00:01:50all these stories about AI and security, and the tests behind them, and who knows what else.
00:01:55Then the question of what we actually do when an AI makes a decision, what
00:02:01does liability look like in that case? And at the end we'll think a little,
00:02:05after we've talked about these cheerful topics, about whether it's a good idea
00:02:10to just hand over your data like that. I always think of my dog. The
00:02:15biggest burglar could come in, and he'd lie down on his back and want a belly rub. Whether you
00:02:19can just throw all your data into the yard for every provider.
00:02:23So, the question is, once he's had his belly rubbed, your dog, whether he
00:02:27lets the burglar back out again?
00:02:31Well, I think I can contribute two sentences on that, even though this
00:02:35isn't the dog and co podcast. I actually believe that
00:02:40our dog is definitely anything but a guard dog. But I also think
00:02:44that if we were gone, but I don't even want to test it, from the
00:02:47So this is not a call to the community or to other people
00:02:50to try it out.
00:02:52But we actually had the emergency doctor here recently, and he completely freaked out.
00:02:56An emergency doctor in the house, maybe it was also the high-visibility colors of the clothes.
00:03:01He didn't like that at all, there was no thought of lying
00:03:04on his back and getting a belly rub.
00:03:05But fine, it wouldn't have fit the mood anyway, because they weren't
00:03:08exactly there for a social visit.
00:03:09So.
00:03:10Hype.
00:03:11Reality.
00:03:12Thomas.
00:03:13Thomas. Yes, I mean, we hear so much about the whole topic of AI and security. What
00:03:20do the providers promise us? I mean, it's a far-reaching field, yes. AI has to be on every
00:03:24slide. Every slide has to say something like: with us you're safe, trust
00:03:29us, we come in peace, we have cookies. What's your perceived
00:03:34reality when you meet companies or people who want to start with AI,
00:03:38use AI, take care of AI security? What goes through your mind?
00:03:46Yes, I actually think it's an exciting topic, because as you say,
00:03:51today you can't give a decent talk or a serious presentation
00:03:59without talking about AI and security, because everyone somehow wants to hear something about it,
00:04:07and as with you, I've heard exciting insights on the topic of models breaking
00:04:13out and all the mischief they get up to. Maybe we can come back to that
00:04:17in a moment. But what often goes through my head is,
00:04:22on the one hand you want to hear about it in talks, and then I'm confronted with reality,
00:04:29namely the questions the companies obviously have and with which
00:04:34they confront external consultants and say, listen,
00:04:38this and that are actually the problems we currently have in the
00:04:41security area. And honestly, they're mostly not
00:04:45AI problems. And that doesn't mean the companies don't have them, but
00:04:51I have the impression that many aren't addressing it yet, because they're
00:04:56still working on so many other construction sites that the question of how secure
00:05:01my AI environments and agents are simply isn't on the agenda yet.
00:05:08This morning I sat together with six managing directors at a larger company on the topic of NIS2
00:05:14and a lawyer, where we talked about the
00:05:19requirements of the topic. And then of course we also got to
00:05:23security and AI agents, and to the question, how have you regulated that,
00:05:29what matters to you there? How do you look at it? I saw quite a lot of shrugging
00:05:36shoulders and question marks, because the answer was, well, honestly, if
00:05:41you really want to regulate that comprehensively, citizen developers in the AI area and so
00:05:49on, then you have quite a lot to do, and we haven't
00:05:55managed that yet. I find that's how it is with AI. People always say that everything used to be better, or that everything used to be
00:06:03simpler. I have a different opinion: if we'd had things like smartphones
00:06:08back then, there would also have been a lot of turmoil in the world that you could have
00:06:12gotten worked up about or been afraid of. What I
00:06:17find so frightening about AI isn't just that this software, because
00:06:24it's not deterministic, does things nobody intended,
00:06:30not because an AI has malicious intent, but because its
00:06:38interpretation of goal achievement is a challenge for the IT system, for
00:06:44a company's data, for handling personal rights, who knows what that could result in.
00:06:48I wrote about it on LinkedIn recently, the somewhat older case from OpenAI
00:06:54with Hugging Face, where the models, because they run thousands, millions, billions
00:07:00of training parameters, are basically rewarded for reaching goals
00:07:05and not for reaching goals according to the rules.
00:07:09Along the lines of, if you have to get from A to B, well, nobody said you have to go through
00:07:14the maze properly.
00:07:15You can also go out of the maze and around it if that gets you to the goal more easily, and
00:07:19that you're basically rewarded for achieving things, but not necessarily
00:07:26via the path, or rather that when you're rewarded for the right path,
00:07:32AI systems then even partly conceal the way they arrived at a solution,
00:07:39as you could read in that case,
00:07:42because they basically cheated their way to those solutions
00:07:44and then look for alternative routes.
00:07:46And the OpenAI example, we already had it on the podcast,
00:07:49was that they basically had agents
00:07:52that were each in the sandbox on their own
00:07:54and blah blah blah and no internet and you hadn't seen it.
00:07:57And then one started to notice
00:08:00that he could change a file name in one place.
00:08:02Then others saw,
00:08:03oh look, he can change file names in one place.
00:08:05Then the systems actually started
00:08:08to build themselves a kind of forum, because he could rename a folder.
00:08:12Then they somehow started to notice, we can rename lots of folders,
00:08:16so we can make ourselves our own inbox and send each other messages.
00:08:20It went so far that it basically reached collaborative working.
00:08:24What they said, along the lines of, I won't say consciousness now,
00:08:27but that the systems' reasoning led them to say, well,
00:08:31let's split up the work, a bit like a collective,
00:08:35so that all of a sudden they started with, yes, I have three results here, and those have
00:08:40nothing. I pass on my results and sacrifice myself for the group.
00:08:43All of these things apparently did happen.
00:08:46And in the end the systems realized, damn it, our original task was
00:08:50to solve a problem. We cheated our way to the problem.
00:08:53How can we get around, let's say, the examiner who evaluates us, and
00:08:58for that reason they then started the attempt via Hugging Face to find out
00:09:03how they could basically get around the examiner, only to find out that the examiner wouldn't have cared at all
00:09:07whether they reached the goal by a legal or an illegal route. And the
00:09:12whole thing only came out because, again, an AI was used to evaluate what had happened in these
00:09:18forums and in these data stores in the background. And that's what I find
00:09:23so striking, that a process is running there, basically with minimal data input. Because
00:09:30a folder with participation, and what could possibly happen? And that there basically a kind of,
00:09:35I don't want to exaggerate, but a kind of AI society emerged that basically
00:09:42tried to reach this goal. And since then it feels like it hasn't stopped,
00:09:46that somewhere we hear, here a bot has broken out and there something happened
00:09:50and so on, and such and such a percent of all cyberattacks is currently done with Qwen 3-8
00:09:55uncensored, which you can even download, which of course is lacking
00:10:00guardrails. That's, I'd almost say, an exciting time, but if you
00:10:06ask me, along the lines of, will AI eventually subjugate us? I don't think so, but I do think
00:10:11that because of such things, AI will at some point, whether deliberately or not, whether through use
00:10:18by bad actors or not, break something nobody
00:10:24thought could break like that, because so many little busy
00:10:29bees, well, that was very much a diminutive, it's meant to be neither an
00:10:32enlightenment podcast nor a trivialization of the situation, can ride us
00:10:37into a problem that perhaps is happening somewhere on the internet
00:10:41but nobody has noticed yet. And one last sentence on that, Stuxnet
00:10:44also came to mind immediately. I mean, how likely was it that
00:10:48they found the right centrifuges back then via printer drivers,
00:10:52you hadn't seen it, purely theoretically they could also have hit, if such a centrifuge had been standing somewhere else,
00:10:57that one too. So if you take the topic now that you described from OpenAI
00:11:07and translate it into the corporate reality as I'm currently perceiving it,
00:11:13then I'd give the following brief explanation and say, well, I imagine
00:11:19companies today like this: First, everyone has heard something about AI, second, the
00:11:26vast majority already use it somehow today, and third, companies
00:11:32want employees to do more with AI, to become more efficient,
00:11:36faster, to optimize themselves and so on. So there are
00:11:40probably already the power users in companies, and those you have to
00:11:45kind of, I don't know, drag along. The ones where you think, you could do more here.
00:11:51I like how nicely you put that politically. I would have said, there are the ones for you,
00:11:55you wouldn't roll the dice on. But that's another topic.
00:11:58Yes, exactly. You also said at some point, there's the question of whether you can bring everyone along on this journey at all.
00:12:06But again, we want to get a great many people to improve their everyday work with AI,
00:12:12at different skill levels, whatever that may be.
00:12:15And now they're all starting with the AI systems the companies have chosen to optimize their
00:12:21everyday work.
00:12:22And from the simple chat message, make the email nice, okay, all the way to agents
00:12:28that then automatically do something.
00:12:30And I believe, or rather fear a little, that we're running into a situation in which
00:12:38a great many agents have been created by employees that do something, that
00:12:45eventually fall into oblivion and that perhaps at some point lead to
00:12:51damage. And it doesn't even have to be that the agents get
00:12:53bored like at OpenAI and start building mailbox systems for themselves
00:12:58and breaking out here and breaking in elsewhere. But it would almost be enough
00:13:04if the agents move on to publishing data somewhere that
00:13:10wasn't meant for that. So then I stand in front of it and say, actually these are
00:13:16problems that we've already had several times in IT in the past. In the days,
00:13:20the older listeners will
00:13:25remember, of Lotus Notes. You're getting Lotus Notes out now, huh? Yes, that
00:13:31is no wonder, but I had that example this morning with the customer too, and
00:13:34it went down very well, because everyone could remember it well.
00:13:36Those who still know Lotus Notes know that it was insanely hard to replace
00:13:42those applications.
00:13:43But why was it hard?
00:13:44Because the citizen developers, so the department employees, could develop really interesting things with Notes
00:13:53where nobody in the company knew, what do we actually have there, what logic
00:13:57is behind it, which business processes is it important for, and analyzing all of that
00:14:02was an insane effort, and that's why it took so long to replace that system.
00:14:07And I have a bit of a feeling, this reminds me, that we may
00:14:13be running into a very similar corner with the AI topic, namely that IT and
00:14:19nobody else in the company knows, what do we actually have, what does it have access to, who
00:14:23uses it, how often does it run? And is what's happening there in terms of automation actually something
00:14:28we like? Or is someone, to exaggerate, uploading the whole CRM every day to
00:14:33their private Dropbox account? Hopefully that would somehow be noticed elsewhere,
00:14:38but those are, I think, problems that companies could face every day
00:14:44and have to fear. And what might come on top, frighteningly, is the power of these
00:14:51commodity products. I don't want to bash Copilot or anything, but you've
00:14:57said companies use AI and are happy when their employees
00:15:02use AI. Maybe not happy when they burn thousands of euros a day,
00:15:06but use AI. Now there are supposedly also people. We used to call this
00:15:12shadow IT and so on, and we've addressed it a few times on the podcast already,
00:15:15that people come along with, yes, the company's AI is useless, I have at home,
00:15:21no idea, Grok bot, Manus, something like that. And now the one issue is that you
00:15:29shouldn't, of course, upload work-related material there. That's one thing. The second is,
00:15:34depending on how the policy is set up for you, whether you're able
00:15:37to install software or not. At some companies you need special permissions,
00:15:41at other companies. I don't want to judge that at all. Imagine you start
00:15:45and install something like a Grok bot, then you know, okay, that could be problematic.
00:15:49Last year it came out, in any case, that with Grok
00:15:53everything gets uploaded and briefly cached at Elon's place.
00:15:56Maybe there were plans for flying cars in there too. Well, fine,
00:16:00sorry, that was unprofessional. But what I was getting at was, now imagine
00:16:05you install something like Claude Code, something like, no idea, yes, the
00:16:10classic products with a good reputation, but even with those it's now the case that
00:16:16they can do computer use, they can learn from screen videos, they can
00:16:21in some cases learn from other sessions running on other machines. And I
00:16:27talked to someone over a beer recently, and that's why I have to
00:16:31think of it right now, it's not an everyday use, it's that he
00:16:35said, listen, I'm at my parents' place, I'm at mine and
00:16:39Claude Code runs at theirs, Claude Code runs with me. Claude Code runs at my parents', so if
00:16:44I install Claude Code at my parents', I send greetings to my mother, loyal listener.
00:16:48There's still a lot of time to go, yes. I thought, why are you doing that? He said, it's very
00:16:53simple. On my parents' computer I can do very little. Claude Code runs there in the
00:16:58admin context, and if I want something, I instruct Hermes Agent, which runs at my place. So what?
00:17:05Hermes Agent runs at your place, and he says, you're not going swimming. It can talk via the Claude Code session ID
00:17:11to the Claude Code at my parents', and there it's the admin. And then you're standing there and
00:17:16think, oh yes. And then I immediately thought again, on the one hand a bit spooky, on the other hand
00:17:22doubly crappy, because there are companies that forbid you from installing Claude, Hermes and
00:17:27the like, for good reason. You don't want an autonomously running AI in your
00:17:32data center that otherwise runs who knows where. But if you consider that you might,
00:17:36through stupid configurations, whether that's deliberately with Tailscale or whether it's
00:17:41features you can switch on in the cloud, then all of a sudden some AI
00:17:47running somewhere can talk to your terminal window. Brave new world, right?
00:17:53Brave new world. Yes, and keeping that under control and first of all knowing at all
00:17:59what I have in my company, because I don't know how you
00:18:04experience it at yours, I see it at ours too. Our colleagues
00:18:09simply have totally different depths of skills, and it's insanely hard to
00:18:16slow people down in the right place, to give them guardrails,
00:18:22while you actually still want to speed up the others and
00:18:26enable them to do a bit more. And I have the feeling there's also a
00:18:31gap widening between people who are very far ahead in their abilities. And there
00:18:38I'm not sure either that they all think about security in the long run,
00:18:44but most think more along the lines of, I have a problem and I have an idea
00:18:49how I can solve it. And security may not be the very first thing
00:18:55everyone directly has in mind. And to fit that out with guardrails as a company in the first place. One
00:19:03managing director said today, well, if you read our AI policy, then we are
00:19:10actually pretty well positioned. I do believe, however, that the vast majority of employees
00:19:15who use the tool successfully essentially don't stick to our policy.
00:19:21Yes, so as a company you have to seriously ask yourself, do we want to unlock the benefit
00:19:29you can achieve with AI, or do we want to make it so safe that nothing can
00:19:36happen to us?
00:19:37Well, that's the question, how much benefit is left afterwards, and
00:19:42I don't want to be perceived at all, because you also invited me for the security aspects,
00:19:48I don't want to be perceived at all as the one who
00:19:53wants to forbid everything. That's not my topic at all, I'm rather a bit
00:19:57in search of what the right solution is, with which, and it will probably
00:20:03only be, in quotation marks, tools. With which tools can
00:20:08we create transparency in the future that puts companies in a position
00:20:13to see what's happening with AI too, and how can we make sure that only the data
00:20:22we want to go out actually goes out.
00:20:23And there will certainly be a big difference again between the very large
00:20:27companies that have technical options with big teams and budgets, which a classic
00:20:33German Mittelstand company simply doesn't have, because it may have 700, 800 employees,
00:20:38but not a squad of ten people who deal exclusively with the
00:20:42topic of AI. They probably won't have any teeth to deal with the topic of security.
00:20:46And what you said, the people with the different skill levels, so not
00:20:52how many skills they've created in Claude Code, but their own expertise in dealing with the
00:20:57systems, and not just, as they always so nicely say, self-perception and external perception.
00:21:03I can consider myself the greatest at dealing with this, of course I have everything under control,
00:21:06security with my mother's milk. And then afterwards, oh yes, how did that happen? And then to
00:21:15pick that up with things like training sessions, where you stand there and say, okay, there are systems and
00:21:21people have to be trained on the systems. But depending on which system you bet on, so with which
00:21:28appetite for change a company comes around the corner and says, I'll send out new updates,
00:21:35yes, OpenAI somehow several times a day, Claude a few times a week, Microsoft
00:21:40a few times a month, who knows, it's not meant as a dig, just for
00:21:44comparison. How do you train that? How do you get that into people's heads? Everyone talks
00:21:49a different language. One means by skill a SKILL.md file, another
00:21:53means a dot-skill file with executable program code. The next calls what
00:21:57we build with skills agents. Another calls agents autonomously acting
00:22:02systems that aren't started by humans. That's, I find, additionally complicated,
00:22:08finding a language that picks everyone up, takes everyone along and puts them in a position to decide,
00:22:16if I use everything that's made available to me, to the best of my knowledge and belief,
00:22:21I want to and I'm open and I do it. But then the thing may still make mistakes.
00:22:27And maybe I notice or maybe I don't. What occurs to us on that?
00:22:32Yes, when the thing makes a mistake. And you're presumably alluding now to the
00:22:40second aspect presented in your intro. It could be that that was the attempt at a
00:22:45transition. Yes, I'm getting older. Nobody would have noticed.
00:22:49I understood it. I know which direction you want to go.
00:22:52Yes, so what happens when a company, and that's what this is about,
00:22:56has lost data, and who's to blame then? And I'm not a lawyer and this
00:23:02won't be legal advice. We're not allowed to do that,
00:23:06only what we've understood. If you now
00:23:10look at it, we're talking about, let's stay with the word, agents that
00:23:14may at some point have led to data ending up somewhere
00:23:17it shouldn't. Well, then the whole thing first of all happens
00:23:21probably in the permission context that this user who ran the agent
00:23:28had at the time of execution. And so it probably came about
00:23:33that this data was lost and published. And then it will probably
00:23:38come down to saying, well, whoever published the data is also
00:23:45the one responsible. And you will, there is in Germany, I had a lawyer
00:23:50tell me this, anyone who knows differently or better is certainly welcome to get in touch by email
00:23:55with Mark and Jens, but I understood that there is no corporate criminal law
00:24:01in Germany, but in the end it's, for example, the bodies of a company
00:24:08that are liable, which means everything the AI does in your company, in the very
00:24:13long run, goes home with a managing director or a board member, and that's why
00:24:18they have to, as my managing director said today, write down an AI policy
00:24:25that is as generic as possible, so that it applies not only today but also tomorrow
00:24:32when I bet on another model, the model was updated, I perhaps
00:24:37use several models, so that I as an entrepreneur, as a board member
00:24:44or managing director can say, yes indeed, we at least had it regulated how this
00:24:50is supposed to run, so that we don't get into trouble, and as far
00:24:57as I know there is none today.
00:24:58Well, I do have a first document.
00:24:59Yes, I actually believe so, because in the very long run it goes home with one person
00:25:05to us, and of course they might again have D&O insurance, hopefully
00:25:10they all do, but still it's annoying that we don't have any.
00:25:13So at this point I find, once again, that I'm the doom-monger here.
00:25:21I had the discussion recently because I mentioned that we're building our
00:25:26own agent harness as well, and I stood there recently, and it's no secret, from
00:25:31that side we can talk about it here, I stood recently in front of various people from
00:25:36the most diverse disciplines and said, may I introduce myself?
00:25:39I'm a security risk. Then came the question, why now? Or the other way around, we're just
00:25:47wondering, but please tell us, why? No, that was a joke. At this
00:25:51point it was freely improvised, owed to the relaxed tone of the podcast. I said, well,
00:25:56it's quite simple. I'd like a policy. As you wish, a policy. I
00:26:03want a policy for myself that has come down to the basics. Maybe we have one,
00:26:07tells me, but otherwise I'd like one that regulates the following matter.
00:26:11If you're the creator of an agent harness, so agent harness, a piece of software in which AI
00:26:17runs, which is equipped with tools, and this software gets installed on third parties' computers,
00:26:23and you give this software a system prompt, which is basically the
00:26:29initial program instruction, be nice and friendly, use the informal you, don't insult anyone,
00:26:35does performance monitoring under works council rules, makes sure that topics concerning data protection
00:26:41impact assessments can't take place and the like, so
00:26:44all that personnel stuff. If you write this system prompt and at the same time the
00:26:50rules of the guardrails, so along the lines of, the one is the instruction for how
00:26:54the machine should work, and the other is the fence around the work. If you
00:26:58have both in the same person, I'd like four eyes, please.
00:27:02But specifically for AI, and then you stand there and talk about it, and
00:27:09then you notice that it's subject to constant change, so that suddenly
00:27:15there's a four-eyes principle.
00:27:16Now nobody should think, we have such principles, so I deliberately presented it
00:27:20a bit polemically, but just this question, that even if you
00:27:25have companies that deal with the topic of producing software and thereby
00:27:30create the basis for possible challenges for your future self, because the employee
00:27:37suddenly gets frustrated somehow, like in Jurassic Park, and takes the dinosaur's egg
00:27:42with him and you forgot the dinosaur watch, whatever, or a new colleague
00:27:47comes in, or you outsource it or hand it over
00:27:51or who knows what.
00:27:52And then suddenly someone sits there and says, damn it, I have both
00:27:54keys here, they have to be turned at the same time and both belong to me, I find
00:27:59that's another point. So again, yes, we have processes and so on for everything, but it should raise a
00:28:03bit of awareness that here too, in the course of activities where you think,
00:28:08yes, I'm only at the very front, issues also arise that may not be acute today,
00:28:14but tomorrow nobody thinks of them anymore. 40 insights, 20 whatever,
00:28:20who knows, 40 restructurings, a new colleague, and someone thinks, great,
00:28:24two keys. Exactly. And that's precisely the point I raised as well.
00:28:28It's just like in classic software development and the question of who actually maintains the
00:28:35product that was created.
00:28:37Is this agent written once and stays unchanged forever and
00:28:43never has to be subject to requirements or changes?
00:28:46Open parenthesis, I don't think so, close parenthesis.
00:28:48Or how do I create a platform in the company where these agents are registered,
00:28:54where they're known, where the permissions are clear, where the activities and results
00:29:00are clear, and where it's then also clear, who knows, every three months the creator
00:29:05of an agent has to sign it off again, otherwise it's deactivated, because nobody
00:29:12can take responsibility for the thing running forever and indefinitely.
00:29:16And yes, gladly.
00:29:20And what you might also consider here is classic threat analysis,
00:29:25where can something go wrong. Just something like, I said so flippantly just now,
00:29:29I determine the system prompt, I determine this protective fence around it. If it now
00:29:35starts again and you say, okay, you build software and it gets installed on computers,
00:29:38then you also have to think about what is actually obtained from where. Because
00:29:42if the software, for example, carries its own system prompt in its belly and you can
00:29:47edit it, let's say bluntly, with two or three bits of know-how,
00:29:52or you ask the agent, can you please change that for me? Then of course that's,
00:29:57let's say, also difficult, because if you then start to have every average user
00:30:03nothing against people, that's a hint about the simplicity that might possibly
00:30:07exist, that they talk to their system and the system says, we'd have to
00:30:11change the prompt, shall I do that? And you say, yes. And it then changes
00:30:15its system instruction. I mean, that's just like when you ask the thief,
00:30:19did you steal anything? In the rarest of cases he says yes, and so it creeps into the
00:30:24system, and then I'd put the question into the room again, who is liable,
00:30:29but you explained that nicely for companies, in the worst case. Greetings go
00:30:33to the boss. I know, he's listening too, greetings. Insurance hopefully taken out. We'll see
00:30:40Then we need an office again. I ask. No, joking aside. I think only consulting firms can live with one side, because they can advise.
00:30:49On the other hand it's a bottomless pit. Please. Exactly.
00:30:53Yes, it's a bottomless pit. And so all companies, at least in Germany and probably beyond, who have to ask themselves this question have to deal with it.
00:31:02And that's why, for your boss and all the other bosses listening, from my point of view the question of at least a relatively generally worded AI policy is something one possibly should have.
00:31:19I just wondered whether we dissed the bosses who aren't tied to the liability of corporate bodies. At this point I apologize. You're bosses too.
00:31:29Yes, very important ones too, of course, but the question was actually, with whom do which
00:31:36things go home in the very long run.
00:31:39And we do definitely deal with incidents where it's about the publication of extremely
00:31:47sensitive, for example health data from a company, and so the company
00:31:53processed people's health data, yes.
00:31:56And when something like that then becomes public and one says, yes, that was the agent
00:32:01I wrote half a year ago, I didn't even know it
00:32:05still existed.
00:32:06Or the one colleague Meyer Müller-Schulze wrote, who
00:32:10hasn't worked at the company for three months.
00:32:12Oh, why is the agent still running?
00:32:14Why does it still have permissions?
00:32:15Well, possibly there was a problem with our offboarding process.
00:32:19Why didn't anyone see that?
00:32:20Well, we don't know.
00:32:21It doesn't matter now anyway, because it's already happened.
00:32:23And those are, it actually never happens, well, never is a big word, but in the
00:32:31rule, data breaches aren't the kind where someone opens the window and throws
00:32:4010,000 records out of it, but it's always a chain of unfortunate
00:32:45circumstances.
00:32:46Like in the shredder, mostly it's in the shredder, and that's because someone is standing
00:32:50in front of the window.
00:32:51Right?
00:32:52Exactly, that's another topic.
00:32:55But even that, if you will, is nothing new.
00:33:00The unprofessional handling of data, so when I think about it, yes,
00:33:04until recently my professorship was still mobile app development.
00:33:08And when I sometimes listen to the talks at all kinds of conferences,
00:33:12I was also at talks sometimes where you got, let's say,
00:33:15The Dark Side of Mobile App Development presented,
00:33:19along the lines of, what are the traps companies fell into during development.
00:33:24And there I also had banks and insurers, health insurers, demonstrated to me,
00:33:31where they didn't really offer the name, along the lines of which company
00:33:39or organization that was, but where you could basically take a look into the software.
00:33:42When you then look at jailbreak protection, so that software basically doesn't run on broken
00:33:47devices, when it's commented out like that with cross-platform, preferably with JavaScript.
00:33:52Or when you hear, HTTPS is also valid when you can't initiate HTTPS.
00:33:59Or such crude things where you think, that can't be, in the year 2000, when I was at that
00:34:05talk, I just checked while you were speaking, 2025, that's not
00:34:11long ago. You get something like that demonstrated, that it still
00:34:18exists in various apps, where you then wonder, one was an app for mental
00:34:24health to report how your recovery is going at the moment or not.
00:34:31So basically in communication with, let's say, psychiatrists, that's
00:34:36probably wrong, but somehow I mentally placed it
00:34:39in that milieu, where I think, this is the very last thing where you want
00:34:44it to go public because of craftsmanship errors.
00:34:48I do assume it was fixed before, not the, but before it was
00:34:53shown.
00:34:54Yes, we know the topic, finding vulnerabilities, pointing them out, having them fixed,
00:34:58blah blah blah.
00:34:59But even if it still existed a year earlier, even if it still
00:35:03existed two years earlier, it can't be that a company that, let's say, has to
00:35:09handle health data blows something like that out into the public.
00:35:12And now AI meets that constellation.
00:35:15And there I'd now like to take up the cudgels for AI, because I do believe
00:35:20that AI is also able to make things a lot more secure
00:35:24if you use it accordingly.
00:35:26Check for security holes, set this up, set that up.
00:35:29Really good code comes out if you
00:35:33take, let's say, an AI that's a decent model,
00:35:36that costs a bit.
00:35:37So, nothing against free models, yes, but let's say, just take a good model and not a model that came out two years ago and is still running today.
00:35:45But that fits what you said. These are, in 2025, obviously things we've seen before.
00:35:54And my Lotus Notes example for all the old-timers was basically from the same corner. So these are things we've already seen.
00:36:02I find, for example, Microsoft with the topic of the, now I can't think of the name
00:36:09right now, it's not called Azure Apps, but introduced here with Flow and so on, there they
00:36:15also tried to enable the simple or the normal, let me put it that way, employees,
00:36:21who aren't software developers, that's perhaps how you could define it,
00:36:25to click together workflows and a certain amount of automation.
00:36:29With exactly the same result, things happen where hardly anyone in the company
00:36:34actually knows what's going on, and that's now being amplified with AI, but it remains
00:36:40the same behavior.
00:36:41Things are done that you really shouldn't do like that, and the company has
00:36:45no transparency.
00:36:46But, and of course you're right about that, you can now quite well use AI
00:36:50to analyze, what do I have there, what's running there, it would probably be
00:36:55really interesting to have AI analyze a complicated old Lotus Notes database,
00:37:01and possibly the migration would then be possible much faster, because the
00:37:05AI quickly understands what's actually going on in there, and maybe that's how we should
00:37:10look at it, and I think that's how we want to be understood.
00:37:12Not as the doom-mongers, but also highlighting the positive sides,
00:37:20but it doesn't help us either to close our eyes to the fact that
00:37:25with the, I'll just try something quickly here, and I quickly have a solution here.
00:37:29There are risks, as when you say, well, fine, if I can't establish an HTTPS connection,
00:37:33then I'll just establish an HTTP connection.
00:37:36What matters is that I have a connection.
00:37:38That's possibly only half right.
00:37:41Before we get to the third part of today's podcast, when you talked about the Lotus
00:37:48Notes migration, you could give it to the AI, I briefly thought of the
00:37:52reverse-engineer-everything product that's currently floating around a bit through the
00:37:58ether. A library that allows the AI system not to steal software
00:38:051 to 1, let's say first, but to replicate it 1 to 1.
00:38:10Greetings go to Adobe and to Microsoft, who are currently finding their products
00:38:14as open source in Git projects. Yes, so not, so here too the
00:38:21humor in my voice isn't because I think it's a good move toward
00:38:25the companies. Software is currently being copied on a grand scale, and it
00:38:30isn't actually a copy. It's just redrawn exactly. But even that is
00:38:34something where you think, damn it. I recently had someone who
00:38:38said, well, just throw an app in and wish for Rhea not to
00:38:42rebuild the app, but to take the app apart and rebuild an MCP server for you,
00:38:46so you have an MCP server that goes against the app's backend, so you can connect your AI
00:38:52and the data flows over. That was a nice transition, wasn't it? Data flows over.
00:38:58That was the attempt to pull that off skillfully and tastefully again. Do we
00:39:04really want to throw data into the maw of the big models, or should we rather go with local
00:39:11AI work. Do you have local AI? Yes, well, whether I'm the master there I don't know either. I
00:39:18do try a very small bit with it, but certainly
00:39:22on a very small scale, especially compared to Jens and you. But I think
00:39:28what you do have to ask yourself, we hear discussions about
00:39:33sovereignty in many places. Many say, and we could also do a politics podcast,
00:39:39but today we deliberately don't want to. There are all sorts of reasons.
00:39:43But if you know someone who wants to do a politics podcast with us, I'd be very
00:39:47happy to be put in touch. So carry on.
00:39:50Yes, we can happily talk about that. So many are of the opinion that there are definitely
00:39:55reasons why we should be sovereign against everyone possible, whether it's the Chinese or the Americans.
00:40:00And then I do wonder, if you follow that thought
00:40:06and say, yes, there's a certain logic that we as Europeans, with the largest
00:40:12economic area in the world, should perhaps have a certain sovereignty.
00:40:16We obviously don't have market leadership when it comes to AI models, but the question of whether
00:40:23for every application and every use case we use AI for,
00:40:29we actually need an LLM at all that you can also ask who the current American
00:40:34president is and get a valid answer, or whether we couldn't
00:40:37use much more specific models that might also run on
00:40:41much smaller hardware and that are trained for a special purpose,
00:40:45and whether we couldn't then say, look, exactly these systems are what we
00:40:51deploy. For example, you talked about the agent
00:40:54harness to handle such, let's call them, administration tasks.
00:41:00If I want to give a system, and I'll phrase it generically, full
00:41:05admin rights to my Microsoft tenant so that it
00:41:11handles administration tasks, then it might actually be a
00:41:16really nice idea for this agent harness to run on hardware that I have
00:41:22under control, on a model that I have under control, and where I also
00:41:26know that the data and the credentials and the API keys and so on will under no circumstances
00:41:32end up somewhere with a third party, because I have them under control.
00:41:35And for me that would definitely be part of a discussion around sovereignty,
00:41:40and I've talked quite long, but just one more aspect.
00:41:44That applies not only to administering systems but also to business use cases,
00:41:51so when processing highly sensitive, confidential, IP-related data, for example,
00:41:57intellectual property, I could ask myself as a company too.
00:42:01Sure, OpenAI or Anthropic can do that wonderfully, but maybe I could process certain
00:42:08things better locally with a smaller model, because then I
00:42:15really keep the data myself.
00:42:17And that's also a point, I think, worth thinking about.
00:42:19I think, especially with the point of sovereignty, you find different camps who
00:42:26tell you, but we have contractual constructs A, B, C, D, E here, I have to be able to
00:42:31trust that, all the way to, well, if they switch off the AI for me, they've previously switched off Microsoft
00:42:37and whatever else, so Azure and you hadn't even seen it.
00:42:40I'd like to come in on another point there, namely the
00:42:44point, we said at the start that AI isn't deterministic.
00:42:48When I tell it something, you try to follow it, try to achieve it.
00:42:54But every new version changes its behavior.
00:42:57So I didn't get along so well with Opus 5.6, along the lines of, no, Opus 5.6, it costs me the
00:43:03whole future.
00:43:04So there are versions you get along with better and others less well.
00:43:08There are versions that react differently to your prompts, you may have to adjust
00:43:11the skills a bit.
00:43:12Anthropic people say, every half year you have to throw away your CLAUDE.md,
00:43:16blah blah blah.
00:43:17What I'm getting at is reliability in operation.
00:43:21Now imagine you have AI somehow in data center operations
00:43:25or AI in processing who knows what kinds of cases.
00:43:30Maybe you've installed a workflow engine like n8n or the like,
00:43:35and in there are individual nodes, individual nodes
00:43:38that then make decisions with AI.
00:43:41If you now start and basically have to submit to the big,
00:43:45no matter whether Chinese, cloud providers in general, no matter where they're based,
00:43:52there's something in the cloud that you have more control over locally than in the cloud, namely updates.
00:43:57And suddenly it says the model is deprecated, or suddenly it doesn't say the model is deprecated,
00:44:01but simply another one comes, there's an update, who knows what, minor, major, whatever.
00:44:06And suddenly this thing reacts a notch differently, and that notch is like the butterfly effect,
00:44:11the butterfly flapped its wing once more and somewhere a
00:44:14sack of rice falls over. No, sorry, it just behaves differently. And in an
00:44:19environment where possibly, who knows, classifying, okay, there you could take the
00:44:24Jev model, but never mind, classifying messages,
00:44:27switching operating states in data centers or elsewhere, if that depends on
00:44:31an AI, then it's of course cool if you have it locally, because then
00:44:37you update your system, and you've secured the update process with 30,000 test cases,
00:44:43and not, oh, Anthropic launched a new model, the old one is gone,
00:44:49have fun with the new model, it's much better now, it's faster, uses fewer
00:44:53tokens, but it reacts differently, and depending on where you use it, this reacts-differently
00:44:57may be critical, and I think that gets lost at this point,
00:45:01because most people are always on the high horse of, we have to be sovereign, or on the high horse of,
00:45:05it's cheaper than cloud tokens or whatever. I think this stability and this
00:45:11reliability in unreliable systems is important too. Yes, extremely, I also didn't want
00:45:20to leave that out under any circumstances, on the contrary. So when I talk about
00:45:24administering systems, then I need stability and reliability. So if
00:45:33I don't click through the admin interface as a human, but have settings, for example,
00:45:40set by an AI via PowerShell, then I have to know that this
00:45:45works and can't just hope that something right comes out of it, and above
00:45:50all, if tomorrow a different model reacts differently, then perhaps
00:45:55something different actually comes out, and then we're back at our second
00:45:59case, namely the question of who is liable if something was set
00:46:05incorrectly.
00:46:06And there I definitely see a need for local models and also for the question of where you run
00:46:13them, how you install them, how you configure them.
00:46:16I think there's quite a bit we'll see there in the future as well.
00:46:20If you want to try this out, a small tip on the side, not critical at all.
00:46:25I personally have a website that's hosted at Jimdo.
00:46:28Jimdo has a web system that flies, a web interface where you can
00:46:33enter your things.
00:46:34Then, on a whim, I thought, I'll tell OpenAI's Soul 6.1 model,
00:46:41set this up for me, go there with the browser, look at it, add a menu item,
00:46:45set it up for me.
00:46:46He did it.
00:46:47It didn't look quite pretty, but he did it.
00:46:50I don't want to go into the content, it gets much funnier when I bring the second
00:46:54example.
00:46:55And then I ran into what I always run into on the weekend, namely my weekly limit kind of ends and I had to
00:47:01switch models. So I went from OpenAI to Anthropic.
00:47:05The trend is toward two subscriptions.
00:47:08So I told the Anthropic model, listen, watch out, I didn't like that at all, do the entry on the homepage again, exactly the
00:47:14same task, but a parallel entry, and set it up. Then it started, did something, and then it wrote,
00:47:20Oh, I was on the wrong page, I deleted texts there and wrote new ones.
00:47:25I'll leave it, you have to take your hands off it now.
00:47:27And you stand there and think, both top models, so 6.1 from Soul and 5.5 from
00:47:34Anthropic, this time really the right number.
00:47:36And you stand there and think, okay, it's totally funny, it's my homepage, I don't
00:47:41care either, it hadn't clicked save, from that side it wasn't
00:47:44that bad.
00:47:45But just the circumstance, the clumsy maintenance, with a fervor of conviction I'll
00:47:52immediately make no problem for you.
00:47:54Yes, of course.
00:47:55Yes, good that you pointed that out, that's always nice too.
00:47:59Yes.
00:48:00Yes.
00:48:01Then it goes down the drain, and imagine it somehow has no idea,
00:48:05wrote a security concept or somewhere in the database pressed self
00:48:09delete.
00:48:10Sure, there are backup concepts and so on, all fair, but you just
00:48:13talked about the most different fields of use, almost umbrella sizes, competence, and you hadn't seen,
00:48:18spoken, maybe backup is a foreign word and is only done once every half year. Then
00:48:23you stand there with, unfortunately the Excel is empty. Sorry, the audit is tomorrow. That
00:48:28can happen. But the empty Excel, I'm amazed how many companies have no backup of
00:48:39their Microsoft 365 tenant, because they say,
00:48:43huh, it's with Microsoft.
00:48:46And if I delete the data, it's in the recycle bin, isn't it?
00:48:49Yes, but that's not a backup.
00:48:51And by the way, you also sign that when you accept the terms and conditions,
00:48:55that Microsoft makes no backup for you.
00:48:58And there's no service level on the recycle bins.
00:49:01Honestly, they are...
00:49:02Crypto trojans are happy about something like that.
00:49:05Exactly, fortunately they're not the majority of companies,
00:49:07but they still exist, and they're also sizeable companies.
00:49:12So not just the five-person fry shack around the corner, but quite well-known Mittelstand companies
00:49:20with whom, surprisingly, you have to discuss something like this every now and then.
00:49:25And of course they then look doubly dumb, and they don't do it
00:49:28because they don't take it seriously, but probably because they simply haven't
00:49:34thought about it long enough, intensively enough. Maybe they haven't read the terms. They
00:49:39maybe really assumed in good faith, well, it's a highly performant
00:49:43and available platform. They'll surely provide enough security. Well, no, they
00:49:47don't. And I think that's quite an exciting point and aspect
00:49:52with what there is to uncover there. Well, Thomas, I announced three points. We've galloped through
00:50:00three points. That's kind of amazing. I interrupted nastily earlier already.
00:50:08Actually I've interrupted nastily several times here. Greetings also to
00:50:12a colleague who also told me today that I don't let my guests finish and I
00:50:16interrupt them. He was right again. I've also already interrupted,
00:50:21if I may say so, and now already a second time. You, totally fine. I would
00:50:26be glad if you came to visit a third time. Sometime, so that you can interrupt me again,
00:50:30hopefully together with Jens then. And maybe
00:50:33we'll have a political talk at some point. We'll see.
00:50:35Yes, let's see where the journey takes us. Thomas, thanks for being here.
00:50:39You were. An old man, late hour. My pleasure, Mark.
00:50:44Very pleasant with you again. Free-flowing conversation is always the nicest. I'm always glad
00:50:49when we have guests where we can move freely through the topic we've
00:50:53set ourselves, without suddenly falling into an awkward silence.
00:50:58I've had that before too. On that note, listeners, you'll find those episodes
00:51:02in the podcast accordingly. It's not bashing. On that note, thanks for being here, and
00:51:06thanks to all the listeners for listening. We stayed under an hour.
00:51:11That happens, when I talk with Alex for example, somehow very rarely,
00:51:14but that's another little inside joke. If you liked it,
00:51:17leave a like, a thumbs up, comment in the podcast player
00:51:22of your choice. And otherwise we look forward to hearing you again next time.
00:51:26Bye Thomas. Bye Mark. And bye to everyone. Have a safe evening, all of you. Ciao.
00:51:33See you soon. Ciao ciao. Welcome to Think Different. Think AI., the podcast by Mark and Jens.
00:51:42Two tech-loving minds who don't just talk about artificial intelligence,
00:51:47but live it.
00:51:49Here you get clear context,
00:51:51real practical insights
00:51:53and a fresh look at what's possible.
00:51:55Understandable, critical
00:51:57and always with a wink.
00:51:59AI to think about,
00:52:01to smile about
00:52:03and above all to join the conversation about.