Matt Hilbert

7 March 2023

Can ChatGPT be your next Technology Writer?

In the second of a series of articles about how ChatGPT and other AI tools like DALL-E will change the way we work with words, code and images, we give ChatGPT a job … as a Technology Writer. And we invite the new Bing to the party as well.

The now infamous AI chatbot, ChatGPT, is grabbing headlines and prompting a lot of chatter about how it will change the way blog posts, press releases, short stories, novels and film scripts are written.

Ask it to write an article, a lyric, a haiku, a story, and it will. In seconds. So why employ a writer when ChatGPT can do it all for you? No need for salary or benefits, or lunch breaks, or pesky holidays. Sit ChatGPT in a corner and wait for it to create a ton of content for you, for free.

All of which prompts a big question for me personally: Is ChatGPT after my job?

I’m a Technology Writer. I sit in a corner, writing press releases, articles, white papers, and blog posts for Redgate. It’s like playing chess with words each day. It’s fun. Every game of chess is different.

But with ChatGPT and now Bing, ‘Your AI-powered answer engine’, walking through the door and glancing in my direction, are things about to change?

Let’s put them to the test.

“Write a press release for me while I have a nap.”

Part of my job at Redgate is to write press releases, so why not make my life a lot easier and have ChatGPT or Bing write them for me?

Sounds good. I fired up the PC and logged into ChatGPT. In the first article in this series, I anthropomorphized ChatGPT as Charlie. Seeing as Charlie now wants my job, let’s continue being friendly. I asked Charlie to write a press release about Redgate opening a new office on the East Coast of the US. Before I could even sit back and take that nap, Charlie had written the following draft:

Oh. And oh again. If I show Charlie’s work to our Chief Revenue Officer, Cassi Roper, or Prashant, our new VP of Americas Sales, I’ll be out of a job. Charlie has decided Redgate’s new office will be based in Boston MA, the four paragraphs repeat a broad general message about expansion and support without actually saying why we’re opening a new office, and the ‘About Redgate Software’ section reads like it’s been written by … a robot … and there are several inaccuracies in there.

More importantly, who on earth is ‘Nina Johnson’, the Public Relations Manager named at the end of the press release? I didn’t even know Redgate employed anyone called Nina Johnson, never mind someone who is a PR Manager. Where did Charlie find her? I also tried to call the number Charlie provided, and it doesn’t exist … like ‘Nina’.

Never mind. I’ve got a backup. Enter the new kid on the block, ‘the new Bing’. It’s powered by ChatGPT but it does things a little differently. It’s also called ‘Sydney’ by the Bing team, so let’s stick with tradition.

I turned to Sydney and asked the same question. Sydney took a little longer, because Sydney ponders for a while before it responds. But sure enough, I soon had another press release, hot off the press:

Oh, and oh again. This one could actually get me fired. Prashant previously worked for IBM, Oracle and Micro Focus? Simon Galbraith is still the CEO of Redgate?

Just to be clear here. Prashant didn’t work for IBM, Oracle or Micro Focus. And while Simon was the Co-Founder of Redgate, he’s now a director and the CEO is Jakub Lamik.

I decided not to follow the advice of Charlie or Sydney. Instead, a press release with the title The Great Return Prompts Redgate Software to Open East Coast Office was issued through our newswire service.

“Find out the business benefits of database DevOps for me while I have a nap.”

Okay, so Charlie and Sydney are rubbish at writing press releases. I decided to make things a little easier by presuming I’d been asked to write an article about the business benefits of database DevOps. I still wanted that nap I was promised, so I asked Charlie and Sydney to find out what the benefits are.

I turned to my old friend Charlie first and, sure enough, I get the business benefits of database DevOps in seconds:

Not great, but not the car crash we saw with Charlie earlier. It was enough to get the ball rolling.

I brought in Sydney and asked the same question:

A slightly different response but better, and I liked the reference sources it provided. Redgate’s in there because we write a lot about database DevOps, by the way. What’s really interesting are the buttons on the bottom right of the screen, so you can dig a little deeper.

So both ChatGPT and Bing can be used to answer general business questions. What about deeper technology stuff? Could they help me there as well?

“Research how to stop a SQL injection attack while I have a nap.”

I’ll be honest here. I don’t know much about SQL injection attacks, but if I was asked to write an article or blog post about how to stop one, I’d do it. The old-fashioned way. You do the research, then you do some more research, then you confirm the research. And then you let the research tell the story.

But hang on – wouldn’t it be nice to get someone else to do all that heavy lifting of the research for me?

I asked Charlie how to stop a SQL injection attack:

I was pretty pleased with Charlie. I didn’t understand it all at first read, but I had something I could base an article on. That is, until I asked Redgate’s Security Lead to check it for me.

He wasn’t happy:

I would argue that #4, #5 and #7 are not valid ways to stop injection attacks at all:

• #4: While this limits the damage, it doesn’t do anything to stop the attack itself, and in most systems, even a least-privileged account would be able to cause extensive damage in most cases where an unmitigated SQL injection vulnerability existed.
• #5: Does almost nothing to help. If you can inject arbitrary SQL, you can most likely bypass most validation done in sprocs. You might get a slight win by being able to avoid granting INSERT / UPDATE / DELETE permissions on the data but wow, it’s a thin case at best.
• #7: Might help you detect it after the fact, but it does nothing to stop it happening – and in most cases, the damage is done…

Never mind. He didn’t know I had an ace up my sleeve. I brought in Sydney:

Once again, Sydney provided a shorter response, but with references and focused options to find out more. With those references, I was pretty sure Redgate’s Security Lead would be a lot happier.

He wasn’t:

The first two bullets are valid, and the fourth is potentially an option but error prone.

So not much use either. A start, perhaps, but it still left me in the dark about the correct way to stop a SQL injection attack.

Where does that leave us?

Like many people, I’ve been down the rabbit hole of finding out what ChatGPT and its cousin, Bing, can and can’t do. I like this Tweet from Mike Solomon on 14 February 2023, which summarizes it perfectly:

There’s good reason for that, which is the dataset ChatGPT, Bing and Bard, the upcoming new entrant on the scene from Google, all use. It’s not the world’s biggest library, The Library of Congress, where 1,350 kilometers of bookshelves hold more than 173 million items including books, manuscripts, sheet music, recordings, photographs and maps. And where thousands of new items are added each working day. Wow – imagine having access to that as a dataset.

Instead, rather than using a curated collection of edited and published material from reputable sources, the new chatbots just scraped the internet. The good bits, the average bits, the bad bits, all of the bits.

So we can’t blame Charlie for inventing Nina Johnson, a PR Manager at Redgate who doesn’t exist, or Sydney for making up a CV and not knowing who the CEO of Redgate is. They’re searching through a mass of data points, some true, some not, and computing answers based on probability and, as other commentators have observed, ‘hallucinating’ sometimes. Or, in more prosaic and honest language: ‘They make sh** up’.

So what do we do with ChatGPT and Bing?

It turns out that Charlie and Sydney aren’t the rock stars the headlines are making them out to be. Hold onto your horses, though, and don’t dismiss them. I actually like Charlie and Sydney because they both live up to the Microsoft promise of being ‘Your AI-powered answer engine’ and they can help you in a couple of neat ways:

If you want to research something, use them

Both Charlie and Sydney have a lot of knowledge under the hood that can be accessed quickly and easily, in seconds. So if you want to research a topic you’re unfamiliar with, they’re a good starting point. If you were researching the business benefits of database DevOps, for example, we’ve already seen how easy it is to find them.

This does come with a caveat, however, because the results are an amalgamation of what’s already been written. They don’t add anything new to the conversation, or provide unique insights. It gives you a baseline of what’s out there and levels the playing field for you. They’re also unreliable and you will need to fact-check everything so that you don’t repeat something they made up. Go technical, as we’ve also seen, and the unreliability of the results increases the deeper you dive.

If you find writing difficult, use them

I love writing, but some people find it difficult. If that’s you, get Charlie or Sydney to do the legwork for you. Rather than simply asking what the business benefits of database DevOps are, for example, you can ask for them to be provided in the form of an article by changing the question to something like: Write an article, with a headline, about the business benefits of database DevOps.

Enter that request into ChatGPT or Bing and you’ll have an article covering broadly the same points they produced to the previous request. The caveat about the results still applies, however. You will have an article, but it will be database-DevOps-word-soup, created from what’s been written before, and there will likely be some factual errors in there. It will also be the same bland content anyone can produce (and probably will).

It will miss out, for example, how database DevOps enables and improves the speed, quality and security of remote working. Along with how it helps to retain and attract staff because developers love DevOps. And the big one: it won’t mention how it standardizes workflows across multiple database types, whether on-premises or in the cloud.

It will score a C- at best, and C- doesn’t cut it because it doesn’t enlighten anyone. It simply adds to the mountain of click-bait clutter already out there. So use it as your starting point to provide a structure and the basis for a story, and then add to it, revise it, update it, improve it.

Finally, have fun

Do use Charlie and Sydney, but verify and fact-check what they produce. I know I’m repeating myself here, but I can’t repeat it enough. Treat them like unruly and often mischievous interns who don’t actually care whether what they provide is accurate or not.

Both are great at searching through the mass of content from the internet they’ve been given and providing a summary of what you’re looking for. But, like the internet, not all of the content they find is true or accurate and they deliver an algorithm-driven response based on the preponderance of data points they find. They don’t think, they compute, and compute doesn’t recognize the truth from an untruth.

They are, however, fun to work with. They make you think, they inspire ideas, and they’re a handy springboard to help with research and writing. They can also make you smile.

I asked Charlie one day to explain what database DevOps is. I write about it all the time, and I thought it would be interesting to find out the broad consensus view from all that Charlie could find in its vast database:

That’s actually not bad.

Then I asked it to explain what database DevOps is in the style of Alistair Cooke, the last true gentleman of BBC radio. He hosted the longest ever running radio program, Letter from America on the BBC World Service from 1946 to 2004. It was beautiful radio. A weekly fifteen-minute wry and perceptive insight into what was happening in US politics, media, sport and society, told with grace, humor, and style.

So let’s end with the way Charlie gives us the same explanation of database DevOps, in the style of the great man himself:

If you’d like to learn more about ChatGPT, read the first post in this series: How a Viking Princess defeated ChatGPT.