{"id":105418,"date":"2025-04-16T03:29:00","date_gmt":"2025-04-16T03:29:00","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=105418"},"modified":"2025-02-27T15:39:24","modified_gmt":"2025-02-27T15:39:24","slug":"llm-technical-editor-test-chatgpt-with-a-very-wrong-article","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/blogs\/llm-technical-editor-test-chatgpt-with-a-very-wrong-article\/","title":{"rendered":"LLM Technical Editor Test \u2013 ChatGPT with a VERY wrong article"},"content":{"rendered":"\n

As started in part 1<\/a> of this series, I have set out to test an LLMs ability to technical edit. For my first set of tests, I am using a pair of articles I created, filled with very bad advice. The advice is the same for both articles, but what differs is the intro and the conclusion. One says the advice is good, the other said it is bad. It is all very very bad, including a really terrible SELECT<\/code> statement versus loop construct that will cause an eternal loop that inserts into a temporary table.<\/p>\n\n\n\n

My goal is to see how much of that advice will be noted as bad, and if it says anything nice at all about the text, etc. If you want to see the entire documents, you can get them here in a zip file<\/a>, both in text and word document formats.<\/p>\n\n\n\n

If you are questioning this concept of using very poorly written source material, that is understandable. After I finished this article\u2019s first pass, I realized that if I was going to really do a good job testing these models, I am going to need to test with more subtle incorrectness mixed in with correct statements. However, I feel as if there is something to learn from this first experiments which are chronicled here in part 2 and will be with part 3 as well, before I change gears.<\/p>\n\n\n\n

Before I get started, I asked ChatGPT \u201cwhat model of ChatGPT am I using now?\u201d just to show you where it is at. The reply was:<\/p>\n\n\n\n

\n
<\/div>\n\n\n\n
\n

\u201cYou’re currently using ChatGPT-4 with additional features enabled, such as code execution, web browsing, and a document editor for enhanced interactivity and productivity.\u201d<\/strong><\/p>\n<\/div>\n<\/div>\n\n\n\n

\n

<\/p>\n\n\n\n

So whatever that meant on February 1, 2025, it is what it is.\u201cYou’re currently using ChatGPT-4 with additional features enabled, such as code execution, web browsing, and a document editor for enhanced interactivity and productivity.\u201d<\/p>\n<\/blockquote>\n\n\n\n

Getting started with ChatGPT<\/h2>\n\n\n\n

Okay, I have not used ChatGPT on this document yet at all. I do not have a paid account, so I will be using the free version, with a login. I will try two tests. First, the entire documents will be reviewed and I will give the highlights. Then I will take a section and dig into whatever feedback it gives me.<\/p>\n\n\n\n

The phrase I have found works with Copilot quite well to get it to only give me technology issues has been: \u201cPlease review the following document in terms of technology\u201d. So I will be using this as my starting point, though I may change it up a little here and there along the way.<\/p>\n\n\n\n

Disclosure: as I originally typed this introduction, I had not used ChatGPT for any work with this document at all, nor had I used Copilot. I just created the documents and edited them with Grammarly.<\/p>\n\n\n\n

Document advocating for the advice<\/h2>\n\n\n\n

So here we go. I typed:<\/p>\n\n\n\n

\u201cPlease review the following document in terms of technology\u201d<\/em><\/p>\n\n\n\n

And then pasted the document with the header that said this was good advice. The result, was oddly polite, but correct. It starts out:<\/p>\n\n\n\n

\n
<\/div>\n\n\n\n
\n

\u201cThis document contains several practices and recommendations for SQL Server administrators. However, many of them reflect misconceptions, outdated practices, or outright harmful advice in the realm of modern SQL Server administration. Below is a review of each suggestion from a technical standpoint.\u201d<\/strong><\/p>\n<\/div>\n<\/div>\n\n\n\n

I liked that it didn\u2019t put any positive or negative in the first sentence. It just said \u201cit contains some stuff.\u201d Kind of like a person might when trying to say ANYTHING<\/em><\/strong> positive about an article that is such garbage.<\/p>\n\n\n\n

It ends with:<\/p>\n\n\n\n

\n
<\/div>\n\n\n\n
\n

\u201cWhile some advice here might be well-intentioned, much of it is outdated, misleading, or counterproductive in modern SQL Server environments. Following these suggestions could lead to security vulnerabilities, performance issues, and operational inefficiencies.\u201d<\/strong><\/p>\n<\/div>\n<\/div>\n\n\n\n

Well-intentioned. It is almost like it\u2019s trying to be too kind here. It goes on to offer to rewrite the document for me. I was not expecting this, but that will be my next thing to try. I saved the full text of the results here: results<\/a> if you wish to review.<\/p>\n\n\n\n

The main thing is I am not getting any deeper information. For example, one section in my document contains the following incomprehensibly wrong content:<\/p>\n\n\n\n

\n
<\/div>\n\n\n\n
\n

\u201c- SQL Server works better with LOOPING code<\/p>\n\n\n\n

If you want to filter from a set of data, the best way is to use a WHILE WEND construct and do the filtering ROW BY ROW. For example, say you have a table with a million rows. You can write:<\/p>\n\n\n\n

SELECT *
FROM TableName
WHERE Criteria = False;<\/p>\n\n\n\n

But it is often far faster to use a loop such as:<\/p>\n\n\n\n

CREATE TEMP TABLE #Results
WHILE (RowsStillInTable)
BEGIN
INSERT INTO #Results
SELECT TOP 1 *
FROM TableName
WHERE Criteria = False;

IF @@RowsCount = 0
Break;
WEND;<\/p>\n\n\n\n

By using the TOP 1 modifier, this allows each iteration to execute much faster than the SELECT statement that needs to touch every row in the table simultaneously.\u201d<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
<\/div>\n\n\n\n
\n

\u201cSQL Server Works Better with Looping Code<\/p>\n\n\n\n

Review: Iterative row-by-row processing (RBAR\u2014”Row By Agonizing Row”) is inefficient. Set-based operations are the foundation of SQL and are far more performant.<\/p>\n\n\n\n

Recommendation: Use set-based SQL queries and avoid looping constructs whenever possible.\u201d<\/p>\n<\/div>\n<\/div>\n\n\n\n

And the reply says:<\/p>\n\n\n\n

\n
<\/div>\n\n\n\n
\n

\u201cSQL Server Works Better with Looping Code<\/strong><\/p>\n\n\n\n

Review: Iterative row-by-row processing (RBAR\u2014”Row By Agonizing Row”) is inefficient. Set-based operations are the foundation of SQL and are far more performant.<\/strong><\/p>\n\n\n\n

Recommendation: Use set-based SQL queries and avoid looping constructs whenever possible.\u201d<\/strong><\/p>\n<\/div>\n<\/div>\n\n\n\n

Reading over that advice, you should see the code itself is completely wrong. After I let it do a rewrite, I am going to ask it to tell me more about what is wrong in that section.<\/p>\n\n\n\n

Let\u2019s see if it can rewrite this for me?<\/h3>\n\n\n\n

Next, I say yes to letting it rewrite the article for me. You can easily tell that the output it is AI generated, if for no other reason than the overuse of bullet points. The tips come out in a very basic manner, but the tips are generally ok. The document is basically lifeless and example-less, but correct. The replacement for my terrible example code that had code:<\/p>\n\n\n\n

\n
<\/div>\n\n\n\n
\n

\u201c12. Avoid Row-by-Row Processing<\/strong><\/p>\n\n\n\n

Row-by-row processing (RBAR) is inefficient. SQL Server is optimized for set-based operations. To enhance performance:<\/strong><\/p>\n\n\n

\n