NeoLoad and ChatGPT

Artificial Intelligence (AI) is coming, whether we like it or not. It is likely to transform every aspect of what it means to be human and represents an existential opportunity – or potential threat – to everyone living and yet to be born. Hyperbole aside, ChatGPT software intelligence has been a hot news topic of late. While it has some limitations, to many of us it’s the first real glimpse of AI’s capabilities.

What is ChatGPT?

ChatGPT is a long-form answering AI engine that produces human-readable text in response to questions. What makes it stand out is that it understands and remembers context, and the responses are easily understood by people regardless of their technological background. More specifically, it’s a large language model chatbot developed by OpenAI based on its GPT-3.5 neural network engine architecture.

Why is it relevant to NeoLoad?

Excellent question! One of the more common customer engagements that I support in my role here at Tricentis is assisting our customers as they implement NeoLoad into a DevOps toolchain. For many performance engineers, this presents an entirely new set of software solutions and scripting languages that they’re not familiar with.

There are a number of features built into NeoLoad that enable support for rapid automation, including creating YAML files. However, many performance engineers have never created or interacted with YAML before. Enter ChatGPT.

ChatGPT offers a glimpse of what truly intelligent software looks like. For example, if I ask it to write a NeoLoad YAML file to run a ten-user test against demowebshop.tricentis.com, it creates the following output:

In this scenario ChatGPT read my request, understood that I was interested in creating a performance scenario, took my time and user inputs as well as the website URL, and built a YAML file that I can immediately drop into NeoLoad and execute.

More than that, it looked at and understood the demowebshop.tricentis.com website. It built a full test scenario designed to navigate to the homepage, search for a product called test product, view the details on that product, add it to the cart, click the proceed to checkout button, fill in the billing information form with sample data, and then click the confirm order button.

On its own. Without any further prompting from me. With a helpful “Copy Code” button at the top of its response, in case I’d like to take and use that code myself.

It gets much better! What if, since this YAML file meets my needs, I want to include this in a Jenkins pipeline?

This lays out the complete series of steps to execute this using Jenkins pipeline code and the NeoLoad CLI. It does make a note to ensure the CLI is installed, however … so …

How does ChatGPT know all this?

The short answer is that it read our documentation, specifically the portions located on our public GitHub repositories.

The long answer is that the OpenAI team has spent a tremendous amount of time and effort to teach ChatGPT using human reinforcement. At the same time the team allowed it internet access to develop its ability to understand human speech and intuit – like you or I would – what is meant by a text prompt.

This seems too good to be true!

Because it is, for now.

ChatGPT is an amazing prototype of what these systems may turn into. It is capable of remarkable feats of linguistic appreciation. However, it is far from perfect, and this is the real danger in relying on it at the present time.

Don’t take my word for it – on the OpenAI website it explicitly lists limitations, like its ability to write plausible-sounding but incorrect answers. Let’s examine that install for the NeoLoad CLI for a moment. Here’s what the NeoLoad documentation on our GitHub page provides for NeoLoad CLI installation:

You might notice that this is a very different set of installation instructions than what ChatGPT provided above, and completely in line with the expressly published limitations that OpenAI themselves talk about when discussing how to use ChatGPT.

However, as a learning and assistive tool, it’s fantastically useful! ChatGPT can provide a lot of guidance and support to new team members as long as you check what it is saying and don’t just blindly accept it. It also provides the potential to automate many tedious tasks, or at least build 90% of what you require. From my own personal experience it works like many computer programs, and slight rephrasing of the prompts can lead ChatGPT to provide much more refined – and accurate – answers to your queries.

ChatGPT is open to anyone who signs up for a free OpenAI account.