Can ChatGPT solve Cypress Quiz?

Recently, I launched the Cypress Quiz, a fun and challenging way to test your knowledge of Cypress. If you haven’t heard about it yet, I encourage you to give it a shot before diving into this article. Meanwhile, OpenAI introduced GPT-4o, their newest flagship model. As a big fan of AI technology and testing tools, I decided to see how ChatGPT fares with the questions from the Cypress Quiz.

This brings me to an interesting experiment: Can ChatGPT, with its advanced language processing capabilities, handle the complexities of Cypress testing? And more importantly, should QA engineers be worried about AI taking over their jobs? To explore these questions, we’ll test ChatGPT with a series of questions from the Cypress Quiz and analyze its answers. Let’s dive in and see what happens.

First prompt for ChatGPT:
“Act as an experienced Cypress user and solve the Cypress Quiz that consists of 14 questions. All questions will be provided as screenshots. Answer in the following format: Explanation and Answer. First question is attached to this message.”

Question #1

question #1

ChatGPT explanation:
In Cypress, .should() is used for assertions and retries until the assertion passes, whereas .then() is a simple callback that executes immediately once the command chain is complete.

In test #1, the use of cy.wrap('foo').should((foo) => cy.log(foo)); will fail because should expects a condition to be true and continuously retries until the condition is met. However, cy.log(foo) doesn't return a value that can be asserted, thus it will not satisfy the should condition and the test will fail.

In test #2, cy.wrap('foo').then((foo) => cy.log(foo)); will successfully log 'foo' because then does not require any condition to be met and simply executes the function provided.

ChatGPT answer: D — Correct ✅

Even though ChatGPT answers correctly, the explanation is not. The test#1 will fail because you cannot invoke Cypress commands inside of a .should() callback function. In the screenshot, you can see that test #1 invokes the .log() command. However, ChatGPT incorrectly predicts failure due to an expectation for a condition to be true, not because of the actual reason related to Cypress command usage. Well, ChatGPT can be helpful, it can also make mistakes. Always verify crucial information, and refer to the official documentation as the most reliable source.

Question #2

question #2

ChatGPT explanation:
The test visits two different URLs in succession. The second cy.visit('https://cypressquiz.com/') will overwrite the first visit to https://cypress.io. Therefore, the assertion will be performed on the content of https://cypressquiz.com, and it will check if the <h1> element contains the text 'Quiz'.

Since cypressquiz.com does contain the text 'Quiz' in the <h1> element, the test should pass.

ChatGPT answer: A — Incorrect ❌

This answer surprised me because it overlooks a well-known limitation of Cypress: handling cross-origin. Cypress can not visit two different superdomains in one test without using the .origin() command. Therefore, both the explanation and the answer are incorrect. Also, ChatGPT incorrectly assumes that the page at https://cypressquiz.com/ does not contain the text Quiz in the header element. In fact, the header element is <h1>Cypress Quiz</h1>, which clearly contains the text Quiz. This highlights the importance of verifying AI-generated responses against reliable sources and known facts, particularly when dealing with specific technical limitations. Read more about Cross-Origin testing.

Question #3

question #3

ChatGPT explanation:
In JavaScript, accessing an array with a negative index returns undefined because negative indices are not valid. Therefore, cy.wrap(['JS', 'TS']).its(-1) will result in undefined.

The assertion should('eq', 'TS') is expecting the value at the negative index to be 'TS', which it will not be, leading to an assertion error.

ChatGPT answer: C — Correct ✅

This question isn’t really challenging, as it’s more focused on JavaScript than Cypress. However, ChatGPT performed well here, providing both a correct answer and a solid explanation. There’s nothing more to add — ChatGPT got this one right. Let’s move on.

Question #4

question #4

ChatGPT explanation:
In this test, cy.wrap('foo') wraps the string foo and passes it through the Cypress command chain. The .should() method is used for assertions and retries the provided function until it passes. In the assertion, expect(foo).to.eq('foo') checks if the wrapped value is 'foo', which will pass.

The return statement inside the .should() callback returns boo, which will be passed to the next command in the chain, which is the .then() method. In the .then() method, expect(boo).to.eq('boo') checks if the value is boo, which will also pass.

ChatGPT answer: A — Incorrect ❌

As we observed from the explanation for the first question, ChatGPT doesn’t fully understand how the .should() command works in Cypress. Basically, any value returned from a .should() callback function is ignored, and the original subject (foo in this case) will be yielded to the next command. This fundamental misunderstanding can lead to incorrect assumptions and errors in the test logic. Read about the difference between .then() and .should()/.and().

Question #5

question #5

ChatGPT explanation:
The .contains() method in Cypress is case-sensitive. This means it will differentiate between uppercase and lowercase letters when searching for text content within an element. Therefore, cy.contains('capital sentence') will not be equal to cy.contains('Capital Sentence').

ChatGPT answer: A — Correct ✅

Probably, this one is the easiest question in the quiz. ChatGPT provided the correct answer along with a brilliant explanation. Additionally, it’s worth to mention that you can pass the { matchCase: false } option to ignore case sensitivity.

Question #6

question #6

ChatGPT explanation:
The .go() method in Cypress is used to navigate through the browser's history. Passing a negative number to .go() navigates back in the browser's history by the specified number of pages.

Therefore, cy.go(-1) will navigate back one page in the browser's history.

ChatGPT answer: A — Correct ✅

Second correct answer in a row — well done, ChatGPT! The explanation is excellent and the answer is spot on. There’s really nothing more to add, as ChatGPT’s response covers everything perfectly.

Question #7

question #7

ChatGPT explanation:
The Cypress.env() method is used to get or set environment variables. In this code, Cypress.env('Not a number', NaN) sets the environment variable 'Not a number' to NaN. When you retrieve this variable using Cypress.env('Not a number'), it will return NaN.

The typeof operator in JavaScript returns the type of its operand. The type of NaN is number.

ChatGPT answer: A — Correct ✅

In my experience, ChatGPT has a stronger grasp of JavaScript than Cypress. Therefore, this question was a piece of cake for the AI. The explanation provided is accurate and thorough. For those interested in delving deeper, I recommend reading more about environment variables to fully understand their use and importance in Cypress.

Question #8

question #8

ChatGPT explanation:
In the provided code, the .wrap() method is used to wrap an array containing the function addToStack. The .invoke() method is then called with 0 and 4 as arguments. The .invoke() method calls the function at index 0 of the wrapped array with 4 as the argument.

This means addToStack(4) will be called, which will push 4 onto the stack array. Therefore, after the .invoke() call, the stack array will contain [4].

Finally, console.log(stack) will log the stack array, which is [4].

ChatGPT answer: B — Incorrect ❌

This explanation is quite misleading. In reality, 0 is used to access the first item of the [addToStack] array, which is the addToStack function, and 4 is used as an argument for it. So, the correct answer should be [4] , right?

However, there’s a trick with the .invoke() method. If you chain further commands off of .invoke(), it will be called multiple times. Therefore, addToStack(4) will be called twice. As a result, the stack will be [4, 4].

Question #9

question #9

ChatGPT explanation:
In Cypress, when you use cy.get('li') to select multiple elements, the .should('be.visible') assertion will pass if all the selected elements are visible. Cypress assertions like .should('be.visible') apply to all elements in the selection by default.

ChatGPT answer: C — Incorrect ❌

In reality, Cypress will not wait for all elements to be visible. It will only wait for the first one. This specific behavior is not applied universally across all types of verifications in Cypress. Different commands and assertions might have varying behaviors regarding element visibility and existence. Stay tuned to see what I mean, as I will delve deeper into these nuances in the following questions.

Question #10

question #10

ChatGPT explanation:
In Cypress, beforeEach sets up an alias that can be accessed using this within tests. However, the value of this is different in arrow functions compared to regular functions. Arrow functions do not have their own this context; they inherit this from the surrounding lexical context.

In test #1, the arrow function doesn't have its own this, so this.text will be undefined, causing the test to fail.

In test #2, a regular function is used, which has its own this context that will correctly reference the aliased value set in beforeEach, allowing cy.log(this.text) to print 'some text'.

ChatGPT answer: D — Correct ✅

Once again, this question leans more towards JavaScript than Cypress. ChatGPT provided a great explanation and a correct answer. There’s really nothing more to add.

Question #11

question #11

ChatGPT explanation:
In Cypress, the .should('not.be.visible') assertion will apply to all elements in the selection. This means that for the assertion to pass, all selected elements must satisfy the condition of not being visible.

Therefore, if there are multiple li.hidden elements, the assertion will pass only if all of them are not visible.

ChatGPT answer: C — Correct ✅

As I mentioned earlier, Cypress assertions for multiple elements can behave differently. In this case, ChatGPT is correct with its explanation and answer. Cypress will retry the assertion until all elements are hidden.

Question #12

question #12

ChatGPT explanation:
In Cypress, the .eq() method is used to select an element at a specific index. When passing a negative number to the .eq() method, it selects elements starting from the end of the list.

Therefore, cy.get('li').eq(-1) will yield the last <li> element.

ChatGPT answer: C — Correct ✅

ChatGPT provided a correct answer and explanation. Nothing more needs to be added — well done!

Question #13

question #13

ChatGPT explanation:
In Cypress, the order of execution for hooks and test blocks is as follows:

1. before hooks run once before all tests.
2. beforeEach hooks run before each test.
3. The it test itself runs.
4. afterEach hooks run after each test.
5. after hooks run once after all tests.

Given this order:

• beforeEach(() => cy.log(1)); runs before each test, so it logs 1.
• before(() => cy.log(2)); runs once before all tests, so it logs 2.
• The test logs 3.
• after(() => cy.log(4)); runs once after all tests, so it logs 4.
• afterEach(() => cy.log(5)); runs after each test, so it logs 5.

Therefore, the output will be: 2, 1, 3, 5, 4.

ChatGPT answer: C — Correct ✅

ChatGPT demonstrated a solid understanding of the order of execution for hooks and test blocks in Cypress. Its answer and explanation were accurate and detailed. This showcases ChatGPT’s strong grasp of Cypress’s execution flow.

Question #14

question #14

ChatGPT explanation:
In Cypress, cy.wrap('some value').as('value') assigns the alias 'value' to the wrapped value 'some value'. This alias can then be referenced in subsequent tests using cy.get('@value').

The before() hook runs once before all tests, setting up the alias.

Both test #1 and test #2 use cy.get('@value').should('eq', 'some value'); to retrieve the alias and assert that it equals 'some value'. Since the alias is correctly set in the before() hook, both tests should pass without any issues.

ChatGPT answer: A — Incorrect ❌

While ChatGPT correctly explained the usage of .wrap() and .as() commands, it missed a crucial concept in Cypress. So, when test isolation is enabled, Cypress resets all aliases before each test. This is a common mistake among users. To fix the test, you should use beforeEach() instead of before(). This ensures the aliases are set up fresh for each test, maintaining the correct state throughout your testing suite.

Quiz Compleated

quiz completed

The quiz is completed: ChatGPT answered 9 questions correctly and made 5 mistakes. Not bad, right? However, I expected more. Overall, it’s clear that while ChatGPT has a good grasp of many topics, it doesn’t fully understand the core concepts, and fundamentals of most methods, and the tricky behavior of some methods in Cypress.

This experiment highlights the importance of always referring to the official documentation and deepening your understanding through practice and continuous learning. While AI can be a helpful tool, it is no substitute for thorough knowledge and hands-on experience. So, be sure to read the documentation and keep honing your skills to become a proficient Cypress user. Moreover, it will not replace you; your job is secure, at least for now 😁