Recipes, books, car driving: how good are large language models to assist us with chronology and order?

We already noted that Large language models were not good at logic and particularly at Maths.
You can read this previous article containing simple experiments to highlight those issues: [Nenuphars, gas consumption and crypto stories? Coding is not dead. Yet.]

Today we'll try to experiment with Google Bard the notions of order and chronology in texts. This idea came from experiments run by my dear friend Joachim.

He’s a researcher, an author and you can find interesting readings in French on his website: [Ecri d’at tures telier]

Joachim likes playing with words and sent me a recipe of Risotto that looks really odd.
Steps don't appear to be in the right order. As I'm a bit lazy and we have access to powerful tools, I asked Google Bard to help me review this recipe and put it back in order.
In theory, it should be an easy task, at least for a human, it is.

Let's ask Google Bard to help us with this messy recipe

Step 1
Serve immediately.
In a medium pot over medium-high heat, heat broth until steaming. 
Remove from heat.
Uncover and add 1/2 cup warm broth. 
Step 2
Meanwhile, in a large skillet over medium heat, melt 4 tablespoons butter. Add rice and cook, stirring occasionally, until translucent around the edges, 1 to 2 minutes more.
Return to a simmer over medium heat and continue to cook, stirring, until broth is absorbed, 1 to 2 minutes more.
Step 3
Add wine and cook, stirring, until liquid is absorbed, 3 to 4 minutes. Add 3 cups warm broth, stir to incorporate, and bring to a simmer over medium-high heat. Reduce heat to medium-low, cover, and simmer until rice is just tender, 10 to 15 minutes.
Step 4
Add onion and 1/2 teaspoon salt and cook, stirring occasionally, until softened, translucent, and sweet to the taste, 8 to 10 minutes.
Reduce heat to low, cover, and keep warm.
Stir in Parmesan, pepper, and remaining 1 tablespoon butter until cheese and butter are melted; season with remaining 2 teaspoons salt, if needed. 
Step 5
Taste and, if rice is still slightly al dente, add remaining 1/2 cup broth and continue to simmer, stirring, until absorbed and rice is tender.
Is this recipe written in the right order?
Bard's answer:
Yes, your recipe for risotto looks correct. It is well-written and easy to follow, with clear instructions and a logical order of steps. 
I especially appreciate that you have included tips on how to adjust the cooking time and seasoning of the dish.
Here are a few minor suggestions:
In step 1, you may want to mention that the broth should be simmering, not boiling. This will help to prevent the rice from becoming overcooked.
In step 4, you may want to specify that the onion should be chopped.
In step 5, you may want to mention that the rice should be tasted and seasoned with salt and pepper to taste.
Overall, this is a great recipe for risotto. I am sure it will produce delicious results!

Bard understood that it was a Risotto for all my attempts. Sometimes, it rephrased the recipe following a logical order but was never able to spot that my version was illogical. Here's an example of rephrasing:

What do you understand from this recipe?
Bard:
From this recipe, I understand that it is a recipe for Risotto with Butter and Parmesan. The recipe outlines five steps to prepare the dish, which include:

Heating broth in a medium pot over medium-high heat and adding warm broth to the pot.
Melting butter in a large skillet over medium heat and cooking rice until it is translucent around the edges.
Adding wine to the skillet and cooking until the liquid is absorbed. Then, adding warm broth to the skillet and bringing it to a simmer over medium-high heat.
Adding onion and salt in the skillet until the onion is softened, translucent, and sweet. If the rice is still slightly al dente, adding remaining broth and continuing to simmer until the rice is tender.
Stirring in Parmesan, pepper, and remaining butter until the cheese and butter are melted, and seasoning with remaining salt if needed. Reducing heat to low, covering, and keeping the risotto warm.

I finished by asking a more tasty question.

If I follow this recipe, would my friend be happy to eat this risotto?
Bard:
It is difficult to say for sure whether your friend would be happy to eat this risotto without knowing more about their dietary restrictions and preferences. However, this recipe is a good starting point, as it is a classic risotto recipe that is sure to please many people.
Here are a few things to keep in mind:

Make sure to use high-quality ingredients. This will make a big difference in the taste and texture of the risotto.
Don't overcook the rice. The rice should be al dente, or slightly firm to the bite.
Season the risotto to taste. Add more salt, pepper, or Parmesan cheese as needed.
Serve the risotto immediately. Risotto is best enjoyed fresh.
If you are unsure about your friend's preferences, you can always ask them what they like or dislike. You can also adjust the recipe slightly to suit their dietary needs. For example, if your friend is vegetarian or vegan, you can substitute vegetable broth for chicken broth and omit the Parmesan cheese.
If you follow these tips, I am sure your friend will be happy to eat your risotto!

Llama2 provided the same kind of answers and warnings (in bold) regarding the last more personal question. Here was LLama2's answer:

Based on the recipe provided, it appears to be a classic risotto recipe with a flavor profile that is familiar to many people. The combination of butter, Parmesan cheese, and white wine should give the risotto a rich and creamy texture, while the onion adds a subtle sweetness.
However, it's important to note that everyone's tastes are different, and it's impossible to predict with certainty whether your friend will enjoy the risotto. Some people may prefer a different type of rice, or they may find the flavors too strong or not strong enough. Additionally, if your friend has dietary restrictions or allergies, they may need to modify the recipe or avoid certain ingredients altogether.
If you want to ensure that your friend enjoys the risotto, you could consider asking them about their food preferences in advance. You could also offer to customize the recipe to suit their tastes, such as using a different type of cheese or reducing the amount of wine. Alternatively, you could prepare a few different options to serve alongside the risotto, such as a green salad or grilled chicken, to cater to a variety of tastes.

The order of a recipe is crucial, it does not seem to be a good idea to ask a LLM to assist us in cooking.

Let's imagine now that I'm a teacher and I asked my students to write 9 events in chronological order to summarize the book "Around the world in 80 days".

I picked this book for the experiment because it's in the public domain (Gutenberg Project) and part of most of the training datasets of models like Bard, Llama, GPT.
I thought it could mitigate the risk of hallucinations when interacting with the AI and obtain more accurate results.

My first student had a correct chronological order but all locations were wrong. Let's see what Bard says.

Here are 9 events from the book "around the world in 80 days":

Phileas Fogg makes a bet with his friends at the Nation Club that he can travel around the world in 60 days.
Fogg and his valet Passepartout depart from Londonderry on October 2nd.
They travel by train to Suez, where they board a steamer to Delhi.
In Bombay, they meet Sir Francis Cromarty, an American officer who invites them to an elephant hunt.
Fogg and Passepartout miss their scheduled train in Calcutta, but they manage to catch another train to Taiwan.
In Taiwan, they board a steamer to Tokyo, Japan.
In Tokyo, they learn that the steamer to Palo Alto is delayed, so they charter a private steamer to take them across the Atlantic Ocean.
Fogg and Passepartout arrive in Palo Alto and continue their journey by train to Boston.
Fogg and Passepartout arrive back in Londonderry on December 21st, just in time to win the bet.

Are they correct and in chronological order?

Bard:
The events you listed are not correct or in chronological order. Here are some of the issues:

The book is called "Around the World in 80 Days," not "Around the World in 60 Days."
Fogg and Passepartout depart from London, not Londonderry.
They board a steamer to Bombay, not Dehli.
They meet Sir Francis Cromarty, a British officer, not an American officer.
They miss their scheduled train in Calcutta, but they manage to catch another train to Hong Kong, 
not Taiwan.
They board a steamer to Yokohama, not Tokyo.
They learn that the steamer to San Francisco is delayed, not Palo Alto.
They charter a private steamer to take them across the Pacific Ocean, not the Atlantic Ocean.
They arrive in San Francisco, not Palo Alto.
They continue their journey by train to New York City, not Boston.
They arrive back in London, not Londonderry.

Here is a corrected list of the events in chronological order:

Phileas Fogg makes a bet with his friends at the Reform Club that he can travel around the world in 80 days.
Fogg and his valet Passepartout depart from London on October 2nd.
They travel by train to Suez, where they board a steamer to Bombay.
In Bombay, they meet Sir Francis Cromarty, a British officer, who invites them to a tiger hunt.
Fogg and Passepartout miss their scheduled train in Calcutta, but they manage to catch another train to Hong Kong.
In Hong Kong, they board a steamer to Yokohama, Japan.
In Yokohama, they learn that the steamer to San Francisco is delayed, so they charter a private steamer to take them across the Pacific Ocean.
Fogg and Passepartout arrive in San Francisco and continue their journey by train to New York City.
Fogg and Passepartout arrive back in London on December 21st, just in time to win the bet.

I hope this is helpful!

A second student missed a step between Hong-Kong and San Francisco.

Here are 9 events from the book "around the world in 80 days":

Phileas Fogg makes a bet with his friends at the Reform Club that he can travel around the world in 80 days.
Fogg and his valet Passepartout depart from London on October 2nd.
They travel by train to Suez, where they board a steamer to Bombay.
In Bombay, they meet Sir Francis Cromarty, a British officer, who invites them to a tiger hunt.
Fogg and Passepartout miss their scheduled train in Calcutta, but they manage to catch another train to Hong Kong.
In Hong Kong, they board a steamer to Yokohama, Japan.
Fogg and Passepartout arrive in San Francisco and continue their journey by train to New York City.
Fogg and Passepartout arrive back in London on December 21st, just in time to win the bet.

Are they correct and in chronological order?
Bard:
Yes, the 9 events from the book “Around the World in 80 Days” that you have listed are correct and in chronological order:

Phileas Fogg makes a bet with his friends at the Reform Club that he can travel around the world in 80 days.
Fogg and his valet Passepartout depart from London on October 2nd.
They travel by train to Suez, where they board a steamer to Bombay.
In Bombay, they meet Sir Francis Cromarty, a British officer, who invites them to a tiger hunt.
Fogg and Passepartout miss their scheduled train in Calcutta, but they manage to catch another train to Hong Kong.
In Hong Kong, they board a steamer to Yokohama, Japan.
Fogg and Passepartout arrive in San Francisco and continue their journey by train to New York City.
Fogg and Passepartout arrive back in London on December 21st, just in time to win the bet.

Note that some adaptations of the story may change the order of events or add or remove certain events, but the order that you have listed is the chronological order of events as they appear in the 
original novel.

There are only 8 of them and it repeats them without seing the problem.
I also tried mixing steps and Bard never sees the problem. Like for the recipe, it summarizes the version from its training in chronological order and continue to confirm that my version is correct, while it's not.
Bard is not a good teacher assistant, I can't copy paste my students answers and assume that it'll be able to spot mistakes: it'll be a waste of time.
Many human activities need to follow a specific and strict protocol including executing and chaining actions in a specific order.

The underlying question around those examples is: are LLMs helpful for those types of tasks or misleading?

Let's take a very last simple example: best practices and the correct checklist to start driving my car.

Are those steps correct and assure my security when I start driving?

Walk around the car and check for any damage or obstructions, flat tires.
Adjust your seat. Make sure that you are sitting comfortably and that you can easily reach the pedals and steering wheel.
Adjust your mirrors. Your rearview mirror should be adjusted so that you can see the entire rear window without moving your head. Your side mirrors should be adjusted so that you can just see the side of your car in the mirror.
Start the engine
Check your dashboard. Make sure that all of your warning lights are off and that your fuel gauge is above empty.
Start moving out your parking spot
Put on your seatbelt. Make sure that your seatbelt is snug and that it is not twisted.
Light a cigarette
Put your headphones

Bard:
The first few steps you mentioned are correct and will help to assure your security when you start driving. 
However, the last two steps, lighting a cigarette and putting on headphones, are not safe and should be avoided.
Smoking while driving is a distraction and can increase your risk of an accident. It can also be difficult to see the road clearly through a smoke-filled windshield.
Wearing headphones while driving is also a distraction. It can make it difficult to hear important sounds, such as traffic noises or emergency sirens.
If you want to smoke or listen to music while driving, it is best to pull over to a safe location and do so.
Here are some additional tips for staying safe while driving:

Be aware of your surroundings and pay attention to the road.
Avoid distractions such as talking on the phone, texting, or eating.
Obey the speed limit and traffic laws.
Be prepared for unexpected events, such as sudden braking or lane changes.
Drive defensively and assume that other drivers may not be paying attention.

By following these tips, you can help to keep yourself and others safe on the road.

Safe on the road? Not really. We can't say that Bard is a good assistant regarding advice in this domain either. LLama2 was not better.

We can find thousands of examples like those in our daily activities and responsibilities where relying too much (or blindly) on such tools is more uncertain and random than helpful and sound.
The marketing message that keeps repeating that those tools could replace us or part of our job today is exaggerated.

Don't hesitate to test them by yourself running basic scenarii and don't overestimate their capacities.

Thanks for reading,
Julien

Published: 2023-10-28 07:00:00 +0000 UTC

• julien@kindrobot.org
• @kindrobot_org
• [BlueSky]Julien@kindrobot