From a79c06f3b9e30cce407490843f21c6d0165fb628 Mon Sep 17 00:00:00 2001 From: Elvis Saravia Date: Thu, 9 Feb 2023 00:09:35 +0000 Subject: [PATCH] more examples --- README.md | 1 + guides/prompts-advanced-usage.md | 106 ++++++++++++++++++++++++++++++- 2 files changed, 106 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index cc8f549..4a60940 100644 --- a/README.md +++ b/README.md @@ -95,6 +95,7 @@ The following are a set of guides on prompt engineering developed by us. Guides - [Crawling the Internal Knowledge-Base of Language Models](https://arxiv.org/abs/2301.12810) (Jan 2023) - [Legal Prompt Engineering for Multilingual Legal Judgement Prediction](https://arxiv.org/abs/2212.02199) (Dec 2022) - [Investigating Prompt Engineering in Diffusion Models](https://arxiv.org/abs/2211.15462) (Nov 2022) + - [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://arxiv.org/abs/2209.09513v2) (Sep 2022) - [Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language](https://arxiv.org/abs/2210.15157) (Oct 2022) - [Piloting Copilot and Codex: Hot Temperature, Cold Prompts, or Black Magic?](https://arxiv.org/abs/2210.14699) (Oct 2022) - [Plot Writing From Scratch Pre-Trained Language Models](https://aclanthology.org/2022.inlg-main.5) (July 2022) diff --git a/guides/prompts-advanced-usage.md b/guides/prompts-advanced-usage.md index 99e6383..96ce2ee 100644 --- a/guides/prompts-advanced-usage.md +++ b/guides/prompts-advanced-usage.md @@ -206,4 +206,108 @@ Output 3: When I was 6 my sister was half my age, so she was 3. Now I am 70, so she is 70/2 = 35. The answer is 35. ``` -Computing for the final answer involves a few steps (check out the paper for the details) but for the sake of simplicity, we can clearly see that there is already a majority answer emerging so that would essentially become the final answer. \ No newline at end of file +Computing for the final answer involves a few steps (check out the paper for the details) but for the sake of simplicity, we can clearly see that there is already a majority answer emerging so that would essentially become the final answer. + +--- +## Generated Knowledge Prompting + +LLMs continue to be improved and one popular technique include the ability to incorporate knowledge or information to help the model make more accurate predictions. + +Using a similar idea, can the model also be used to generate knowledge before making a prediction? That's what attempted in the paper by [Liu et al. 2022](https://arxiv.org/pdf/2110.08387.pdf). In particular, how helpful is this for tasks such as commonsense reasoning? + +Let's try a simple prompt: + +``` +Part of golf is trying to get a higher point total than others. Yes or No? +``` + +Output +``` +Yes. +``` + +This type of mistakes reveal the limitations of LLMs to perform tasks that require more knowledge about the world. How do we improve this with knowledge generation? + +First we generate a few "knowledges": + +``` +Input: Greece is larger than mexico. +Knowledge: Greece is approximately 131,957 sq km, while Mexico is approximately 1,964,375 +sq km, making Mexico 1,389% larger than Greece. + +Input: Glasses always fog up. +Knowledge: Condensation occurs on eyeglass lenses when water vapor from your sweat, breath, +and ambient humidity lands on a cold surface, cools, and then changes into tiny drops of liquid, +forming a film that you see as fog. Your lenses will be relatively cool compared to your breath, +especially when the outside air is cold. + +Input: A fish is capable of thinking. +Knowledge: Fish are more intelligent than they appear. In many areas, such as memory, their +cognitive powers match or exceed those of ’higher’ vertebrates including non-human primates. +Fish’s long-term memories help them keep track of complex social relationships. + +Input: A common effect of smoking lots of cigarettes in one’s lifetime is a higher than +normal chance of getting lung cancer. +Knowledge: Those who consistently averaged less than one cigarette per day over their lifetime +had nine times the risk of dying from lung cancer than never smokers. Among people who smoked +between one and 10 cigarettes per day, the risk of dying from lung cancer was nearly 12 times +higher than that of never smokers. + +Input: A rock is the same size as a pebble. +Knowledge: A pebble is a clast of rock with a particle size of 4 to 64 millimetres based on the +Udden-Wentworth scale of sedimentology. Pebbles are generally considered larger than granules +(2 to 4 millimetres diameter) and smaller than cobbles (64 to 256 millimetres diameter). + +Input: Part of golf is trying to get a higher point total than others. +Knowledge: +``` + +Knowledge 1: +``` +The objective of golf is to play a set of holes in the least number of strokes. A round +of golf typically consists of 18 holes. Each hole is played once in the round on a standard golf +course. Each stroke is counted as one point, and the total number of strokes is used to determine +the winner of the game. +``` + +Knowledge 2: +``` +Golf is a precision club-and-ball sport in which competing players (or golfers) use +many types of clubs to hit balls into a series of holes on a course using the fewest number of +strokes. The goal is to complete the course with the lowest score, which is calculated by +adding up the total number of strokes taken on each hole. The player with the lowest score +wins the game. +``` + +We are using the prompt provided in the paper by [Liu et al. 2022](https://arxiv.org/pdf/2110.08387.pdf). + +Next step is to integrate the knowledge and get a prediction. I reformatted the question into QA format to guide the answer format. + +``` +Question: Part of golf is trying to get a higher point total than others. Yes or No? + +Knowledge: The objective of golf is to play a set of holes in the least number of strokes. A round of golf typically consists of 18 holes. Each hole is played once in the round on a standard golf course. Each stroke is counted as one point, and the total number of strokes is used to determine the winner of the game. + +Explain and Answer: +``` + +Answer 1 (confidence very high): +``` +No, the objective of golf is not to get a higher point total than others. Rather, the objective is to play a set of holes in the least number of strokes. The total number of strokes is used to determine the winner of the game, not the total number of points. +``` + + +``` +Question: Part of golf is trying to get a higher point total than others. Yes or No? + +Knowledge: Golf is a precision club-and-ball sport in which competing players (or golfers) use many types of clubs to hit balls into a series of holes on a course using the fewest number of strokes. The goal is to complete the course with the lowest score, which is calculated by adding up the total number of strokes taken on each hole. The player with the lowest score wins the game. + +Explain and Answer: +``` + +Answer 2 (confidence is a lot lower): +``` +Yes, part of golf is trying to get a higher point total than others. Each player tries to complete the course with the lowest score, which is calculated by adding up the total number of strokes taken on each hole. The player with the lowest score wins the game. +``` + +Some really interesting things happened with this example. In the first answer, the model was very confident but in the second not so much. I simplify the process for demonstration purposes but there are few more details to consider when arriving to the final answer. Check out the paper for more. \ No newline at end of file