💡 The Align AI Research Review is a weekly dive into the most relevant research papers, AI industry news and updates
Caltech's new groundbreaking study investigates how effectively designed input prompts can significantly impact LLM outcomes, changing unlikely predictions into likely ones!
This blog discusses a novel paper “What’s the Magic Word? A CONTROL THEORY OF LLM PROMPTING”. The study conceptualizes large language models (LLMs) as discrete stochastic dynamical systems and uses control theory to understand and modify their outputs. The researchers investigate the 'reachable set' of outputs for an LLM, showing that prompt engineering can have a major impact on LLM behavior.
Let’s explore the limitations of the optimizing prompts.
Limitations in Existing Work
The performance of LLMs is substantially impacted by the specific prompt tokens used, implying that even minor variations in prompt text can result in distinct outcomes (Brown et al., 2020).
Prompt optimization algorithms rely on heuristics to select high-value swaps during the sampling phase, which may not always ensure the best prompts and could potentially overlook more effective alternatives (Wen et al., 2023).
In AutoPrompt and its derivatives, token exchanges within the prompt are significantly influenced by gradient information at the token embedding layer. This dependence could restrict the algorithm's efficacy if the gradient information is insufficiently informative or leads to suboptimal prompt modifications (Shin et al., 2020).
(Soatto et al., 2023) theoretical analysis of LLM controllability is restricted to 'meaningful sentences,' preventing the consideration of out-of-distribution inputs and outputs. This limitation confines the practical applicability of their findings.
The proposed work mainly focuses on a practically oriented exploration of LLM controllability. Unlike previous studies, the research did not eliminate "meaningless sentences" from the state or input space.
Proposed System
The paper takes the following steps to build the area of LLM control theory:
Formalize LLMs as a type of discrete stochastic dynamical system.
Analyze the reachable set of system outputs Ry(xo), for which there occurs a control input steps u for each y € Ry(xo) that directs the LLM from its initial condition.
Determine an upper bound for the controllability of token representations in self-attention.
The research empirically inspects how controllable a panel of open-source language models (Falcon-7b, Llama-7b, Falcon-40b) are when beginning with states from the Wikitext dataset, and establish a tractable statistical metric (such as k – ϵ controllability) for estimating LLM steerability (see fig.1).
The research employed the identical Back-off Prompt strategy (see algorithm 1) to analyze the controllability of k − ϵ for all experiments, while adjusting the dataset D's features for each experiment.
Empirical findings indicate that reaching a specific output is significantly more complex than the prior likelihood or the prior rank attributed to a given token by the LLM. Although prompt optimization-based k − ϵ controllability experiments can provide a lower bound on the content of the reachable set, it is possible to frequently manipulate even the least likely token to become the most likely token with only a few input tokens
In addition, the paper revealed that true next token y is accessible over 97% of the time across all models with prompts of tokens.
Furthermore, the analysis shows that the top 75 most likely next tokens, as evaluated by the LLM, are accessible at least 85% of the time with prompts of k.
Short prompt sequences can significantly change the chance of specific outputs, even transforming the least likely tokens into the most likely ones.
Final Words
The novel research introduces a control theory framework for understanding and building LLM-based systems.
LLMs are significantly being used as components within software systems, but their unpredictable nature poses challenges for system design.
The paper formalizes notions of reachability, controllability and stability for LLM-based systems and analyzes the control features of chain-of-thought reasoning and distributional control.
The work also analyzes the learnability of control for LLMs and the composability of LLM systems.
