Same Logic, Different Words: Why AI Systems Contradict Themselves, and What We Did About It

Created on November 01, 2026

2026   ·   LLM   reasoning   logic   NeurIPS   ·   ai

Ask a large language model the same question two ways and you may get two different answers. In a paper accepted at NeurIPS 2026, my coauthors and I measured how often this happens, asked why, and tested a fix.

A simple test that strong models fail

Here is a small reasoning problem of the kind we gave the models, dressed in business clothing. A retailer’s shipping policy has two rules, and we know two facts about a customer named Avery.

The facts:

  • Avery does not hold a premium account.
  • Avery made a purchase this year.

The rules:

  • Rule 1. If a customer holds a premium account or is an active customer, that customer qualifies for free shipping.
  • Rule 2. Any customer who has referred a friend or made a purchase this year counts as an active customer.

Does Avery qualify for free shipping? Yes. Avery made a purchase this year, so by Rule 2 Avery is an active customer. An active customer satisfies the “or” in Rule 1, so Avery gets free shipping. The missing premium account does not matter. It is there as a distraction, and a careful reader ignores it.

Written in the notation of logic, with $P$ for “holds a premium account,” $A$ for “is an active customer,” $F$ for “qualifies for free shipping,” $R$ for “referred a friend,” and $B$ for “bought something this year,” the two rules are:

\[\text{Rule 1: } (P \lor A) \rightarrow F \qquad\qquad \text{Rule 2: } (R \lor B) \rightarrow A\]

Here the symbol “or” is written as a small v, “and” as an upside-down v, “not” as a hook, and “if … then” as an arrow. The facts are “not $P$” and “$B$.” The chain of reasoning is three short steps: $B$ holds, so $A$ holds by Rule 2; $A$ holds, so “$P$ or $A$” holds; “$P$ or $A$” holds, so $F$ holds by Rule 1.

Now here is Rule 1 again, saying exactly the same thing in different words:

If Avery does not both lack a premium account and fail to be an active customer, then Avery qualifies for free shipping.

That sentence is hard to read, but it is not a different rule. “Not both lacking X and lacking Y” is just a roundabout way of saying “having X or Y.” Logicians call this De Morgan’s law, and it is one of a handful of rewriting rules that every introductory logic course teaches:

\[\neg(\neg P \land \neg A) \;\equiv\; P \lor A\]

So the rewritten Rule 1 is

\[\neg(\neg P \land \neg A) \rightarrow F\]

which has precisely the same meaning as the original. Any customer who qualifies under one phrasing qualifies under the other. A system that understands the rule, rather than the wording, should give the same answer about Avery either way.

The paper uses three such rewriting rules, applied one at a time and in combination. The other two are even simpler. One replaces “if X then Y” with “not X, or Y.” The other inserts a double negative, turning “X” into “not not X.” Each produces a sentence that sounds different and means the same.

Rule 1 of the Avery example. The diagram asks about the premium account first, then active status.

The picture above is the point of the paper in one glance. However Rule 1 is worded, the questions it asks about a customer, and the order it asks them in, are the same. The diagram is the rule. The sentences are just three ways of describing it.

The Avery problem is adapted from one in the paper, with the subject changed from film history to a shopping account and the logic left untouched. We built a dataset of 1,500 problems with this kind of structure, rewrote each one in several logically equivalent ways using those three rules, and asked some of the most capable models available, including OpenAI’s o3 and Google’s Gemini, to solve both the originals and the rewrites.

They changed their answers often. On the hardest set of rewrites, o3 gave a different verdict on 37 percent of problems compared to the original wording. Gemini flipped on 35 percent. Some of that variation is ordinary randomness in how models generate text, so we measured that too, by asking the same question twice. Randomness accounted for only a few percentage points. The rest came from the rewording.

Figure 3b of the paper. Three models; inconsistency rate on the mixed rewrites versus run-to-run randomness.

The grey bars are the models disagreeing with themselves when asked the identical question twice. The blue bars are what happens when the logic is reworded. The gap between them is the effect of wording.

Accuracy fell as well, by up to 20 percentage points on the rewritten problems. The models were not just inconsistent. They were worse.

Why this matters outside the lab

Most public discussion of AI reasoning focuses on benchmark scores: how many math problems a model solves, how it does on a bar exam. Those numbers matter, but they answer a different question from the one a business faces when it puts a model to work.

In practice, organizations increasingly ask language models to apply rules. Does this claim meet the policy? Has this contract clause been triggered? Should this transaction be flagged for review? Is this applicant eligible? The rule is fixed. What varies is how it is written down, who wrote it, and how the facts of each case are described.

Consider a procurement team that uses a model to decide which vendors need an audit. Again two rules and a few facts, this time about a supplier called Bennett Supply.

The facts:

  • Bennett Supply is a certified vendor.
  • Bennett Supply has no overdue invoices.
  • Bennett Supply missed a delivery deadline last quarter.

The rules:

  • Rule 1. If a vendor is certified and has been flagged for review, the vendor must undergo an audit.
  • Rule 2. Any vendor with overdue invoices or a missed delivery deadline is flagged for review.

The model is asked whether the following statement is true: “Bennett Supply is not required to undergo an audit.” The correct answer is false. The missed deadline triggers Rule 2, so Bennett is flagged. Bennett is certified and flagged, so Rule 1 applies and an audit is required. The clean invoice record is the distraction this time.

With $C$ for “certified,” $G$ for “flagged,” $U$ for “must undergo an audit,” $O$ for “has overdue invoices,” and $M$ for “missed a deadline”:

\[\text{Rule 1: } (C \land G) \rightarrow U \qquad\qquad \text{Rule 2: } (O \lor M) \rightarrow G\]

The facts are $C$, “not $O$,” and $M$. The statement to judge is “not $U$,” and it is false because $U$ follows. This problem too is adapted from one in the paper, which uses the same two-rule shape: an “and” in the first rule, an “or” in the second, and one irrelevant fact.

Now suppose the compliance department rewrites Rule 2 as “a vendor is flagged for review unless it has neither overdue invoices nor a missed deadline,” or Rule 1 as “a vendor escapes audit only if it is not certified or has not been flagged.” Nothing about Bennett’s situation has changed. In our experiments, the models’ answers often did.

A system that gives one answer when a policy is written as a list of conditions and a different answer when the same policy is written as a list of exclusions is not reliable, no matter how well it scores on a benchmark. The customer who is denied a refund because the rule was phrased one way, when a colleague with identical facts was approved under the other phrasing, does not care about benchmark scores.

This is a question about trust, and it is a different question from accuracy. A model can be right most of the time and still be untrustworthy if its answers depend on wording rather than substance.

Why models trip

The simplest explanation is that models learn to read wording rather than logic. They are trained on enormous amounts of text, and in text, certain phrasings tend to go with certain conclusions. A model can get very good at exploiting those patterns without ever representing the underlying rule as a rule.

We checked a few alternative explanations. Perhaps the rewritten problems were simply harder? If so, we would expect the models to go from right to wrong far more often than from wrong to right. Instead the flips went in both directions at nearly equal rates. The rewording was not making problems harder. It was scrambling the model’s reading of them.

We also tried the obvious fix: tell the model about the rules of logic. We put the replacement rules, with explanations and examples, directly in the prompt. This did almost nothing. One model improved by under three percentage points. Another got slightly worse. Knowing that two forms are equivalent, in the sense of having been told so, is not the same as treating them as equivalent.

Giving the model a map

If the problem is that models read words instead of logic, one answer is to hand them the logic directly, in a form that does not depend on wording.

Computer scientists have a tool for exactly this. Any rule built from “and,” “or,” and “not” can be drawn as a compact flowchart called a decision diagram. You start at the top, check one fact at a time, follow the branch for yes or no, and arrive at the answer at the bottom. The useful property is that two rules which mean the same thing produce the same diagram, however differently they are written. The refund rule and its rewording above would draw identically.

Our method converts each reasoning problem into its decision diagram, then trains a small companion network to turn that diagram into a numerical summary the language model can consult. We train the companion so that problems with the same logical structure land close together in its summary, regardless of surface wording. The language model then reads this summary alongside the text while it reasons, through a small added component that lets it pay attention to the logical structure when that helps and ignore it when it does not.

Method sketch, simplified from Figure 4 of the paper.

The top row is an ordinary language model. The bottom row is what we add: the same question, turned into its decision diagram, summarized by a small network, and handed to the model as a second input.

The analogy I find useful is a translator who, when given a contract in two languages, is also handed a diagram of what the contract actually commits each party to. The words differ. The diagram is the same. A good translator uses both.

One detail matters for anyone thinking about deployment. Showing the model the diagram as plain text, written out in the prompt, did not work. The gains came only from training the model to use the structure, not from describing the structure to it.

What we found

We tested the approach on two open models, Llama 3.1 and Qwen 2.5, under three different training recipes, so that the result would not depend on one particular setup.

Self-contradiction roughly halved. Under two of the three training recipes, the rate at which a model changed its answer on a logically equivalent rewrite fell from above 30 percent to about 15 percent. Under the third, it fell by several points. The improvement held for every kind of rewriting we tried, not just one.

Table 2 of the paper. Llama 3.1 8B, mixed rewrites; lower is better.

The effect is largest under the two reinforcement recipes, which are the ones most often used to train models for reasoning today.

Accuracy went up, and it went up most on logical forms the model had never seen in training. This is the result I care most about. The model was trained on one family of rule phrasings and evaluated on others. It did better on the new ones because it had learned something about structure, not because it had memorized more examples.

The gains carried over to real-world reasoning benchmarks written by human experts, not just to our constructed data. The companion network trained on our synthetic problems helped on those benchmarks without any retraining.

The approach is also cheap. The companion network is small, and the added component inside the language model adds negligible latency. This is not a bigger model. It is the same model with a better sense of what the question is asking.

One result for the technically curious. When we looked inside the companion network’s summaries, we found that adding a reasoning step corresponded to moving in a consistent direction in the summary space, across different problems. The network had learned to represent “one more step of logic” as a reusable operation, which helps explain why it generalizes.

What this means for anyone deploying AI

Three things, I think.

Consistency is measurable. Before adopting a model to apply a rule, you can test it the way we did: write the rule several equivalent ways and see whether the answers agree. Most teams test accuracy. Few test this, and the two can come apart.

Prompting has limits. Telling a model to reason carefully, or spelling out the logical rules in the prompt, did not fix the problem in our experiments. Some failures are in how the model was trained, and have to be addressed there.

Structure helps. Giving a model an explicit, wording-independent representation of the logic it is supposed to follow made it more consistent and better at generalizing. For high-stakes rule application, that is worth the engineering effort.

This is one paper, on one kind of reasoning, with open models of moderate size. I would not claim more than that. But it points toward a standard for deployed AI that is easy to state: one rule, one answer, however the rule is phrased.

About the paper

“When Form Changes but Logic Doesn’t: Building Logic-invariant LLMs through Structures,” by Xuyuan Liu (Dartmouth), Xinshuai Dong (Carnegie Mellon), Elynn Chen (NYU Stern), and Yujun Yan (Dartmouth). NeurIPS 2026, the 40th Conference on Neural Information Processing Systems.

This work was supported in part by the National Science Foundation under grant DMS-2412577.