[Models] Seeing markdown in a Smart Window chat
Categories
(Core :: Machine Learning: General, defect, P1)
Tracking
()
People
(Reporter: rrando, Unassigned)
References
(Regression)
Details
(Keywords: regression, Whiteboard: [genai][models])
Attachments
(4 files)
n3.jpg
|
||
Updated•6 months ago
|
The last chat history sent from the user is attached as a json. Theoretically it should trigger the error.
|
||
Comment 4•6 months ago
|
||
:gazhibekova, since you are the author of the regressor, bug 2015111, could you take a look? Also, could you set the severity field?
For more information, please visit BugBot documentation.
Root Cause:
The "No Tables" instruction in the system prompt was insufficient for multi-turn conversations where the model had comparison data from search results. The model correctly used headers + bullets in the main body but reverted to Markdown table syntax in "Key Differences" summary sections at the end of responses.
Fix:
Strengthened the "No Tables" instruction across all model prompts with:
- Explicit callout of summary/wrap-up sections as failure points
- A negative example showing what NOT to do (table in a comparison section)
- Two correct alternative formats (headers+bullets and labeled comparison list)
- A reinforcement reminder at the end of the prompt
Verification:
- Reproduced the bug locally using the reported conversation flow
- Ran automated eval (no_tables suite) with 35 test cases including 5 new multi-turn cases that mirror the exact bug scenario
- Results across all 3 models:
- Qwen3-235B: 0/35 tables, 89% alternative formatting
- gpt-oss-120b: 0/35 tables, 91% alternative formatting
- gemini-2.5-flash-lite: 0/35 tables, 91% alternative formatting
- Confirmed fix locally -- the reported conversation no longer produces tables
PR for updating the prompt in remote-settings: Strengthened the No Tables instruction
PR in eval repo: automated eval PR
|
||
Updated•6 months ago
|
Description
•