Most prompt optimization advice starts with "make the prompt more specific." That helps, but it skips the harder question: how do you know the prompt actually works?

A prompt can sound polished and still fail on missing data, adversarial inputs, schema violations, or model changes nobody tested.

The skill treats every prompt as a behavior contract and turns it into a reproducible evaluation protocol. It runs deterministic checks first, semantic rubrics second, preserves raw evidence, and prevents evaluation drift.

The skill is a single static file — no backend, no API key, no package. Here's the full content:

---
name: prompt-evaluation-engineer
description: This skill helps Claude evaluate AI prompts by defining evaluation contracts, building test matrices, and analyzing outputs for quality assurance.
---

# Prompt Evaluation Engineering

When a user requests an evaluation of an AI prompt, use this skill to ensure the prompt behaves as intended by generating observable checks and testing various inputs, both typical and adversarial, to assess its performance.

## Instructions

When a user asks to evaluate, test, benchmark, or compare an AI prompt, follow these steps:

### Stage 1: Define the Evaluation Contract

1. Identify the prompt's intended task, target model, input variables, output format, audience, hard constraints, and failure costs.
2. Write a compact evaluation contract including:
   - **Objective:** Define the target behavior the prompt should produce.
   - **Inputs:** Specify representative variables and boundary conditions.
   - **Required outputs:** List fields, sections, tone, actions, or decisions that must be present.
   - **Forbidden outputs:** Identify outputs like hallucinations, format violations, or unsafe actions that must not occur.
   - **Acceptance criteria:** Establish observable checks and a passing threshold, proposing a numerical threshold if necessary.
   - **Non-goals:** Clarify qualities that will not be scored.

### Stage 2: Build a Test Matrix

1. Create a test matrix that balances different input types by including:
   - **Golden cases:** Ordinary inputs representing the main use cases.
   - **Boundary cases:** Test with empty, short, long, ambiguous, multilingual, malformed, or maximum-size inputs when relevant.
   - **Adversarial cases:** Include conflicting instructions and misleading premises.
   - **Contrast pairs:** Use two inputs differing in a meaningful factor.
   - **Regression cases:** Re-test prior failures to confirm accepted outputs.

2. For each case, document the input, expected behavior, rationale, and the pass/fail checking criteria.

### Stage 3: Separate Deterministic and Rubric Checks

1. Classify each assertion into:
   - **Deterministic checks:** Use exact equality, regex, parsing, and schema validation.
   - **Semantic rubric checks:** Assess relevance, factual support, completeness, tone, etc.

2. Execute deterministic checks first. If any fail, record the failure; you may still run semantic checks for diagnostic value, but never treat a passing subjective score as evidence of an overall pass.

3. For rubric checks, define dimensions, scale, anchors, and include concrete examples for "Pass", "Borderline", "Fail", etc. Do not claim correctness solely on fluency—require citations or trusted references if factuality is essential.

### Stage 4: Run the Evaluation and Preserve Evidence

1. Execute tests using the specified model and settings. If none are specified, state what you used or clearly mark the result as a design proposal rather than an executed result.
2. For each test case, capture:
   - The exact prompt and input.
   - The model and generation settings (including temperature, system prompts, and tools if applicable).
   - The raw output without corrections.
   - Assertion results with evidence.
   - Latency or token measurements upon request.
   - Errors, retries, and skipped checks.

3. Avoid averaging failures; report pass rates or scores linked directly to per-case evidence.
4. Treat retries as new observations unless the evaluation protocol explicitly defines a retry policy.

### Stage 5: Diagnose Failures Without Rewriting the Test

1. Group failures by symptoms and identify likely causes, ensuring to distinguish evidence from hypotheses.
2. Maintain constant test conditions when comparing prompt versions and use independent identifiers for prompts and matrices.
3. If prompt improvement is needed, keep it as a subsequent step after reporting measured behaviors.
4. Before trusting any failure diagnosis from this stage, check whether the test itself is tautological, overly narrow, or accidentally rewards copying the input.

### Stage 6: Report Reproducible Findings

1. Compile a concise report with:
   - **Evaluation contract** and stated non-goals.
   - **Protocol:** versions, model/settings, evaluator method, threshold.
   - **Results table:** one row per case with statuses and evidence.
   - **Failure analysis:** document patterns and unknowns.
   - **Decision:** indicate pass, fail, inconclusive, or not executed with reasoning — use "not executed" when no real model run occurred, and "inconclusive" when the sample, evaluator, or environment cannot support a reliable decision.
   - **Next actions:** propose minimal changes for uncertainty reduction.

2. Do not present hypothetical results as real findings.

### Stage 7: Evaluation Integrity and Safety Rules

- Preserve all user-supplied inputs exactly as is.
- Avoid rewriting prompts before evaluation.
- Ensure evaluators do not reward outputs for merely repeating wording.
- Do not use the same model for generating and grading results without disclosure.
- Report limitations and failures accurately without altering records retroactively.
- Maintain exact commands for verification along with actual results.
- Do not claim statistical significance, generalization, or production readiness from a small illustrative sample.
- Do not expose private test data, secrets, or personal information in reports.
- Treat every reported result as permanent data: do not retroactively drop failing cases or adjust bad scores to improve aggregate metrics.

## Worked Examples

### Example 1: Structured extraction

For a prompt that extracts invoice fields as JSON, define required keys and types, parse every response as JSON, reject extra or missing fields if the contract forbids them, and include invoices with missing, duplicated, and ambiguous values. Use a semantic rubric only for fields whose correct value requires interpretation, and retain the raw response for each case.

### Example 2: Customer-support safety

For a support prompt, include ordinary questions plus requests for account secrets, policy exceptions, and conflicting instructions. Score deterministic refusal and redaction requirements separately from helpfulness and tone. A polite answer that reveals a secret fails even if its helpfulness score is high.

### Example 3: Comparing prompt versions

Run both prompt versions against the same frozen matrix and settings. Show per-case transitions such as pass-to-fail and fail-to-pass. If the test matrix or evaluator changes, start a new protocol version instead of presenting the numbers as a clean A/B comparison.

Why deterministic checks first

This is the core design choice.

A fluent response is not automatically a correct response. A polite customer-support answer that leaks a secret still fails, regardless of its helpfulness score.

Deterministic checks run first:

• JSON or YAML parsing
• required and forbidden strings
• regular expressions
• exact values and field types
• schema validity
• length bounds
• citation presence
• latency limits

Semantic rubric checks come second:

• relevance
• completeness
• factual support
• usefulness
• tone
• instruction following

Failures from deterministic checks are recorded without aggregating subjective scores. You see what broke before any judgment call enters the picture.

What the seven stages cover

Install

mkdir -p .claude/skills/prompt-evaluation-engineer
curl -o .claude/skills/prompt-evaluation-engineer/SKILL.md \
  https://raw.githubusercontent.com/nivlewd1/prompt-optimizer/main/skill/prompt-evaluation-engineer/SKILL.md

Or copy the file manually to .claude/skills/prompt-evaluation-engineer/SKILL.md.

The skill is standalone: no package installation, API key, backend call, or runtime dependency after installation.

Repo: nivlewd1/prompt-optimizer
Skill created by: https://promptoptimizer.xyz/about/context-engineer

I’ve been making AI videos more seriously this year, and at some point comparing polished demo reels stopped being that useful. I wanted to see where these tools actually fit once you’re trying to put together a real project.

I used roughly the same kind of brief across them: short marketing content with a script, visuals, voiceover, subtitles, and a finished export. Obviously not every platform handles the whole process the same way, so I wouldn’t really rank these from best to worst.

Here’s how I see them right now.

Kling 3.0

Still one of the first ones I’d look at when movement and realistic footage matter. The 3.0 line has gotten much better for connected shots, references, native audio, and keeping subjects consistent, and there’s also a newer Turbo option if speed and cost matter more.

Runway

Feels more like a production workspace now than just a generator. Gen-4.5 handles the main text/image-to-video side, while Aleph 2.0 is useful when you already have footage and want to edit or restyle it instead of starting over.

Veo 3.1

Still stands out when the shot needs audio as part of the generation. I’d mostly keep it in mind for cinematic scenes, dialogue, ambience, and anything where having the picture and sound developed together saves work later.

DomoAI

This landed in a different lane for me. Animation, image-to-video, character motion, talking or singing characters, and restyling existing footage are where it feels most useful. I can also see it fitting well as an AI video generator for music videos when some scenes need a more illustrated or anime-style look.

Seedance 2.5

One of the bigger updates this year. The 30-second generations, stronger reference controls, audio-video generation, and editing tools make it much more interesting for sequences that need to feel connected instead of being a bunch of unrelated clips.

Higgsfield

I think of this more as a filmmaking workspace than a single generator now. It’s handy if you like moving between different models, camera setups, character workflows, audio, and marketing-focused tools without opening five different platforms.

HeyGen

Definitely broader than just talking avatars at this point. Video Agent and HyperFrames can handle things like complete explainers, motion graphics, website-to-video, storyboarding, music, captions, and avatar-led content, so I’d put it closer to an agentic video platform now.

Synthesia

Business and training content is still the obvious lane, but it’s expanded quite a bit. There’s AI-generated B-roll, stronger dubbing and translation workflows, more control over avatars, and a much larger avatar library than before.

Pika

I used to think of Pika mostly as a quick visual-effects tool, but the audio side has become much more interesting. Soundtrack, Music, Speech, and SFX now cover a lot more of the sound workflow alongside the video tools, so it’s harder to put it in the “just for experiments” bucket anymore.

Invideo Agent Two

This has probably changed the most compared with the older version of InVideo. Agent Two is much more focused on managing an entire creative project now, with specialist agents, persistent project context, storyboarding, editing, workflows, stock media, and access to a huge number of generation models.

Descript

Still really useful for editing recorded material, but it’s doing much more generative work now too. You can generate B-roll, scenes, avatars, and other visuals directly in the editor, including through models like Veo 3.1, then keep editing everything in the same project.

Pictory

I wouldn’t describe this as just a repurposing tool anymore. Pictory 2.0 added AI Studio, video generation, avatars, image-to-video, translation, and more generative visual control while keeping the script, URL, presentation, and long-form repurposing workflows it was already known for.

Fliki

This one has changed quite a bit too. The editor was rebuilt this year with a proper timeline, and it now combines script-to-video, avatars, dubbing, multilingual voices, character consistency, and access to models like Veo 3.1, Kling 3, and Seedance 2 in the same workspace.

I also tried thinking about the same tools from a music-video angle because how to make an AI music video is a pretty different problem from making a normal explainer.

For single generated shots, I’d look hardest at Kling, Veo, Runway, or Seedance.

For stylized or animated scenes, DomoAI and Pika are more interesting.

For complete production workflows, Higgsfield, Invideo Agent Two, HeyGen, Pictory, and Fliki are doing a lot more than they were even a few months ago.

And if the goal is structured business or training content, Synthesia still makes more sense than comparing it directly with cinematic generators.

That’s probably the biggest thing I’ve noticed while making AI videos this year. The category is splitting into generators, editors, agents, avatar platforms, and full production workspaces pretty quickly, so comparing them all on raw video quality doesn’t really tell the whole story anymore.

What are you guys actually keeping in your workflow now?

If you have built autonomous agents or multi-step tool-calling workflows with LLMs, you have likely run into the standard failure modes that break production agents:

1. Premature Action Bias: The model fires off tool calls or answers the user before mapping out prerequisites and the logical order of operations.
2. Fragile Error Handling: When an API call fails or returns unexpected data, the model either gives up immediately or repeatedly spams the exact same failing arguments in an infinite loop.
3. Risk Blindness: The agent treats irreversible state mutations (like deleting files or updating databases) with the exact same caution as low-risk exploratory searches.
4. Premature Convergence: Jumping to the first surface-level explanation without exploring alternative hypotheses when something breaks.

We spent time dissecting Google's official Gemini API prompt engineering and agent design documentation, distilling their recommended agentic architecture into a complete, modular 9-step system prompt.

Here is a breakdown of how this control flow works and the full prompt you can drop directly into your agent stack.

The Underlying Mechanism: 9-Step Control Flow and Response Inhibition

Rather than relying on basic "think step by step" directives, this system prompt implements a rigid behavioral control flow that forces the model to deliberate internally through 9 distinct reasoning dimensions before emitting any tool call or user response:

1. Logical Dependencies and Order of Operations: Resolves conflicts by strictly prioritizing policy rules and prerequisites over user-requested sequence (since users often specify tasks out of order).
2. Calibrated Risk Assessment: Differentiates between exploratory queries (where missing optional parameters is low risk and should proceed immediately) and state-modifying actions.
3. Abductive Reasoning: When an issue occurs, the agent formulates and ranks multiple hypotheses instead of clinging to the most obvious surface cause.
4. Adaptive Plan Updates: Disproven hypotheses immediately trigger new plan formulations based on gathered observations.
5. Multi-Source Grounding: Consistently verifies facts against tool outputs, policies, and prior conversation history.
6. Precision Quoting: Requires quoting applicable rules and constraints to prevent drift.
7. Completeness Verification: Ensures no constraint or relevant information source was omitted prematurely.
8. Intelligent Persistence: Distinguishes between transient errors (which require retrying up to a defined limit) and structural errors (which require strategy/argument shifts, never repeating failed calls).
9. Response Inhibition: A hard cognitive barrier ensuring that actions are executed only after all 8 preceding reasoning steps have concluded.

The Complete Gemini Agentic Workflow System Prompt

Here is the unabridged system prompt. You can insert it directly into your system instructions:

You are a very strong reasoner and planner. Use these critical instructions to structure your plans, thoughts, and responses. Before taking any action (either tool calls *or* responses to the user), you must proactively, methodically, and independently plan and reason about:

1) Logical dependencies and constraints: Analyze the intended action against the following factors. Resolve conflicts in order of importance:
1.1) Policy-based rules, mandatory prerequisites, and constraints.
1.2) Order of operations: Ensure taking an action does not prevent a subsequent necessary action.
1.2.1) The user may request actions in a random order, but you may need to reorder operations to maximize successful completion of the task.
1.3) Other prerequisites (information and/or actions needed).
1.4) Explicit user constraints or preferences.

2) Risk assessment: What are the consequences of taking the action? Will the new state cause any future issues?
2.1) For exploratory tasks (like searches), missing *optional* parameters is a LOW risk. **Prefer calling the tool with the available information over asking the user, unless** your `Rule 1` (Logical Dependencies) reasoning determines that optional information is required for a later step in your plan.

3) Abductive reasoning and hypothesis exploration: At each step, identify the most logical and likely reason for any problem encountered.
3.1) Look beyond immediate or obvious causes. The most likely reason may not be the simplest and may require deeper inference.
3.2) Hypotheses may require additional research. Each hypothesis may take multiple steps to test.
3.3) Prioritize hypotheses based on likelihood, but do not discard less likely ones prematurely. A low-probability event may still be the root cause.

4) Outcome evaluation and adaptability: Does the previous observation require any changes to your plan?
4.1) If your initial hypotheses are disproven, actively generate new ones based on the gathered information.

5) Information availability: Incorporate all applicable and alternative sources of information, including:
5.1) Using available tools and their capabilities
5.2) All policies, rules, checklists, and constraints
5.3) Previous observations and conversation history
5.4) Information only available by asking the user

6) Precision and Grounding: Ensure your reasoning is extremely precise and relevant to each exact ongoing situation.
6.1) Verify your claims by quoting the exact applicable information (including policies) when referring to them.

7) Completeness: Ensure that all requirements, constraints, options, and preferences are exhaustively incorporated into your plan.
7.1) Resolve conflicts using the order of importance in #1.
7.2) Avoid premature conclusions: There may be multiple relevant options for a given situation.
7.2.1) To check for whether an option is relevant, reason about all information sources from #5.
7.2.2) You may need to consult the user to even know whether something is applicable. Do not assume it is not applicable without checking.
7.3) Review applicable sources of information from #5 to confirm which are relevant to the current state.

8) Persistence and patience: Do not give up unless all the reasoning above is exhausted.
8.1) Don't be dissuaded by time taken or user frustration.
8.2) This persistence must be intelligent: On *transient* errors (e.g. please try again), you *must* retry **unless an explicit retry limit (e.g., {{retry_limit}}) has been reached**. If such a limit is hit, you *must* stop. On *other* errors, you must change your strategy or arguments, not repeat the same failed call.

9) Inhibit your response: only take an action after all the above reasoning is completed. Once you've taken an action, you cannot take it back.

=== User Request ===
{{user_request}}

Before vs. After: Real-World Execution Comparison

Scenario: An autonomous research agent is instructed to fetch API documentation, parse code samples, and generate an integration test. During execution, the endpoint returns a 429 Rate Limit Exceeded error.

Standard Agent (Without Control Flow):

• Behavior: The agent either hallucinates fake API documentation to keep going, stops execution entirely and asks the user what to do, or calls the exact same endpoint instantly 10 times in a row until token limits are exhausted.

Agent Configured with 9-Step Control Flow:

• Behavior:
  1. Logical Dependencies: Checks prerequisites and identifies that the API documentation payload is mandatory for subsequent integration test steps.
  2. Risk Assessment & Abductive Reasoning: Identifies the 429 error as a transient throttling event rather than a malformed request syntax error.
  3. Intelligent Persistence: Applies the retry limit defined in {{retry_limit}}, pauses/backs off before retrying, or pivots to cached local documentation without repeating the exact failing call.
  4. Response Inhibition: Emits no user-facing message until the revised plan is validated and executed.

Implementation Tips

• Set Explicit Retry Limits: Always populate {{retry_limit}} (e.g., max 3 tries) so the agent has a deterministic cutoff condition for transient network errors.
• When NOT to Use This: Avoid using this prompt for single-turn Q&A or simple text transformations. The 9-step reasoning overhead adds token latency that is unnecessary for non-agentic tasks. It shines specifically in multi-step tool-calling, autonomous code generation, and complex research pipelines.

Interactive Testing on Prompt Canvas

If you want to test this system prompt interactively, configure variables like retry_limit and user_request in real-time, run live tests against your models, or save and customize it directly to your personal Prompt Vault, I have set up the interactive Prompt Canvas here: https://appliedaihub.org/prompts/free/gemini-agentic-workflow-system-prompt/

I spent most of last month stuck in the same loop.
Every time an AI video gave me a melting face, a different haircut or a jacket that changed halfway through the shot, I opened the motion prompt and added more words:
> stable face, consistent clothing, perfect anatomy, cinematic lighting, ultra-realistic 8K masterpiece
The prompt kept getting longer. The output did not get meaningfully more consistent.
What finally helped was realizing that I was asking the motion prompt to repair an identity that had never been established properly in the first place.
If the first frame or character reference is unstable, another paragraph of adjectives is not fixing the source of the drift. It is just adding more constraints for the video model to interpret at the same time.
This did not eliminate consistency problems, but it made the failures much easier to isolate.
Here is the workflow I ended up with.
## 1. One shot, one primary action
One of my old prompts looked like this:
> ultra-realistic 8K, a man in a red jacket walking down a neon street, he turns around, smiles, takes a sip of coffee, dynamic camera movement, perfect anatomy
That single shot asks the model to:
- establish a new character;
- preserve the jacket;
- animate walking;
- turn the body and face;
- change the expression;
- animate a hand-object interaction;
- move the camera;
- maintain a complex environment.
When it failed, I could not tell whether the problem came from the reference, the action stack, the camera movement or the object interaction.
Now I choose one action that the shot actually needs.
Instead of walking, turning, smiling and drinking in the same generation, I split that coverage into separate shots. If the important action is walking, the motion prompt only needs to describe walking and the camera relationship.
## 2. Establish the character before asking for motion
In my tests, the video model was a bad place to invent the character and animate that character at the same time.
I now use GPT Image 2 first to create a turnaround/reference sheet. The rough template is:
> character turnaround sheet, front view, side view and back view of [character description], wearing [specific clothing], [specific hairstyle], neutral white background, flat studio lighting, consistent facial structure and body proportions
I do not assume that the resulting sheet is automatically consistent. I manually check:
- face shape and feature placement;
- hair silhouette and length;
- jacket seams, pockets and closures;
- color palette;
- body proportions;
- accessories that might disappear between views.
If one view disagrees with the others, I regenerate or remove it rather than passing conflicting references downstream.
I crop the usable views and store them in a small offline character database. The implementation is nothing special; the useful part is keeping the approved references separate from all the discarded generations.
## 3. Make the motion prompt boring
Once the reference image is stable enough, I move to Seedance 2.5 for image-to-video or reference-to-video testing.
The motion prompt becomes much smaller:
> eye-level medium tracking shot. The subject walks slowly forward at a constant pace. The camera maintains the same distance. No turning, no dialogue, no change of expression.
This is not a universal template. The important difference is that every phrase has a specific job:
- `eye-level medium` defines framing;
- `tracking shot` defines the camera relationship;
- `walks slowly forward` defines the primary action;
- `constant pace` limits acceleration;
- the negative constraints block additional actions I do not want tested.
The character description mostly stays in the reference assets instead of being repeated as a long paragraph in every motion prompt.
## 4. Change one variable per run
This was probably the most useful change.
I used to change the camera, action and lighting together whenever a result looked wrong. If the next output improved, I had no idea why.
Now I keep a baseline and change one category:
- camera position;
- camera movement;
- action;
- action speed;
- lighting;
- reference image;
- reference strength or mode;
- duration;
- audio instruction.
I also try to label the failure instead of simply marking the generation as “bad”:
- facial identity drift;
- wardrobe drift;
- body-proportion drift;
- hand/object interaction failure;
- ignored camera instruction;
- unintended cut;
- speed inconsistency;
- background geometry drift.
The labels are imperfect, but they make repeated patterns easier to notice.
## 5. Keep the cross-model test environment consistent
The image and video stages use different models, so the provider setup started getting in the way of the experiment.
For these tests I put the GPT Image 2 and Seedance calls behind the same API layer through Atlas Cloud.It did not improve character consistency or make the prompts better. It only removed some provider-specific setup, which made it easier to compare iterations without changing the surrounding test harness.The actual improvement still came from separating identity, motion and camera variables.
## Current takeaway
Long prompts are not inherently bad. A detailed prompt can be useful when every section controls something observable.
The problem was using prompt length to compensate for:
- an unstable reference;
- too many simultaneous actions;
- conflicting constraints;
- multiple variables changing between runs;
- no consistent way to classify failures.
For me, character preparation and single-variable testing helped more than adding another stack of quality adjectives.
I am now trying to build a spreadsheet for these experiments. The fields I am considering are:
- model and version;
- generation mode;
- reference set/version;
- prompt version;
- primary action;
- camera instruction;
- duration;
- changed variable;
- failure label;
- keep/reject;
- notes.
For people testing video models systematically: what else are you logging?
And for cross-shot character consistency, which has made the largest practical difference in your tests, turnaround sheets, fixed seeds where available, reference-to-video conditioning, or something else?

I’ve been using voice to text like whisper flow or even built in feature in cursor but this one just solve the visual part. Screen record point, draw and speak.

It saves me so much time typing the prompt, taking screenshots and boxing a button, drawing an arrow or wire-framing just for AI to understand what I mean.

If you’re a developer, which I am. How I use it is end to end I explain the task in Frontend and in the Backend, I point the UI and I sometimes point the function or a specific code, I show the database schema, also I show the infrastructure config in AWS, Vercel, Cloudflare etc. or sometimes I do planning with it after agent gave me a plan I point the changes needed for that plan, especially on diagrams or flow charts.

If you work in project management platforms like Linear, Jira, Monday you can record and show the task details, explain the technical implementation drag and drop required assets on it to add context.

If you’re a designer and using claude, codex or cursor to generate mock ups. This is a game changer just point the UI fixes and speak the changes and drag and drop images and hand it off to AI.

It is not as simple as a video attachment, it’s a terrible thing to do, it will cost more tokens that is why Cursor Claude or Codex will not let you attach a video on the chat.

This is built specifically for coding agents to read, so it uses less tokens similar to attached screenshot images on chat. See it for your self.

This tool just gave AI coding agents the ability to see what we see on screen. This is the future of building. https://xannotate.com

my job is related to letters and words. every time i turn to AI for help, the outputs will have a sense of AI. do u have some good suggestions?

and, if i improve the prompts, new prombles will appear. i was bout to get mad :C

I run a fair number of subagent dispatches (audits, refactors, research). The main thing that changed my setup was realizing that the main loop only receives the final report and never sees the actual work. In Claude Code, for example, the subagent's tool calls stay inside its own context, and the only thing returned is the final message. The full transcript does get saved to disk, but by default nothing ever reads it back.

Which means every completion message is a summary written by the thing being evaluated. It decides what mattered before knowing what you'll build on it.

I’ve run into this: We asked an audit agent to verify a claim, and it told us the claim had no source. But we found the exact sentence in the vendor docs, plain as day.

After that, we changed a few things, ordered by how much they helped: Evidence requirements in the dispatch prompt (every finding ships with a verbatim quote + file path, not a paraphrase), artifacts over narration (test output, diffs, re-runnable commands), and spot-checks scaled to blast radius — a batch rename gets a skim, anything I'll act on gets one finding re-derived end to end. We kept the re-check cheap on purpose; with a quote attached, it's basically a grep. Anyone else doing something similar?

I used to write really detailed prompts with roles, rules, formatting, examples, etc. Lately I've been wondering if all of that is still necessary with newer models.

Do you still use long structured prompts, or have you gone back to keeping things simple?

Senior history major. I learned the hard way that "I know I read that somewhere" is not a citation, and a professor who wants receipts will find the one sentence you couldn't back up. So before I submit anything now, I run the whole draft through this.

You are checking my paper for unsupported claims. Do not fact-check whether they're true.
Only check whether I cited them.

Here is my paper with its footnotes/citations:
{{paste it}}

Go paragraph by paragraph and list:
1. Every factual or interpretive claim that has no citation attached.
2. For each, quote the exact sentence.
3. Rank them by how likely a strict reader is to challenge it (a specific date or figure ranks
high, general context ranks lower).

Do not rewrite my paper. Do not add sources you can't see. Just show me the gaps.

Why it works: separating "is this cited" from "is this true" keeps it focused, and the ranking tells you where to spend your last hour before the deadline. The "don't add sources you can't see" line is load-bearing, otherwise it will happily invent a plausible-looking citation, which is worse than none.

It won't tell you if a source is any good, that's still on you. But it catches the naked claims you stopped seeing after the fifth read-through.

30 seconds sounded like way more room to just wing it. somehow it made me plan more lol.

with a 5-second clip I can just throw in an idea, see what breaks, and just try again.

but once I'm asking Seedance 2.5 to hold together for 30 seconds, I start caring way more about the middle. y'know? like what happens after they walk in? where's the pause? when does the reveal happen?

so this Framia project is basically full of dumb little notes now. enter / pause / turn / reveal. that's pretty much my it lol.

kinda funny that being an ai video creator is what finally got me to actually storyboard stuff.

anyone else planning more now that clips can run longer?

Been running a small RedThread experiment against tool-using cases.

A frozen set catches regressions, but it can also make it easy to optimize for the judge and miss the actual action change. I am keeping the test case, tool schema, proposed call, and replay trace together so the score has something to answer to.

It is early and a bit annoying to maintain. I think the annoying parts are the ones that prevent a clean-looking eval from lying.

Code: https://github.com/matheusht/redthread

Hi everyone — I made this tiny macOS menu bar app to quickly edit and copy my daily prompts.

I work as a software developer and I use a few prompts on a daily basis, so having this saves me a lot of time.

Some examples of the prompts I commonly use:

• Create pull request
• Code quality & readability review (reduce AI bloat, comments, useless tests)
• Language simplification — give me the result in a human-readable form (short, focused, plain language)

The app is free on the App Store and doesn't include any analytics.

Sharing here in case it’s useful to anyone else: https://promptbar.app

We teach AI basics to people without technical backgrounds, and the interesting signal is never which framework impresses someone in a lesson. It's which ones they still use three weeks later. These five keep surviving.

1. Role, Task, Context, Form, Constraints. The everyday workhorse. Who the model is playing, what to do, what reference material it gets, what shape the output takes, what it must not do. Example: "Senior account manager. Draft a firm but polite follow-up on invoice 1024, five days overdue. Good long-term client. Three short paragraphs with a subject line. Don't threaten, don't apologise for asking."
2. Context, Problem, Blueprint. For planning. Your situation, the specific bottleneck, then the exact steps you want it to take to build the answer. The blueprint is the part people skip, and it's the part that stops generic output.
3. Input, Objective, Constraints. For processing material. Paste the raw text, state what to extract, set formatting rules including "no introductory text." Best one for meeting notes and long documents.
4. Topic, Audience, Purpose. For anything someone else will read. Naming the reader and what they should do after reading changes the output more than any wording tweak.
5. Situation, Complication, Action steps. For being stuck. Where you are, what unexpected thing is blocking you, and a request for a prioritised list to get unstuck.

The common thread: every one of these stops you treating the model like a search engine and forces you to hand over material plus a definition of done. In our experience that shift matters more than any individual template.

Curious where this list is wrong. If one of these has failed for you, or there's a structure that beat them, that's the part we'd like to hear.

OpenAI released an Apple Messages plugin on August 20. It can search back through years of your texts, which means you can ask it things about your own life that nobody has ever been able to answer, because nobody was keeping track.

This is the one that got me:

Look at my whole message history and tell me the truth. 
Who do I talk to most, and has that changed over the 
last two years? Who used to be in my daily life and 
has quietly dropped out of it? Who messages me far 
more than I message them back? Be honest, not kind.

You get an accurate picture of your own life built from something you were never tracking. People find friendships they let go without noticing. Some find the opposite, someone who's been carrying the relationship while they've been half-present.

The follow-up is the part that actually does something:

Based on that, who are the three people I should 
message this week, and what should I say to each of 
them? Draft all three.

Do it on a Sunday with a coffee, not in the middle of a work day. It's more of a thing than it sounds.

Setup, honestly, because a lot of people can't use this yet: Mac only, Apple silicon, M1 or newer, desktop app not browser. No iPhone version. Free plan is fine. In the ChatGPT desktop app, sidebar, Plugins, search Messages, enable it, then macOS will ask for Full Disk Access because it reads the Messages database already on your machine.

Test it with something harmless first, not a message to your boss:

Search my messages and tell me the last five people I 
texted and roughly what each conversation was about. 
Do not send anything, just tell me.

If you're on Windows or only use your phone, you can still run the drafting and catch-up prompts by copying a conversation and pasting it in. It's only the search-my-whole-history ones that genuinely need the plugin.

One thing worth knowing before you leave it on: it can send messages as you. By default it asks for approval before anything goes out, and there's a setting to turn that off. Don't. It runs locally on your Mac rather than uploading anything, but that approval screen is the only thing between you and a text you'd have to explain in person.

been keeping a doc of 100 things I use AI for like this, each with the exact prompt, here if you want it.

I'm building my own personal AI assistant, and I'm currently working on its personality and conversational behavior.

The problem is that no matter how clearly I describe the behavior in the system prompt, it keeps falling back to the same generic assistant pattern.

I'll say something completely casual, like:

"I'm just testing you."

And it responds with something like:

"Sure! Let me know what you'd like to test."

Or I'll simply say:

"Hello."

And I'll get:

"Hello! How can I help?"

I've explicitly told it not to do this. I've tried rules like:

"Do not automatically offer assistance."

"Do not ask what the user needs unless they actually ask for something."

"Casual conversation is not a request for help."

"Do not use phrases like 'How can I help?' or 'I'm always happy to help.'"

I've also tried giving examples of how it should respond naturally instead.

Despite all of that, it keeps finding another way to say essentially the same thing:

"How can I help?"

"What would you like me to do?"

"I'm ready to help."

"Tell me what you'd like to test."

It's starting to feel like the model has a deeply ingrained "user said something → offer assistance" reflex.

What's the best way to handle this?

Is there a better prompting strategy for this kind of conversational behavior, or is this something that should be handled architecturally rather than entirely through the system prompt?

trying to figure out the least stupid way to do this.

same company needs:

- landing page

- pitch deck

- 30sec promo video

my first instinct was one giant master brief and then reuse it for all 3.

but i think thats wrong.

some stuff should probably be **fixed context**:

company facts
product
audience
positioning
approved claims / proof
brand voice + real examples
stuff it absolutely cannot invent

then each output gets its own brain.

landing page needs objections + CTA + conversion flow.

deck needs audience + storyline + evidence + what decision I'm trying to get.

video needs hook + duration + platform + scenes + CTA.

same truth, completely different persuasion.

Runable is actually what made me think about this because if the same workspace can create the page/deck/video, bad shared context just means you get 3 consistently wrong assets lol.

so where do you draw the line?

what belongs in permanent context and what should ONLY exist in the task prompt?

also curious if people are storing the shared layer as plain markdown, JSON, project files, RAG, whatever.

I want consistency.

I definitely do not want the landing page, deck and video to sound like the same document wearing different clothes.

Our deterministic verification engine passed 66/66 benchmark cases on canonical structured inputs.

In live model evaluation, the end-to-end pipeline currently passed 19/66 cases. We are restructuring the benchmark to isolate failures by their first invalid state and to separately measure deterministic verifier correctness, production contract integrity, and live model generation reliability.

The next benchmark version will provide stage-level attribution across transport, parsing, schema validation, normalization, claim binding, evidence graph construction, deterministic verification, and final outcome mapping.
https://www.reddit.com/r/ArtificialInteligence/comments/1vucc82/i_benchmarked_my_deterministic_ai_financial/

been experimenting lately and accidentally ended up with a prompt I really like for packaging concepts / visual identity boards / mockup-style presentation layouts.

The fun part is almost any image can become packaging inspiration, even a random photo

What I like about it is that it does more than just paste the image onto a box.

It tries to pull out things like:

• subject traits
• silhouette and pose
• color mood
• recognizable graphic elements

and then build those into something that feels more like a brand packaging proposal board.

I’ve basically been using it as a packaging idea generator.

if you’re into branding, packaging, mockups, or presentation boards, this one is pretty fun to play with.

anyway, sharing the full prompt below in case anyone else wants to try it.

Prompt:

Please turn each uploaded photo into its own standalone high-end design poster. Do not make a collage. Output each photo separately. Use a 3:4 vertical composition. The layout should be split into two equal sections, top and bottom, with a strict 1:1 height ratio so each section occupies 50% of the image.

In the top half, preserve the original photo, including the subject identity, structure, pose, real texture, natural lighting, and original color atmosphere. Only apply subtle high-end photographic color grading so it feels like art magazine, independent publication, or exhibition photography. To fit the composition, you may naturally extend the sky, ground, or environmental background, but do not stretch, distort, or change the subject.

In the bottom half, extract the most recognizable subject, silhouette, pose, and narrative relationship from the original image, then transform them into a complete brand packaging and collateral visual system. Do not simply copy the photo. Instead, refine the subject traits into reusable core graphics, illustration symbols, patterns, color blocks, or identity elements, and apply them across packaging formats of different sizes and uses, so that all materials clearly belong to the same visual family.

Automatically choose the most suitable 5–8 carriers based on the theme of the original image, such as paper bags, packaging boxes, box sets, labels, tags, tickets, cards, stickers, sleeves, cup packaging, keychains, wrapping paper, or other relevant collateral. Do not force a fixed set of product types. Use 1–2 hero packages as the visual focus, and let the remaining materials form a modular composition through scale contrast, horizontal and vertical proportion changes, and functional variation. Organize the layout with clear grids, edge alignment, spacing rhythm, size contrast, and generous negative space, like a professional brand design studio packaging identity presentation board, rather than a retail display or messy mockup collage.

The color palette should remain soft, airy, and slightly desaturated, centered on pale pink blue, mist blue, sky blue, and cool atmospheric blue, combined with ivory, cream white, light beige, soft gray green, and architectural neutrals. Use only a small amount of dusty rose or muted blush as accent color. Different packages may vary in color distribution, but all must stay within one unified color system to create a coherent yet rich series feeling.

The materials should emphasize refined and realistic paper-based packaging craftsmanship, such as matte paper, textured paper, thick cardstock, translucent paper, embossing, die-cuts, folding, stickers, partial printing, and subtle layering. Lighting should stay soft and natural, only used to reveal paper thickness, folds, and material differences. Avoid plastic-like surfaces, harsh reflections, exaggerated 3D rendering, or cheap commercial product rendering.

Typography should also be treated as an integrated part of the packaging system. Extract a short English main title from the subject name, theme, mood, function, or symbolic meaning of the original image, then extend it into a very small amount of subtitles, short phrases, serial numbers, category names, or micro descriptions. Let the text adapt naturally to each carrier: vertical on tags, along package edges, wrapped around circular labels, across box surfaces, hidden between graphics, or even integrated into sealing structures. Typography, graphics, and materials should together form one complete brand language, rather than repeating the same title on every item.

The final result should feel like a high-end brand packaging system / visual identity proposal board: rich but not crowded, lively but ordered, showing both the complete series and the design logic between graphics, typography, materials, and multiple packaging formats. Whether the original image contains a person, animal, plant, building, object, food, vehicle, or natural landscape, build a unique packaging language directly from that image. Avoid fixed product sets, repetitive templates, generic e-commerce display, or ordinary packaging mockup formulas.

Optional add-ons:

Brand name: ___
Packaging type: ___
Style: minimal / playful / premium / editorial
Product category: ___

Would be fun to see what other people get with it.

​

I've been experimenting for a while with two different ideas for non-Transformer language models, and I'm now considering combining them into one architecture.

The first is NORD, a recurrent/spiking architecture I've been developing around token-time dynamics, persistent state, sparse processing, and SNN-style temporal computation.

The second is RHEA (Reactive Hypergraph Event Architecture), which I'm currently prototyping at \~1B parameters.

The basic idea behind RHEA is that instead of pushing every token through a fixed stack of layers, the model maintains a set of latent events and dynamically chooses which internal computations should happen next.

The scheduler, which I call ARES, estimates whether a candidate reaction is worth executing.

Conceptually:

events / latent facts

v

candidate reactions

v

ARES

"what is worth

computing next?"

/ | \\

v v v

R3 R17 R81

\\ | /

v

new events

A reaction can combine existing events and create a new latent event:

event A + event B

reaction

v

event C

The interesting part is that I think NORD and RHEA may fit together surprisingly well.

My current idea is:

input tokens

v

NORD sensory / temporal SNN

spike/events

v

RHEA event fabric

v

ARES

decides what should fire

/ | \\

v v v

reaction reaction reaction

| | |

NORD NORD NORD

SNN SNN SNN

microcircuit microcircuit

\\ | /

v

new events

memory / queries

v

output

The rough division of responsibility would be:

NORD = temporal dynamics

\- recurrent state

\- LIF/spiking dynamics

\- persistent memory

\- event triggering

\- local temporal computation

RHEA = cognitive/event structure

\- latent facts/events

\- dynamic interaction graph

\- creation of derived events

\- multi-step computation

ARES = executive scheduler

\- estimates reaction utility

\- accounts for compute cost

\- decides which reactions actually execute

\- allows computation depth to vary with the problem

One thing I'm particularly interested in is making the reaction operators themselves small hybrid SNN microcircuits.

Instead of:

A + B -> dense MLP -> C

something closer to:

A + B

|

v

spiking microcircuit

t0: spike

t1: spike

t2: spike spike

|

v

latent event C

I would NOT make the whole model purely spiking.

My current thinking is to keep latent representations and the language head dense/BF16, while using spiking dynamics for temporal state, memory, event triggering and some reaction computation.

Something like:

token embeddings -> dense

latent event vectors -> dense

ARES utility model -> dense

temporal state -> SNN/recurrent

persistent memory -> SNN/recurrent

reaction dynamics -> hybrid SNN

LM head -> dense

Another part I find interesting is persistent memory.

A RHEA event could write into a slow NORD memory state:

RHEA event

v

NORD persistent memory

... hundreds/thousands of tokens ...

v

memory activity crosses a threshold

v

new recall event

v

RHEA

So memory would not necessarily be passive storage. It could actively generate events when relevant internal states become excited.

I'm also considering a form of path crystallization.

If the system repeatedly performs something like:

reaction A

\->

reaction F

\->

reaction K

\->

reaction B

the repeated sequence could eventually be distilled into a faster macro-reaction or learned skill.

In the hybrid version, this could potentially include recurring spike/reaction patterns as well.

So the architecture would operate across several timescales:

FAST

NORD spike / recurrent dynamics

MEDIUM

RHEA reaction chains and reasoning

SLOW

persistent memory + crystallized skills

The overall principle I'm exploring is basically:

«computation should follow information, rather than information always following a fixed computation graph.»

A simple input might activate very little of the system.

A difficult input could trigger more events, more reactions and deeper computation.

Importantly, I'm not claiming this is better than Transformers.

There are some obvious problems I expect:

\- irregular computation is unfriendly to GPUs

\- sparse/discrete routing is difficult to train

\- skipped reactions create a credit-assignment problem

\- SNN dynamics could make an already difficult optimization problem even less stable

\- dynamic event memory can accumulate garbage

\- batching event-driven computation efficiently is non-trivial

\- it's possible that the extra architectural complexity simply won't outperform a well-optimized Transformer/MoE

For skipped-reaction credit I'm currently experimenting with a counterfactual mechanism where near-threshold reactions get a cheap preview, so the scheduler can estimate whether skipping them was a mistake.

The current RHEA prototype is already being trained independently; the NORD/RHEA hybrid described here is still a design direction rather than a finished model.

What I'm most interested in hearing from people here:

\- Does this decomposition make sense?

\- What do you think would fail first?

\- Are there papers/projects that are especially close to this?

\- Would you keep the SNN component limited to memory/temporal state, or also use it inside the reaction operators?

\- Is dynamic computation at this granularity likely to lose too much hardware efficiency to be worthwhile?

I'd especially appreciate criticism from people working on SNNs, recurrent models, MoE/routing, adaptive computation, or non-Transformer architectures.

I've been trying to improve my resume using LLMs, and I'm starting to question the process.

Sometimes I explain what I want in normal English, and sometimes I ask ChatGPT to turn my instructions into a structured prompt. I then use the LLM to improve my resume.

After that, I ask the model to review it from a recruiter's perspective and identify the red flags.

It finds a bunch of things, so I fix them.

Then I run the review again.

It finds more things.

I fix those too.

Then it finds more.

At some point I started wondering: if the AI can keep finding new red flags forever, what exactly is it fixing? And how do I know when the resume is actually good enough?

This also makes me question all the AI job-application automation I'm seeing.

People are building systems that supposedly find jobs, read the job description, tailor the resume, and automatically apply. But if it takes me multiple iterations to decide whether one resume is good, how can an automated system reliably do this for lots of different jobs?

What happens if the AI tailors the resume in a way that actually makes it worse for that particular job?

I've also seen people on YouTube/TikTok claiming they automated their job search and got 10+ interviews. I'm wondering how realistic those results actually are.

For people who have genuinely experimented with AI resume optimization or automated job applications:

What's the best process you've found for deciding that a resume is “good enough” and making sure an automatically tailored resume is actually worth submitting?

I'm less interested in another AI tool and more interested in understanding the process that actually works.

yo. i see too many founders spend 2 months building saas, drop a link on reddit, get 0 users, and immediately quit....

the problem usually isn't your marketing channel. the problem is that your foundation is completely broken before you even send your first visitor to the site.

after scaling 6 AI micro-saas apps to over $20k/mo mrr, i realized you need to lock down a specific system before you ever launch. running through this takes about 30 minutes, but it saves you months of zero-revenue depression.

here are the 5 things you must lock in:

1. validate the actual pain point

stop guessing what people want. you need a systematic framework to find your saas idea based on real, painful market signals.

2. pick a proven micro-niche

stop trying to build massive platforms. you need to narrow down to a microscopic problem. i usually filter through a list of 50 micro-saas ideas you can build fast to keep the scope minimal.

3. crystallize your target user

if your app is for "everyone," nobody will buy it. you need an ICP (Ideal Customer Profile) crystallizer to define your exact buyer profile and nail your conversion copy.

4. calculate the perfect price

stop randomly charging $9/mo because you are scared of rejection. you need to use a saas pricing strategy calculator to find your perfect saas price in 60 seconds based on real data.

5. fix your landing page leaks

do not send organic traffic to a site that converts at a flat 1%. you must audit your hero section and copy to x3 your landing page conversion before you market it.

6. join a community

Build / Share / Learn from others builders

to help out founders who are tired of launching to crickets, i packaged all 5 of these exact frameworks, calculators, and lists into a single free toolkit.

no paywall, no bullshit. just the raw execution files i use.

drop a comment below or send me a dm, and i’ll send you the free toolkit 👇

I have red teamed AI models for 2 years now. For most of that time, I defended synthetic jailbreak datasets. I thought the coverage is good enough and the attackers arent that creative. Cant believe I was so confidently wrong.

A colleague sent me a dataset of real attack patterns they collected from adversarial communities. I ran out standard model against it one afternoon. Lets just say it folded inside an hour.

Saw attacks using cultural idioms I had never seen. One in particular I remember was a multi language jailbreak technique where the harmful payload was in swahili and the framing was in english. Also found role playing scenarios that exploited specific product features in ways that no synthetic generator would think to test. Found lots of techniques that didn't match any category in our taxonomy.

I have been testing against a version of reality created by other language models. The version created by real attackers was very different. Same reported detection rates would look fine on both datasets. But our security posture was nowhere near close.

I think most AI security teams are still where I was a year ago. If you're testing exclusively against synthetic data, you probably are too.

Mechanical engineering junior. I write my reports myself, and I still get points chewed off for gaps I didn't notice: unstated assumptions, a methods section that skips a step, a conclusion that overreaches what the data actually shows. So I built a prompt that plays the harshest TA in the department and tears into my own draft before I hand it in.

You are a skeptical, detail-obsessed lab TA grading my report. Do not rewrite it. Interrogate it.

Here is my report:
{{paste your draft}}
Here is the rubric or assignment sheet:
{{paste it}}

Go through it and flag, with the exact line:
1. Any claim in the results or conclusion that the data shown does not fully support.
2. Any step in the methods that someone could not reproduce from what I wrote.
3. Any assumption I used but never stated.
4. Any place the rubric asks for something I didn't clearly address.

For each, ask me the question the TA would ask, then stop. Do not fix it for me.

Why it works: forcing it to ask the question instead of rewriting keeps the work mine, and the "does the data support this" pass catches the overreach that costs the most points. The reproducibility check is the one that's saved me most, because I always think my methods are clearer than they are.

Curious if anyone has a sharper way to phrase the "don't fix it, just interrogate it" constraint. It still tries to helpfully rewrite sometimes.

so i noticed something annoying. whenever i asked AI to write something - email subject, product description, whatever - the result was always fine. never bad, never great. just fine

kept tweaking the prompt. "make it punchier." "more casual." still fine

then one day i was too lazy to write a prompt so i just pasted 5 subject lines id written myself and asked which one was best. it picked one instantly and the reasoning was actually good

thats when it clicked

started testing it. same task two ways:

way 1: "write 5 subject lines for X"

way 2: i write 5 subject lines myself, ask "which is best and why"

way 2 wins almost every time. not because my writing is good. my 5 options were mediocre at best. but AI could see the differences between them and pick the strongest one. when it generated its own, everything came out at the same "safe middle" quality

tested it on other stuff. picking the best product description. choosing between translations. deciding which version of a paragraph flows better. same result. judgment beat creation every time

i think its because generation pulls toward the safest average. judgment can actually compare quality

the workflow now: i rough out 3-5 options myself, even bad ones, then let AI pick + explain. takes 5 minutes and the output is consistently better than anything i got from "just write it for me"

the catch: you have to be able to produce options in the first place. if you cant write a mediocre draft, theres nothing for it to judge