Keeping AI Agent Skills in Order
The Idea
Skills are the plain Markdown files that tell AI agents how to behave. They pile up fast. Teams copy each other’s, people write personal variants, and sooner or later a personal skill quietly disagrees with a team or enterprise policy.
That is a small governance problem, and it looks more like a document problem than a code problem. I wanted to see how much of it a simple tool could make visible before it causes confusion.
The interesting question was not whether a model can spot a contradiction. It was which questions deserve a model at all, and which can be answered by something cheaper and more predictable.
You point SkillsCop at your folders of skills, and it tells you which ones say nearly the same thing and which lower level skills break the rules above them.
The similarity check compares every pair of skills on both their full text and their description, and lists the pairs that cross the threshold. It uses no model, so it is fast and gives the same answer on every run.
The Build
Skills belong to one of three levels, and the higher level always wins: enterprise, then team, then personal. That hierarchy shapes the whole tool.
Finding near duplicates is a question about wording, so SkillsCop does not use a model for it. It compares every pair of skills with TF-IDF and cosine similarity, once on the full text and once on the description, and reports any pair that crosses a threshold. It is fast, free and gives the same answer every time.
Finding conflicts is a question about meaning, so this is where a model earns its place. SkillsCop only compares across levels, higher against lower, because “does the lower one contradict the higher one?” has a clear answer. A local model first screens each pair with a closed question and returns a probability. Only the pairs above the threshold go to a second model, which writes a short explanation of what conflicts and why.
The results land in a timestamped CSV, with the score and the explanation next to the two file paths, so a reviewer can sort it, filter it and go straight to the files. Every model call is written to a log.
Everything runs locally through Ollama. Skill files are internal policy, exactly the kind of text an organisation does not want to send to a third party, so nothing leaves the machine and there is no per-token cost.
Behind the Build
Designed and built end-to-end by me, from the hierarchy model and the two-stage conflict check to the prompts, settings and reports. It is a personal project with no commercial deployment, and it is open source under the Apache 2.0 licence.
The design choice I would point to is the order of the work. Cheap and deterministic first, the model only where judgement is needed. A lot of duplication is plain copy and paste, and a method that costs nothing should catch it before anything expensive is involved. For conflicts, the model is asked a closed question that returns a number, because a number can be thresholded, sorted and compared across runs, and a paragraph cannot. The long explanation is only paid for on the few pairs that look like a problem.
Two smaller decisions made a difference. The prompts give the model a precise definition of a conflict, and of what is not one, because models over-flag if you let them: two skills about the same topic are not in conflict, and a stricter rule that also satisfies a looser one is not either. And the tool checks its settings before it starts, so a wrong path fails in a second and not an hour into a run.
The conflict check only compares skills across levels, higher against lower. The progress bar shows how many pairs are left, because each one is a call to a local model. Only the pairs a first model flags go on to get a written explanation.
What I Took From It
Conflict detection is a screening aid, not a verdict. A local model can miss a real conflict and flag a compatible pair, so the default thresholds are starting points and not tuned values. Similarity compares words, not meaning, so a rewritten duplicate can score low. It is also slow on large collections, because every cross-level pair is a model call and there is no cache yet.
What does carry over to product work is the pattern. Use the cheapest method that is good enough first, and keep the model for the question that really needs judgement. Ask it a closed question, and let a threshold you can see and change decide what goes forward. And since the person reading the result is the one who has to act on it, give them a spreadsheet of evidence, not a verdict.