PR Comments as a Training Loop
This is kind of a sidecar to my previous post about kicking off the Voice CLI project—maybe not really a follow-up, or maybe just going one level deeper on something that worked really well.
Treating the LLM Like a Smart Junior Dev
As I mentioned in that post, the initial Voice CLI phase we were working on ended up needing a fair number of corrections. What I ended up doing was working with a flow I’m very familiar with: going in and making inline comments on the PR.
This fits into a core principle of this work: treat the LLM like a smart junior dev. And one of the main workflows anyone would be familiar with is the pull request review process—giving feedback, making suggestions on how to make changes, things like that.
What’s nice about this approach is that those comments are embedded in the PR and can be easily fetched with the GitHub API—whether that’s through the API itself, the gh command line tool, or one of the SDKs out there.
I ended up creating a small Python script that calls out to the GitHub API by calling gh. I’m sure there’s a Python SDK for this stuff, but I liked the zero-code-dependency approach that just invoking the command line gets you. To get the specific comments, I needed to hit the GraphQL endpoint, which you can actually call with gh api graphql. You can check out the full script below, which was generated by Claude after a bit of back and forth. (I had initially wanted to have it be way more command line + jq focused).
Determinism Over Flexibility
I think this is actually a great example of when actual code’s determinism can beat out the flexibility you get with agent processing. Obviously agent is still in the loop here, but this gets at another core principle: when you can add determinism to your the system, you almost always should. Or similarly, the less you need to the LLM to do, the less it can F it up.
Giving my “junior dev” the ability to read and act to my comments worked really well. I could prompt it to just read the comments and make the changes, and it made the plan, executed it all, and did a great job getting everything into a mergeable state pretty easily. It worked really well.
And it all felt really familiar, working a workflow I’ve done hundreds of times before.
Learning
Continuing with the idea that the LLM is a smart junior dev who takes feedback on their changes well. The expectation I’d have with a new team member is that they don’t need to many passes on a given concept to bring it into how they operate on the codebase. I’d expect the dev to be taking the feedback, internalizing it, and apply that new knowledge for the next time they encounter it.
How do we create a similar feedback loop? I’m not totally sure, but my working hypothesis is to crystalize the review into the LLM’s memory / context. This prompted me to generate a new doc in my docs directory that was the initial pieces of something akin to a style guide.
So now the last piece of the puzzle when prompting Claude to make changes based on PR comments is to inspect the current guide and make any updates that are missing based on the PR feedback. Basically, whenever you encounter a gap in how the LLM operates, encode a fix for it in the repo’s context so that it can have a shot of not stepping in it in the future.
Appendix: format_pr.py
1#!/usr/bin/env python3
2"""Format GitHub PR inline comments as Markdown.
3
4Uses GitHub GraphQL API to fetch review threads with resolved status.
5
6Usage with gh CLI:
7 ./format_pr.py [PR_NUMBER]
8
9If PR_NUMBER is not provided, uses the current PR from the branch.
10"""
11
12import json
13import subprocess
14import sys
15
16def run_gh_command(args):
17 """Run a gh CLI command and return the output."""
18 result = subprocess.run(
19 ['gh'] + args,
20 capture_output=True,
21 text=True,
22 check=True
23 )
24 return result.stdout
25
26def get_repo_info():
27 """Get the owner and repo name from the current git repository."""
28 # Get remote URL
29 result = subprocess.run(
30 ['git', 'config', '--get', 'remote.origin.url'],
31 capture_output=True,
32 text=True,
33 check=True
34 )
35 url = result.stdout.strip()
36
37 # Parse owner/repo from URL (handles both HTTPS and SSH)
38 # Example: git@github.com:owner/repo.git or https://github.com/owner/repo.git
39 if 'github.com' in url:
40 parts = url.split('github.com')[-1].strip(':/')
41 parts = parts.replace('.git', '')
42 owner, repo = parts.split('/')[:2]
43 return owner, repo
44
45 raise ValueError(f"Could not parse GitHub repo from URL: {url}")
46
47def get_pr_number():
48 """Get the PR number for the current branch.
49
50 Returns None if no PR exists for the current branch.
51 """
52 try:
53 output = run_gh_command(['pr', 'view', '--json', 'number', '-q', '.number'])
54 return int(output.strip())
55 except subprocess.CalledProcessError:
56 return None
57
58def fetch_review_threads(owner, repo, pr_number):
59 """Fetch review threads using GitHub GraphQL API."""
60 query = """
61 query($owner: String!, $repo: String!, $pr: Int!) {
62 repository(owner: $owner, name: $repo) {
63 pullRequest(number: $pr) {
64 reviewThreads(first: 100) {
65 nodes {
66 isResolved
67 isOutdated
68 comments(first: 100) {
69 nodes {
70 id
71 author {
72 login
73 }
74 body
75 path
76 line
77 diffHunk
78 url
79 }
80 }
81 }
82 }
83 }
84 }
85 }
86 """
87
88 # Use gh api graphql command
89 output = run_gh_command([
90 'api', 'graphql',
91 '-f', f'query={query}',
92 '-F', f'owner={owner}',
93 '-F', f'repo={repo}',
94 '-F', f'pr={pr_number}'
95 ])
96
97 return json.loads(output)
98
99def format_threads(data):
100 """Format review threads as Markdown."""
101 threads = data['data']['repository']['pullRequest']['reviewThreads']['nodes']
102
103 # Count resolved vs unresolved
104 resolved_count = sum(1 for t in threads if t['isResolved'])
105 unresolved_count = len(threads) - resolved_count
106
107 print(f"# PR Review Comments ({len(threads)} threads)\n")
108 print(f"- **Unresolved:** {unresolved_count}")
109 print(f"- **Resolved:** {resolved_count}\n")
110
111 # Group by resolved status
112 unresolved_threads = [t for t in threads if not t['isResolved']]
113 resolved_threads = [t for t in threads if t['isResolved']]
114
115 # Show unresolved first
116 if unresolved_threads:
117 print("## Unresolved Comments\n")
118 for thread in unresolved_threads:
119 format_thread(thread, resolved=False)
120
121 if resolved_threads:
122 print("## Resolved Comments\n")
123 for thread in resolved_threads:
124 format_thread(thread, resolved=True)
125
126def format_thread(thread, resolved):
127 """Format a single review thread."""
128 comments = thread['comments']['nodes']
129 if not comments:
130 return
131
132 # First comment is the main one
133 first_comment = comments[0]
134 author = first_comment['author']['login'] if first_comment['author'] else 'Unknown'
135 path = first_comment.get('path', 'unknown file')
136 line = first_comment.get('line', '?')
137 body = first_comment.get('body', '')
138 diff_hunk = first_comment.get('diffHunk', '')
139 url = first_comment.get('url', '')
140
141 status = "✅ RESOLVED" if resolved else "⚠️ UNRESOLVED"
142 if thread.get('isOutdated'):
143 status += " (outdated)"
144
145 print(f"### [{status}] {author} on `{path}:{line}`\n")
146
147 if url:
148 print(f"[View on GitHub]({url})\n")
149
150 # Include the code diff context if available
151 if diff_hunk:
152 print(f"```diff\n{diff_hunk}\n```\n")
153
154 print(f"{body}\n")
155
156 # Show replies if any
157 if len(comments) > 1:
158 print("**Replies:**\n")
159 for reply in comments[1:]:
160 reply_author = reply['author']['login'] if reply['author'] else 'Unknown'
161 reply_body = reply.get('body', '')
162 print(f"- **{reply_author}:** {reply_body}")
163 print()
164
165 print("---\n")
166
167def main():
168 try:
169 # Get PR number from argument or current branch
170 if len(sys.argv) > 1:
171 pr_number = int(sys.argv[1])
172 else:
173 pr_number = get_pr_number()
174 if pr_number is None:
175 # Get current branch name for the message
176 result = subprocess.run(
177 ['git', 'branch', '--show-current'],
178 capture_output=True,
179 text=True,
180 check=True
181 )
182 branch = result.stdout.strip()
183 print(f"No pull request found for branch '{branch}'", file=sys.stderr)
184 print("\nTo view comments for a specific PR, "+
185 "use: make pr-comments PR=<number>", file=sys.stderr)
186 print("Or run: ./scripts/format_pr.py <PR_NUMBER>", file=sys.stderr)
187 sys.exit(0) # Exit cleanly, not an error condition
188
189 # Get repo info
190 owner, repo = get_repo_info()
191
192 # Fetch and format review threads
193 data = fetch_review_threads(owner, repo, pr_number)
194 format_threads(data)
195
196 except subprocess.CalledProcessError as e:
197 print(f"Error running command: {e}", file=sys.stderr)
198 print(f"Output: {e.stderr}", file=sys.stderr)
199 sys.exit(1)
200 except Exception as e:
201 print(f"Error: {e}", file=sys.stderr)
202 sys.exit(1)
203
204if __name__ == '__main__':
205 main()
This post was transcribed from a voice memo by the author and later edited for clarity by the author with help from AI.