Investigating the Reliability of GitHub Traces for Coding-Agent Detection
Independent research project in progress
GitHub repository
I chose "Agentic Much? Adoption of Coding Agents on GitHub" because I found its topic interesting and it matches my research interests. Since this is my first independent research project (if it can even be called that), I wanted to start with a published study that provides both a research method and open-source code. My first step was to run the authors' pipeline, understand its outputs, and use the initial results to define the next research questions.
The next steps for this research study are to investigate how reliably public GitHub traces can identify coding-agent adoption, and how detection results change when traces are missing or modified.
Sample and execution
I selected repositories from my own GitHub starred list. I expected about 63 repositories, but the GitHub API returned 59. I have not confirmed why the counts differed. The request saved the list to data/sina-starred.csv, so the URLs did not appear in the terminal. wc -l showed 60 lines, meaning one header and 59 repository URLs. I did not measure the time spent fetching the list separately.
I deliberately did not apply the original paper's repository-selection filters before running the pipeline. The authors required at least 10 stars, 5,000 lines of code, 100 commits, and recent activity, and excluded forks. I wanted to include smaller repositories as well, including projects that might not meet those thresholds, so I could explore how the detection pipeline behaves on them. I did not pre-filter my list by these criteria. This was a deliberate choice for an exploratory test, but it means my sample-selection method differs from the original study and is not representative of GitHub as a whole.
I set up Python 3.13 and installed the dependencies for the original project. An early import check failed because I ran it from the wrong working directory. Running it from code/ fixed the relative-path issue. A one-repository test on ifixai-ai/iFixAi completed in 40.47 seconds.
I then ran the original agent-impact code on all 59 repositories:
The local default analysis start date was 1 January 2025. The full run took 1 hour, 54 minutes, and 33 seconds. This was the analysis time, not the time needed to fetch the starred list. At first, the progress indicator stayed at 0%, and CPU and network activity did not seem to show much work. The first task completed after 10 minutes and 12 seconds. The log does not establish what caused the delay, but the run later completed normally.
Preliminary results
The run produced 56 successful main-report rows and 3 failures.
Pipeline signal | Result |
|---|---|
File-based signal ( | 19 of 56 |
Commit-based signal ( | 38 of 56 |
| 15 of 56 |
All three flags were false | 16 of 56 |
File and commit signals disagreed | 23 of 56 |
Of the 23 disagreements, 21 had a commit signal without a file signal, while 2 had a file signal without a commit signal. These are outputs of the published heuristics, not verified counts of actual coding-agent use.
The ratio field had 38 numeric values, with a median of 11.818% and a range from 0.081% to 100%. It measures AI-classified commits divided by AI- plus Human-classified commits, excluding BOT-classified commits. It does not measure the percentage of code written by AI.
Problems and limitations
Three repositories failed: labri-progress/agent-impact returned a git show-ref subprocess error, diegosouzapw/OmniRoute returned a GitHub GraphQL HTTP 502, and SinaMombeiny/sinamombeiny.github.io had a file-list retrieval error. I have not confirmed the underlying causes. The pilot also contained future timestamps and -1 values that need investigation. The supplementary script for summarizing results required fixes for a copy-and-paste syntax error and extra CSV columns.
This is a small, self-selected sample. I have not completed manual validation, so disagreement alone cannot show which heuristic is more accurate.
Next steps
I will review repositories where the two signals disagree, along with selected cases where they agree or no signal was detected. I will record the public evidence behind each judgment and note where evidence remains uncertain. After that, I plan to run controlled tests on isolated repository copies or test fixtures, removing or changing selected trace markers to see how the detection results respond. These experiments will help assess possible false positives and missed detections where independent evidence is available.
Paper: Agentic Much? Adoption of Coding Agents on GitHub
Original code: labri-progress/agent-impact
Published dataset: Zenodo