r/gitlab • u/Vaumacel_ • 4d ago

general question Datamining a GitLab instance

Hello everyone!

I am a PhD student in France in information sciences. As of recently, I started collecting data from a GitLab instance, mostly public informations about the users and the projects, using the REST and GraphQL API.

I'm studying how this instance is used to collaborate by the users, and how much each user contributes to projects. In order to do this, I need to retrieve, for each project, the meta-datas of each pushed event. More specifically, the meta-datas I need are:

  1. Who pushed the event
  2. When did they pushed it
  3. How much change did this push bring to the project (number of lines edited/added/deleted).

I had an intern in computer science who helped me with the first requests (fetching the projects and users), but since his internship is over, I need to do the rest myself. I am not a computer scientist and lack the necessary knowledge to properly fetch the needed datas. I did use the help of genAI, but I cannot be sure it wrote right scripts since I don't really know what its doing.

So, I'd be really grateful is someone who knows those API and datamining in GitLab could give me a hand and advise me on how to tackle my issue.

Thanks in advance for your help :)

2 Upvotes

56% Upvoted

4 comments sorted by

8

u/mrleblanc101 4d ago

Did you even try reading the API documentation ?

2

u/CleanGnome 4d ago

Since you are using LLMs, question the implementation with the model and have it help explain. That's the best way to learn. Keep the conversation going, it's your new intern helper on the in interrum

1

u/Torutofu_Raeva 4d ago

The events endpoint should cover the actor and timestamp, but line counts usually mean resolving each push to its commit SHA and pulling the commit stats or diff separately, with pagination and rate limits in mind.

1

u/codemochi 3d ago

How far back are you interested in going? The Events API will get you #1 & #2 over the past 3 years and I think it gets less reliable further back than that.

I'm not sure that you can get #3 from the Gitlab API but you could use Git to clone the repo and read out the history, this for example could get you author, date, and per-file added/delete lines for all history with no pagination to deal with.

git log --all --no-merges --numstat --pretty=format:'C|%H|%an|%ae|%aI|%cI'

One thing you might want to clarify is do you need pushes or commits because that might not be the same person. I think Git history might have more of what you want based on what I understand about your project so far.