Status and plans about

SVN to GitLab migration

Zsolt Kovari
zsolt.kovari@cern.ch
BE-CO-APS, Java DevTools team
accsoft-devtools-support@cern.ch

11/10/2018 BE-CO Technical Meeting 1

 Outline
• Introduction and motivation
• Status & overview of progress
• Current status of tooling, migrations
• Outstanding requests towards IT
• GitLab@CERN day outcome
• Plans for migration
• Life with GitLab after migration
• Proposal for coherence and roles concerns
• Summary

11/10/2018 BE-CO Technical Meeting 2

 History of VCS in acc-co
Thanks for the
input for
• Before 1998, started using Razor Alastair, Niall,
and Vito
• In 2003, started using CVS
• In 2008, moved acc-co to SVN
• By 2013 – limited number of rules (pre-commit hooks)
• Consequence: 45 GB of content
•  Had to cut history and start fresh (old history is still available in acc-co-old)
• Present: current acc-co SVN have rules (pre-commit)
• SVN acc-co is relatively clean, only 5 GB of content

• We always change VCS

• Keeping acc-co clean is important
• This is the first time we migrate from a monorepo to separate ones

11/10/2018 BE-CO Technical Meeting 3

 Current life with SVN: acc-co
• Currently, one huge SVN monorepo
• ~20 millions LOC, ~1000 projects
• SVN is easy to use for simple workflows
• For branching it’s more cumbersome
• Easy administration & maintenance
• Centralized configurations, easy integrations/tooling
• We have set up our own Atlassian suite over
10 years ago (now maintained by IN)
• Fisheye&Crucible (sources), JIRA (issues), Confluence (wikis), Bamboo (builds)

11/10/2018 BE-CO Technical Meeting 4

 Trigger/motivation to move to GitLab

• Trigger to move away from SVN

• SVN closure was announced by IT

11/10/2018 BE-CO Technical Meeting 5

 Reminder from
IT Technical
Users Meeting

It was extended
for acc-co

Still the case

11/10/2018 BE-CO Technical Meeting 6

 Trigger/motivation to move to GitLab

• Trigger to move away from SVN

• SVN closure was announced by IT
• Motivation to move to IT’s GitLab
• Use IT services (as we always try)
• Abandon some of our own services (Fisheye&Crucible)
• Some teams would benefit from Git(Lab)
• They either already use it or will use it
• Better to keep acc-co together and support 1 VCS only
• One common VCS at CERN is the preferred solution

11/10/2018 BE-CO Technical Meeting 7

 GitLab in a nutshell
• Every Git repository is part of a GitLab project
• 1 GitLab project = 1 Git repo
• A GitLab project contains more than a repo:
• issue tracking, merge requests, code review, wiki, push rules, CI, webhooks, etc.
• Projects can be organized to GitLab groups
• A group can contain further groups = subgroups
• With GitLab groups and subgroups, we can create any tree hierarchy

11/10/2018 BE-CO Technical Meeting 8

 How to split acc-co? Reminder from
previous TMs

• Have one acc-co main group on GitLab

• https://gitlab.cern.ch/acc-co
• Map the current SVN layout (trunk) to a tree of subgroups and
projects
• 1 project (so 1 Git repo) might contain
• 1 product (e.g. japc-monitoring)
• Multi-products (e.g. whole lsa stack, lsa-core, lsa-server, etc.)

11/10/2018 BE-CO Technical Meeting 9

Status of tooling and migration

11/10/2018 BE-CO Technical Meeting 10

 SVN to GitLab Migration Tool
• Run by us, user has to provide only the preferred SVN to GitLab mapping
• Automatically migrates N products to M GitLab projects
• Either 1 product (1 SVN directory)  1 GitLab project
• Or multi-products  1 GitLab project
• Keeps the history
• Automatically applies:
• same settings, visibility, permissions, push rules, access rules, integrations, etc.
• same .gitignore file
• access to the last contributors from the last X years

It’s ready for migration. We will run it, user

only has to provide the preferred mapping

11/10/2018 BE-CO Technical Meeting 11

How a migrated project looks like

JIRA
integration Unique path

Points to our wiki space Full history

11/10/2018 BE-CO Technical Meeting 12

 Release from Git with CBNG
• Available since June 2018 https://wikis.cern.ch/display/DVTLS/CBNG+Git+Release
• Comparing to SVN, no much new info to say…
• 1 new safety check to avoid accident releases:
• Cannot release if current branch…
• …is not ‘master’, or
• …doesn’t start with ‘release’
• (But it’s possible otherwise, just further verification is necessary (given vcsBranch property))
• ~1000 releases were done in different projects of LSA, Stream, DevOps, OP

CBNG is stable and ready for migration

11/10/2018 BE-CO Technical Meeting 13

 Replacing Atlassian? 1/2
Confluence, JIRA, Bamboo
Responsibility
• Confluence (wikis), JIRA (issues) will remain of BE-CO-IN
• No reason to move away, too many users are involved
• GitLab is not a satisfactory replacement
• Bamboo (builds)
• Under discussion with IN to have Jenkins
• For CBNG users, we will provide auto-generated pipeline

11/10/2018 BE-CO Technical Meeting 14

 Replacing Atlassian? 2/2
Fisheye&Crucible
Responsibility
• Want to move to GitLab and use its functionalities of BE-CO-IN

• Thoroughly evaluated it with the users and

gave feedback to IT RQF1098567, INC1779923, INC1795173
• But at the moment we are not yet convinced
• Certain features are missing/buggy
• Some are just coming (see next section)
• Only when they work, we will be able to move
• And eventually abandon our own Fisheye&Crucible services

11/10/2018 BE-CO Technical Meeting 15

 Status of migrations
• Follow progress on https://wikis.cern.ch/display/DVTLS/Current+progress+of+migrations
• Already migrated
• LSA, Stream, some DevOps and OP products
• In-progress
1. JAPC
• ~400k LOC, 26 modules
2. mpe-java repository (and some acc-co products) of TE-MPE-MS
• 2+ million LOC, ~220 products

• Challenges:
• How to split SVN to Git repositories
• Some GitLab features/integrations are not yet available

11/10/2018 BE-CO Technical Meeting 16

Outstanding requests towards IT

11/10/2018 BE-CO Technical Meeting 17

 Global code search 1/2
• End of July 2017: We first requested from IT-CDA
• End of July 2018: Code search was available on gitlab.cern.ch (after 1 year)
• Indexing the whole instance took 7 weeks
• For testing, we synchronize (hourly) acc-co to GitLab (acc-co-test)
• Collect feedback on wiki

One feedback: “I'm really not convinced by the

GitLab global search. It basically doesn't work, or
it's super slow...”

11/10/2018 BE-CO Technical Meeting 18

 Global code search 2/2
• “super slow”
• General GitLab (or service) problem (see next slide)
• “doesn't work”
• In special scenarios, we have elasticsearch indexing problems INC1779923, gitlab-ce-51689
• We know the problem, there’s a workaround
• We cannot search for special characters, not even for dots INC1795173, gitlab-ee-7098
• Biggest limitation at the moment
• This might be a showstopper for us

Issue is reported to GitLab, it’s already followed-up. I’m

optimistic about a resolution

11/10/2018 BE-CO Technical Meeting 19

 Performance improvements
• GitLab is slow RQF1102675
• Instead of providing “SLOW”/”NOT SLOW” feedback, we proved it with a
benchmark (courtesy of Balint, repo)
• Measures simple daily actions such as project navigations or search
• Runs hourly, every day (see image)

11/10/2018 BE-CO Technical Meeting 20

 GitLab code search in one group: acc-co-test

Big impact of GitLab

maintenance behind
the scenes

GitLab
Median is 20s on Fisheye
GitLab, 5s on
Fisheye
60 sec

20 sec
5 sec

11/10/2018 BE-CO Technical Meeting 21

 Performance improvements
• GitLab is slow RQF1102675
• Instead of providing “SLOW”/”NOT SLOW” feedback, we proved it with a
benchmark (courtesy of Balint, repo)
• Measures simple daily actions such as project navigations or search
• Runs hourly, every day (see image)
• Collaboration between BE-IT
• They improve performance, and we report time series (wiki)
• Good news: we can anticipate improvements
• Due to IT’s Git repo migrations from NFS to S3
• Objective from IT: “bring average search time from 20s to 5s”

11/10/2018 BE-CO Technical Meeting 22

 Code reviews
• Currently, most of the teams want to stay on Crucible
• No tree view in code reviews, hard to review bigger changes (wiki)
• Improvement in next GitLab release
• Upcoming feature: “File browser in merge request”
• Solves tree view problem
• End of November: deployed on gitlab.cern.ch
• Allow code reviews without merge-requests?
• To comment and start discussion
• on 1 commit is available
• on the current state of files is on GitLab’s roadmap
• Since 1/10/2018 gitlab-ce-19976, Milestone: Next 4-7 releases

11/10/2018 BE-CO Technical Meeting 23

 Code reviews: Bottom line

• Promising, we have to re-evaluate again after

the next release
• But we might have to keep Crucible for some
time

11/10/2018 BE-CO Technical Meeting 24

 GitLab Premium features
• Different subscriptions are available: https://about.gitlab.com/pricing/#self-managed
• IT’s GitLab is at Starter subscription
• Next candidate: Premium
• More features, 24/7 emergency support from GitLab

• Some Premium features would…

1. Provide better features for developers (better code reviews, better JIRA integration)
2. Ease administration (replace some of our in-house tooling in the future)
• Update for the whole instance is not an option
• 3 millions $/year
• Discussions for are ongoing

11/10/2018 BE-CO Technical Meeting 25

 Other GitLab related requests

• Custom server side hooks RQF1112454

• Allow external contributors to develop on our code base RQF0845250
• Organize GitLab training

11/10/2018 BE-CO Technical Meeting 26

 Keep SVN service alive
1. Keep SVN service alive after planned closure (June 2019) RQF1123625
• For FESA and BE-ICS repositories
• Expected to have, but without Kerberos and SSH authentication
2. Postpone READ-ONLY deadline (original plan was February 2019)
3. Provide READ-ONLY access to some SVN repositories after closure (June
2019)
• For acc-co, (acc-co-old), mpe-java, acc-cpp
• Reason: we don’t plan to migrate everything for clean-up purposes

IT already acknowledged the justified need of these

requests.

11/10/2018 BE-CO Technical Meeting 27

 GitLab@CERN day

11/10/2018 BE-CO Technical Meeting 28

 GitLab@CERN day
• IT organized a 1 day event https://indico.cern.ch/event/749059/
• Members of GitLab company visited CERN
• Toby Thorslund - Strategic Account Leader
• Job van der Voort - VP
• Kamil Trzciński – Staff developer
• I also made a presentation of our use cases
• “Keeping the illusion of a monorepo”

11/10/2018 BE-CO Technical Meeting 29

 About GitLab company
• 4th Top Software Company
• (huge potential vendor
• Open source lock-in threat by the way)
• 2000+ contributors to GitLab Community Edition • They seem a bit too
ambitious
• Each 22nd of the month: new minor release • Focusing on too many
• (deployed on gitlab.cern.ch 1-2 months later) things at the same time
• GitLab’s vision
• A single application for the whole DevOps lifecycle
• From VCS, CI, binary repository, Issue tracker, Portfolio, etc. (Dev)
• …to CD, monitoring, logging, security, etc. (Ops)

11/10/2018 BE-CO Technical Meeting 30

 Outcome of GitLab@CERN day (1/2)
• Discussing Premium feature possibilities
• Without updating the whole instance for each user (14000)…
• They try to come up with alternative strategies to allow some Premium features
• …for certain groups/users
• Must be followed-up with IT
• Performance improvements
• This seems a general GitLab problem
• Next releases should mitigate the problem
• They are not interested in money contributions anymore (paying for features)
• Which means we cannot pay for delivering certain features for us

11/10/2018 BE-CO Technical Meeting 31

 Outcome of GitLab@CERN day (2/2)

• We (CERN) expressed that we feel a bit “helpless” about GitLab,

namely:
• “Is this pending feature will be developed at some point?”
• “Is this bug will be resolved in the near future?”
• “What is the rational direction of GitLab at the moment?”
• As an answer, Job (VP of GitLab) advised:
• We shouldn’t hesitate to contact him directly or product managers of
GitLab
• …with problems that affect a bigger community at CERN
• So I did! (next slide)

11/10/2018 BE-CO Technical Meeting 32

 Get insight from product manager(s)
• I contacted James Ramsay (product manager of SCM, code reviews, etc.)
• I presented our situation, use cases, and code review concerns
• Next day he replied, opened a proposal to better support our use cases
• Communication is still on-going, followed-up by their colleagues as well
• I hope I can:
• present our use cases and concerns directly to GitLab
• make them listen
• get a better insight of their rational expectations about their short-term plans

GitLab@CERN event was a good opportunity to represent

our use cases, and maybe develop some connection

11/10/2018 BE-CO Technical Meeting 33

 Plans for migration

11/10/2018 BE-CO Technical Meeting 34

 What do we migrate?

• We migrate known active products

• It’s also a good chance for clean-up
• Obsolete, or seemingly obsolete products can remain in SVN
• We can migrate them later even after “SVN closure” (SVN acc-co remains
READ-ONLY)
• Migrate only the trunk by default
• Old branches/tags are polluting
• No technical limitations however
• If you need to migrate SVN branches/tags as well, let me know in
advance

11/10/2018 BE-CO Technical Meeting 35

 Workflow
• Developer sends us a JSON config Not trivial,
• With the preferred SVN to GitLab mapping (wiki) requires a lot
of discussion
• I will verify it, give remarks, recommendations
• I will check all the 1000 product mappings one by one
• If necessary, I will kindly ask for a reconsideration 
• But final decision always belongs to the developers
• Test migration to gitlab-dev.cern.ch
• Verify it on gitlab-dev (layout, history, author mapping, settings, etc.)
• During the final migration
• Set SVN directories read-only
• Migrate to gitlab.cern.ch/acc-co/*
• No SVN-Git synchronization after migration

11/10/2018 BE-CO Technical Meeting 36

 Schedule of migrations
• On-demand migrations now
• If you’d like to migrate something, let us know
• Start “mass” migration after LS2 Baseline
• Not really a mass, rather an incremental migration
• Start with CO projects
• Order is not important
• Finish with OP projects (applications)
• Probably 1 application 1 Git repo mapping
• This has to be done by end of Q2 2019
• Schedule will be here: https://wikis.cern.ch/display/DVTLS/Plans+and+schedule

11/10/2018 BE-CO Technical Meeting 37

 Training, guides are necessary
• IT acknowledged the need for GitLab specific trainings
• Also, some acc-co specific training by us
• Related to our development environment
• e.g.: Eclipse, CBNG
• Probably in more sessions to find available time slots for everyone
• Guides will be here
• https://wikis.cern.ch/display/DVTLS/Git+and+GitLab+guides

11/10/2018 BE-CO Technical Meeting 38

Life with GitLab after migration
(vision and proposal)

11/10/2018 BE-CO Technical Meeting 39

 Coherence concerns
• Imagine we migrated acc-co, everyone is happy*
• What’s next?
• How do we keep it coherent?
• …for the following 5-10 years

*feeling or showing pleasure or contentment

11/10/2018 BE-CO Technical Meeting 40

 We want to do the following in the future
• Keep our clear layout and conventions in acc-co
• Protect top level, apply naming conventions?
• Avoid private/hidden repositories
• Protect our code base (no binaries, no log files, etc.)
• This is a must, see acc-co history
• Use one central solution for the same problem
• Using JIRA instead of GitLab’s issue tracker
• Using Confluence (wikis) instead of GitLab’s wiki
• Make zero hassle for the user to
• Apply the same settings, same .gitignore files, set up CI

11/10/2018 BE-CO Technical Meeting 41

 Problem with GitLab roles
• Gitlab has roles and actions they can do
• Developer, Maintainer, Owner
• Inflexible roles, some of them are not inline with our policies
• For example, Maintainer can:
• Hide source code
• Change pre-receive rules (only SVN admins could do in SVN) – jeopardize keeping the repo
clean
• Developer cannot:
• Edit project’s description
• Add new members
• Set up further minor settings

11/10/2018 BE-CO Technical Meeting 42

Proposal for role and coherence problems

• Since we want to be more flexible

• …the only way we see it now to make a
separate interface which redefines the roles
• This would also allow us to guarantee
coherence

11/10/2018 BE-CO Technical Meeting 43

 Summary

11/10/2018 BE-CO Technical Meeting 44

 Upcoming work before LS2 Baseline
• Re-evaluation of code reviews by teams and us
• Follow up the following matters
• Code search and performance problems
• Code review for different workflows, ongoing communication with GitLab
• Keep SVN service after planned closure
• Start organizing training with IT
• Finish documentations, guides
• Think about SVN to Git mappings
• Continue on-demand migrations
• Discuss proposal for coherence/role problems

11/10/2018 BE-CO Technical Meeting 45

 Conclusion
• We have to and want to migrate from SVN to GitLab
• Done extensive work
• Evaluating GitLab with users and providing feedback to IT
• Insisting on our needs
• We have prepared for migration
• Tooling
• Discussions with teams
• There are still some limitations and missing features

11/10/2018 BE-CO Technical Meeting 46