Set up your own Git server

According to an analysis report Version Control Systems Popularity conducted by RhodeCode in 2016, Git has almost become a dominant player in today’s VCS (Version Control Systems) field. When choosing your own VCS, 87% of the people choose to use Git, while SVN is the second with only 6%. Whether you look at Google Trends or the questions on Stack Overflow, you can see the explosive growth of Git. In addition, according to the Eclipse Community Survey, the utilization rate of SVN was much higher than other VCS around 2010. Since 2010, the utilization rate of SVN began to decline rapidly. Accordingly, Git usage rose rapidly and surpassed SVN in 2014.

Now, Git has become a must-have skill for programmers, and more and more enterprises are adopting Git. In the open source world, Github is a place for programmers to gather and party, but it is not suitable for private projects. Although Github also supports private projects, setting up your own Git server can be a better choice in many cases. This blog will introduce and learn several ways to set up a Git server.

Git supports four different transport protocols: Local, HTTP(S), Secure Shell (SSH), and Git. These four protocols have different uses in different situations and each has its own advantages and disadvantages. You can choose them according to the actual situation.

I. Local agreement

The local protocol is the most basic Git protocol, which is useful when you want to experiment with Git locally. We first set up two directories: /git/repo as a remote repository and ~/working as a local working directory.

aneasystone@little-stone:~$ sudo mkdir -p /git/repo aneasystone@little-stone:~$ sudo git init --bare /git/repo/test.git An empty Git repository has been initialized at/Git /repo/test.git/Copy the code

We create a bare repository (a repository that does not contain the current working directory) in the /git/repo directory with git init –bare. Next we clone the repository in the working directory:

aneasystone@little-stone:~$ cd ~/working/
aneasystone@little-stone:~/working$ git clone/git/repo/test.git is being cloned to'test'. Warning: You seem to have cloned an empty repository. To complete.Copy the code

We can then use pull and push just like any other repository.

aneasystone@little-stone:~/working$ cd test/
aneasystone@little-stone:~/working/test$ touch 1
aneasystone@little-stone:~/working/test$ touch 2
aneasystone@little-stone:~/working/test$ git add .
aneasystone@little-stone:~/working/test$ git commit -m 'first commit'[master (root commit) 4983f84] 2 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 1 create mode 100644 2 aneasystone@little-stone:~/working/test$sudo git push [sudo] aneasyStone git push [sudo] Delta compression using up to 8 threads. Compress object: 100% (2/2), done. Written to the object: 100% (3/3), 205 bytes | KiB/s, 205.00. Total 3 (delta 0), reused 0 (delta 0) To /git/repo/test.git * [new branch] master -> masterCopy the code

Local protocol not only useful when the Git experiment, if your team has a Shared file system, on this Shared file system can create a remote repository, hang the Shared file system in the local team members, can directly use the local agreement for collaborative development, completely don’t need to set up a dedicated Git server.

SSH protocol

Although the local protocol is simple, it is generally not applicable, because you cannot control the user’s operation on the shared file system. The user has push permission, which means that the user has full Shell permission on the remote directory, and they may inadvertently or even intentionally modify or delete Git internal files and damage the Git repository.

It is safer to use a dedicated Git server. If you have a server that can connect to Git using SSH, setting up Git services is very simple. The first step is to ensure that the SSH service (SSHD) is running on your server. Most Linux server versions include this service by default. If not, you can start by installing Openssh-Server. Then create a Git remote repository on the server:

root@myserver:~# mkdir -p /git/repo
root@myserver:~# git init --bare /git/repo/test.gitAn empty Git repository has been initialized at/Git /repo/test.git/Copy the code

Then clone the library locally:

aneasystone@little-stone:~/working$ git cloneSSH: / / root @ myserver/git repo/test. The git is cloned'test'. root@myserver's password: Warning: You seem to have cloned an empty warehouse.Copy the code

With clone, SSH ://root@myserver is used before the URL. You can also use SCP:

$ git clone root@myserver:/git/repo/test.git
Copy the code

Another difference is that you need to enter the root password of the remote server each time you pull or push. Obviously, having every Git user use root to access a server is not a very secure practice, and there are several ways to solve this problem:

The most obvious method is to create a separate account for each Git user and assign them read and write permissions to the repository. This method works well, but it is very troublesome to manage the account. You can try it when there are not many team members, but it is not recommended.
Another approach is to configure the SSH server to use an existing authentication system to manage users, for exampleLDAPThis is very common in many enterprises and can be savedadduserThe hassle of manually managing server accounts;
Another option is to create a single account, such as git, that has read and write access to the repository and is used by everyone. The advantage of this approach is that it is easier for users to manage and can be used as described belowauthorized_keysThe file manages the user’s public key.

Let’s try a third method. Create an account on your server called git:

root@myserver:~# adduser git
Adding user `git'... Adding new group `git'(1000)... Adding new user `git' (1000) with group `git'. Creating home directory `/home/git'... Copying files from `/etc/skel'. Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully Changing the user informationfor git
Enter the new value, or press ENTER for the default
	Full Name []: git
	Room Number []:   
	Work Phone []: 
	Home Phone []: 
	Other []: 
Is the information correct? [Y/n] Y
Copy the code

Then set up git repository permissions (by default, git repository permissions for rwxr-xr-x mto, only the creator root have write permissions, means using the git account only clone pull, can’t push) :

# chmod a+w -R /git/repo/test.git
Copy the code

We’re using chmod a+w to make the Git repository writable to everyone. What if we wanted to make the repository read-only for some users?

Then you can happily do git locally:

$ git clone git@myserver:/git/repo/test.git
Copy the code

Everything seems fine, but after a few attempts at git, you’ll find that you have to enter your password once for every git operation. First of all, we need to know that as long as we can SSH into the server, we can operate Git, so if SSH supports secret free login, we can “commit code without secret”. Fortunately, SSH supports public key authentication, which requires no password login. In Linux, each user can have one or more key pairs (public and private keys come in pairs). These keys are usually stored in the ~/.ssh directory. Before starting, make sure that you have already generated a public key. Pub = id_dsa.pub = id_rsa.pub = id_rsa.pub

aneasystone@little-stone:~/.ssh$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/aneasystone/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/aneasystone/.ssh/id_rsa.
Your public key has been saved in /home/aneasystone/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:4Ulpufuhs/AgDMb0VXnqMUTw6bD/HrAOI2z9c1cod9I aneasystone@little-stone
The key's randomart image is: +---[RSA 2048]----+ | .oo. | | oo+. | | . o.Oo | | o . . B++ | | + . .. So o | | . + . .. +. + E | | * * + oo + | | . o Oo+.o. | | **+. | +----[SHA256]-----+Copy the code

So we generate two files in the ~/. SSH directory, id_rsa is your private key and id_rsa.pub is your public key. For details on the principles of private and public keys and the RSA encryption algorithm, see my previous article on HTTPS and certificates.

Your Git server is maintained and managed by a dedicated server administrator. After you have generated your public key, you can send an email to the server administrator to apply for Git services, along with your public key. After receiving your request, the server administrator can do the following if he or she agrees:

First copy the public key file to the server:

# scp id_rsa.pub root@myserver:/home/git
Copy the code

SSH: authorized_keys: authorized_keys: authorized_keys: authorized_keys: authorized_keys: authorized_keys: authorized_keys:

root@myserver:/home/git# cat id_rsa.pub >> /home/git/.ssh/authorized_keys
Copy the code

If you want to apply for Git service, you can follow this step. Once this is done, the server administrator will reply to your email notifying you that the Git service is up and running, so you can perform Git operations without entering a password. For more detailed steps in Connecting to Github with SSH, see this guide on Github.

As a server administrator, there is one more thing to consider about SSH, and that is SSH security. You can use SSH to pull and submit code, which means that you can use SSH to connect to the server. Doing anything with a Git repository is a big worry.

Therefore, we also need to put some restrictions on git accounts. By default, our login shell for new accounts is /bin/bash, which is configured in /etc/passwd:

git:x:1000:1000:git,,,:/home/git:/bin/bash
Copy the code

You can use the CHSH command to change the login shell of the user so that he cannot access the server through SSH. How to change? Take a look at the /etc/shells file, which defines all the available login shells. You can change /bin/bash to any of these:

root@myserver:~# cat /etc/shells 
# /etc/shells: valid login shells
/bin/sh
/bin/dash
/bin/bash
/bin/rbash
Copy the code

Obviously, these shells are not what we want. Is there a shell that only allows users to do Git and nothing else? Git-shell: /usr/bin/git-shell Git login shell:

root@myserver:~# chsh git
Changing the login shell for git
Enter the new value, or press ENTER for the default
	Login Shell [/bin/bash]: /usr/bin/git-shell
Copy the code

Git will be rejected if the user attempts to SSH to the server.

Git protocol

The SSH protocol solves the problem of users having direct access to a Git repository, but if we want to give read-only access to a Git repository to everyone except the repository maintainer, which is often common in open source projects and within the enterprise, anyone can look at the repository code. It is very troublesome for the administrator to configure SSH keys for each user. Workarounds can be used to achieve this effect, but they are tedious. Here are the specific steps:

useg+wSet the permission for your Git repository so that the group of users that the repository creator belongs to has write permission, not everyone. (This step can also be done in thegit initWhen you add--sharedParameters);
Then add the Git account to the repository creator’s user group;
Create a git_ro account that has only read-only permissions on the warehouse.
Finally, create a key pair for the GIT_ro account and make the gIT_ro private key public for everyone to use.

As you can see, using SSH ultimately leads to authorization, and exposing the private key is not very elegant. In fact, Git provides another way to make this easier: the Git protocol. To use the Git protocol, you must run a Git daemon on the server. The Git command comes with a daemon parameter:

root@myserver:~# git daemon --reuseaddr --base-path=/git/repo/ /git/repo/
Copy the code

Refer to the documentation of git-Daemon for the above parameters. Git-daemon listens on port 9418. If your server has a firewall, add this port to the whitelist. If you are using Aliyun, add a security group rule like this:

To make our repository accessible to all users, we also need to create a file named git-daemon-export-ok in the repository directory:

root@myserver:~# cd /git/repo/test.git/
root@myserver:/git/repo/test.git/# touch git-daemon-export-ok
Copy the code

Git ://myserver/test.git ://myserver/test.git ://myserver/test.git

aneasystone@little-stone:~/working$ git clone git://myserver/test.git
Copy the code

There are several other things that the server administrator can do, such as setting the Git daemon to start automatically when the server restarts. There are several ways to do this, as described in the Git daemon section of Pro Git.

HTTP(S) protocol

Generally, unauthorized access is carried out through Git protocol and authorized access is carried out through SSH protocol. If your project is an internal project and only aims at some authorized users, SSH protocol is sufficient. However, if both authorized access and unauthorized access are required, SSH and Git protocol may be used together. This is expensive to maintain. This is where our grand finale comes in, the HTTP protocol, which supports both access methods.

HTTP is the most widely used way to access Git services. It supports two modes: Dumb HTTP of the old version and Smart HTTP of the new version. Dumb HTTP is rarely used. Unless otherwise specified, the HTTP protocol mentioned below is Smart HTTP. The advantage of using HTTP is that you can use various HTTP authentication mechanisms, such as username/password, which is much simpler than configuring SSH keys and more acceptable to the average user. If you are concerned about data transfer security, you can also configure THE HTTPS protocol, which is the same as normal Web services.

Let’s try to build a Git server based on HTTP protocol. Pro Git provides an apache-based configuration example, if you are using Apache as a Web server, you can refer to it. Here we use Nginx as a Web server, the principle is essentially the same. HTTP requests are received through a Web server and forwarded to a CGI script named Git-http-Backend that comes with Git.

First we install the required software:

# apt-get install -y git-core nginx fcgiwrap apache2-utils
Copy the code

As a Web server, Nginx itself cannot execute external CGI scripts, which need to be passed through fcGIWrap, just like using php-fPM to execute PHP scripts. Apache2-utils is a Web server tool set provided by Apache. It contains some useful gadgets, such as htpasswd, which can generate Basic authentication files.

Start nginx and fcGIwrap and visit http://myserver to test whether the Web server is accessible:

# service nginx start
# service fcgiwrap start
Copy the code

Then we open and edit Nginx configuration file (/etc/ Nginx /sites-available/default) :

location / {
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend;
        fastcgi_param GIT_HTTP_EXPORT_ALL "";
        fastcgi_param GIT_PROJECT_ROOT /git/repo;
        fastcgi_param PATH_INFO $uri;
        fastcgi_param REMOTE_USER $remote_user;
        fastcgi_pass unix:/var/run/fcgiwrap.socket;
}
Copy the code

Here we set a bunch of FastCGI parameters with fastCgi_param as follows:

SCRIPT_FILENAME: specifies a CGI scriptgit-http-backendRepresents that each HTTP request will be forwarded to the CGI script;
GIT_HTTP_EXPORT_ALL:git-http-backendBy default, only directories can be accessedgit-daemon-export-okIf GIT_HTTP_EXPORT_ALL is specified, all Git repositories are allowed to access.
GIT_PROJECT_ROOT: the root directory of the Git repository;
REMOTE_USER: passes the authenticated user information to the CGI script, if any;

After the change, we restart Nginx and clone the repository via HTTP:

aneasystone@little-stone:~/working$ git clone http://myserver/test.git
Copy the code

4.1 Enabling Identity Authentication

So far, everything is OK, but when we push the code, we get the following error 403:

aneasystone@little-stone:~/working/test$ git push origin master
fatal: unable to access 'http://myserver/test.git/': The requested URL returned error: 403
Copy the code

To resolve this error, you can find a description of git-http-backend in the official documentation:

By default, only the upload-pack service is enabled, which serves git fetch-pack and git ls-remote clients, which are invoked from git fetch, git pull, and git clone. If the client is authenticated, the receive-pack service is enabled, which serves git send-pack clients, which is invoked from git push.

The first time you read this sentence, it may be confusing because we are not familiar with the concepts of upload-pack, fetch-pack, receive-pack and send-pack mentioned here. By default, only authenticated users can push code. If a Git repository wants all users to have permission to push code, you can set http.receivePack for the repository:

root@myserver:/# cd /git/repo/test.git/
root@myserver:/git/repo/test.git# git config http.receivepack true
Copy the code

Of course, the best way is to enable authentication for push operation. There is a configuration of Lighttpd on the official website that we can use for reference:

$HTTP["querystring"] = ~"service=git-receive-pack" {
	include "git-auth.conf"
}
$HTTP["url"] = ~"^/git/.*/git-receive-pack$" {
	include "git-auth.conf"
}
Copy the code

This configuration seems simple enough, but to understand why it’s done, you need to take a look at Git’s internals. As described in git-http-backend, when git fetch, git pull, and Git clone is performed on the git client, the uploat-pack service is invoked. When git push is performed, The receive-pack service is invoked. To make this clear, let’s take a look at the Nginx access log.

Git clone

[27/Nov/2018:22:18:00] "GET /test.git/info/refs? Service = git - upload - pack HTTP / 1.1"200, 363,"-" "Git / 1.9.1"
[27/Nov/2018:22:18:00] "POST/test. The git/git - upload - pack HTTP / 1.1"200, 306,"-" "Git / 1.9.1"
Copy the code

Git pull:

[27/Nov/2018:22:20:25] "GET /test.git/info/refs? Service = git - upload - pack HTTP / 1.1"200, 363,"-" "Git / 1.9.1"
[27/Nov/2018:22:20:25] "POST/test. The git/git - upload - pack HTTP / 1.1"200, 551,"-" "Git / 1.9.1"
Copy the code

Git push:

[27/Nov/2018:22:19:33] "GET /test.git/info/refs? Service = git - receive - pack HTTP / 1.1"401, 204,"-" "Git / 1.9.1"
admin [27/Nov/2018:22:19:33] "GET /test.git/info/refs? Service = git - receive - pack HTTP / 1.1"200, 193,"-" "Git / 1.9.1"
admin [27/Nov/2018:22:19:33] "POST/test. The git/git - receive - pack HTTP / 1.1"200, 63,"-" "Git / 1.9.1"
Copy the code

Clone request /info/refs? Service =git-upload-pack, then request /git-upload-pack; Push request /info/refs first? Service =git-receive-pack, then request /git-receive-pack, so in the lighttpd configuration above we see two records. If we want to do push access, we should restrict both requests. For Git transfer principles, see the Git Internal Principles – Transport Protocol section of Pro Git.

The Nginx configuration file looks like this:

location @auth {
        auth_basic "Git Server";
        auth_basic_user_file /etc/nginx/passwd;

        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend;
        fastcgi_param GIT_HTTP_EXPORT_ALL "";
        fastcgi_param GIT_PROJECT_ROOT /git/repo;
        fastcgi_param PATH_INFO $uri;
        fastcgi_param REMOTE_USER $remote_user;
        fastcgi_pass unix:/var/run/fcgiwrap.socket;
}

location / {
        error_page 418 = @auth;
        if ( $query_string = "service=git-receive-pack" ) {  return 418; }
        if ( $uri ~ "git-receive-pack$" ) { return 418; }

        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend;
        fastcgi_param GIT_HTTP_EXPORT_ALL "";
        fastcgi_param GIT_PROJECT_ROOT /git/repo;
        fastcgi_param PATH_INFO $uri;
        fastcgi_param REMOTE_USER $remote_user;
        fastcgi_pass unix:/var/run/fcgiwrap.socket;
}
Copy the code

The same configuration can also be put in a common configuration file with include directive, so that the user name and password need to be filled in when pushing. We use the Nginx auth_basic_user_file directive for authentication. The username and password are stored in the /etc/nginx-passwd file, which can be generated using htpasswd in the apache2-utils package mentioned above:

root@myserver:/# htpasswd -cb /etc/nginx/passwd admin 123456
Copy the code

Unable to create temporary object directory Unpack failed: Unable to create temporary object directory

aneasystone@little-stone:~/working/test$ git push origin master
Counting objects: 3, done.
Writing objects: 100% (3/3), 193 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
error: unpack failed: unable to create temporary object directory
To http://myserver/test.git
 ! [remote rejected] master -> master (unpacker error)
error: failed to push some refs to 'http://myserver/test.git'
Copy the code

/ Git /repo: / Git /repo: / Git /repo: / Git /repo: / Git /repo We can make the Git repository directory readable and writable to all, or we can set its owner as a www-data user like this:

root@myserver:/# chown -R www-data:www-data /git/repo
Copy the code

4.2 Certificate Management

We solved the problem of user authentication from the administrator’s point of view, but from the user’s point of view, it is a pain to enter the user name and password every time you submit code. Before introducing SSH, we can use the public key authentication mechanism of SSH to save the trouble of entering a password. Is there a similar method for HTTP? The answer is yes, Git’s credential.helper.

For example, store username and password information in the cache as follows:

$ git config --global credential.helper cache
Copy the code

This is reserved for 15 minutes by default. If you want to change the retention time, you can use the –timeout parameter, or save the password in a file as follows:

$ git config --global credential.helper store
Copy the code

This ensures that your password does not expire, but keep in mind that your password is stored in plain text in your home directory. You can solve this problem by using a Credential management tool that comes with your operating system, such as OSX Keychain or Git Credential Manager for Windows. For more information, see the Pro Git credential Store section.

There is also a more crude and simple way:

aneasystone@little-stone:~/working$ git clone http://admin:123456@myserver/test.git
Copy the code

Fifth, comprehensive comparison

This section provides a comprehensive comparison of the four Git protocols.

Local agreement
- Advantages: Easy to set up, does not depend on external services, directly uses the existing file and network permissions, commonly used for file system sharing
- Disadvantages: The shared file system is inconvenient to configure and use, and cannot protect the warehouse from accidental damage, resulting in low transmission performance
SSH protocol
- Advantages: easy to set up, all data encrypted by authorization, data transmission is very safe, high transmission performance
- Disadvantages: Does not support anonymous access, and the SSH key configuration has certain thresholds for white users
The Git protocol
- Advantages: Suitable for open projects, no authorization, highest transmission performance
- Disadvantages: Lack of authorization mechanism, more trouble to set up, enterprises generally do not default to open 9418 port need to be added
HTTP/S agreement
- Advantages: Supports both authorized and unauthorized access, providing high transmission performance and ensuring data security with HTTPS
- Disadvantages: It is troublesome to set up HTTP services and difficult to manage authentication credentials

6. More advanced tools

This is the most basic way to set up a Git server. If you just want to replace your SVN with a version control system, this is probably enough. But if you want your version control system to have a more user-friendly UI, better manage your users and permissions, support more modern Pull Request functionality, and better integration with CI/CD systems, you need a more advanced tool. You can try GitWeb, Gitolite, Gitlab, Gogs, Gitea, or, if you like, put your code on popular code hosting platforms like Github, Bitbucket, etc.

reference

Version Control Systems Popularity in 2016
Pro Git 2nd edition
Git-http-backend Official document
Connecting to GitHub with SSH
Nginx fastcgi configuration
Remember username and password when pushing Git remotely
Set up Git server – official website of Liao Xuefeng
Configuring the Nginx Git server on an Ubuntu system – Technical column by Zhimin Zhang
Git server HTTP (s) based on the nginx configuration protocol | Yvan の parallel space and time

I. Local agreement

SSH protocol

Git protocol

HTTP(S) protocol

4.1 Enabling Identity Authentication

4.2 Certificate Management

Fifth, comprehensive comparison

6. More advanced tools

reference

Related Posts

Large front end (environment & software) installation memorandum – Efficiency section

Tip Tips | Flink sink schema field design

What if GitHub access is not smooth? Wait online, urgent