Every troubleshooting is a treasure, all kinds of dog blood after the table, the kind of satisfaction after solving the problem is irreplaceable.
background
The release system architecture diagram is simplified as follows:
The administrator calls the “publishing program (code-named Varian, hereinafter referred to as Varian)” by Jenkins, and the publishing program will perform a series of initialization operations. After completion, the Docker image will be generated and uploaded to the Docker warehouse. The container cluster will update the image, and users will access our container cluster through load balancing.
The old Varian was developed by shell+ Python and released with Jenkins(JDK1.7). Due to many internal projects, many compatible scripts were written and the code was chaotic. We plan to reconstruct Varian, develop it completely in Python, modularize various functions, and deploy and release different types of projects by assembling modules based on Lego ideas to reduce coupling. Update Jenkins to the latest version and the JDK to 1.8. Now that the new Varian has been developed and deployed for testing, the story begins.
In order to reduce the impact on the existing project, we decided to redeploy a new environment. After passing the test, the old environment was abandoned and the new environment was directly enabled. The new environment information is as follows:
- System: Debian8
- Language: Python3.4
- JDK1.8 + Jenkins2.134
Troubleshooting Process
Fixed nginx access to 403
Jenkins called Varian to normally deploy a static project (pure HTML, CSS, JS and other static resources), visited container cluster through load balancing (refer to the above architecture diagram), and found that the page style could not be loaded. The browser called out the console by F12 and found that a CSS file returned 403 status
Nginx for web services, a quick mental review of the circumstances under which nginx returns 403:
- Nginx is whitelisted, but the IP addresses accessed by the client are not whitelisted
Allow 192.168.0.152; deny all;Copy the code
- The path to access is a directory, and Nginx has a prohibited column directory configured
When accessing a directory, you can list the contents of the directory
autoindex off;
Copy the code
- The access path is a file, but the nginx service configured users and user groups do not have access to the file
This configuration in #nginx specifies the users and user groups for the nginx service
user www-data www-data;
Copy the code
- The directory index is incorrectly configured. For example, if you only have index. HTML in your directory, you have index. SHMTL or index. PHP configured
index index.shtml index.php;
Copy the code
Nginx has configured the user and user group as www-data, but the owner and owner group of CSS files are root, and other users do not have any permissions
# cat /etc/nginx/nginx.conf
user www-data www-data;
# ls -lh csl.css
-rw-r----- 1 root root 7.9K Jul 24 12:34 csl.css
Copy the code
Here is a detailed explanation of file permissions in Linux. The ccl. CSS file in the upper side is used as an example:
-rw-r----- 1 root root 7.9K Jul 24 12:34 csl.css
Copy the code
Split by space
- The first paragraph
-rw-r-----
The 10 characters define the permissions of a file- The first character, here, is
-
It means this is a file, and you’ll see something liked
Represents the directory,l
On behalf of the connection - The remaining nine characters are grouped in groups of three. Characters 2-4 represent ownership limits, characters 5-7 represent ownership group permissions, and characters 8-10 represent permissions of other users
- The characters r=4, w=2, and x=1 represent the read, write, and execute permissions respectively. If this character has a value, it indicates that the owner has the RW read and write permission. For example, the owner has the RW read and write permission, and the owner group has only the R permission, but other users have no permission
- The first character, here, is
- The second paragraph is a number that represents the number of connections to the file
- Root indicates that the owner of the user is root
- Root indicates that the user belongs to the root group
- The fifth paragraph indicates the file size
- The following three paragraphs are the modification time
- The last paragraph is the file name
[root@localhost] [root@localhost] [root@localhost] [root@localhost] [root@localhost] [root@localhost] [root@localhost] Is that the end of the matter? Of course not, think carefully why all other file permissions are ok, only this file permissions wrong? And find out why
tomcat8 UMASK
After repeated tests, IT was found that I had normal permissions to publish and deploy the final file directly by executing python scripts on the console under Linux, but the permissions were incorrect after Jenkins executed the same script.
What is the difference between a console execution and a Jenkins execution? The Jenkins project and Tomcat files are changed to the owner and the owner group are root. The result is still the same.
This CSS file is procedurally generated. The generated file has incorrect permissions. Umask! The word popped up, umask, which controls the permissions to create new files.
A brief introduction to umask: The umask value is used to set the default permission of a user when creating a file. It is opposite to the chmod command to set the file permission. There are a total of four digits, but we usually use only the last three digits, which also correspond to the permissions of the owner owner group and other users. The default file permissions you create are 644 (the maximum initial permissions are 666, umask is 022, and the final permissions are: 6-0,6-2,6-2=644). The default permissions are determined by umask. If umask is set to 000, the file permissions are 666 and the folder permissions are 777. If umask is set to 000, the folder permissions are 755. Then the final permission is 7-0,7-2,7-2=755.
Check the umask of the root user and the umask of the Jenkins user, both of them are 0022, no problem, and the permission of the login to these two accounts to create a new file is normal, there is only one situation left Jenkins!
Jenkins has no place to configure UMASK, Jenkins runs in the Tomcat container, and the old version of Varian also has similar processing logic, which has been no problem. This time, Tomcat8 has been upgraded, does tomcat8 update UMASK? Half a doubt to see, sure enough! The umask of Tomcat8 is changed to 0027 by default and 0022 by default
# vi tomcat/bin/catalina.sh
if [ -z "$UMASK" ]; then
UMASK="0027"
fi
Copy the code
Finally solved the case and revealed the truth to the world!