preface

The environment

MacOS

scenario

Use Python to grab a dynamically rendered JS page using Scrapy, scrapy-splash, and Doker containers.

To the pit

No1, Jenkins Scrapy, Pip, etc Command not found error

** Case Analysis: ** Go to the Jenkins home page -> Mange Jenkins -> System Information, find the table of Environment Variables, and find the value of PATH in the table. The default is/usr/bin: / bin: / usr/sbin, / sbin, look at your PIP, scrapy installed where?

$ which pip
/usr/local/bin/pip

$ which scrapy
/usr/local/bin/scrapy
Copy the code

So. Why command not found? Because may be it.

** Solutions: ** Is to modify the Environment variables for Jenkins. I tried to check the Environment variables in Configura System -> Global Properties, add the key value pair Name and fill in the PATH. Value to fill in/usr/local/bin: / usr/bin: / bin: / usr/sbin, / sbin, (no attempt only fill in/usr/local/bin, has a spirit of the students can have a try).

No2, execute in the build script that introduces a third party frameworkPythonError ImportError: No module named **

Case analysis: The executable file of my crawler reads as follows

main.py

#! /usr/bin python
# -*- coding: utf-8 -*-
# @Time : 2018/11/29 15:58
# @Author : Mars

from scrapy import cmdline

cmdline.execute(['scrapy'.'crawl'.'Here is crawler entry program name'])
Copy the code

In Terminal, python main.py is perfectly fine, but when deployed to Jenkins, the following error occurs

 $ /bin/sh -xe /Users/Shared/Jenkins/tmp/jenkins4266129241674795365.sh
+ python main.py
Traceback (most recent call last):
  File "main.py", line 20, in <module>
    from scrapy import cmdline
  File "/ Library/Python / 2.7 / site - packages/scrapy/set p y", line 27, in <module>
    from . import _monkeypatches
  File "/ Library/Python / 2.7 / site - packages/scrapy / _monkeypatches. Py." ", line 20, in <module>
    import twisted.persisted.styles  # NOQA
ImportError: No module named twisted.persisted.styles
Copy the code

Don’t wonder if there is something wrong with your twisted bag. I have searched online for 2 or 3 days, but I can’t find the same problem with the same style, let alone the same answer. I’ve tried,

  • Upgrade Jenkins
  • Uninstall reshipmenttwisted
  • Increase the PYTHONPATH
  • Increase the PATH
  • .

I ended up putting the program on the desktop by writing a paragraph in Jenkins

cd /Users/Mars/Documents
Copy the code

There was an error when building Jenkins

$ /bin/sh -xe /Users/Shared/Jenkins/tmp/jenkins3643154797228771836.sh
+ cd /Users/Mars/Documents
/Users/Shared/Jenkins/tmp/jenkins3643154797228771836.sh: line 2: cd: /Users/gatscm2/Documents: Permission denied
Copy the code

I immediately typed two letters id in Jenkins’ shell script and built it again.

$ /bin/sh -xe /Users/Shared/Jenkins/tmp/jenkins4686859629267551489.sh + id uid=267(jenkins) gid=267(jenkins) egid=1(daemon) groups=267(jenkins),12(everyone),20(staff),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin), 702(com.apple.sharepoint.group.2),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.acce ss_ftp),101(com.apple.access_screensharing-disabled),102(com.apple.access_ssh-disabled)Copy the code

All operations in Jenkins’ system were performed under the identity of the computer user Jenkins, so why could there be various cases of module failure? Now that we’ve found the source of the problem, let’s solve it.

Solution: Change Jenkins system execution identity

#Stop Jenkins
$ sudo launchctl unload /Library/LaunchDaemons/org.jenkins-ci.plist

#Modify group and user
$Sudo vim +1 +/daemon + '/daemon/staff/' +/daemon + '/daemon/Mars' +wq org.jenkins-ci.plist

#Modify the owning permission of the previous file
$ sudo chown -R Mars:staff /Users/Shared/Jenkins/
$ sudo chown -R Mars:staff /var/log/jenkins

#Start Jenkins
$ sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist
Copy the code

The second step can also be directly open the file to modify,

Find the string tag corresponding to GroupName for key and replace daemon with staff

Find the string tag where the key tag is UserName and replace the daemon with the user name you normally use on your computer

Verify that the modification is successful and try using the id command again

$ /bin/sh -xe /Users/Shared/Jenkins/tmp/jenkins1779228406794725443.sh + id uid=502(Mars) gid=20(staff) groups=20(staff),702(com.apple.sharepoint.group.2),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appser veradm),98(_lpadmin),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),101(c om.apple.access_screensharing-disabled),102(com.apple.access_ssh-disabled)Copy the code

Well, problem solved. == Here, it is suggested to change Jenkins’ user group and user name to the account used by the computer, which can avoid many unknown pits ==

No3, Jenkins use Clone code after GIT plug-in, SSH Key problem

Git clone ** can be used with a local shell command, but it is not possible to use git clone ** in Jenkins.

Solution: Replace the.ssh file in the Jenkins directory with the.ssh file on your PC. Yes! Jenkins had his own set of private and public keys.

Ending

This is the first time in the platform is serious by writing experience output, there are better solutions, welcome to leave a message to attack, grow together.

Talk about the process of writing, how to encounter the problem, how to analyze, how to solve. In particular, how to analyze may feel silly after solving the problem, but in the absence of external help, how to solve their own, after recording.

Turn around and look, HMM! This is growing up!