To keep track of what we’re on, the company has changed the network to require proxy access. Specific performance is to enter the account and password before the Internet.

So our crawler code has to be rewritten.

Uri target = newUri(sUrl); HttpWebRequest resquest = (HttpWebRequest)WebRequest.Create(target); ProxySetting(resquest); ...// Set the proxy
private static void ProxySetting(WebRequest request)
{
    WebProxy proxy = WebProxy.GetDefaultProxy();// Get the default Settings of IE
    if(proxy.Address ! =null)// If the address is empty, no proxy server is required
    {
        try
        {
            string dm = System.Configuration.ConfigurationManager.AppSettings["strDomain"];
            string name = System.Configuration.ConfigurationManager.AppSettings["strUserName"];
            string pwd = System.Configuration.ConfigurationManager.AppSettings["strPassWord"];
            proxy.Credentials = new NetworkCredential( name,pwd,dm );
            request.Proxy = proxy;
        }
        catch (Exception e)
        {
            throwe; }}}Copy the code

In this way, a problem is to judge, read and set the proxy every time the crawler, which is a great waste of multi-threaded programs, resulting in greatly prolonged grasping time.

Can we set up the Proxy once and then assign it to webrequest.proxy every time we fetch it? Or some network environments do not need a proxy at all, so it is not necessary to set up the proxy at all?

The simplest way to do this is to configure the proxy on the caller and pass the proxy in as a parameter (proxy == null if there is no proxy) and assign it directly to webrequest.proxy. However, in this case, the parameters of the function will change, which will cause many changes in the use of the function, and the cost is too high. What if the caller doesn’t know to set up the proxy?

Another problem is that our fetching function is static, which means we can’t construct an instance in which we initialize a Proxy member variable and then call the fetching function.

I can’t do this, I can’t do that.

October revolution a cannon, then thought of static variables. With proxy as a static variable, you only need to initialize it once (the first time you call it), and then you don’t need to initialize it again for the life of the process.

\

// Static variables
static WebProxy _proxy = null;
// Static constructor
static CSocket()
{
    WebProxy proxy = WebProxy.GetDefaultProxy();// Get the default Settings of IE
    if(proxy.Address ! =null)
    {
        try
        {
            string dm = System.Configuration.ConfigurationManager.AppSettings["strDomain"];
            string name = System.Configuration.ConfigurationManager.AppSettings["strUserName"];
            string pwd = System.Configuration.ConfigurationManager.AppSettings["strPassWord"];
            proxy.Credentials = new NetworkCredential(name, pwd, dm);// Created from the configuration encapsulation parameters
            _proxy = proxy;
        }
        catch (Exception e)
        {
        }
    }
}

// grab the function
Uri target = new Uri(sUrl);
HttpWebRequest resquest = (HttpWebRequest)WebRequest.Create(target);
resquest.Proxy = _proxy;
Copy the code


\

\

\