(From Lebyte)
To introduce myself
Hello everyone, I am a programmer. Crawler programmer, now working at Lybyte; I graduated from a college in 2015. Climbed taobao, jingdong, stem wechat, in short, you usually use almost every platform have been my crawler patronize.
In 2016, I resigned from a large factory to start my own business with the vice president of the factory at that time. The plan was to earn tens of millions of yuan within five years, but the startup failed. In 2019, I left the company after three years and six months and started my job hunting and working life again.
The Internet layoffs in 2019, we are not mixed well. I must be the same. I searched for a job for nearly a month before I found the right one. I remember it only took two weeks to find a job when I first graduated.
However, god pays off. On December 2, 2019, I joined a small company (about 100 people), not that I despise this small company. The thing that he does behind is really too great, I did not expose him is to give him face.
My title is senior Golang engineer, responsible for the recommendation system research and development of a department of the company. After 3 months of efforts, I put the recommendation system from 0 to 1 dry out.
As soon as the 1.0 version of the project was finished, the business division leader suddenly called me to the office on February 25 and said that I had to leave due to the recent personnel adjustment of the company. I was only three months into a six-month probationary contract.
I told the leader that I would complete the handover as soon as possible and try to complete the handover within two weeks. He said, ‘Just two days… MD, at that time, the personnel officer put the resignation application in front of me and urged me to sign it, but I didn’t sign it… I said it’s not over yet. I can put it off as long as I can. The brother I was exchanging with was very cooperative and said it was too difficult. Two more days.
Hr said: “You quit first, and then wechat will ask you if there are any problems later. You should not refuse to help, right?”
My heart: “XXX (sensitive words), you drive people away so you still want me to help you follow-up to solve the problem!” . It took me another day to sign my resignation papers.
Later I learned that if I was dismissed during the probation period, I could get half a month’s compensation. At that time, I should not have signed the voluntary resignation application form given by personnel. I thought it was normal to be dismissed without compensation during the probation period. Also blame me before did not have a good general law, so I hope you remember to beware of small company operation, probation was resigned must at least half a month to compensate or arbitration, small companies generally eat soft afraid of hard, as long as you open his mouth will give.
I would like to thank my brother who took over from me. He helped me hold up the transition process for almost a week, giving me time to find my next job. But he asked the boss to leave me one day after I left because he didn’t want to work in this stupid company either haha… However, because of his illness, he has not gone to work for half a year and can only recuperate at home. Why don’t the good guys get what they want?
After the outbreak of many companies this round of layoffs, I found that I am not suitable for pure backend development. Because I have been working for 5 years, all the knowledge I have been exposed to is around the technology stack of crawler and reverse.
Crawler is always adversarial work, anti-crawler technology is constantly upgraded, crawler also need to upgrade. Others do encryption, you have to do decryption, others do shell, you have to shell. Crawlers can easily move to the back end to build bricks. The back end cannot be easily transferred to the crawler.
In short, the odds of a top reptile engineering firm getting laid off are slim. I think good, I still continue to do reptile, after all, our education is not good, must engage in some choke technology company to value you, otherwise you will be fired.
It’s hard being unemployed
I left the small company on March 3rd. Since then, I have been sending out resumes every day. It is very difficult to find jobs during the epidemic. Boss, hook after the sea are very few calls back, or several outsourcing companies.
During the epidemic, in addition to BATJ, such a giant company, it was able to maintain regular wages for employees and not to lay off employees. Other companies are laying off employees, cutting wages, suspending salaries, and companies that can’t afford compensation but want to cut staff say they’re moving to Huairou, leaving workers to work out the commute, and then leaving them.
I was embarrassed when I didn’t have a job, because I had two mortgages and I couldn’t stop paying them. At ordinary times did not store grain consciousness, so a passive dimission whole I still really quite passive.
At that time, I had more than 20,000 yuan left in my bank account, and the rigid output of mortgage + rental + living expenses was at least 1.8W per month. That means my next job will make at least $1.8 million after taxes even if I take a pay cut.
The pressure is enormous, so save as much as you can. I used to drink mineral water, but now I go to the self-service water machine to drink. I used to take taxis, but now I take the subway and bus.
In order to save money, I eat an apple every morning and noon, and then go shopping and cook by myself in the evening. This can save about 50-70 per day on meals.
During the interview is also very embarrassed, most small companies can not afford to pay, I do not want to reduce too much salary. However, I’ve found that many companies can’t do APP reverse/scraping because it’s a low-tech and obscure technology. Then I reflected on whether I should directly crack the app of one or two big factories, record a video tutorial and upload it to a web disk, and then put the viewing address in my resume. In this way, my resume will be so distinctive that HR will send it to the interviewer.
Once I meet the interviewer, I can overwhelm him with technology. Niu force of junior college and 985, 211 first-class university of comprehensive ability are about the same. The only difference is that people get more interviews, and if you’re not exceptional enough to take advantage of the limited opportunities available, you’re in a tough spot.
So can enter the factory of junior college sometimes his shining point is very general is very strong. Otherwise the same ability company is more willing to high education, can ask you to explain you have irreplaceable.
Shanghai successful entry, education is despised by the boss
I found that Beijing was under very strict control during the outbreak, which caused companies in Beijing to not function normally. So I started sending resumes to Shanghai and Hangzhou…
I don’t know whether it is god’s favor or my own efforts. Finally, a company in Shanghai called me. After 1 or 2 interviews, every question of the interviewer was fooled by me. But they were still upset, mainly because I thought I was charging too much.
The technical director of the other party gave me an assignment to reverse xx APP. I worked from 9 a.m. to 11 p.m., and I didn’t leave the computer except to go to the bathroom and have a meal. I sent the reverse result to the interviewer.
HR called me on Thursday, April 9th and asked me when I could start working. I said next Monday (April 13).
Even though the company is more than 1,300 kilometers away in Shanghai, I don’t think I can afford to wait. I have had no income for nearly a month and a half, and I bought two houses with too much leverage.
I have calculated how much I save and spend each month, so there is hardly any food in stock. I need to get in, pack up a few quick changes of clothes and buy a plane ticket to Shanghai leaving on Saturday. The entry went well…
About my second week on the job, the owner of the company was at a high-level meeting. Directly rebuked the TECH director who hired me:
“Who are you hiring? From now on we only need 985, 211, we don’t want this kind of school. What kind of school is this?” .
Then the TECHNICAL director told me about it, and I was furious. Although MY education is low, at least MY skills are OK. Crawler, 985, 211, I can do it. I can do what I can’t do.
Stand on technology
I’m just not productive yet. You’ll see what I can do when I help you build some apps… In fact, a person’s real power comes from anger, the kind of anger that settles deep in the heart and unleashes all its energy on something.
The business model of this company is very limited by crawler technology, and it’s up to you to analyze your data. And data analysis is based on your ability to pull data. It’s the kind of scenario where you need to monitor the entire web for video, articles and data, and the company has been doing pretty well for the last few years.
Not in recent years, business is more and more difficult to do. The main reason is that the emerging data represented by the short video industry are all concentrated on THE APP. The technology of catching APP is completely different from that of making web crawlers on websites before.
My first task is to grab xx video number, thief nima XX. Due to the high concentration of APP work, otherwise it is very bad to interrupt the reverse thinking. Every night at 9 o ‘clock, I go home from work and collapse into bed after taking a shower.
After a month of struggle, I finally got the XX video number done. Business immediately take the research and development results to brag to party A company, see we can pull some APP data, other bidding companies can not do. Then the company successfully got a cooperation contract with a large company. That’s 30 million!! I made several apps in the back. Anyway, they are all the apps we use every day on our mobile phones.
Say “No” to “42K, 1 million shares”
Now that I’m on my feet, my next step is to teach the boss how to behave. Only 985, 211, right? OK! I’m on a horse hunt. Keep sending resumes. Keep looking. I’ve talked to a lot of companies, but of course a lot of companies haven’t. The reason we want to, in front of I also give you a popular crawler technology stack mistakes, there are other reasons for me.
I made sure that a few days after the departure, the boss met with the director to determine the next data acquisition strategy. The director told my boss THAT I had offered to leave, adding, ‘You got him right’. My boss called me to his office and asked why I left?
Of course I can’t because of your saying, ‘We only want 985, 211’, I told him his mind was made up… Leave out a lot of… Boss: “I give you rise to 42K +100W stock (cent four years take clear), I think you do my technology partner. At present, the company’s market value is X billion, the company will be listed in gem in three years… I have been running this company for 9 years, and I have never met anyone who can do crawler like you. If you can stay here, you will report directly to me (even promoted two levels), and a crawler team of 8 people will lead you.”
My heart: “you don’t want 985, 211?” . I refused, being my boss could be incompetent, without background, or even “without money”. But if you only look at the degree, you can’t.
That’s my bottom line.
Let me give you an analogy. You are doing forward engineering in front and back, client, Java, Python, C++. You are driving all kinds of frameworks every day. Feel free to study the algorithm, see how to code the brick more beautiful.
Crawler/reverse, the nature of our work means we don’t have to worry about JVM memory models, GC tuning, sorting algorithms, data structures, B + trees, red black trees… And we don’t have time to know.
These requirements are more than sufficient for each programming language’s standard library.
Crawler engineers are not the “engine engines” that you forward development engineers need to understand development languages and frameworks. As a result, I sometimes encounter interviewers who tend to ask about the underlying principles of algorithmic development frameworks.
You can crawler technology garbage, but crawler technology does not have a correct understanding, that must be you recruit people are also very garbage ah. Because you use the way of backstage assessment to recruit crawlers, then you recruit people who are also OK with engineering ability, but know a little about crawlers.
In fact, the crawler/reverse engineer is most important to understand the design rules of each operating system and some application loading mechanism. For example, you do web crawler, you have to understand browser kernel rendering mechanism, JS anti-machine recognition principle? How to go to machine fingerprint? How to locate the encryption function? AST syntax tree unscrambles code? How to break dynamic fonts? How does the slider break?
What non-reptilian engineers know about IP pool is the basic skills. We can solve the problem simply by building proxy pool. Why do we need professional reptilian engineers?
If you’re working on Android reverse, you need to know something about Android app development. What are the four components of Android? Don’t start looking at the source code of your app using Jadx, you’ll just be confused. How does Android anti-debug work? How to find the signature method?
A young man: “direct source code search keywords on the line?” What if the source code gets mixed up? How to shell? Even if you have Java layer signature functions figured out, you will find that your knowledge is not enough later.
After others put the signature function in so, you have to understand assembly bar. Code register, segment address register, offset address register, jump instruction, return instruction, ollVM, etc. These knowledge, that is really obscure.
By the time you’ve learned 95% or more of the techniques of reverse analysis, people suddenly stop playing HTTP (s). They make their own protocol, MMTLS have you heard? You now use wechat every day is to walk this agreement. You still want to grab a bag? To be beautiful!
The young man said, “Why don’t you use tcpdump and Wireshark?” . Those who can say this are amateurs, and it’s still the Internet 20 years ago. What protocol is not encrypted? All you can see with these raw TCP packets is encrypted 1010.
As a result, APP reverse/crawler technology has evolved to include sandboxes (actually virtual machines on Android), injection calls, hooks, and more. In a conceptual sense, the development of these technologies is to replace the lower level of inefficient pure reverse engineering with engineering capabilities.
So you have the halfline protocol, which is the encryption method that’s hard to break and restore and I call it by injection call, get the encryption result and then go through the interface. But slowly somebody else call link detection came up, you call directly also bad.
I once did reverse discovery of XXX App, and XXX App would detect the calling link. In short, the nature of a reptile is a kind of cheating behavior, and cheating has high and low points. Low level crawler, is to play automation, make a similar button Sprite to dot dot. The mobile phone is equipped with a MITmProxy to intercept traffic, and the lower-level one reads information directly from elements on the screen. With a large amount of grabbing, the technique becomes difficult.
PS: recommended to watch: Java eight actual combat project collection self-study course B station: BV1Ch411v7fP