preface

The Spring Festival 2020 is destined to be an extraordinary one as the whole country fights COVID-19. Besides the usual routine of not going out, washing hands, wearing masks, we thought, what else can we do in this context? Extended quarantine periods and telecommuting may be the biggest contribution ordinary people can make to the country’s campaign, given the looming end of the Lunar New Year holiday and the increased risk of infection on the return journey. However, in our country, regardless of other industries, at least in our high-tech industry, remote working culture has not been popularized, so we hereby introduce the engineer remote working experience of PingCAP practice for nearly five years to you. This article will try to describe less ideas, but more from the practical aspects of our landing experience, in order to help more friends and companies to take action as soon as possible in such a special moment, for the country and society to contribute our humble strength.

We have been through the practice has proved that in this day and age, at least for similar software engineering mainly mental and creative work, the methodology and the infrastructure, has enough work for remote more efficiency than the traditional model is poor, sometimes even better output (believe that already have students think of the traffic in the morning to the mood and thinking of side effects). Let’s talk about some specific landing experience.

The management philosophy of telecommuting

Telecommuting is nothing new in foreign countries. In Silicon Valley, especially the new generation of tech companies almost all have Remote working genes. There are many reasons behind this that are not covered in this article. If you are interested, check out Remote by David Heinemeier Hansson from 37 Signals. For us, PingCAP has began to practice at the beginning of the company to establish the culture, mainly for this reason: on the one hand, including me, several co-founder is engineer, itself in comparison with the pursuit of efficiency, work form of freedom to improve the efficiency of our work, at the same time we hate inefficient formalism; On the other hand, as a startup company, we don’t want talent to be unable to join us because of geographical limitations.

A good example is our lead architect siddontang, our first hire, who didn’t want to move to Beijing for family reasons and has been working remotely from home in zhuhai for the past few years (this blog details his personal experience). Another very important reason is that we have a global workforce based on an open source development model, so remote working genes have been instilled from the beginning.

Software engineering is a work with brain power as the main resources. Supported by the highly developed Internet technology, it is actually natural for remote work. But why do we feel that remote work is not as efficient as centralized work most of the time? ** In addition to the communication and cooperation barriers brought by distance, we believe that the most fundamental difference is in management philosophy, which is inclined to traditional management thinking or self-driven management thinking. In PingCAP, we have always advocated the latter in corporate culture. ** To do this, we need to establish a strong vision driving force and implement it in our various details, while creating as free, open and sharing a working environment as possible for our colleagues.

Fortunately, it also fits perfectly with the direction we’re working in in the open source space. In PingCAP, for example, we never make any form of clock in, every Friday we all have our routine but no employees TGIF share issues, any one colleague has a chance to stand on the stage to share their work achievements and experience, we send you even peripheral products are again and again on the design, select material selected, And limited supply, in order to let every partner feel warm and respect. All this work seems to have nothing to do with our telecommuting, but little by little we are becoming a powerful remote organization that exists out of form and above form.

02 Goal and plan management

If you ask the question, when do engineering teams need to communicate the most? I think it’s time to make plans and goals.

The first thing we need to address in telecommuting is that we need to establish more clear and efficient management of goals and plans that can be operated remotely. At the macro level, we rely on OKR for company and team objective management at PingCAP. OKR is an increasingly popular objective management tool in Silicon Valley and many Domestic Internet companies. Through exploration, we think OKR is a more suitable for remote work team goal management tool, because compared OKR KPI, in the first place more emphasis on jointly by the team members to develop team goals, so that is easy to make the benefits of the whole team to agree on goals and key results, always keeping the team’s direction. Secondly, OKR enables team members to better understand the significance of doing what they are doing for the team and the company. This is particularly important for remote teams, which greatly promotes the collaboration between departments and people and makes the team more integrated. Finally, OKR has another important feature: Transparency. In our practice, each team has access to the other team’s OKRs, and once you’ve developed your team’s OKRs, you need to announce them at the company level to make sure everyone understands them.

At the micro level, for example, a specific project plan development and execution tracking requires the same transparency. Our practice is that the project leader establishes a global project “map” for each major project, and strives to make sure that even students who join the project halfway can clearly know what the situation is now, where the link of the resources needed, who is in charge and where the risk points are. This is especially important for managers of remote engineering teams. Here’s an example:

Item tracking list for a project

Telecommuting took off when our goals and plans were clearly, efficiently, and transparently formulated, published, and managed throughout the company.

If a workman would do his work well, he must sharpen his tools

Since we are more hands-on here, let’s focus on the tools we use in software engineering telecommuting environments. Enterprise culture and management by objectives we need a relatively long time to gradually establish, the introduction of tools is relatively quick, that is, as the saying goes, to do a good job, must first sharp tools, the use of good tools will make things twice the result with half the effort.

PingCAP’s main product, TiDB, is an open source database. The main workflows we develop are built on Github and fully open to the community. Therefore, our tool chain is also centered on Github, connecting other tools. The following is a complete list of tools (many of these tools have domestic alternative tools. If the company is not like PingCAP with global distribution of employees, it can choose according to actual needs) :

  • GitHub: Code hosting, public RFC, community Issue feedback, product release, Code Review, etc.
  • Zoom: Online meeting.
  • Slack: I don’t know.
  • Wechat, Corporate wechat: INSTANT messaging (yes, we use both, but mostly corporate wechat).
  • Online documentation: document collaboration, slides, tables.
  • Email, calendar.
  • Confluence: Internal documentation, including established design documentation (e.g. internal RFC documentation), wikis, etc.
  • Jira: Bug and Milestone tracking.
  • Trello: Kanban, memos for important clients and events.
  • Jenkins: Continuous integration, Daily Build.

Here is a small example of the workflow we developed:

  1. Assuming that we need to make a new function, the first tool we may use from conception is online document. The colleague in charge will draft a document, describe the general idea in it, and then Share it with relevant colleagues through the function of Share. Most of the time these design documents will be shared in a mailing list that all engineers can share and anyone can comment on or edit.

  2. After the final version of the document is finalized through communication and discussion (I will focus on the communication in the next section), it will be synchronized to Confluence and GitHub (English version will be posted on GitHub if it can be made public).

  3. The project will then be divided into multiple sub-projects, assigned to specific people via JIRA and submitted directly to GitHub. A Reviewer (also Maintainer) of the module of the project will participate in Code Review. Two LGTMS (Looks Good to Me) were collected and tested by various continuous integration tools before being merged into the trunk.

The process of fixing bugs is similar. It is worth mentioning that we developed a bot to synchronize the issues from the community in GitHub to the internal JIRA.

The creativity of good engineers is limitless, especially in the context of remote work, and we encourage engineers to make their own tools to improve their productivity.

In addition to the Issue bot mentioned above, our Chaos testing (recently opened source), CI/CD, even simple dynamic public opinion monitoring on social networks, has automated tools. Some students optimize their work process by automatic means. Here are some interesting examples: Disksing uses App Script to automatically generate her own weekly newspaper (disksing.com/review-reco…). ; Siddontang wrote his own small tool github- CLI (github.com/siddontang/…) To efficiently track the Github projects you’re following; I wrote a little script in Python to collect daily updates of cards on a given Board on Trello and send me a summary email… There are a lot of examples of this, and sometimes it’s amazing how imaginative and hands-on you are, and we strongly encourage you to do these things.

Our IFTTT bot is collecting tweets mentioning TiDB and PingCAP

Github Issue synchronized by our bot

Daily dynamic report automatically generated before the end of the day

Weekly Report automatically generated before Weekly meetings

A weekly personal report generated from the template in advance

Remind everyone to prepare weekly report enterprise wechat robot

SRE robots automatically Merge PR and cherry-pick to the Release branch

With all this fun stuff, the point I’m trying to make is that automation is crucial for remote work when it comes to leaving things to a machine rather than a human. Especially for online collaboration, one more person’s participation means one more communication cost. I have a few suggestions for engineering teams choosing to develop relevant productivity tools:

  1. Choose tools with open APIS to facilitate bot writing and form synergies.

  2. Cut taboo too much too miscellaneous, choose a few good, will be fully used.

  3. Message Hubs, which use IM like Slack as tools, try to keep things in one place as much as possible.

Additional as mentioned above, because we also have some overseas colleagues, customers, and the demand of the overseas community communication, so the main selecting tools are relatively common in the international, if your company’s business in the domestic, these tools can find basic domestic or proprietary deployment of alternatives, such as catalog, Tower, Gitlab, etc.

Remote friendly communication and collaboration mechanism

If the tools discussed above are just the basics, the biggest challenge of working remotely is communication. For a mature team, good communication must be essential, and even the quality of communication determines the quality of doing things. This is not to say that there is less or no communication due to the constraints of remote work. On the contrary, communication in this environment may be more and more detailed, but the form is not limited to face-to-face meetings.

Before talking about our communication practice, I would like to talk about the meaning of communication. First of all, I think the most important meaning of communication is: information leveling. In plain English, for a remote team, everyone needs to know what they are doing, what the team is doing, and what to do. In many companies, this is done in meetings, or even in a loud voice. But in a remote team, communication needs to be more transparent.

Even if it is remote, meetings are still the most efficient way of information leveling in most cases. Video conferencing tools like Zoom have provided a good platform, and the popularity of smart phones and mobile Internet has made it more convenient to participate in meetings. One more thing to mention here is that PingCAP is a heavy user of Zoom (and also an enterprise customer), and the user experience of Zoom is really excellent. Even our meetings at the whole company level are completed through Zoom. (We just learned an exciting news yesterday, and made a friendly advertisement for Zoom. At present, domestic users can use Zoom.com.cn free of charge and for any length of time until the epidemic is effectively controlled.

In PingCAP, by default, all meetings are online because remote students will participate in them.

In terms of content, there are probably two kinds of meetings, one is regular meeting, the other is specific business communication meeting. I believe it is no different from other companies. Here are some practices that we think are better:

In PingCAP, teams (including virtual teams) of all sizes have regular meetings, usually weekly, and for important and urgent projects, daily. The duration and length of meetings may vary. The weekly meeting is a good opportunity for team members to know each other what they are doing, and also for Manager to know whether the direction is wrong and whether the progress is normal.

Another point, to share some practices about meetings:

  1. It is better for the Manager not to make decisions directly in meetings where the information is leveled, such as regular meetings. Because a lot of information may be just received and decisions may not be made after deep thinking. On the other hand, information may not be comprehensive and further cross-team communication is needed.

  2. Use the Calendar. I suggest the R&D team to make Calendar visible to others (financial, business and senior management teams need to consider this). It is a good channel for information levelling by subscribing to the Calendar of colleagues related to you.

  3. The Agenda will be issued before the Meeting, and the Meeting notes will be formed after the Meeting and sent to participants and recorded in the Wiki.

  4. Keep big meetings to a minimum. Amazon’s “two Pizzas principle” also applies to meetings (this is easier said than done, especially when it comes to cross-team collaboration, which we’re struggling with).

Here are a few of our own experiences. At PingCAP we have always required as much documentation as possible because of remoting, and we did practice this rigorously in the early days, but at that time we relied heavily on online documentation and ran into a problem: collaboration was great, but indexing was weak. Many times, the documentation for a particular issue is missing, or there are multiple documents (many of them even discussion papers) describing the same issue.

To address this issue, we later introduced Confluence as a team Wiki, which is mainly used for indexing and searching information. We rely heavily on Confluence and play around with it a lot. Here are just a few examples of best practices:

  1. Create a team Page for each team (similar to the “map” concept mentioned earlier) that indexes everything related to the team so that newcomers can see it at a glance. For example, here’s an example from the TiKV team:

  1. Team weekly report and individual weekly report, the weekly report of each team will be summarized and summarized layer by layer, and issued before the weekly executive meeting. All weekly papers are indexed in Confluence.

  2. Logbook, this is the interesting thing for us, for some exploratory work, such as small experiments, performance testing, optimization of some special scenarios, etc. We will also record them in detail and form a series of experimental logbooks. These logbooks can be retrieved by the keyword Confluence. First, they can be used as materials for future implementation or output into articles, and second, they can be used to prevent repeated work in the future.

With the full introduction of Internal collaboration at Confluence, our document fragmentation issues have been greatly alleviated. The only problem is that Confluence mobile is not working as well as Atlassian would like to see it improved in the future.

Another pitfall comes from the choice of IM tools. This may not be a pit, but more due to the wechat platform itself is not designed for the office scene. Since most of our domestic customers tend to use wechat as a communication channel, as a toB company, we have to adapt to this reality (for example, I have thousands of wechat groups on my phone), which leads to fragmentation of our IM communication, and the remote working environment further amplifies this problem. Maybe the same thing, some students are looking at wechat, some students are looking at Slack, and that leads to message asymmetry. Moreover, wechat is a closed system, without Open API, it is difficult to synchronize to a platform through technical means. Another problem is that the use of wechat makes it difficult and sometimes distressing to separate work from life. This problem has been alleviated to some extent through the introduction of enterprise wechat, but because enterprise wechat is a new IM and also a closed system, the problem of information fragmentation and overseas colleagues’ use habits still exist. In this regard, we are still exploring.

Self-management in a remote office environment

Another important aspect of telecommuting is personal psychological development and self-management. For example, when I work remotely, I have to leave home and go to a coffee shop or WeWork or something, but our lead architect can use his study as an office every day for five years. People are definitely the most important part, but I don’t want to go into too much detail in this article, but I want to focus as much as possible on a more general approach.

In remote Settings, it takes a worker to overcome loneliness and a strong sense of self-discipline to overcome burnout without colleagues around. At this point, I recommend using personal time management tools such as tomato clocks, calendars, etc. But as the company chooses tools, do not take too much, choose a fully used best. It also helps to have a regular routine in your life, as outlined in the blog post by Siddontang cited above.

Another important point is that many engineers tend to be introverted and tend to get caught up in difficulties, especially in remote environments. In such cases, it is important to take the initiative to ask for help and communicate, perhaps even more frequently than in face-to-face situations.

For managers, to achieve this, they need to break down the tasks in detail. In the early stage of communication, they need to repeatedly confirm whether they have reached an agreement with the students who work remotely, which needs to pay great attention to.

06 Online and Offline

PingCAP is not a completely telecommuting company, we still have offices in major cities (Beijing, Shanghai, Guangzhou, Shenzhen, Chengdu, Hangzhou, Silicon Valley). As mentioned in the last section, remote working may look great, but it may not be for everyone. People are social animals, and many times we still need to go from online to offline to have a meal and chat with colleagues. Because of this, our solution is that PingCAP uses a remote work style and culture, but still maintains offices all over the place, so we have an unwritten convention that when you have more than four employees in each city, you can find an office.

In the operation of Office in various places, it is also characterized by PingCAP. Many local subsidiaries of traditional companies are usually located in special offices such as sales, testing, r&d, etc. However, due to our telecommuting culture, our offices are actually more like a virtual organization, that is to say, the classmates of the same team may belong to different offices, or vice versa, each branch may have students of different functions and teams.

There are advantages to this:

  1. As a toB company, our domestic customers are mainly distributed in several major cities, and it is more convenient to carry out customer support and market activities with local branches.

  2. Having colleagues from different departments in the same city office helps to build a more diverse and healthy culture, as well as smoother cross-team collaboration.

The Manager of the Office is more inclined to the management and organization of specific affairs and activities of the local Office. In addition, each Office has an administration to deal with daily affairs. Therefore, our Team building usually has several types:

  1. The local Office will have TB funds and can organize activities by itself.

  2. Whenever the team travels to the same place, organize the team’s TB (of course, most of us are programmers and basically just eat and eat).

Business trips are mentioned here. By the way, we suggest that Managers in remote R&D teams should try to meet most of the team members Face to Face once a month, and these trips can usually be arranged together with customer visits. Offline communication can make online communication smoother.

In general, telecommuting is not perfect, and technology companies like ours have a natural cultural and regulatory soil, but there is still plenty of room for improvement. We welcome your suggestions. I am very glad to share my experience with you at such a long time before telecommuting culture was popularized in China. In this special period, we do not move at home at the same time labor to create value, but also for the society to do a little tiny contribution!