Every database has a variety of logs, and MongoDB is no exception. MongoDB has four types of logs, including system logs, Journal logs, oplog primary/secondary logs, and slow query logs. These logs keep track of different aspects of the MongoDB database. The following describes these types of logs.

The system log

System logs are very important in the MongoDB database. They record the startup and stop operations of MongoDB, as well as any exception information that occurs during the running of the server.

To configure the system log, specify the logPath parameter when starting Mongod

mongod -logpath=/data/log/mongodb/serverlog.log -logappend
Copy the code

System logs are continuously appended to the file specified by logPath.

Journal log

The journaling(diary) logging function is a very important function in MongoDB, which ensures the data integrity of the database server in the case of accidental power failure, natural disasters and so on. It adds additional reliability to MongoDB with pre-written redo logs. When this function is enabled,MongoDB creates a Journal when a write is performed, which contains the disk address and bytes changed during the write operation. Thus, in the event of a sudden server outage, the diary can be replayed at startup to re-perform writes that could not have been flushed to disk prior to the outage.

MongoDB configures the WiredTiger engine to use memory buffers to hold journal records. WiredTiger synchronizes cached log records to disk based on the following intervals or conditions

  1. Starting with MongoDB 3.2, cached journal data was synchronized to disk every 50ms
  2. If the write operation is set to j:true, WiredTiger forces the log file to be synchronized
  3. Because the journal file size used by MongoDB is limited to 100MB, WiredTiger creates a new log file approximately every 100MB of data. When WiredTiger creates a new Journal file, WiredTiger synchronizes the previous Journal file

When MongoDB reaches the above commit, it logs the update operation. This means that MongoDB commits changes in batches, meaning that each write is not flushed to disk immediately. However, by default, you cannot lose more than 50ms of written data in the event of a crash.

Data files are flushed to disk every 60 seconds by default, so the Journal file only needs to record about 60 seconds of written data. The log system pre-allocates several empty files, which are stored in /data/db/journal directories named _J.0 and _J.1. After running MongoDB for a long time, files similar to _J.6217 and _J.6218 will appear in the log directory. These are the current log files. The value in the file increases as MongoDB runs longer. After a normal database shutdown, the diary files are cleared (because they are no longer needed after a normal shutdown).

To write data to mongodb, memory is first written, and then disk is flushed every 60 seconds. Journal is also written, and the corresponding buffer is also written first. Then WiredTiger is used to flush journal files from disk to disk every 50ms. MongoDB can also restore from the last checkpoint (think of it as a mirror); However, to undo changes made since the last checkpoint, you still need to use Journal

In the event of a system crash or forced termination of the database using the kill -9 command, Mongod will replay the Journal file at startup and display a large amount of validation information.

The above mentioned is for WiredTiger engine, MMAPv1 engine is a little different, first, it is every 100ms flush, second, it writes journal file through the private View, data file through the shared view. I won’t go into more detail here, because MongoDB 4.0 doesn’t recommend using this storage engine. As of MongoDB 3.2, WiredTiger is the recommended default storage engine for MongoDB

Note that if the client writes faster than the journal refresh rate, Mongod restricts the write operation until the journal is written to disk. This is the only case where Mongod would restrict writing.

Capped Collection

Introduction to Capped Collection before I cover the next two types of logging.

Normal collections in MongoDB are created dynamically and can grow automatically to hold more data. There is a different type of collection in MongoDB called fixed collections. The fixed collection needs to be created beforehand, and its size is fixed. The behavior type of a fixed collection is the same as that of a circular queue. If there is no space, the oldest document is deleted to free up space, and the newly inserted document takes up the space.

Create a fixed collection:

db.createCollection("collectionName", {"capped":true."size": 100000,"max": 100})Copy the code

A fixed size collection of 100,000 bytes with 100 documents is created. Whichever limit is reached first, new documents inserted later will push the oldest document out of the collection: the number of documents in a fixed collection cannot exceed the document limit, nor can it exceed the size limit.

Once a fixed set is created, it cannot be changed. You cannot convert a fixed set to an unfixed set, but you can convert a regular set to a fixed set.

db.runCommand({"convertToCapped": "test"."size" : 10000});
Copy the code

Fixed collections can perform a special sort called natural sort, which returns the documents in the result set in the order they are on disk. Natural order is the insertion order of documents, so the resulting documents are sorted from old to new. From new to old:

db.my_capped_collection.find().sort({"$natural": 1});Copy the code

Oplog primary/secondary logs

Replica Sets Replicate Sets are used to back up data between multiple servers. The MongoDB replication function is implemented using oplog operation logs, which contain every write operation on the primary node. Oplog is a fixed set in the local database of the primary node. The backup node can query this collection to know which operations need to be replicated.

All databases in a mongod instance use the same Oplog, meaning that operation logs (inserts, deletes, and changes) for all databases are logged to Oplog

Each backup node maintains its own Oplog, which records every operation of copying data from the primary node. In this way, each member can be used as a synchronization source for other members. As shown, the backup node retrieves the operations it needs to perform from the currently used synchronization source and performs them on its own data set. Finally, these operations are written to its own Oplog, and if an operation fails (which can only happen if the data of the synchronous source is corrupted or inconsistent with the primary node), the backup node stops copying data from the current synchronous source.

Oplog stores all the write operations performed in sequence. Each member in replica sets maintains its own Oplog. Oplog of each member should be exactly the same as oplog of the main node (there may be some delay).

If a backup node hangs for some reason, but it is restarted, synchronization starts automatically from the last operation in Oplog. During the replication operation, data is copied to be written into the Oplog. Therefore, the backup node may replicate data that has been synchronized. MongoDB was designed with this in mind: executing the same operation in Oplog multiple times has the same effect as executing it only once.

Because oplog is a fixed size, it can only hold a certain number of operation logs. Typically, oplog space usage grows at about the same rate as the system processes write requests: if 1KB of write requests are being processed on the primary node per minute, then Oplog will probably also write 1KB of operation logs per minute.

However, there are some exceptions: If a single request can affect multiple documents (such as deleting multiple documents or updating multiple documents), multiple operation logs can appear in Oplog. If multiple documents are affected by a single operation, each affected document will correspond to one oplog log. Therefore, if 10W documents were removed by executing db.student.remove(), then there would be 10W action logs in Oplog, one for each deleted document. If a large number of batch operations are performed, oplog can fill up quickly.

Slow Query logs

MongoDB uses system profilers to find operations that take too long. The system profiler logs operations in the fixed set System.profile and provides a lot of information about operations that take too long, but the overall performance of the corresponding Mongod suffers. So we tend to open profilers periodically to get information.

By default, the system analyzer is turned off and does not record anything. You can turn on the profiler by running db.setProfilingLevel() in the shell

db.setProfilingLevel(level,<slowms>) 0=off 1=slow 2=all
Copy the code

The first parameter specifies the level. Different levels have different meanings. 0 indicates shutdown, 1 indicates default recording of operations that take more than 100 milliseconds, and 2 indicates recording all operations. The second parameter is a custom “too long” criterion, such as recording all operations that take 500ms

Db. SetProfilingLevel (1500);Copy the code

If the profiler is enabled and the System.profile collection does not exist, MongoDB creates a capped collection of several Megabytes for it. If you want your analyzer to run longer, you may need more space to record more operations. At this point, you can close the profiler, delete and recreate a new fixed collection named System.profile, and make it fit the required capacity. Then re-enable the analyzer on the database.

You can check the maximum size of the collection with db.system.profile.stats().

Like can focus on WeChat public number: programmers sooner or later A programmer who insist on writing original articles, here a person also can realize an orgy of a group of people, is now focusing on distributed, Spring, directing, Kafka, HBase, the JDK source technology such as sharing, learn from each other.