This is the second day of my participation in Gwen Challenge

Node forwards formData data

Recently, a thousand years old pit has been filled: enterprise wechat robot SDK compatible with Node, the process of solving the problem is a little difficult, fortunately, all solved one by one..

Background: Since the interface for uploading files on enterprise wechat uses formData format, both the client and the node code acting as the proxy need to use formData format. But I don’t want to use another library, it’s just a small request.. Encapsulate it yourself. And so began my trample pit road…

background

Client: browser environment code

Formdata (formData); formData (formData);

form.setAttribute("method"."POST");
if(format === "formdata") {// If you want to send the formData type, assign the input from the page directly to the input of the form
    form.setAttribute("action"."http://localhost:3000/postFile");
    form.setAttribute("enctype"."multipart/form-data")
    let input = document.createElement("input");
    input = msg;
    form.appendChild(input);
    let keyInput = document.createElement("input");
    keyInput.name = "rootKey"
    keyInput.value = this.key
    form.appendChild(keyInput);
}
Copy the code

Client: Node environment code

I need to package formData myself. I refer to the HTTP format code in Postman. After debugging many times, I get the following code:

const resObj = {
    postData: "".headers: {} as any
}
// Format the message body
if(format === "formdata") {// Select a key
    const boundaryKey = "515897053453198140930459"
    resObj.postData = getFormData(boundaryKey, '1.txt', msg, this.key)
    resObj.headers = {
      hostname: 'localhost'.port: 3000.path: '/postFile'.method: 'POST'.headers: {
        'Content-Type': 'multipart/form-data; boundary=--------------------------515897053453198140930459'.'Content-Length': checkLength(resObj.postData)
      }
    };
}
Copy the code

// Encapsulates the message body
export function getFormData(boundaryKey:string, filename: string, msg: string, key: string) {
    const extType = getMimeType(filename)
    return ` -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --${boundaryKey}\r
Content-Disposition: form-data; name="media"; filename="${filename}"\r
Content-Type: ${extType}\r
\r
${msg}\r
----------------------------${boundaryKey}\r
Content-Disposition: form-data; name="rootKey"\r
\r
${key}\r
----------------------------${boundaryKey}--\r
`
}

function getMimeType(filename: string){
    const mimes = {
        '.png': 'image/png'.'.gif': 'image/gif'.'.jpg': 'image/jpeg'.'.jpeg': 'image/jpeg'.'.txt': 'text/plain'
    };
    const extname = filename.match(/\.(\w+)$/)
    returnextname? mimes[extname[0]]:mimes['.txt'] / / the default TXT
}

Copy the code

Here are a few bugs that I found after debugging many times:

The Length of content-length cannot be too long. If the Length of content-length is too short, the message will be cut short.
Content-type must be written. The boundary in it should not copy postman. It should be long —–.
The boundarykey can be filled with anything and can be fixed without affecting requests.
Note \r\n, be careful, otherwise it will cause message body parsing failure, the request can not be sent directly.
Pay attention to the order between keys, otherwise it will be troublesome to parse them manually on the Node side.

After formatting, send the request code as follows:

// Send the formData request
const {headers, postData} = this.getFormatBody(sendMsg[i], format)
const req = http.request(headers, function(res) {
    res.setEncoding('utf8');
    const resData = [] as any;
    res.on('data'.function (chunk) {
      resData.push(chunk);
    });
    res.on('end'.function() {
      allRes.push(resData.join(""))
      if(i===msg.length-1){ resolve(allRes); }})}); req.write(postData);Copy the code

Server code

When doing the server side, it is a bit troublesome. At the beginning, I did a simple string parsing:

const postFile = async (ctx: any) => {
    const postMsg = ctx.request
    delete postMsg.header.host
    // Get the key from the client
    const keyReg = / (? <="rootKey"\r\n\r\n)(\w|\W)+(? =\r\n-----)/
    const keyStr = allBodyArr.pop()
    key = keyStr.match(keyReg)[0]
    / / send rawBody
    const allBody = postMsg.rawBody // You need to set the bodyParser extension form to FormData to get it
    // Get the delimiter in formData (you can also get it in the header)
    const splitKey = allBody.match(/------(\w|\W)+? (? =\n)/) [0]
    // Get the body of each part of the message
    const allBodyArr = allBody.split(splitKey)
    // The first is a newline character
    allBodyArr.shift()
    for(let i = 0; i < allBodyArr.length; i++){
        // Splice formData body
        let bodyStr = `\n${splitKey}${allBodyArr[i]}${splitKey}\n`
        // Call the enterprise wechat interface to get the id of the file
        let fileInfo = await postAxios(`https://qyapi.weixin.qq.com/cgi-bin/webhook/upload_media?key=${key}&type=file`, 
            postMsg.header, bodyStr)
        if(fileInfo.errcode=="0") {// Send the file
            let res = await postAxios(`https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=${key}`.null, botSendFile(fileInfo["media_id"]))
            if(res.errcode! ="0"){
                ctx.body = sendText(0.200."Failed to send file")}}else{
            ctx.body = sendText(0.200."Failed to get file ID")
        }
        ctx.body = sendText(0.200."ok")}}Copy the code

In fact, it can run, but a few more files will find the problem:

Text files can be transferred normally, but there is a problem in transferring files and pictures. The pictures sent to the wechat of the enterprise are found to be inconsistent with the specifications after downloading.
And some small pictures clearly meet the length requirements, but the client error.

To find the bug

Start from the request first, found that postmsg, node, browser three methods to get the same, is wrong file format, so confirm the backend code problems. In fact, I am not very confident about what I have encapsulated. I suspect that I have written too much or lost some characters accidentally. Then I get the picture that cannot be displayed and save it, right click it to open it in text form, and then open the original picture in text format for comparison. It is found that the image that cannot be displayed is utF-8 encoding, the original image is not this encoding. Bodyparser converts the data directly to UTF-8 format, check, the documentation does not say how to obtain buffer format.

Okay, so I’m gonna have to figure this out myself

solution

Idea 1: Save the configuration and forward it. The blog on the Internet is all this method, considered whether to use, but still feel too cumbersome.
Idea 2: Direct forwarding.

In fact, I used the formData library on the Internet to try, and it is possible to directly forward the read file using Axios. I couldn’t figure out why, so I looked at the source code, and found that others were using the form of stream, so I thought how to use stream to do forwarding.
1. Want to configure proxy directly forward client request, check for a long time too troublesome.
2. If you want to see if pipe can be used for forwarding, it will not be useful. How to forward depends on the data format.
3. And finally how to get the native request body.

To solve the problem

Req: ctx.req: ctx.req: ctx.req: ctx.req: ctx.req: ctx.req: ctx.req: ctx.req Tried axios to see if it could send it, and hey god, it worked, so solved the problem.
After solving the problem, it was found that the previous requests such as images were faulty and bodyParser was still needed. Therefore, the process of converting buffer was written in the form of middleware in front of BodyParser and attached to the property of REQ.

The final code is as follows:

// Middleware code
app.use(async (ctx, next) => {
    // Process formData as buffer
    if (ctx.request.url === '/postFile') {
        const buffer: any = []
        ctx.req.on('data'.(chunk: any) = > {
            buffer.push(chunk)
        })
        ctx.req.on('end'.(chunk: any) = > {
            const bufferRes = Buffer.concat(buffer)
            const keyReg = / (? <="rootKey"\r\n\r\n)(\w|\W)+(? =\r\n-----)/
            const key = bufferRes.toString('utf-8').match(keyReg)[0]
            ctx.myrequest = {
                bufferRes,
                key,
            }
        })
    }
}))
    await next()
}).use(
    bodyParser({
        formLimit: '20mb'.extendTypes: {
            form: ['multipart/form-data'].// will parse application/x-javascript type body as a JSON string}}),)Copy the code

// Send the file code
const postFile = async (ctx: any) => {
    const postMsg = ctx.request
    delete postMsg.header.host
    const {bufferRes, key} = ctx.myrequest
    let fileInfo = await postAxios(
        `https://qyapi.weixin.qq.com/cgi-bin/webhook/upload_media?key=${key}&type=file`,
        postMsg.header,
        bufferRes,
    )
    if (fileInfo.errcode == '0') {
        // Send the file
        let res = await postAxios(
            `https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=${key}`.null,
            botSendFile(fileInfo['media_id']),if(res.errcode ! ='0') {
            ctx.body = sendText(0.200.'Failed to send file')}}else {
        ctx.body = sendText(0.200.'Failed to get file ID')
    }
    ctx.body = sendText(0.200.'File sent successfully')}Copy the code

Successfully solved ~

The client Node environment code is faulty

Post and the browser were done, and I found that the formData I wrapped in Node was still having coding problems because I was concatenating it directly with strings

Solution: Split code into headers, message bodies, and tails.

The message body is a buffer because it is read directly from the file
Both the header and the tail are strings, so they are converted to buffers.

Use buffer.from (head/tail).
Finally, buffer. concat is used to concatenate the three parts, and buffer. byteLength(allBuffer) is used to calculate the length of bytes

The code is as follows:

const boundaryKey = "515897053453198140930459"

// Use buffer to avoid coding problems
export function getFormData(filename: string, msg: any, key: string) {
    const extType = getMimeType(filename)
    const head = ` -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --${boundaryKey}\r
Content-Disposition: form-data; name="media"; filename="${filename}"\r
Content-Type: ${extType}\r
\r\n`
    const tail = `\r
----------------------------${boundaryKey}\r
Content-Disposition: form-data; name="rootKey"\r
\r
${key}\r
----------------------------${boundaryKey}--\r
    `// All are converted to buffer
    const headBuffer = Buffer.from(head)
    const tailBuffer = Buffer.from(tail)
    const allBuffer = Buffer.concat([headBuffer, msg, tailBuffer])
    return {
        body: allBuffer,
        head: {
            'Content-Type': `multipart/form-data; boundary=--------------------------${boundaryKey}`.'Content-Length': Buffer.byteLength(allBuffer)
        }
    }
}
Copy the code

Solve problems ~~

Node forwards formData data

Node forwards formData data

background

Client: browser environment code

Client: Node environment code

Server code

To find the bug

solution

To solve the problem

The client Node environment code is faulty

Related Posts

How does React implement useState

An exclusive interview with DevUI component Library

Click on the page element and the Vite plugin opens the Vue component for me 🚀