goaccessCloudFunctions

为了方便分析云函数的运行状况，我让云函数用nginx的格式把访问日志输出到log文件中，这样就可以用nginx常用的分析工具比如goaccess来分析日志了。

goaccess是一个命令行日志分析工具，可以分析nginx、apache等web服务器的访问日志，并生成HTML格式的报告。

但是它需要运行的在一个服务器上，而我就是因为服务器刚刚过期没续费被回收了才把程序迁移到云函数上的，专门为了一天只运行几次的goagent再去弄台服务器就没必要了。用云函数直接运行goagent虽然略微折腾一点，但是现在有腾讯云的codebuddy帮忙折腾，这点麻烦就不怎么麻烦了。

首先在codebuddy里面明确的给draft讲，我需要把goagent打包成一个符合lambda规范的layer，然后我还要用云函数来调用这个layer。它吭哧吭哧折腾了一会儿，说这步骤太多了，搞得有点乱，我写个程序然后运行它来完成这个工作吧。经过一番自动的折腾、调试，以及给他错误信息（一开始没有吧geoip打包进去），它最终折腾出来了这个

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

#!/bin/bash

# 创建目录结构
mkdir -p goaccess-layer/bin
mkdir -p goaccess-layer/lib64
mkdir -p goaccess-layer/usr/share/GeoIP

# 下载GeoIP数据库
wget -O goaccess-layer/usr/share/GeoIP/GeoIP.dat.gz https://dl.miyuru.lk/geoip/maxmind/country/maxmind4.dat.gz
gunzip goaccess-layer/usr/share/GeoIP/GeoIP.dat.gz

# 复制goaccess二进制文件
cp /bin/goaccess goaccess-layer/bin/

# 复制依赖库
cp /lib64/libncursesw.so.6 goaccess-layer/lib64/
cp /lib64/libtinfo.so.6 goaccess-layer/lib64/
cp /lib64/libGeoIP.so.1 goaccess-layer/lib64/
cp /lib64/libcrypto.so.1.1 goaccess-layer/lib64/
cp /lib64/libssl.so.1.1 goaccess-layer/lib64/
cp /lib64/libpthread.so.0 goaccess-layer/lib64/
cp /lib64/libc.so.6 goaccess-layer/lib64/
cp /lib64/libdl.so.2 goaccess-layer/lib64/
cp /lib64/libz.so.1 goaccess-layer/lib64/
cp /lib64/ld-linux-x86-64.so.2 goaccess-layer/lib64/

# 创建bootstrap文件
cat > goaccess-layer/bootstrap << 'EOF'
#!/bin/bash
export LD_LIBRARY_PATH=/opt/lib64:$LD_LIBRARY_PATH
EOF

# 设置权限
chmod 755 goaccess-layer/bin/goaccess
chmod 755 goaccess-layer/bootstrap
chmod 755 goaccess-layer/lib64/*
chmod 644 goaccess-layer/usr/share/GeoIP/GeoIP.dat

# 创建打包脚本
cat > package-layer.sh << 'EOF'
#!/bin/bash
cd goaccess-layer
zip -r ../goaccess-layer.zip ./*
cd ..
echo "Layer has been packaged to goaccess-layer.zip"
EOF

chmod 755 package-layer.sh

echo "Setup completed. Run ./package-layer.sh to create the layer package."

注意到脚本的最后一部分，还又生成了一个package-layer.sh脚本，总之运行完得到了一个layer文件。下载下来上传到云函数的层里面，就可以使用了：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
'use strict';
const fs = require('fs');
const path = require('path');
const { execSync } = require('child_process');

exports.main_handler = async (event, context) => {
    console.log("Starting log file processing");
    // fs.rmdirSync('/mnt/log/', { recursive: true });
    // return;
    try {
        // 列出/mnt/log目录下的文件
        const files = fs.readdirSync('/mnt/log/');
        console.log('Files in /mnt/log/ directory:', files);

        if (files.length === 0) {
            console.log('The directory is empty');
            return 'Directory is empty';
        }

        // 筛选符合格式的日志文件
        const logFiles = files.filter(file => 
            file.match(/^(access|error)\.log\.\d{4}-\d{2}-\d{2}$/));
        
        console.log('Found log files:', logFiles);

        // 处理每个日志文件
        for (const logFile of logFiles) {
            const logPath = path.join('/mnt/log/', logFile);
            const htmlReport = path.join('/mnt/log/', `${logFile}.html`);
            
            // 检查HTML报告是否已存在
            if (!fs.existsSync(htmlReport)) {
                console.log(`Generating HTML report for ${logFile}`);
                
                // 根据日志类型选择合适的日志格式
                const logFormat = logFile.startsWith('access') ? 
                    'COMBINED' : 'COMMON';
                
                // 设置环境变量以确保goaccess可以找到其依赖库和GeoIP数据库
                process.env.LD_LIBRARY_PATH = '/opt/lib64:' + (process.env.LD_LIBRARY_PATH || '');
                process.env.GEOIP_DB_PATH = '/opt/usr/share/GeoIP/GeoIP.dat';
                
                // 构建goaccess命令，使用layer中的goaccess，并明确指定GeoIP数据库位置
                const geoipDbPath = '/opt/usr/share/GeoIP/GeoIP.dat';
                const command = `/opt/bin/goaccess ${logPath} -o ${htmlReport} --log-format=${logFormat} --date-format=%Y-%m-%d --time-format=%H:%M:%S --geoip-database=${geoipDbPath}`;
                
                try {
                    execSync(command);
                    console.log(`Successfully generated HTML report for ${logFile}`);
                } catch (error) {
                    console.error(`Error generating report for ${logFile}:`, error.message);
                }
            } else {
                console.log(`HTML report already exists for ${logFile}`);
            }
        }

        return {
            statusCode: 200,
            message: 'Log processing completed',
            processedFiles: logFiles
        };
        
    } catch (error) {
        console.error('Error processing log files:', error.message);
        return {
            statusCode: 500,
            message: 'Error processing log files',
            error: error.message
        };
    }
};

运行了一下，完美

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
Found log files: [

  'access.log.2025-05-22',

  'access.log.2025-05-23',

  'error.log.2025-05-22'

]

Generating HTML report for access.log.2025-05-22

Successfully generated HTML report for access.log.2025-05-22

 [PARSING /mnt/log/access.log.2025-05-22] {0} @ {0/s}

Generating HTML report for access.log.2025-05-23

Successfully generated HTML report for access.log.2025-05-23

 [PARSING /mnt/log/access.log.2025-05-23] {0} @ {0/s}

Generating HTML report for error.log.2025-05-22

Successfully generated HTML report for error.log.2025-05-22

 [PARSING /mnt/log/error.log.2025-05-22] {0} @ {0/s}

配一个自动运行计划，就可以看监控日志了。