zhangyang-variable-monitor/README_en.md

Variable Monitor

Monitor numerical variables (given address, length), and print system stack information when the set conditions are exceeded.

Number of simultaneous monitoring

• Monitoring with the same timing length will be grouped into one group, corresponding to one timer.
• A set of up to 32 variables, after which a new timer is allocated.
• The global maximum number of timers is 128.
• The above quantity limit is defined in the watch_module.h header macro.

Currently, monitoring is limited to the same application, and simultaneous calls from multiple applications are not currently supported.

• Multiple applications can work normally if only one program calls cancel_all_watch();.

Usage

Example: helloworld.c

• Add #include "watch.h"
• Set each variable that needs to be monitored: name && address && length, set threshold, comparison method, timer interval (ns), etc.
• start_watch(watch_arg); Start monitoring
• Call cancel_all_watch(); when you need to cancel monitoring

When the set conditions are exceeded, the system stack information is printed and viewed with dmesg, as shown in the following example:

• Within a timer, if multiple variables exceed the threshold, the stack information will not be output repeatedly;
• The timer restart time after printing the stack is 1s, and the next round of monitoring will start after 1s.

[86245.364861] -------------------------------------
[86245.364864] -------------watch monitor-----------
[86245.364865] Threshold reached:
                 name: temp0, threshold: 150 
[86245.364866] Timestamp (ns): 1699589000606300743
[86245.364867] Recent Load: 116.65, 126.83, 151.17
[86245.365669] task: name lcore-worker-4, pid 803327
[86245.365672] task: name lcore-worker-5, pid 803328
[86245.365673] task: name lcore-worker-6, pid 803329
[86245.365674] task: name lcore-worker-7, pid 803330
[86245.365676] task: name lcore-worker-8, pid 803331
[86245.365677] task: name lcore-worker-9, pid 803332
[86245.365679] task: name lcore-worker-10, pid 803333
[86245.365681] task: name lcore-worker-11, pid 803334
[86245.365682] task: name lcore-worker-68, pid 803335
[86245.365683] task: name lcore-worker-69, pid 803336
[86245.365684] task: name lcore-worker-70, pid 803337
[86245.365685] task: name lcore-worker-71, pid 803338
[86245.365686] task: name lcore-worker-72, pid 803339
[86245.365687] task: name lcore-worker-73, pid 803340
[86245.365688] task: name lcore-worker-74, pid 803341
[86245.365689] task: name lcore-worker-75, pid 803342
[86245.365694] task: name pkt:worker-0, pid 803638
[86245.365702]  hrtimer_nanosleep+0x8d/0x120
[86245.365709]  __x64_sys_nanosleep+0x96/0xd0
[86245.365711]  do_syscall_64+0x37/0x80
[86245.365716]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[86245.365718] task: name pkt:worker-1, pid 803639
[86245.365721]  hrtimer_nanosleep+0x8d/0x120
[86245.365724]  __x64_sys_nanosleep+0x96/0xd0
[86245.365726]  do_syscall_64+0x37/0x80
[86245.365728]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[86245.365730] task: name pkt:worker-2, pid 803640
[86245.365732]  hrtimer_nanosleep+0x8d/0x120
[86245.365734]  __x64_sys_nanosleep+0x96/0xd0
[86245.365737]  do_syscall_64+0x37/0x80
[86245.365739]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[86245.365740] task: name pkt:worker-3, pid 803641
[86245.365743]  hrtimer_nanosleep+0x8d/0x120

Parameter Description

start_watch passes in the watch_arg structure. The meaning of each field is as follows

• name limit MAX_NAME_LEN(15) valid characters

typedef struct
{
    pid_t task_id;               // current process id
    char name[MAX_NAME_LEN + 1]; // name (15+1)
    void *ptr;                   // virtual address
    int length_byte;             // byte
    long long threshold;         // threshold value
    unsigned char unsigned_flag; // unsigned flag (true: unsigned, false: signed)
    unsigned char greater_flag;  // reverse flag (true: >, false: <)
    unsigned long time_ns;       // timer interval (ns)
} watch_arg;

An initialization example

watch_args = (watch_arg){
    .task_id = getpid(),
    .ptr = &temp,
    .name = "temp",
    .length_byte = sizeof(int),
    .threshold = 150 + i,
    .unsigned_flag = 0,
    .greater_flag = 1,
    .time_ns = 2000 + (i / 33) * 5000
};

demo

In the main project directory:

make && insmod watch_module.ko
./watch

You can see the printed stack information in dmesg

# Unload module and clean compile files
rmmod watch_module.ko && make clean

Only tested on kernel 5.17.15-1.el8.x86_64.

Other

The program is divided into two parts: character device and user space interface, both of which communicate through ioctl.

User space address access

• The variable virtual address passed in by the user program, use get_user_pages_remote to obtain the memory page where the address is located, and kmap maps it to the kernel.
  • In the 192.168.40.204 environment, the HugeTLB Pages test mounts normally.
• The memory page address + offset is stored in the kernel_watch_arg corresponding to the timer, and hrTimer accesses kernel_watch_arg when polling to get the real value.

timer grouping

• The hrTimer data structure is defined in the global array kernel_wtimer_list. When allocating a timer, it will check the traversal kernel_wtimer_list to compare the timer interval.
• Watches with the same timing interval are assigned to the same group and correspond to the same hrTimer.
• If the number of variables monitored by a timer exceeds TIMER_MAX_WATCH_NUM (32), a new hrTimer will be created.
• The total number of hrTimers (kernel_wtimer_list array length) limit is MAX_TIMER_NUM(128).

Memory page mount/unmount

• get_user_pages_remote/ kmap will increase the corresponding count and requires the equivalent put_page/kunmap.
• A global linked list in the module watch_local_memory_list stores the page and kt corresponding to each successfully mounted variable. When performing the close operation of the character device, it is traversed and unloaded.

Stack output conditions: The conditions are referenced from diagnose-tools::load.c

• TASK must satisfy TASK_RUNNING and __task_contributes_to_load.
• __task_contributes_to_load corresponds to the kernel macro task_contributes_to_loa.

// https://www.spinics.net/lists/kernel/msg3582022.html
// remove from 5.8.rc3,but it still work
// whether the task contributes to the load
#define __task_contributes_to_load(task)                                                                               \
    ((READ_ONCE(task->__state) & TASK_UNINTERRUPTIBLE) != 0 && (task->flags & PF_FROZEN) == 0 &&                       \
     (READ_ONCE(task->__state) & TASK_NOLOAD) == 0)