系统版本信息(该版本较低,不支持sched_ext),本文基于bclinux / RHEL / CentOS,如果用的是ubuntu,替换成对应的apt命令即可。
[root@hadoop02 ~]# uname -a Linux hadoop02 5.10.0-200.0.0.131.30.oe2203sp3.bclinux.x86_64 #1 SMP Tue Sep 24 08:43:22 CST 2024 x86_64 x86_64 x86_64 GNU/LinuxLinux内核从6.12版本开始正式支持sched_ext调度程序,所以需升级内核至6.12或其他高版本。
本文将内核升级至6.15.7版本,sched_ext调度器基于6.15.7实现。
一、pahole 版本 ≥ 1.16
打开SCHED_CLASS_EXT配置项时,依赖关系要求pahole的版本 ≥ 1.16。
下载pahole源码
[root@hadoop02 ~]# git clone git://git.kernel.org/pub/scm/devel/pahole/pahole.git [root@hadoop02 ~]# cd pahole/ [root@hadoop02 ~]# git checkout v1.30在编译pahole的过程中,pahole/.gitmodules文件配置了git libbpf的地址,如果不能访问外网git hub,可通过下面命令将地址换成国内镜像。
pahole根目录执行: sed -i 's|https://github.com/libbpf/libbpf|https://gitee.com/mirrors/libbpf.git|' .gitmodules git submodule sync git submodule update --init安装cmake等工具
[root@hadoop02 ~]# yum install cmake gcc make elfutils-libelf-devel -y编译及安装
[root@hadoop02 pahole]# mkdir build && cd build [root@hadoop02 build]# cmake -D__LIB=lib .. [root@hadoop02 build]# make -j$(nproc) [ 1%] Building C object CMakeFiles/bpf.dir/lib/bpf/src/bpf.c.o [ 3%] Building C object CMakeFiles/bpf.dir/lib/bpf/src/bpf_prog_linfo.c.o [ 5%] Building C object CMakeFiles/bpf.dir/lib/bpf/src/btf.c.o [ 6%] Building C object CMakeFiles/bpf.dir/lib/bpf/src/btf_dump.c.o [ 8%] Building C object CMakeFiles/bpf.dir/lib/bpf/src/btf_iter.c.o [root@hadoop02 build]# make install Consolidate compiler generated dependencies of target bpf [ 37%] Built target bpf Consolidate compiler generated dependencies of target dwarves [ 57%] Built target dwarves Consolidate compiler generated dependencies of target dwarves_emit安装
[root@hadoop02 build]# make install -- Installing: /usr/local/include/dwarves/elfcreator.h -- Installing: /usr/local/include/dwarves/elf_symtab.h -- Installing: /usr/local/include/dwarves/hash.h -- Installing: /usr/local/include/dwarves/libctf.h -- Installing: /usr/local/share/man/man1/pahole.1 -- Installing: /usr/local/bin/ostra-cg -- Installing: /usr/local/share/dwarves/runtime/python/ostra.py -- Installing: /usr/local/bin/btfdiff -- Installing: /usr/local/bin/fullcircle -- Installing: /usr/local/share/dwarves/runtime/Makefile -- Installing: /usr/local/share/dwarves/runtime/ctracer_relay.c -- Installing: /usr/local/share/dwarves/runtime/ctracer_relay.h -- Installing: /usr/local/share/dwarves/runtime/linux.blacklist.cu修改环境变量
echo '/usr/local/lib64' | sudo tee /etc/ld.so.conf.d/dwarves.conf sudo ldconfig安装完成
[root@hadoop02 linux-6.15.7]# pahole --version v1.30二、6.15.7内核编译选项(sche_ext相关)
在开始设置config选项前,一定要按第一节内容,安装pahole,否则部分config将不会显示。
make menuconfig设置以下配置项(务必注意,需要按顺序设置)
CONFIG_LOCALVERSION="填入自己设置的内核版本名称"
"General setup" -> "Local version - append to kernel release"
CONFIG_BPF_SYSCALL=y,允许用户态通过bpf()系统调用加载 eBPF 程序到内核
"General setup" -> "BPF subsystem" -> "Enable bpf() system call"
CONFIG_BPF_JIT=y,启用BPF即时编译加速
"General setup" -> "BPF subsystem" -> "Enable BPF Just In Time compiler"
勾选后,CONFIG_HAVE_EBPF_JIT 将自动激活(依赖架构支持,x86_64/ARM64 通常支持)
CONFIG_DEBUG_INFO=y,生成完整的调试符号信息
“Kernel hacking” -> “Compile-time checks and compiler options” -> “Debug information",选择Generate DWARF Version 4 debuginfo
CONFIG_DEBUG_INFO_SPLIT=n,禁用调试信息拆分
“Kernel hacking” -> “Compile-time checks and compiler options” -> "Produce split debuginfo in .dwo files"
CONFIG_DEBUG_INFO_REDUCED=n,禁用缩减版调试信息
Kernel hacking” -> “Compile-time checks and compiler options” -> "Reduce debugging information"
CONFIG_DEBUG_INFO_BTF=y,生成BPF类型格式(BTF)调试信息,提供内核数据结构元数据
“Kernel hacking” -> “Compile-time checks and compiler options” -> “Generate BTF type information ”
CONFIG_SCHED_CLASS_EXT=y,启用可扩展的 BPF 调度类(sched_ext)
"General setup" -> "Extensible Scheduling Class (NEW)”
CONFIG_FUNCTION_ERROR_INJECTION=y,开启故障注入测试
“Kernel hacking” -> "Kernel Testing and Coverage" -> "Fault-injections of functions"
CONFIG_BPF_KPROBE_OVERRIDE=y,控制 eBPF 程序是否能够修改内核函数的返回值
“Kernel hacking” -> "Tracers" -> "enable BPF programs to override a kprobed function"
可通过检测脚本检查配置是否正确(放在linux内核源码的根目录):
[root@hadoop02 linux-6.15.7]# cat check_config.sh #!/bin/bash # check_config.sh # 功能:验证内核配置是否满足 sched_ext 要求,并检查 pahole 版本兼容性 CONFIG=${1:-.config} # 支持自定义配置文件路径 # 检查配置项函数(支持必须启用或禁用的选项) check_config() { local opt=$1 local expected_val=${2:-y} # 默认检查是否=y if [ "$expected_val" = "n" ]; then # 对于需要=n的选项:不存在或=n都算通过 if ! grep -q "^$opt=" "$CONFIG" || grep -q "^$opt=n" "$CONFIG"; then echo -e "\033[32m[OK]\033[0m $opt not set or =n" else echo -e "\033[31m[FAIL]\033[0m $opt is enabled (should be n)" fi else # 对于需要=y的选项:必须明确存在且=y if grep -q "^$opt=y" "$CONFIG"; then echo -e "\033[32m[OK]\033[0m $opt enabled" else echo -e "\033[31m[FAIL]\033[0m $opt not set" fi fi } version_compare() { local need_ver=$1 local current_ver=$2 [ "$(printf '%s\n' "$need_ver" "$current_ver" | sort -V | head -n1)" = "$need_ver" ] } echo "=== Checking required kernel configs (must be y) ===" check_config CONFIG_BPF_SYSCALL check_config CONFIG_BPF_JIT check_config CONFIG_DEBUG_INFO check_config CONFIG_DEBUG_INFO_BTF check_config CONFIG_SCHED_CLASS_EXT check_config CONFIG_FUNCTION_ERROR_INJECTION check_config CONFIG_BPF_KPROBE_OVERRIDE echo echo "=== Checking conflict kernel configs (must be n or absent) ===" check_config CONFIG_DEBUG_INFO_SPLIT n check_config CONFIG_DEBUG_INFO_REDUCED n echo echo "=== Checking pahole version (need >= 1.16) ===" if command -v pahole >/dev/null 2>&1; then ver=$(pahole --version | grep -oE '[0-9]+\.[0-9]+') echo "Detected pahole version: $ver" if version_compare "1.16" "$ver"; then echo -e "\033[32m[OK]\033[0m pahole >= 1.16" else echo -e "\033[31m[FAIL]\033[0m pahole too old (need >=1.16)" fi else echo -e "\033[31m[FAIL]\033[0m pahole not installed" fi echo echo "=== Checking libdwarves_emit.so availability ===" if ldconfig -p | grep -q libdwarves_emit.so; then echo -e "\033[32m[OK]\033[0m libdwarves_emit.so found" else echo -e "\033[31m[WARN]\033[0m libdwarves_emit.so not in standard paths" echo "Try: export LD_LIBRARY_PATH=/usr/local/lib64:\$LD_LIBRARY_PATH" fi执行check_config.sh ,确保各项检查pass
编译后的内核启动时,如果卡在Loading initial ramdisk ...,用下面的config文件重新编译内核,这个文件已经设置了sched_ext相关的配置项。
6.15.7_sched_ext.config
链接: https://pan.baidu.com/s/1-i5uhL7G5MXcdtnvY8S56w 提取码: htk5
三、6.15.7内核编译选项(签名相关)
不关闭签名,可能产生下面的错误,强烈建议关闭。
- SSL error:0480006C:PEM routines::no start line: ../crypto/pem/pem_lib.c:763 extract-cert: certs/signing_key.pem make[3]: *** [certs/Makefile:74: certs/signing_key.x509] Error 1 make[2]: *** [scripts/Makefile.build:461: certs] Error 2 make[2]: *** Waiting for unfinished jobs.... make[1]: *** [/home/geshifei/linux-6.15.7/Makefile:2003: .] Error 2 make: *** [Makefile:248: __sub-make] Error 2CONFIG_MODULE_SIG=n,关闭模块签名
“Enable loadable module support” -> “Module signature verification”
四、关闭证书
linux-6.15.7源码目录执行:
scripts/config --disable SYSTEM_TRUSTED_KEYS
scripts/config --disable SYSTEM_REVOCATION_KEYS
五、编译6.15.7内核
make -j$(nproc) make modules_install make install grub2-mkconfig -o /boot/grub2/grub.cfg六、重启进6.15.7
重启,启动项中选择刚刚编译的6.15.7 kernel。本文后续操作,都是在6.15.7系统中!!
七、安装工具包
编译clang、全功能bpftool需要用到这些工具包。
[root@hadoop00 ~] yum install valgrind valgrind-devel [root@hadoop00 ~] yum install libedit-devel [root@hadoop00 ~] yum install elfutils-libelf-devel [root@hadoop00 ~] yum install zlib-devel [root@hadoop00 ~] yum install binutils-devel [root@hadoop00 ~] yum install libcap-develubuntu:
[root@hadoop00 ~] apt install valgrind [root@hadoop00 ~] apt install libedit-dev [root@hadoop00 ~] apt install libelf-dev [root@hadoop00 ~] apt install zlib1g-dev [root@hadoop00 ~] apt install binutils-dev [root@hadoop00 ~] apt install libcap-dev八、升级clang≥16+
BPF程序在内核中运行时,可以通过 BTF (BPF Type Format) 来获取类型信息,这些信息是通过clang编码到.BTF section中的。低版本calng会出现不支持新标记的问题(比如btf_type_tag、btf_decl_tag),本文将clang升级到17.0.6。
1)下载及编译clang 17.0.6
[root@hadoop02 ~] cd /opt 下载代码 官方速度很慢:git clone git@github.com:llvm/llvm-project.git 国内gitee速度很快:git clone https://gitee.com/mirrors/llvm-project.git 国内清华需要排队:git clone https://mirrors.tuna.tsinghua.edu.cn/git/llvm-project.git [root@hadoop02 opt] cd llvm-project/ [root@hadoop02 llvm-project] git checkout llvmorg-17.0.6 [root@hadoop02 llvm-project] mkdir build && cd build [root@hadoop02 build] cmake -G "Unix Makefiles" -DLLVM_ENABLE_PROJECTS="clang;lld;compiler-rt" -DCMAKE_BUILD_TYPE=Release ../llvm [root@hadoop02 build] make -j$(nproc) [root@hadoop02 build] make install2)检查clang版本
[root@hadoop02 build]# which clang /usr/local/bin/clang [root@hadoop02 build] /usr/local/bin/clang --version clang version 17.0.6 (git@github.com:llvm/llvm-project.git 6009708b4367171ccdbf4b5905cb6a803753fe18) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /usr/local/bin九、安装libbpf
一定要用内核源码自带的libbpf编译,否则可能有兼容问题。
[root@hadoop02 ~]# cd /home/geshifei/linux-6.15.7/tools/lib/bpf [root@hadoop02 bpf]# make -j$(nproc) && make install十、安装libbpftool
一定要用内核源码自带的bpftool编译,否则可能有兼容问题。
需要按照(六)(七)节安装工具包,才能编译出全功能版的libpftool,在编译bpftool时显示:
Auto-detecting system features: ... clang-bpf-co-re: [ on ] ... llvm: [ on ] ... libcap: [ on ] ... libbfd: [ on ]编译命令
[root@hadoop02 ~]# cd /home/geshifei/linux-6.15.7/tools/bpf/bpftool/ [root@hadoop00 bpftool]# make -j$(nproc) [root@hadoop00 bpftool]# make install十一、运行自带的sched_ext调度器
a)编译
[root@hadoop00 bpftool]# cd /home/geshifei/linux-6.15.7/tools/sched_ext/ [root@hadoop00 sched_ext]# make -j$(nproc)b)编译成功后,调度器在build/bin目录下。
[root@hadoop00 sched_ext]# /home/geshifei/linux-6.15.7/tools/sched_ext/build/bin/scx_simple local=1 global=0 local=143 global=15 local=275 global=24 local=558 global=25 ^CEXIT: unregistered from user space [root@hadoop00 sched_ext]#