第六章实验:虚拟化安全

实验一:使用LibVMI获取虚拟机相关信息

安装libvmi

背景知识

libvmi 库提供了对正在运行中的底层虚拟机的运行细节进行监视的功能。监视的功能是由观察内存细节,陷入硬件事件和读取 CPU 寄存器来完成的。这种方式被称作虚拟机自省 (virtual machine introspection)。

实验步骤

本实验使用 ubuntu20.04 进行安装。

仓库地址: libvmi/libvmi: The official home of the LibVMI project is at https://github.com/libvmi/libvmi.

安装文档: Setup — KVM-VMI 0.1 documentation

注意事项:

(1)根据 Option 2: Bare-metal setup 一节的内容安装。 (2)注意 KVM-VMI/kvm-vmi 中有较多的 submodule,若不使用安装文档中提供的带 –recursive 参数的下载命令,则需要手动补全这些 submodule,以免后续编译出错。

  • 安装KVM-VMI

根据文档安装。

注意事项:

(1)KVM-VMI需要用kvmi-v7分支。

(2)在make之前需要额外执行以下指令(相关issue: Compile Failure: kvm-vmi/kvm/tools/objtool/.fixdep.o.cmd:1: *** Missing separator. Stop. · Issue #57 · KVM-VMI/kvm (github.com) ):

make distclean
make olddefconfig
./scripts/config --enable KVM
./scripts/config --enable KVM_INTEL
./scripts/config --enable KVM_AMD
./scripts/config --enable KVM_INTROSPECTION
./scripts/config --disable TRANSPARENT_HUGEPAGE
./scripts/config --enable REMOTE_MAPPING
./scripts/config --disable SYSTEM_TRUSTED_KEYS
./scripts/config --disable SYSTEM_REVOCATION_KEYS
  • 安装QEMU

根据文档安装。

注意事项:

(1)需要关闭apparmor(或者修改其配置):

sudo service apparmor stop
  • 在虚拟机中添加qemu指令

根据文档添加即可。

其中 http://libvirt.org/schemas/domain/qemu/1.0 这个网址为空,访问不到内容,具体说明在 libvirt: QEMU/KVM/HVF hypervisor driver 中,按照文档添加即可,无需担心。

  • 安装libkvmi

根据文档安装即可。

  • 安装LibVMI

根据文档安装即可。

应用 libvmi 的示例程序获取虚拟机信息

1) 打印 Windows 内核的 GUID 和 PE_HEADER;2)拦截并显示 CR3 事件;3)打印虚拟机中运行的进程列表。

实验步骤

先将 kvm-vmi/libvmi/etc/libvmi-example.conf 复制到 /etc/libvmi.conf,然后将虚拟机名字改为其中对应的名字。这里我们使用 Windows7 虚拟机(32位),所以就将虚拟机名字改为 win7,与配置文件对应(修改配置文件使其与虚拟机名字对应亦可):

win7 {
    ostype      = "Windows";
    win_pdbase  = 0x18;
    win_pid     = 0xb4;
    win_tasks   = 0xb8;
    win_pname   = 0x16c;
}
  • 1)打印 Windows 内核的 GUID 和 PE_HEADER

使用 example 中的示例程序:

➜  build git:(5882bc2) ✗ examples/vmi-win-guid name win7 /tmp/introspector
Windows Kernel found @ 0x3c14000
	Version: 32-bit Windows 7
	PE GUID: 4ce78a09412000
	PDB GUID: 684da42a30cc450f81c535b4d18944b12
	Kernel filename: ntkrpamp.pdb
	Multi-processor with PAE (version 5.0 and higher)
	Signature: 17744.
	Machine: 332.
	# of sections: 22.
	# of symbols: 0.
	Timestamp: 1290242569.
	Characteristics: 290.
	Optional header size: 224.
	Optional header type: 0x10b
	Section 1: .text
	Section 2: _PAGELK
	Section 3: POOLMI
	Section 4: POOLCODE
	Section 5: .data
	Section 6: ALMOSTRO
	Section 7: SPINLOCK
	Section 8: PAGE
	Section 9: PAGELK
	Section 10: PAGEKD
	Section 11: PAGEVRFY
	Section 12: PAGEHDLS
	Section 13: PAGEBGFX
	Section 14: PAGEVRFB
	Section 15: .edata
	Section 16: PAGEDATA
	Section 17: PAGEKDD
	Section 18: PAGEVRFC
	Section 19: PAGEVRFD
	Section 20: INIT
	Section 21: .rsrc
	Section 22: .reloc
  • 2)拦截并显示 CR3 事件

使用 example 中的示例程序:

➜  build git:(5882bc2) ✗ examples/cr3-event-example win7 /tmp/introspector
Waiting for events...
CR3 write happened: Value=0x7ee0d480
CR3 write happened: Value=0x185000
CR3 write happened: Value=0x7ee0d480
CR3 write happened: Value=0x7ee0d220
CR3 write happened: Value=0x7ee0d480
CR3 write happened: Value=0x7ee0d220
CR3 write happened: Value=0x185000
CR3 write happened: Value=0x7ee0d480
CR3 write happened: Value=0x185000
CR3 write happened: Value=0x7ee0d480
CR3 write happened: Value=0x7ee0d220
CR3 write happened: Value=0x7ee0d480
CR3 write happened: Value=0x7ee0d220
CR3 write happened: Value=0x7ee0d480
......
  • 3)打印虚拟机中运行的进程列表

使用 example 中的示例程序:

➜  build git:(5882bc2) ✗ examples/vmi-process-list -n win7 -s /tmp/introspector
LibVMI Suggestion: set win_ntoskrnl=0x3c14000 in libvmi.conf for faster startup.
LibVMI Suggestion: set win_kdbg=0x12ac28 in libvmi.conf for faster startup.
LibVMI Suggestion: set win_kdvb=0x83d3ec28 in libvmi.conf for faster startup.
Process listing for VM win7 (id=4)
[    4] System (struct addr:85c3bc78)
[  224] smss.exe (struct addr:975deae0)
[  304] csrss.exe (struct addr:86d40768)
[  352] wininit.exe (struct addr:92c8dd40)
[  360] csrss.exe (struct addr:86d46030)
[  400] winlogon.exe (struct addr:85cad420)
[  444] services.exe (struct addr:86dca5e8)
[  452] lsass.exe (struct addr:86e124e0)
[  460] lsm.exe (struct addr:86e14828)
[  564] svchost.exe (struct addr:86e47180)
[  632] svchost.exe (struct addr:86e5c780)
[  684] svchost.exe (struct addr:86e68568)
[  808] svchost.exe (struct addr:86e35a58)
[  848] svchost.exe (struct addr:86eb0030)
[  912] audiodg.exe (struct addr:86ebf500)
[  964] svchost.exe (struct addr:86ece030)
[ 1080] svchost.exe (struct addr:86ef2030)
[ 1200] dwm.exe (struct addr:86f26030)
[ 1212] explorer.exe (struct addr:86f289f0)
[ 1256] spoolsv.exe (struct addr:86f34030)
[ 1296] svchost.exe (struct addr:86f51d40)
[ 1308] taskhost.exe (struct addr:86f59868)
[ 1428] svchost.exe (struct addr:86fa8030)
[ 1824] rundll32.exe (struct addr:8703cb18)
[ 1844] rundll32.exe (struct addr:8703ed40)
[ 1864] rundll32.exe (struct addr:870412e8)
[  660] SearchIndexer. (struct addr:870c8958)
[ 1232] SearchProtocol (struct addr:870efb88)
[ 1412] SearchFilterHo (struct addr:87106c70)
[ 1288] wmpnetwk.exe (struct addr:87116cb0)

调用LibVMI的接口实现进程信息获取

**1)修改示例程序中的 cr3-event-example.c;2)修改 示例程序中的 process-list.c,根据需要获取进程、线程信息。 ** libvmi 的 api 文档:https://libvmi.com/api/

实验步骤

我们来看看有哪些事件支持被监控:

// libvmi/events.h

#define VMI_EVENT_INVALID           0
#define VMI_EVENT_MEMORY            1   /**< Read/write/execute on a region of memory */
#define VMI_EVENT_REGISTER          2   /**< Read/write of a specific register */
#define VMI_EVENT_SINGLESTEP        3   /**< Instructions being executed on a set of VCPUs */
#define VMI_EVENT_INTERRUPT         4   /**< Interrupts being delivered */
#define VMI_EVENT_GUEST_REQUEST     5   /**< Guest-requested event */
#define VMI_EVENT_CPUID             6   /**< CPUID event */
#define VMI_EVENT_DEBUG_EXCEPTION   7   /**< Debug exception event */
#define VMI_EVENT_PRIVILEGED_CALL   8   /**< Privileged call (ie. SMC on ARM) */
#define VMI_EVENT_DESCRIPTOR_ACCESS 9   /**< A descriptor table register was accessed */
#define VMI_EVENT_FAILED_EMULATION  10  /**< Emulation failed when requested by VMI_EVENT_RESPONSE_EMULATE */
#define VMI_EVENT_DOMAIN_WATCH      11  /**< Watch create/destroy events */
  • 1)修改示例程序中的 cr3-event-example.c

来看看 vmi_event 中包含了什么,我们可以从中选择想要监控的信息打印出来:

// libvmi/events.h

struct vmi_event {
    /* CONST IN */
    uint32_t version; /**< User should set it to VMI_EVENTS_VERSION */

    /* CONST IN */
    vmi_event_type_t type;  /**< The specific type of event */

    /**
     * IN/OUT/RESPONSE
     *
     * The VMM maintained SLAT ID. Can be specified when registering mem_event (IN).
     * On an event report (OUT) specifies the active SLAT ID on the vCPU.
     * Iff VMI_EVENT_RESPONSE_SLAT_ID is set (RESPONSE), switch the vCPU to this VMM pagetable ID.
     *
     * Note: on Xen this corresponds to the altp2m_idx.
     */
    uint16_t slat_id;

    /**
     * RESPONSE
     *
     * The VMM should switch to this SLAT ID on the occurance of the next event.
     * Iff VMI_EVENT_RESPONSE_NEXT_SLAT_ID is set.
     *
     * Note: on Xen this corresponds to the altp2m_idx and it also enables MTF singlestepping.
     *  The altp2m switch automatically happens in the singlestep handler in Xen after a single
     *  instruction is executed.
     */
    uint16_t next_slat_id;

    /**
     * CONST IN
     *
     * An open-ended mechanism allowing a library user to
     *  associate external data to the event.
     * Metadata assigned to this pointer at any time (prior to
     *  or following registration) is delivered to the callback,
     *  for each matching event. The callback is also free to
     *  modify in any way. The library user assumes all memory
     *  management for this referenced data.
     */
    void *data;

    /**
     * CONST IN
     *
     * The callback function that is invoked when the relevant is observed.
     */
    event_callback_t callback;

    /* OUT */
    uint32_t vcpu_id; /**< The VCPU relative to which the event occurred. */

    /**
     * Reserved for future use
     */
    uint32_t _reserved[7];

    union {
        reg_event_t reg_event;
        mem_access_event_t mem_event;
        single_step_event_t ss_event;
        interrupt_event_t interrupt_event;
        privcall_event_t privcall_event;
        cpuid_event_t cpuid_event;
        debug_event_t debug_event;
        descriptor_event_t descriptor_event;
        watch_domain_event_t watch_event;
    };

    /*
     * Note that the following pointers assume compiler compatibility
     * ie. if you compiled a 32-bit version of LibVMI it will be
     * incompatable with 64-bit tools and vice verse.
     */
    union {
        /**
         * OUT
         *
         * Snapshot of some VCPU registers when the event occurred
         */
        union {
            x86_registers_t *x86_regs;
            arm_registers_t *arm_regs;
        };

        /**
         * RESPONSE
         *
         * Read data to be sent back with VMI_EVENT_RESPONSE_SET_EMUL_READ_DATA
         */
        emul_read_t *emul_read;

        /**
         * RESPONSE
         *
         * Instruction buffer to be sent back with VMI_EVENT_RESPONSE_SET_EMUL_INSN
         */
        emul_insn_t *emul_insn;
    };
};

因为我们这里选择了监控寄存器,所以再来看看 reg_event_t 中包含什么:

// libvmi/events.h

typedef struct {
    /**
     * CONST IN
     *
     * Register for which write event is configured.
     * Hypervisors offering register events tend to
     *  have a limited number available for monitoring.
     * These registers tend to be those defined as
     * 'sensitive register instructions' by Popek and
     *  Goldberg, meaning that the registers trigger
     *  a VMEXIT, trap, or equivalent.
     *
     * Note for MSR events on Xen: up to Xen 4.7 only MSR_ALL is supported.
     *  Starting with Xen 4.8 the user has the option to subscribe to specific
     *  MSR events, or to continue using MSR_ALL. However, in this case MSR_ALL
     *  only corresponds to common MSRs that are defined by LibVMI in libvmi.h.
     *  To subscribe to MSR events that are NOT defined by LibVMI, the user can specify
     *  MSR_UNDEFINED here and then set the specific MSR index in the 'msr' field
     *  below.
     */
    reg_t reg;

    /**
     * CONST IN
     *
     * Event filter: callback triggers IFF register==<equal>
     */
    reg_t equal;

    /**
     * CONST IN
     *
     * IFF set to 1, events are delivered asynchronously and
     *  without pausing the originating VCPU
     * Default : 0. (i.e., VCPU is paused at time of event delivery).
     */
    uint8_t async;

    /**
     * CONST IN
     *
     * IFF set to 1, events are only delivered if the written
     *  value differs from the previously held value.
     * Default : 0. (i.e., All write events are delivered).
     */
    uint8_t onchange;

    /**
     * CONST IN
     *
     * Type of register event being monitored.
     * Hypervisors offering register events do so only for those that trigger a
     *  VMEXIT or similar trap. This predominantly means that only write events
     *  are supported by the corresponding LibVMI driver
     */
    vmi_reg_access_t in_access;

    /**
     * OUT
     *
     * Type of register access that triggered the event
     */
    vmi_reg_access_t out_access;

    uint32_t _pad;

    /**
     * OUT
     *
     * Register value read or written
     */
    reg_t value;

    /**
     * OUT
     *
     * Previous value of register (only for CR0/CR3/CR4/MSR)
     */
    reg_t previous;

    /**
     * CONST IN/OUT
     *
     * MSR register operations only
     *
     * CONST IN: Starting from Xen 4.8 the user can use this field to specify an
     *  MSR index to subscribe to when the MSR is not formally defined by LibVMI.
     *
     * OUT: holds the specific MSR for which the event occurred
     *  when the user registered with MSR_ALL.
     * Unused for other register event types.
     */
    uint32_t msr;
} reg_event_t;

选择其中一些感兴趣的数据,并修改 call back 函数:

// cr3-event-example.c

...

event_response_t my_callback(vmi_instance_t vmi, vmi_event_t *event)
{
    (void)vmi;
    printf("CR3 write happened: Value=0x%"PRIx64"\n", event->reg_event.value);
    printf("\tvcpu_id=0x%"PRIx64"\n", event->vcpu_id);
    printf("\trbx=0x%"PRIx64"\n", event->x86_regs->rbx);
    printf("\tcr0=0x%"PRIx64"\n", event->x86_regs->cr0);
    printf("\tprevious CR3 value=0x%"PRIx64"\n", event->reg_event.previous);
    return VMI_EVENT_RESPONSE_NONE;
}

...

int main (int argc, char **argv)
{
	...
    vmi_event_t my_event = {0};
    my_event.version = VMI_EVENTS_VERSION;
    my_event.type = VMI_EVENT_REGISTER;
    my_event.callback = my_callback;
    my_event.reg_event.reg = CR3;
    my_event.reg_event.in_access = VMI_REGACCESS_W;
    ...
}

重新 make,然后运行:

➜  build git:(5882bc2) ✗ examples/cr3-event-example win7 /tmp/introspector
Waiting for events...
CR3 write happened: Value=0x7f9740a0
	vcpu_id=0x0
	rbx=0x83d77c00
	cr0=0x8001003b
	previous CR3 value=0x185000
CR3 write happened: Value=0x185000
	vcpu_id=0x0
	rbx=0x83d77c00
	cr0=0x8001003b
	previous CR3 value=0x7f9740a0
CR3 write happened: Value=0x7f9740a0
	vcpu_id=0x0
	rbx=0x83d77c00
	cr0=0x8001003b
	previous CR3 value=0x185000
CR3 write happened: Value=0x185000
	vcpu_id=0x0
	rbx=0x83d77c00
	cr0=0x8001003b
	previous CR3 value=0x7f9740a0
CR3 write happened: Value=0x7f9740a0
	vcpu_id=0x0
	rbx=0x83d77c00
	cr0=0x8001003b
	previous CR3 value=0x185000
......
  • 2)修改 process-list.c

在windows中,EPROCESS 结构属于内核的执行体层,包含了进程的资源相关信息诸如句柄表、虚拟内存、安全、调试、异常、创建信息、I/O转移统计以及进程计时等。32位 win7 系统 EPROCESS 结构如下:

kd> dt _EPROCESS
ntdll!_EPROCESS
   +0x000 Pcb              : _KPROCESS
   +0x098 ProcessLock      : _EX_PUSH_LOCK
   +0x0a0 CreateTime       : _LARGE_INTEGER
   +0x0a8 ExitTime         : _LARGE_INTEGER
   +0x0b0 RundownProtect   : _EX_RUNDOWN_REF
   +0x0b4 UniqueProcessId  : Ptr32 Void
   +0x0b8 ActiveProcessLinks : _LIST_ENTRY
   +0x0c0 ProcessQuotaUsage : [2] Uint4B
   +0x0c8 ProcessQuotaPeak : [2] Uint4B
   +0x0d0 CommitCharge     : Uint4B
   +0x0d4 QuotaBlock       : Ptr32 _EPROCESS_QUOTA_BLOCK
   +0x0d8 CpuQuotaBlock    : Ptr32 _PS_CPU_QUOTA_BLOCK
   +0x0dc PeakVirtualSize  : Uint4B
   +0x0e0 VirtualSize      : Uint4B
   +0x0e4 SessionProcessLinks : _LIST_ENTRY
   +0x0ec DebugPort        : Ptr32 Void
   +0x0f0 ExceptionPortData : Ptr32 Void
   +0x0f0 ExceptionPortValue : Uint4B
   +0x0f0 ExceptionPortState : Pos 0, 3 Bits
   +0x0f4 ObjectTable      : Ptr32 _HANDLE_TABLE
   +0x0f8 Token            : _EX_FAST_REF
   +0x0fc WorkingSetPage   : Uint4B
   +0x100 AddressCreationLock : _EX_PUSH_LOCK
   +0x104 RotateInProgress : Ptr32 _ETHREAD
   +0x108 ForkInProgress   : Ptr32 _ETHREAD
   +0x10c HardwareTrigger  : Uint4B
   +0x110 PhysicalVadRoot  : Ptr32 _MM_AVL_TABLE
   +0x114 CloneRoot        : Ptr32 Void
   +0x118 NumberOfPrivatePages : Uint4B
   +0x11c NumberOfLockedPages : Uint4B
   +0x120 Win32Process     : Ptr32 Void
   +0x124 Job              : Ptr32 _EJOB
   +0x128 SectionObject    : Ptr32 Void
   +0x12c SectionBaseAddress : Ptr32 Void
   +0x130 Cookie           : Uint4B
   +0x134 Spare8           : Uint4B
   +0x138 WorkingSetWatch  : Ptr32 _PAGEFAULT_HISTORY
   +0x13c Win32WindowStation : Ptr32 Void
   +0x140 InheritedFromUniqueProcessId : Ptr32 Void
   +0x144 LdtInformation   : Ptr32 Void
   +0x148 VdmObjects       : Ptr32 Void
   +0x14c ConsoleHostProcess : Uint4B
   +0x150 DeviceMap        : Ptr32 Void
   +0x154 EtwDataSource    : Ptr32 Void
   +0x158 FreeTebHint      : Ptr32 Void
   +0x160 PageDirectoryPte : _HARDWARE_PTE_X86
   +0x160 Filler           : Uint8B
   +0x168 Session          : Ptr32 Void
   +0x16c ImageFileName    : [15] UChar
   +0x17b PriorityClass    : UChar
   +0x17c JobLinks         : _LIST_ENTRY
   +0x184 LockedPagesList  : Ptr32 Void
   +0x188 ThreadListHead   : _LIST_ENTRY
   ......

与 EPROCESS 对应,ETHREAD 结构记录了线程中相关的信息。32位 win7 系统 ETHREAD 结构如下:

kd> dt _ETHREAD
ntdll!_ETHREAD
   +0x000 Tcb              : _KTHREAD
   +0x200 CreateTime       : _LARGE_INTEGER
   +0x208 ExitTime         : _LARGE_INTEGER
   +0x208 KeyedWaitChain   : _LIST_ENTRY
   +0x210 ExitStatus       : Int4B
   +0x214 PostBlockList    : _LIST_ENTRY
   +0x214 ForwardLinkShadow : Ptr32 Void
   +0x218 StartAddress     : Ptr32 Void
   +0x21c TerminationPort  : Ptr32 _TERMINATION_PORT
   +0x21c ReaperLink       : Ptr32 _ETHREAD
   +0x21c KeyedWaitValue   : Ptr32 Void
   +0x220 ActiveTimerListLock : Uint4B
   +0x224 ActiveTimerListHead : _LIST_ENTRY
   +0x22c Cid              : _CLIENT_ID
   ......

我们根据这些信息,修改 process-list.c:

int main (int argc, char **argv)
{
    ...
    // 添加变量
    vmi_pid_t pid = 0, ppid = 0;
    unsigned long tasks_offset = 0, pid_offset = 0, name_offset = 0, pdbase_offset = 0;
	...

    /* init the offset values */
    if (VMI_OS_LINUX == vmi_get_ostype(vmi)) {
        if ( VMI_FAILURE == vmi_get_offset(vmi, "linux_tasks", &tasks_offset) )
            goto error_exit;
        if ( VMI_FAILURE == vmi_get_offset(vmi, "linux_name", &name_offset) )
            goto error_exit;
        if ( VMI_FAILURE == vmi_get_offset(vmi, "linux_pid", &pid_offset) )
            goto error_exit;
    } else if (VMI_OS_WINDOWS == vmi_get_ostype(vmi)) {
        if ( VMI_FAILURE == vmi_get_offset(vmi, "win_tasks", &tasks_offset) )
            goto error_exit;
        if ( VMI_FAILURE == vmi_get_offset(vmi, "win_pname", &name_offset) )
            goto error_exit;
        if ( VMI_FAILURE == vmi_get_offset(vmi, "win_pid", &pid_offset) )
            goto error_exit;
        // 添加获取win_pdbase的步骤
        if ( VMI_FAILURE == vmi_get_offset(vmi, "win_pdbase", &pdbase_offset) )
            goto error_exit;
    } else if (VMI_OS_FREEBSD == vmi_get_ostype(vmi)) {
        tasks_offset = 0;
        if ( VMI_FAILURE == vmi_get_offset(vmi, "freebsd_name", &name_offset) )
            goto error_exit;
        if ( VMI_FAILURE == vmi_get_offset(vmi, "freebsd_pid", &pid_offset) )
            goto error_exit;
    }

    ...

    /* walk the task list */
    while (1) {

        current_process = cur_list_entry - tasks_offset;

        /* Note: the task_struct that we are looking at has a lot of
         * information.  However, the process name and id are burried
         * nice and deep.  Instead of doing something sane like mapping
         * this data to a task_struct, I'm just jumping to the location
         * with the info that I want.  This helps to make the example
         * code cleaner, if not more fragile.  In a real app, you'd
         * want to do this a little more robust :-)  See
         * include/linux/sched.h for mode details */

        /* NOTE: _EPROCESS.UniqueProcessId is a really VOID*, but is never > 32 bits,
         * so this is safe enough for x64 Windows for example purposes */
        vmi_read_32_va(vmi, current_process + pid_offset, 0, (uint32_t*)&pid);

        procname = vmi_read_str_va(vmi, current_process + name_offset, 0);

        // 根据偏移获取各种进程、线程信息
        uint32_t pdbase = 0;
        vmi_read_32_va(vmi, current_process + pdbase_offset, 0, (uint32_t*)&pdbase);

        vmi_read_32_va(vmi, current_process + 0x140, 0, (uint32_t*)&ppid);

        vmi_read_addr_va(vmi, current_process + 0x188, 0, (addr_t*)&ThreadListHead);
        ThreadListHead = ThreadListHead - 0x268;

        int64_t CreateTime = 0;
        vmi_read_64_va(vmi, ThreadListHead + 0x200, 0, &CreateTime);

        uint32_t Cid = 0;
        vmi_read_64_va(vmi, ThreadListHead + 0x22c, 0, (uint32_t*)&Cid);

        if (!procname) {
            printf("Failed to find procname\n");
            goto error_exit;
        }

        /* print out the process name */
        if((pid > 5) && (strcmp(procname, "svchost.exe") != 0)) {
        	printf("[%5d] %s (struct addr:%"PRIx64"), pdbase:%"PRIx64", ppid:%5d\n", pid, procname, current_process, pdbase, ppid);
            // 添加我们需要的输出语句
			printf("ThreadListHead:%"PRIx64"\n", ThreadListHead);
        	printf("thread createtime:%ld\n", CreateTime);
        	printf("Cid:%5d\n", Cid);
        }

    ...
}

输出结果:

  build git:(5882bc2)  examples/vmi-process-list -n win7 -s /tmp/introspector
LibVMI Suggestion: set win_ntoskrnl=0x3c14000 in libvmi.conf for faster startup.
LibVMI Suggestion: set win_kdbg=0x12ac28 in libvmi.conf for faster startup.
LibVMI Suggestion: set win_kdvb=0x83d3ec28 in libvmi.conf for faster startup.
Process listing for VM win7 (id=4)
[  224] smss.exe (struct addr:975deae0), pdbase:7ee0d020, ppid:    4
ThreadListHead:975de808
thread createtime:133255839485781250
Cid:  224
[  304] csrss.exe (struct addr:86d40768), pdbase:7ee0d060, ppid:  296
ThreadListHead:85cc87b0
thread createtime:133255839501718750
Cid:  304
[  352] wininit.exe (struct addr:92c8dd40), pdbase:7ee0d0a0, ppid:  296
ThreadListHead:85cb3770
thread createtime:133255839501875000
Cid:  352
[  360] csrss.exe (struct addr:86d46030), pdbase:7ee0d040, ppid:  344
ThreadListHead:85cad828
thread createtime:133255839502968750
Cid:  360
[  400] winlogon.exe (struct addr:85cad420), pdbase:7ee0d0c0, ppid:  344
ThreadListHead:86c07d48
thread createtime:133255839503125000
Cid:  400
[  444] services.exe (struct addr:86dca5e8), pdbase:7ee0d080, ppid:  352
ThreadListHead:86e0f030
thread createtime:133255839506718750
Cid:  444
[  452] lsass.exe (struct addr:86e124e0), pdbase:7ee0d0e0, ppid:  352
ThreadListHead:86e20030
thread createtime:133255839507656250
Cid:  452
[  460] lsm.exe (struct addr:86e14828), pdbase:7ee0d100, ppid:  352
ThreadListHead:86e144a0
thread createtime:133255839507187500
Cid:  460
[ 1200] dwm.exe (struct addr:86f26030), pdbase:7ee0d280, ppid:  808
ThreadListHead:86f26658
thread createtime:133255839526718750
Cid: 1200
......

读者可以根据示例自行添加需要获取的信息。上述两个结构的具体信息读者可以自行查阅或自行调试获取。

参考6.3小节,获取并输出Linux内核系统调用表sys_call_table的内容

背景知识

符号(Symbols),就是kernel中的变量(Variable Name)或函数名称(Function Name)。在Linux中,System.map文件是内核符号名称及其对应内存地址的映射表。它在内核构建过程中创建,并位于内核源代码树的根目录中(通常是/linux/System.map)。System.map文件由内核的崩溃转储分析工具和一些性能分析工具使用,以将内核地址映射到它们对应的符号名称。

实验步骤

这一个实验我们以 ubuntu14.04 为例。

安装 ubuntu14.04 系统以及修改虚拟机 XML 文件的过程不在此赘述。

虚拟机安装完成后,我们将它的 System.map 文件复制到宿主机 boot 目录下(其他路径也可以),检查一下 sys_call_table 符号是否存在:

➜  libvmi git:(5882bc2) ✗ cat /boot/System.map-3.13.0-24-generic | grep sys_call_table
ffffffff81801400 R sys_call_table
ffffffff81809cc0 R ia32_sys_call_table

然后修改 /etc/libvmi.conf 文件:

ubuntu14.04 {
    sysmap      = "/boot/System.map-3.13.0-24-generic";
    ostype      = "Linux";
}

接着用 map-symbol 程序查看这一块内存的内容:

➜  libvmi git:(5882bc2) ✗ build/examples/map-symbol ubuntu14.04 sys_call_table /tmp/introspector
00000000|  80 9e 1b 81 ff ff ff ff  20 9f 1b 81 ff ff ff ff   ........ .......
00000010|  b0 89 1b 81 ff ff ff ff  20 6b 1b 81 ff ff ff ff   ........ k......
00000020|  90 e9 1b 81 ff ff ff ff  c0 e9 1b 81 ff ff ff ff   ................
00000030|  a0 e9 1b 81 ff ff ff ff  10 e9 1c 81 ff ff ff ff   ................
00000040|  00 90 1b 81 ff ff ff ff  30 86 01 81 ff ff ff ff   ........0.......
00000050|  d0 10 18 81 ff ff ff ff  20 fe 17 81 ff ff ff ff   ........ .......
00000060|  e0 f2 17 81 ff ff ff ff  a0 bb 07 81 ff ff ff ff   ................
00000070|  30 a9 07 81 ff ff ff ff  10 6a 72 81 ff ff ff ff   0........jr.....
00000080|  c0 c8 1c 81 ff ff ff ff  c0 9f 1b 81 ff ff ff ff   ................
00000090|  70 a0 1b 81 ff ff ff ff  10 a5 1b 81 ff ff ff ff   p...............
000000a0|  d0 a5 1b 81 ff ff ff ff  60 7b 1b 81 ff ff ff ff   ........`{......
000000b0|  f0 2d 1c 81 ff ff ff ff  30 e0 1c 81 ff ff ff ff   .-......0.......
000000c0|  80 9a 09 81 ff ff ff ff  e0 1d 18 81 ff ff ff ff   ................
000000d0|  f0 22 18 81 ff ff ff ff  60 ab 17 81 ff ff ff ff   ."......`.......
000000e0|  a0 35 17 81 ff ff ff ff  80 fa 2b 81 ff ff ff ff   .5........+.....
000000f0|  60 02 2c 81 ff ff ff ff  e0 fa 2b 81 ff ff ff ff   `.,.......+.....
00000100|  c0 76 1d 81 ff ff ff ff  80 75 1d 81 ff ff ff ff   .v.......u......
00000110|  40 be 07 81 ff ff ff ff  f0 f4 08 81 ff ff ff ff   @...............
00000120|  40 ad 06 81 ff ff ff ff  60 62 07 81 ff ff ff ff   @.......`b......
00000130|  30 b1 06 81 ff ff ff ff  b0 e3 07 81 ff ff ff ff   0...............
00000140|  d0 ac 1b 81 ff ff ff ff  a0 2d 60 81 ff ff ff ff   .........-`.....
00000150|  60 31 60 81 ff ff ff ff  40 31 60 81 ff ff ff ff   `1`.....@1`.....
......

实验二:嵌套虚拟化环境实验

配置并重新编译KVM虚拟化环境,开启嵌套虚拟化模式,运行启动嵌套虚拟机

实验步骤

检查L0 KVM的嵌套虚拟化支持是否开启:

cat /sys/module/kvm_intel/parameters/nested

如果显示Y说明已经开启。

创建、启动L1、L2虚拟机的步骤详见:第一章实验

注意:使用qemu-system-x86_64命令启动虚拟机时,应加入-cpu host参数,以便上层vCPU继承下层CPU(或vCPU)的VMX特性。

在L0层KVM中捕获L1层KVM对嵌套虚拟机VMCS的写操作

实验思路

在KVM嵌套虚拟化实现中,L1、L2虚拟机实际都由L0物理机启动。

但在逻辑上(即从L1的视角),L2虚拟机由L1虚拟机中的KVM维护、创建、启动。

因此,L0、L1分别维护了如下VMCS数据结构:

VMCS名称 实际维护者 逻辑上的host 逻辑上的guest
vmcs01 L0 L0 L1
vmcs12 L1 L1 L2
vmcs02 L0 L0 L2
shadow_vmcs12 L0 L1 L2

当L2发生VM exit,需要进入L1 KVM处理,并返回L2,该过程步骤如下:

  1. L2 VM exit,控制流进入L0 KVM
  2. L0 KVM读取vmcs02数据,将guest部分内容更新至shadow_vmcs12
  3. L0 KVM将shadow_vmcs12推送至L1虚拟机中维护的vmcs12
  4. L0 KVM切换VMCS到vmcs01,控制流进入L1 KVM
  5. L1 KVM识别到L2虚拟机的陷入并处理,继续执行L2虚拟机,该指令被L0捕获后,控制流回到L0KVM
  6. L0 KVM从L1虚拟机中读取到已经被L1 KVM维护过的vmcs12,存入shadow_vmcs12
  7. L0 KVM将shadow_vmcs12中guest相关数据更新至vmcs02
  8. L0 KVM切换VMCS到vmcs02,控制流回到L2虚拟机

因此,只需在L1处理L2 VM exit的前(步骤3、步骤4之间)后(步骤6、步骤7之间),对vmcs12分别进行读取、对比,即可捕获L1层KVM对嵌套虚拟机VMCS的写操作。

其中,在L1处理L2 VM exit前(步骤3、步骤4之间)加入代码如下:

record_vmcs12_data = vmcs12->guest_cr0;

在L1处理L2 VM exit后(步骤6、步骤7之间)加入代码如下:

if (record_vmcs12_data != vmcs12->guest_cr0)
{
	printk("[cpt6-exp2] before L1 run, guest_cr0=%llx\n",record_vmcs12_data);
	printk("[cpt6-exp2] after  L1 run, guest_cr0=%llx\n",vmcs12->guest_cr0);
}

其中record_vmcs12_data为全局变量。

实验步骤

使用补丁文件exp2/cpt6-exp2-kernel.patch修改Linux内核源码,编译、安装:

make -j
sudo rmmod kvm_intel
sudo insmod kernel-source/arch/x86/kvm/kvm-intel.ko

依次启动L1、L2虚拟机

打开dmesg可以看到L1 KVM对L2虚拟机CR0寄存器的修改,如图:

cpt6-exp2-result