x86 - Constant reboot after setting up the Global Descriptor Table and protected mode

Question

I must have done something wrong with the GDT setup and the switch to protected mode because it keeps constantly rebooting.

Here is my kernel.asm that should setup the GDT and switch to protected mode:

    bits 16

    jmp main
    %include "gdt.inc"

    main:
        cli
        xor ax,ax
        mov ds,ax
        mov es,ax
        mov ax,0x9000
        mov ss,ax
        mov sp,0xffff
        sti

        call InstallGDT

        cli
        mov eax,cr0
        or eax,1
        jmp 08h:Stage3

    bits 32
    Stage3:

        mov ax,0x10
        mov ds,ax
        mov ss,ax
        mov es,ax
        mov esp,90000h

    Stop:

        mov byte [0xb8000],'A'
        cli
        hlt

and there is the gdt.inc:

bits 16
InstallGDT:

    cli
    pusha
    lgdt   [toc]
    sti
    popa
    ret
gdt_data:
    dd 0
    dd 0

    dw 0ffffh
    dw 0
    db 0
    db 10011010b
    db 11001111b
    db 0

    dw 0ffffh
    dw 0
    db 0
    db 10010010b
    db 11001111b
    db 0
end_of_gdt:
toc:
    dw end_of_gdt - gdt_data -1
    dd gdt_data

My bootloader.asm loads 10 sectors to 0x1000:0x000 and then jumps there.

I test the code with the commands:

nasm -f bin -o bootloader.bin bootloader.asm
nasm -f bin -o kernel.bin kernel.asm
cat bootloader.bin kernel.bin>OS.bin
qemu-system-i386 OS.bin

Where is my fault?

Answer 1

Since I can only assume you have read the sectors into memory at 0x1000:0x0000 correctly I can only point out potential problems in kernel.asm and gdt.inc.

---

Issues with the code

If you reached the kernel stage with jmp 0x1000:0x0000 (I suspect this is the case) then in kernel.asm you incorrectly set the DS, and ES segment registers to the wrong value. In that case you will need to set those two registers to 0x1000, not 0x0000. This code:

    xor ax,ax
    mov ds,ax
    mov es,ax

needs to be changed to:

    mov ax,0x1000
    mov ds,ax
    mov es,ax

---

The next major problem you have is that the GDT record (inside toc) takes a linear address. A linear address in real mode is the same thing as a physical address. From the instruction set manual it says:

The source operand specifies a 6-byte memory location that contains the base address (a linear address) and the limit (size of table in bytes) of the global descriptor table (GDT)

You have used an ORG of 0x0000 (since you didn't specify one) for kernel.asm so NASM assumes all offsets generated are from a base of 0x0000 including the label gdt_data. So when you do this:

toc:
dw end_of_gdt - gdt_data -1
dd gdt_data

gdt_data will be some small offset just above 0x0000. In physical memory your GDT record is actually at 0x1000:0x0000+(small offset). 0x1000:0x0000 in physical (linear) memory is (0x1000<<4)+0x0000 = 0x10000 so you need to add that to gdt_data. Your toc should look like this to compensate:

toc:
dw end_of_gdt - gdt_data -1
dd gdt_data+0x10000

---

The next issue you have is that you don't actually switch the protected mode flag on. You have this:

    mov eax,cr0
    or eax,1

It should be:

    mov eax,cr0
    or eax,1
    mov cr0, eax

You need to update the protected mode bit in the CR0 register after you set the bit to 1.

---

Related to the GDT issue, you have created GDT entries for the code segment from an offset of 0x00000000 which encompass the entire 4gb address space. This is correct. Again though, since NASM created offset from 0x0000 and your code is actually loaded at 0x1000:0x0000 (physical address 0x10000) you need to add 0x10000 to the value of stage3 label in the JMP that finally sets up protected mode. As well, because we are encoding a value that is above 0xFFFF we need to force NASM to use a 32-bit operand so we use the dword qualifier on the JMP. You have this:

jmp 08h:Stage3

It should be this:

jmp dword 08h:Stage3+0x10000