4GB of memory and NForce4SLI

Discussion:

Giuseppe Ghibò

2006-03-27 16:59:39 UTC

I tried currently 2006.0 with kernel 2.6.12.18mdksmp-x86_64 into
a MSI K8N Diamond Plus motherboard with Athlon64-X2 4800. The
motherboard has chipset Nforce4-sli and 4 modules of 1GB each.

Currently the BIOS sees 3328MB of memory and in such way
it correctly boots with 2.6.12.18mdksmp and free shows
3350260 kB free mem with the 'free' command.
How to obtain the rest of 700MB of memory? OK, there is the addressing
space for PCI, etc., but for instance other mobos shows more than 3.3GB.
E.g. on Tyan 2885 I see 4096MB in BIOS and 3559132 with 'free'.

The K8N Diamond has an option in BIOS to "Enable Hardware Memory Hole",
which enable memory remapping around the memory hole. The CPU needed
for this option is at least stepping E0. A64-4800X2 is stepping E6
so should be supported. Once the "Hardware Memory Hole" has been
enabled the BIOS sees 4096MB of memory. But the kernel is no
longer able to boot. Latest logs were:

...
Freeing unused kernel memory: 196k freed
Kernel panic - not syncing: No init found. Try passing init= option to kernel

I tried also with kernel-2.6.14.2mdksmp-x86_64 but same results. Even
with kernel-linus-2.6.16.15mdksmp-x86_64 gives the same (plus error
during 3ware modules scsi probing:

scsi: host 0 channel 0 id 0 lun 0x5a5a5a5a5a5a5a5ahas LUN larger than
currently supported.

Bugs in kernel IOMMU?

Bye
Giuseppe.

Thomas Backlund

2006-03-28 14:01:57 UTC

Permalink

Post by Giuseppe GhibÃ²
I tried currently 2006.0 with kernel 2.6.12.18mdksmp-x86_64 into
a MSI K8N Diamond Plus motherboard with Athlon64-X2 4800. The
motherboard has chipset Nforce4-sli and 4 modules of 1GB each.
Currently the BIOS sees 3328MB of memory and in such way
it correctly boots with 2.6.12.18mdksmp and free shows
3350260 kB free mem with the 'free' command.
How to obtain the rest of 700MB of memory? OK, there is the addressing
space for PCI, etc., but for instance other mobos shows more than 3.3GB.
E.g. on Tyan 2885 I see 4096MB in BIOS and 3559132 with 'free'.

Atleast MSI nforce4 boards does only support ~3.5GB memory without the
"memory hole" enabled...

Post by Giuseppe GhibÃ²
The K8N Diamond has an option in BIOS to "Enable Hardware Memory Hole",
which enable memory remapping around the memory hole. The CPU needed
for this option is at least stepping E0. A64-4800X2 is stepping E6
so should be supported. Once the "Hardware Memory Hole" has been
enabled the BIOS sees 4096MB of memory. But the kernel is no
...
Freeing unused kernel memory: 196k freed
Kernel panic - not syncing: No init found. Try passing init= option to kernel

you need to tell it how much memory you have using the "mem=" boot
option...

check this:
http://hardware.mcse.ms/archive20-2005-8-216970.html

at the end one user states:

<quote>

Post by Giuseppe GhibÃ²
Found the solution. Set the BIOS to Fill Memory Hole and then told Linux
it had 5G, i.e. mem=5G on the boot line.
BTW a mem= line is required if when you set the BIOS to Fill Memory Hole,
without it you'll get a kernel panic when you boot. The FC3 installer
just
does a straight boot so it will panic on a machine with the Fill Memory
Hole option set in the BIOS.

</quote>

Post by Giuseppe GhibÃ²
I tried also with kernel-2.6.14.2mdksmp-x86_64 but same results. Even
with kernel-linus-2.6.16.15mdksmp-x86_64 gives the same (plus error
scsi: host 0 channel 0 id 0 lun 0x5a5a5a5a5a5a5a5ahas LUN larger than
currently supported.
Bugs in kernel IOMMU?

you can also try booting with iommu=force

Post by Giuseppe GhibÃ²
Bye
Giuseppe.

--
Regards

Thomas

Giuseppe Ghibò

2006-03-29 16:26:05 UTC

Permalink

Post by Thomas Backlund

Atleast MSI nforce4 boards does only support ~3.5GB memory without the
"memory hole" enabled...

you need to tell it how much memory you have using the "mem=" boot
option...

I tried with mem=... too but up to 4096m you always get only 3.25GB.
With mem=4099m xfs is not starting anymore, and with bigger values such
as mem=4103m, or even mem=5120m, mem=5G it
goes in kernel panic.

Post by Thomas Backlund
http://hardware.mcse.ms/archive20-2005-8-216970.html
<quote>

</quote>

you can also try booting with iommu=force

didn't help.

Post by Thomas Backlund

Post by Giuseppe GhibÃ²
Bye
Giuseppe.

Giuseppe Ghibò

2006-04-04 15:53:50 UTC

Permalink

Post by Giuseppe GhibÃ²

Post by Thomas Backlund

Atleast MSI nforce4 boards does only support ~3.5GB memory without the
"memory hole" enabled...

you need to tell it how much memory you have using the "mem=" boot
option...

Post by Thomas Backlund
http://hardware.mcse.ms/archive20-2005-8-216970.html
<quote>

</quote>

you can also try booting with iommu=force

didn't help.

Post by Thomas Backlund

Post by Giuseppe GhibÃ²
Bye
Giuseppe.

According to this paper:

http://techfiles.de/dmelanchthon/files/memory_hole.pdf

it seems the memory hole can be shrinked by limiting
the AGP Aperture size. Unfortunately in the K8N Diamond I don't see
and BIOS option (furthermore it has PCI-express and not AGP slot)
to do this.

Bye
Giuseppe.

Giuseppe Ghibò

2006-05-09 17:19:45 UTC

Permalink

Post by Thomas Backlund

Atleast MSI nforce4 boards does only support ~3.5GB memory without the
"memory hole" enabled...

you need to tell it how much memory you have using the "mem=" boot
option...
http://hardware.mcse.ms/archive20-2005-8-216970.html
<quote>

</quote>

you can also try booting with iommu=force

Post by Giuseppe GhibÃ²
Bye
Giuseppe.

In case someone might be interested, today was out the BIOS 1.10 for the MSI K8N
Diamond Plus. This seems have improved the handling of the 4G memory-hole, Here
is results:

- with kernel 2.6.12.18mdksmp => "hardware memory hole" enabled in BIOS =>
works and 'free' shows full 4GB (indeed 3.85GB which sound the highest value
obtained right now) of memory:

total used free shared buffers cached
Mem: 4042208 131780 3910428 0 2852 69948
-/+ buffers/cache: 58980 3983228
Swap: 6385796 0 6385796

while with "hardware memory hole" disabled it was before:

total used free shared buffers cached
Mem: 3348972 389712 2959260 0 6892 212548
-/+ buffers/cache: 170272 3178700
Swap: 6385796 0 6385796

What is also pretty strange is that USB port are working only when
hardware memory hole is enabled, otherwise USB (either with Legacy
USB enabled or disabled) remains still not-working
properly (indeed works with kernel 2.6.14-2mdk, but 2.6.14-2mdk doesn't
work with hardware memory hole enabled) with errors (as soon
as for instance an USB pen is plugged):

kernel: usb 2-9: new high speed USB device using ehci_hcd and address 10
kernel: usb 2-9: device descriptor read/all, error -110
kernel: usb 2-9: new high speed USB device using ehci_hcd and address 11
kernel: usb 2-9: device descriptor read/all, error -110
kernel: usb 2-9: new high speed USB device using ehci_hcd and address 12
kernel: usb 2-9: unable to read config index 0 descriptor/start
kernel: usb 2-9: can't read configurations, error -110
kernel: usb 2-9: new high speed USB device using ehci_hcd and address 13
kernel: usb 2-9: unable to read config index 0 descriptor/start
kernel: usb 2-9: can't read configurations, error -110

- with kernel 2.6.14-2mdksmp + hardware hole enabled in BIOS => boot but soon
after a while (i.e. at end of inits) crashes with error:

kernel panic - not syncing: PCI-DMA: high address but no IOMMU

note that in this case I've also tried to boot using iommu=force and
iommu=off, but same error.

- with 2.6.16.11-server-smp-3mdk, works for hardware memory hole enabled (sees
3.85GB too), but doesn't work with USB (errors in /var/log/messages are
similar to the ones above).

- with kernel 2.6.12-20.uc4mdksmp, same as 2.6.12.28mdksmp.

Bye
Giuseppe.

Giuseppe Ghibò

2006-05-12 15:13:56 UTC

Permalink

Post by Giuseppe GhibÃ²
- with kernel 2.6.14-2mdksmp + hardware hole enabled in BIOS => boot but
kernel panic - not syncing: PCI-DMA: high address but no IOMMU
note that in this case I've also tried to boot using iommu=force and
iommu=off, but same error.

using for this case 'iommu=soft swiotlb=65536' works instead, but
AFAIK soft it's an option for Intel CPUs.

Luiz Fernando Capitulino

2006-03-28 17:20:57 UTC

Permalink

On Mon, 27 Mar 2006 18:59:39 +0200
Giuseppe Ghibò <***@mandriva.com> wrote:

| I tried currently 2006.0 with kernel 2.6.12.18mdksmp-x86_64 into
| a MSI K8N Diamond Plus motherboard with Athlon64-X2 4800. The
| motherboard has chipset Nforce4-sli and 4 modules of 1GB each.
|
| Currently the BIOS sees 3328MB of memory and in such way
| it correctly boots with 2.6.12.18mdksmp and free shows
| 3350260 kB free mem with the 'free' command.
| How to obtain the rest of 700MB of memory? OK, there is the addressing
| space for PCI, etc., but for instance other mobos shows more than 3.3GB.
| E.g. on Tyan 2885 I see 4096MB in BIOS and 3559132 with 'free'.

It's a BIOS issue then, isn't it? When we talked via IRC, I thought it
was a kernel problem..

| The K8N Diamond has an option in BIOS to "Enable Hardware Memory Hole",
| which enable memory remapping around the memory hole. The CPU needed
| for this option is at least stepping E0. A64-4800X2 is stepping E6
| so should be supported. Once the "Hardware Memory Hole" has been
| enabled the BIOS sees 4096MB of memory. But the kernel is no
| longer able to boot. Latest logs were:
|
| ...
| Freeing unused kernel memory: 196k freed
| Kernel panic - not syncing: No init found. Try passing init= option to kernel
|
| I tried also with kernel-2.6.14.2mdksmp-x86_64 but same results. Even
| with kernel-linus-2.6.16.15mdksmp-x86_64 gives the same (plus error
| during 3ware modules scsi probing:
|
| scsi: host 0 channel 0 id 0 lun 0x5a5a5a5a5a5a5a5ahas LUN larger than
| currently supported.

That sequence '0x5a5a5a5a' is the kernel's poisoning patter, I can't
say whether it's happening due a kernel issue.

Did you try to update your BIOS? And why do you need to change BIOS'
options? I mean, is it the _right_ way to do it?

| Bugs in kernel IOMMU?

Better to try the simpler first.

--
Luiz Fernando N. Capitulino

Giuseppe Ghibò

2006-03-28 17:51:31 UTC

Permalink

Post by Luiz Fernando Capitulino
On Mon, 27 Mar 2006 18:59:39 +0200
| I tried currently 2006.0 with kernel 2.6.12.18mdksmp-x86_64 into
| a MSI K8N Diamond Plus motherboard with Athlon64-X2 4800. The
| motherboard has chipset Nforce4-sli and 4 modules of 1GB each.
|
| Currently the BIOS sees 3328MB of memory and in such way
| it correctly boots with 2.6.12.18mdksmp and free shows
| 3350260 kB free mem with the 'free' command.
| How to obtain the rest of 700MB of memory? OK, there is the addressing
| space for PCI, etc., but for instance other mobos shows more than 3.3GB.
| E.g. on Tyan 2885 I see 4096MB in BIOS and 3559132 with 'free'.
It's a BIOS issue then, isn't it? When we talked via IRC, I thought it
was a kernel problem..

Why should be a BIOS issue? I have only two kind of setting: Memory
Hole Disabled and Memory Hole Enabled. Other BIOS parameters, like
"MTRR Mapping Mode" (which can be set to "Continuous" or "Discrete") have no
effect. Maybe is a general waeakness of the kernel to get the 4096MB of mem on
NForce4? Note also that I cited Tyan only to show that with 4GB of ECC memory
the PCI addressing space cause 4GB to shrink also on other kind of motherboards,
but the case of the MSI K8NDiamond Plus, IMHO the shrinkage is too much: at
least 200MB of fewer memory. I heard some people were able to get 3.6GB of
memory also on much older motherboards such as based on VIA chipset like ASUS
A8V, which is much more than the MSI one.

Post by Luiz Fernando Capitulino
| The K8N Diamond has an option in BIOS to "Enable Hardware Memory Hole",
| which enable memory remapping around the memory hole. The CPU needed
| for this option is at least stepping E0. A64-4800X2 is stepping E6
| so should be supported. Once the "Hardware Memory Hole" has been
| enabled the BIOS sees 4096MB of memory. But the kernel is no
|
| ...
| Freeing unused kernel memory: 196k freed
| Kernel panic - not syncing: No init found. Try passing init= option to kernel
|
| I tried also with kernel-2.6.14.2mdksmp-x86_64 but same results. Even
| with kernel-linus-2.6.16.15mdksmp-x86_64 gives the same (plus error
|
| scsi: host 0 channel 0 id 0 lun 0x5a5a5a5a5a5a5a5ahas LUN larger than
| currently supported.
That sequence '0x5a5a5a5a' is the kernel's poisoning patter, I can't
say whether it's happening due a kernel issue.
Did you try to update your BIOS? And why do you need to change BIOS'
options? I mean, is it the _right_ way to do it?

It's not possible to upgrade the BIOS, because MSI doesn't have
released any firmware upgrade for this model of motherboard. In other
words default one is the latest.

Post by Luiz Fernando Capitulino
| Bugs in kernel IOMMU?
Better to try the simpler first.