SMP/CMT Broadcom 63xx

An example of SMP initialization on BCM6358 SoC: http://pastebin.com/wV3njK7c taken from linux-2.6.12.tar.bz2, mirror →linux-2.6.12-inv.zip

OpenWrt SMP on BMIPS cores: smp-bmips.c

The main thread is configured by the bootloader (CFE) when the SoC is initialized. It can be checked in the log returned by CFE via serial. Example:

CFE version cfe.d081.5003 for BCM96358 (32bit,SP,BE)
Build Date: Wed Nov 11 10:36:35 CST 2009 (Lihua_68693)
Copyright (C) 2006 Huawei Technologies Co. Ltd.


Boot Address 0xbe000000

Initializing Arena.
Initializing Devices.

@w45260: Flash Manufacture id :c2

@w45260Flash Device id :2201

@w45260flipCFIGeometry:1
Parallel flash device: name , id 0x2201, size 16384KB
*** GetHG556aBoardVersion = <0> ***

CPU type 0x2A010: 300MHz, Bus: 133MHz, Ref: 64MHz
Total memory: 67108864 bytes (64MB)

Total memory used by CFE:  0x80401000 - 0x8052A510 (1217808)
Initialized Data:          0x8041F3C0 - 0x80421B60 (10144)
BSS Area:                  0x80421B60 - 0x80428510 (27056)
Local Heap:                0x80428510 - 0x80528510 (1048576)
Stack Area:                0x80528510 - 0x8052A510 (8192)
Text (code) segment:       0x80401000 - 0x8041F3B4 (123828)
Boot area (physical):      0x0052B000 - 0x0056B000
Relocation Factor:         I:00000000 - D:00000000

*** GetHG556aBoardVersion = <0> ***

Board IP address                  : 192.168.1.1  
Host IP address                   : 192.168.1.35  
Gateway IP address                :   
Run from flash/host (f/h)         : h  
Default host run file name        : vmlinux  
Default host flash file name      : bcm963xx_fs_kernel  
Boot delay (0-9 seconds)          : 1  
Board Id Name                     : HW556  
Psi size in KB                    : 64
Number of MAC Addresses (1-32)    : 14  
Base MAC Address                  : 5c:4c:a9:6e:4a:a2  
Ethernet PHY Type                 : Internal
Memory size in MB                 : 64
CMT Thread Number                 : 1

Here we have

CMT Thread Number                 : 1

Then the main thread will be the core1. This is important since the BCM6358 SoC cores haven't the same features:

This parameter is located between offsets 0x014-0x017 in CFE. We can change it HEX editing the CFE. Setting the value to 0, makes the core0 the main thread. This brings 32kB instead 16kB icache to the operating system and therefore increases the performance.

Some CFEs allow to change the Main thread using the command line interface. This option is probably only present in most recent SoCs such as BCM6368.

BCM6368 SoC cores are identical:

So no benefit using a different core for the main thread.

To know if your CPU has concurrent multi-threading support (CMT) check bit 18 at BRCM Configuration register (read_c0_brcm_config_0):
0 = 1 core
1 = 2 cores, multi-thread supported

Also check the bit 12:
1 = Multicore CPU with split I-cache
0 = Multicore CPU with shared I-cache

read_c0_brcm_cmt_intr();

read_c0_brcm_cmt_ctrl();

read_c0_brcm_cmt_local();

On a CMT CPU, the TLB is shared between the two cores. Since hardware exception serialization must be turned off to allow ipis to reach the other core during operations such as I-cache flushing, we need to use software locking to ensure serialized access to the TLB and the corresponding CP0 registers.

Besides locking, the implementation is slightly different than on a standard SMP, as the CP0_CONTEXT is shared between the cores. Therefore it cannot be used to store the processor number, which is obtained from the CP0 CMT local register instead. It cannot be used to find the faulting address either.

If the lock cannot be taken, we must return from exception to allow software interrupts (of higher priority than TLB exceptions) to be serviced. The TLB exception will be retaken if really needed and we can try again to obtain the lock.

An entry may also be added on one core while the other core enters a TLB handler, so we must ensure the exception is is still valid by probing the TLB to avoid the following race:

		TP0			TP1
	TLB exception
	acquire lock
	...			access Badvaddr corresponding to entry X
	write to tlb entry X	enter TLB exception
	release lock		acquire lock
				...
	<refill:		Badvaddr may be present in the TLB now>
	<mod/load/store:	Badvaddr may have been removed from the TLB>

→ http://pastebin.com/JWCFs0qz

Note: Enable CMT support for the BCM6358 should be possible reviewing the code of linux-2.6.12.tar.bz2 (in the top of the page) and adapting it to a recent kernel.

BCM6362 and BCM6368 have a private TLB for each thread.

• No support in BCM6358, mainly caused by the shared TLB.
• BCM6362 and BCM6368 are supported. Available through the SMP subtarget in trunk versions:
  https://dev.openwrt.org/changeset/36526
  https://dev.openwrt.org/changeset/36527
• Fatal bug causing jffs2 data corruption → temporal workaround https://dev.openwrt.org/changeset/40396

The list of related devices: bcm63xx