OK. on a P570, with an LPAR with two virtual processors, uncapped, and the following definition on en0: to an LPAR with one dedicated CPU with a similiar en0 configuration.
# ifconfig en0
en0: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPR
T,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>
inet 10.6.101.63 netmask 0xffffff00 broadcast 10.6.101.255
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
I ran the following test:
ftp node2
user root
password root
put "| dd if=/dev/zero bs=1m count=3000000" /dev/null
quit
On the nodes I had topas -i 5 running.
I see about 220K (KBPS) and what appears to be a CPU bound situation. I hope this gives you enough to be able to test. Be sure to compare with a MTU size of 1500 to see what, if any affect it has and CPU load and throughput.
Note: I edited the command - there was a typo and the command did not work as entered.