[gem5-users] running fs.py with X86KvmCPU failed

Discussion:

Da Zhang

2018-02-08 04:48:13 UTC

I am trying to run fs.py with kvm support which might help speedup
our simulation in full system mode. I find the cpu type X86KvmCPU which is
a "kvm-based hardware virtualized cpu". But running fs.py failed with the
error information:

panic: KVM: Failed to enter virtualized mode (hw reason: 0x80000021)

Memory Usage: 2416600 KBytes

Program aborted at tick 53418967500

--- BEGIN LIBC BACKTRACE ---

build/X86/gem5.fast(_Z15print_backtracev+0x1f)[0xaee60f]

build/X86/gem5.fast(_Z12abortHandleri+0x34)[0xaee6f4]

/lib64/libpthread.so.0(+0xf5e0)[0x7f5b9ac685e0]

/lib64/libc.so.6(gsignal+0x37)[0x7f5b9901c1f7]

/lib64/libc.so.6(abort+0x148)[0x7f5b9901d8e8]

build/X86/gem5.fast[0x6627df]

build/X86/gem5.fast[0x95d518]

build/X86/gem5.fast(_ZN10BaseKvmCPU13handleKvmExitEv+0x249)[0xc10859]

build/X86/gem5.fast[0xbade3c]

build/X86/gem5.fast(_ZN10EventQueue10serviceOneEv+0x91)[0xad4fc1]

build/X86/gem5.fast(_Z9doSimLoopP10EventQueue+0xa0)[0xb5b110]

build/X86/gem5.fast(_Z8simulatem+0x1f3)[0xb5b563]

build/X86/gem5.fast[0x93867d]

build/X86/gem5.fast[0x939c65]

/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x730a)[0x7f5b9a56b0ca]

/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]

/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x663c)[0x7f5b9a56a3fc]

/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x67bd)[0x7f5b9a56a57d]

/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x67bd)[0x7f5b9a56a57d]

/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]

/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f5b9a56d002]

/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5513)[0x7f5b9a5692d3]

/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]

/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x663c)[0x7f5b9a56a3fc]

/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]

/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f5b9a56d002]

/lib64/libpython2.7.so.1.0(+0x10043f)[0x7f5b9a58643f]

/lib64/libpython2.7.so.1.0(PyRun_StringFlags+0x65)[0x7f5b9a5872a5]

build/X86/gem5.fast(_Z6m5MainiPPc+0x5f)[0xad327f]

build/X86/gem5.fast(main+0x33)[0x5ddf23]

/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f5b99008c05]

build/X86/gem5.fast[0x5df8fc]

--- END LIBC BACKTRACE ---

Aborted (core dumped)

the command is as simple as:

build/X86/gem5.fast -d ~/tmp/output1/ configs/example/fs.py --mem-size=2GB
--disk-image=linux-x86.img --cpu-type=X86KvmCPU

Any idea? thanks in advance.

best,
Da

Jason Lowe-Power

2018-02-08 17:01:02 UTC

Permalink

These patches "fix" the problem. However, they may not apply cleanly to
HEAD and they definitely are not cleanly implemented.

https://gem5-review.googlesource.com/c/public/gem5/+/7362
https://gem5-review.googlesource.com/c/public/gem5/+/7361

Cheers,
Jason

Post by Da Zhang
I am trying to run fs.py with kvm support which might help speedup
our simulation in full system mode. I find the cpu type X86KvmCPU which is
a "kvm-based hardware virtualized cpu". But running fs.py failed with the
panic: KVM: Failed to enter virtualized mode (hw reason: 0x80000021)
Memory Usage: 2416600 KBytes
Program aborted at tick 53418967500
--- BEGIN LIBC BACKTRACE ---
build/X86/gem5.fast(_Z15print_backtracev+0x1f)[0xaee60f]
build/X86/gem5.fast(_Z12abortHandleri+0x34)[0xaee6f4]
/lib64/libpthread.so.0(+0xf5e0)[0x7f5b9ac685e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f5b9901c1f7]
/lib64/libc.so.6(abort+0x148)[0x7f5b9901d8e8]
build/X86/gem5.fast[0x6627df]
build/X86/gem5.fast[0x95d518]
build/X86/gem5.fast(_ZN10BaseKvmCPU13handleKvmExitEv+0x249)[0xc10859]
build/X86/gem5.fast[0xbade3c]
build/X86/gem5.fast(_ZN10EventQueue10serviceOneEv+0x91)[0xad4fc1]
build/X86/gem5.fast(_Z9doSimLoopP10EventQueue+0xa0)[0xb5b110]
build/X86/gem5.fast(_Z8simulatem+0x1f3)[0xb5b563]
build/X86/gem5.fast[0x93867d]
build/X86/gem5.fast[0x939c65]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x730a)[0x7f5b9a56b0ca]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x663c)[0x7f5b9a56a3fc]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x67bd)[0x7f5b9a56a57d]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x67bd)[0x7f5b9a56a57d]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f5b9a56d002]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5513)[0x7f5b9a5692d3]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x663c)[0x7f5b9a56a3fc]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f5b9a56d002]
/lib64/libpython2.7.so.1.0(+0x10043f)[0x7f5b9a58643f]
/lib64/libpython2.7.so.1.0(PyRun_StringFlags+0x65)[0x7f5b9a5872a5]
build/X86/gem5.fast(_Z6m5MainiPPc+0x5f)[0xad327f]
build/X86/gem5.fast(main+0x33)[0x5ddf23]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f5b99008c05]
build/X86/gem5.fast[0x5df8fc]
--- END LIBC BACKTRACE ---
Aborted (core dumped)
build/X86/gem5.fast -d ~/tmp/output1/ configs/example/fs.py --mem-size=2GB
--disk-image=linux-x86.img --cpu-type=X86KvmCPU
Any idea? thanks in advance.
best,
Da
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

Da Zhang

2018-02-08 20:50:36 UTC

Permalink

Hi Jason

The package works (I used the second one)! And it also works with the
package you provided (
https://gem5-review.googlesource.com/c/public/gem5/+/7301) in my another
email to fix the keyboard and mouse issue for running later linux kernel
and ubuntu. (However, there are some conflicts in this package, and it is a
little tricky to merge them.)

Now, I can run gem5 for linux kernel v4.8.13 and ubuntu 16.04.1 with
kvm support. And the speedup is so amazing. It used to take me 20 ~ 30
minutes to boot up the system without the kvm cpu. Now, it takes only
several seconds!!!

Thanks so much!

best,
Da

Post by Jason Lowe-Power
These patches "fix" the problem. However, they may not apply cleanly to
HEAD and they definitely are not cleanly implemented.
https://gem5-review.googlesource.com/c/public/gem5/+/7362
https://gem5-review.googlesource.com/c/public/gem5/+/7361
Cheers,
Jason

Post by Da Zhang
I am trying to run fs.py with kvm support which might help speedup
our simulation in full system mode. I find the cpu type X86KvmCPU which is
a "kvm-based hardware virtualized cpu". But running fs.py failed with the
panic: KVM: Failed to enter virtualized mode (hw reason: 0x80000021)
Memory Usage: 2416600 KBytes
Program aborted at tick 53418967500
--- BEGIN LIBC BACKTRACE ---
build/X86/gem5.fast(_Z15print_backtracev+0x1f)[0xaee60f]
build/X86/gem5.fast(_Z12abortHandleri+0x34)[0xaee6f4]
/lib64/libpthread.so.0(+0xf5e0)[0x7f5b9ac685e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f5b9901c1f7]
/lib64/libc.so.6(abort+0x148)[0x7f5b9901d8e8]
build/X86/gem5.fast[0x6627df]
build/X86/gem5.fast[0x95d518]
build/X86/gem5.fast(_ZN10BaseKvmCPU13handleKvmExitEv+0x249)[0xc10859]
build/X86/gem5.fast[0xbade3c]
build/X86/gem5.fast(_ZN10EventQueue10serviceOneEv+0x91)[0xad4fc1]
build/X86/gem5.fast(_Z9doSimLoopP10EventQueue+0xa0)[0xb5b110]
build/X86/gem5.fast(_Z8simulatem+0x1f3)[0xb5b563]
build/X86/gem5.fast[0x93867d]
build/X86/gem5.fast[0x939c65]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x730a)[0x7f5b9a56b0ca]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x663c)[0x7f5b9a56a3fc]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x67bd)[0x7f5b9a56a57d]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x67bd)[0x7f5b9a56a57d]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f5b9a56d002]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5513)[0x7f5b9a5692d3]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x663c)[0x7f5b9a56a3fc]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f5b9a56d002]
/lib64/libpython2.7.so.1.0(+0x10043f)[0x7f5b9a58643f]
/lib64/libpython2.7.so.1.0(PyRun_StringFlags+0x65)[0x7f5b9a5872a5]
build/X86/gem5.fast(_Z6m5MainiPPc+0x5f)[0xad327f]
build/X86/gem5.fast(main+0x33)[0x5ddf23]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f5b99008c05]
build/X86/gem5.fast[0x5df8fc]
--- END LIBC BACKTRACE ---
Aborted (core dumped)
build/X86/gem5.fast -d ~/tmp/output1/ configs/example/fs.py
--mem-size=2GB --disk-image=linux-x86.img --cpu-type=X86KvmCPU
Any idea? thanks in advance.
best,
Da
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

Da Zhang

2018-02-13 15:51:16 UTC

Permalink

Hey Jason,

The package works.
However, I encountered performance issues with increased CPU number. The
performance was very great with up to 4 CPUs ("-n", I am assuming "number
of CPUs" equal to "number of cores" for testing multithreading workload
later). However, the system fails to boot (or maybe it was just too slow)
when I scale it >= 5 CPUs. Any ideas or suggestions?

I am working on a node with 2 x Intel(R) Xeon(R) CPU E5-2470 v2 @ 2.40GHz,
10 physic cores each (and 40 logical cores in total). I only used the fs.py
script with --cpu-type=X86KvmCPU, --mem-size=8GB and -n 4. Moreover, the
ancient Linux kernel and image from gem5 website have no performance
problem with increased CPUs.

best,
Da Zhang

Post by Da Zhang
Hi Jason
The package works (I used the second one)! And it also works with the
package you provided (https://gem5-review.googlesource.com/c/public/
gem5/+/7301) in my another email to fix the keyboard and mouse issue for
running later linux kernel and ubuntu. (However, there are some conflicts
in this package, and it is a little tricky to merge them.)
Now, I can run gem5 for linux kernel v4.8.13 and ubuntu 16.04.1 with
kvm support. And the speedup is so amazing. It used to take me 20 ~ 30
minutes to boot up the system without the kvm cpu. Now, it takes only
several seconds!!!
Thanks so much!
best,
Da

Post by Da Zhang
I am trying to run fs.py with kvm support which might help speedup
our simulation in full system mode. I find the cpu type X86KvmCPU which is
a "kvm-based hardware virtualized cpu". But running fs.py failed with the
panic: KVM: Failed to enter virtualized mode (hw reason: 0x80000021)
Memory Usage: 2416600 KBytes
Program aborted at tick 53418967500
--- BEGIN LIBC BACKTRACE ---
build/X86/gem5.fast(_Z15print_backtracev+0x1f)[0xaee60f]
build/X86/gem5.fast(_Z12abortHandleri+0x34)[0xaee6f4]
/lib64/libpthread.so.0(+0xf5e0)[0x7f5b9ac685e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f5b9901c1f7]
/lib64/libc.so.6(abort+0x148)[0x7f5b9901d8e8]
build/X86/gem5.fast[0x6627df]
build/X86/gem5.fast[0x95d518]
build/X86/gem5.fast(_ZN10BaseKvmCPU13handleKvmExitEv+0x249)[0xc10859]
build/X86/gem5.fast[0xbade3c]
build/X86/gem5.fast(_ZN10EventQueue10serviceOneEv+0x91)[0xad4fc1]
build/X86/gem5.fast(_Z9doSimLoopP10EventQueue+0xa0)[0xb5b110]
build/X86/gem5.fast(_Z8simulatem+0x1f3)[0xb5b563]
build/X86/gem5.fast[0x93867d]
build/X86/gem5.fast[0x939c65]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x730a)[0x7f5b9a56b0ca]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x663c)[0x7f5b9a56a3fc]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x67bd)[0x7f5b9a56a57d]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x67bd)[0x7f5b9a56a57d]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f5b9a56d002]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5513)[0x7f5b9a5692d3]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x663c)[0x7f5b9a56a3fc]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f5b9a56d002]
/lib64/libpython2.7.so.1.0(+0x10043f)[0x7f5b9a58643f]
/lib64/libpython2.7.so.1.0(PyRun_StringFlags+0x65)[0x7f5b9a5872a5]
build/X86/gem5.fast(_Z6m5MainiPPc+0x5f)[0xad327f]
build/X86/gem5.fast(main+0x33)[0x5ddf23]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f5b99008c05]
build/X86/gem5.fast[0x5df8fc]
--- END LIBC BACKTRACE ---
Aborted (core dumped)
build/X86/gem5.fast -d ~/tmp/output1/ configs/example/fs.py
--mem-size=2GB --disk-image=linux-x86.img --cpu-type=X86KvmCPU
Any idea? thanks in advance.
best,
Da
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

Jason Lowe-Power

2018-02-15 17:17:34 UTC

Permalink

Hi Da,

You likely need to enable gem5's multithreaded mode to get many CPUs to
boot correctly. I've had success with up to 32 cores on a 4-core 8-thread
system. I'm not sure if fs.py automatically does this correctly or not. See
my scripts here:
https://github.com/jlpresearch/gem5/tree/jason/kvm-testing/configs/myconfigs
(note:
I haven't rebased in a couple of months).

Cheers,
Jason

Post by Da Zhang
Hey Jason,
The package works.
However, I encountered performance issues with increased CPU number. The
performance was very great with up to 4 CPUs ("-n", I am assuming "number
of CPUs" equal to "number of cores" for testing multithreading workload
later). However, the system fails to boot (or maybe it was just too slow)
when I scale it >= 5 CPUs. Any ideas or suggestions?
10 physic cores each (and 40 logical cores in total). I only used the fs.py
script with --cpu-type=X86KvmCPU, --mem-size=8GB and -n 4. Moreover, the
ancient Linux kernel and image from gem5 website have no performance
problem with increased CPUs.
best,
Da Zhang

Post by Da Zhang
Hi Jason
The package works (I used the second one)! And it also works with the
package you provided (
https://gem5-review.googlesource.com/c/public/gem5/+/7301) in my another
email to fix the keyboard and mouse issue for running later linux kernel
and ubuntu. (However, there are some conflicts in this package, and it is a
little tricky to merge them.)
Now, I can run gem5 for linux kernel v4.8.13 and ubuntu 16.04.1 with
kvm support. And the speedup is so amazing. It used to take me 20 ~ 30
minutes to boot up the system without the kvm cpu. Now, it takes only
several seconds!!!
Thanks so much!
best,
Da

Post by Da Zhang
I am trying to run fs.py with kvm support which might help speedup
our simulation in full system mode. I find the cpu type X86KvmCPU which is
a "kvm-based hardware virtualized cpu". But running fs.py failed with the
panic: KVM: Failed to enter virtualized mode (hw reason: 0x80000021)
Memory Usage: 2416600 KBytes
Program aborted at tick 53418967500
--- BEGIN LIBC BACKTRACE ---
build/X86/gem5.fast(_Z15print_backtracev+0x1f)[0xaee60f]
build/X86/gem5.fast(_Z12abortHandleri+0x34)[0xaee6f4]
/lib64/libpthread.so.0(+0xf5e0)[0x7f5b9ac685e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f5b9901c1f7]
/lib64/libc.so.6(abort+0x148)[0x7f5b9901d8e8]
build/X86/gem5.fast[0x6627df]
build/X86/gem5.fast[0x95d518]
build/X86/gem5.fast(_ZN10BaseKvmCPU13handleKvmExitEv+0x249)[0xc10859]
build/X86/gem5.fast[0xbade3c]
build/X86/gem5.fast(_ZN10EventQueue10serviceOneEv+0x91)[0xad4fc1]
build/X86/gem5.fast(_Z9doSimLoopP10EventQueue+0xa0)[0xb5b110]
build/X86/gem5.fast(_Z8simulatem+0x1f3)[0xb5b563]
build/X86/gem5.fast[0x93867d]
build/X86/gem5.fast[0x939c65]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x730a)[0x7f5b9a56b0ca]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x663c)[0x7f5b9a56a3fc]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x67bd)[0x7f5b9a56a57d]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x67bd)[0x7f5b9a56a57d]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f5b9a56d002]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5513)[0x7f5b9a5692d3]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x663c)[0x7f5b9a56a3fc]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f5b9a56d002]
/lib64/libpython2.7.so.1.0(+0x10043f)[0x7f5b9a58643f]
/lib64/libpython2.7.so.1.0(PyRun_StringFlags+0x65)[0x7f5b9a5872a5]
build/X86/gem5.fast(_Z6m5MainiPPc+0x5f)[0xad327f]
build/X86/gem5.fast(main+0x33)[0x5ddf23]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f5b99008c05]
build/X86/gem5.fast[0x5df8fc]
--- END LIBC BACKTRACE ---
Aborted (core dumped)
build/X86/gem5.fast -d ~/tmp/output1/ configs/example/fs.py
--mem-size=2GB --disk-image=linux-x86.img --cpu-type=X86KvmCPU
Any idea? thanks in advance.
best,
Da
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

Da Zhang

2018-02-15 21:57:59 UTC

Permalink

Hi Jason,

Thanks a lot! Your scripts work!
I am able to scale up to 40 cores on a compute node with 20 physical cores
/ 40 logical cores with your scripts. So I think the fs.py doesn't do
multithreaded mode correctly for running with kvm cpu. Some of your code
are commented as "required" for running kvm, is that all I need to make
fs.py work?
Moreover, can I use --fast-forward combined with kvm? Our goal is to run a
program with kvm to an interesting point and then switch to a detailed cpu
for experiments. Can we specify the number of instructions to fast forward
(as with --fast-forward option)? So that we can avoid the long wait time
for things like initialization.

best,
Da Zhang

Post by Jason Lowe-Power
Hi Da,
You likely need to enable gem5's multithreaded mode to get many CPUs to
boot correctly. I've had success with up to 32 cores on a 4-core 8-thread
system. I'm not sure if fs.py automatically does this correctly or not. See
my scripts here: https://github.com/jlpresearch/gem5/tree/jason/
kvm-testing/configs/myconfigs (note: I haven't rebased in a couple of
months).
Cheers,
Jason

Post by Da Zhang
Hey Jason,
The package works. However, I encountered performance issues with increased CPU
number. The performance was very great with up to 4 CPUs ("-n", I am
assuming "number of CPUs" equal to "number of cores" for testing
multithreading workload later). However, the system fails to boot (or maybe
it was just too slow) when I scale it >= 5 CPUs. Any ideas or suggestions?
2.40GHz, 10 physic cores each (and 40 logical cores in total). I only used
the fs.py script with --cpu-type=X86KvmCPU, --mem-size=8GB and -n 4.
Moreover, the ancient Linux kernel and image from gem5 website have no
performance problem with increased CPUs.
best,
Da Zhang

Post by Da Zhang
Hi Jason
The package works (I used the second one)! And it also works with the
package you provided (https://gem5-review.googlesource.com/c/public/
gem5/+/7301) in my another email to fix the keyboard and mouse issue
for running later linux kernel and ubuntu. (However, there are some
conflicts in this package, and it is a little tricky to merge them.)
Now, I can run gem5 for linux kernel v4.8.13 and ubuntu 16.04.1 with
kvm support. And the speedup is so amazing. It used to take me 20 ~ 30
minutes to boot up the system without the kvm cpu. Now, it takes only
several seconds!!!
Thanks so much!
best,
Da

Post by Da Zhang
I am trying to run fs.py with kvm support which might help speedup
our simulation in full system mode. I find the cpu type X86KvmCPU which is
a "kvm-based hardware virtualized cpu". But running fs.py failed with the
panic: KVM: Failed to enter virtualized mode (hw reason: 0x80000021)
Memory Usage: 2416600 KBytes
Program aborted at tick 53418967500
--- BEGIN LIBC BACKTRACE ---
build/X86/gem5.fast(_Z15print_backtracev+0x1f)[0xaee60f]
build/X86/gem5.fast(_Z12abortHandleri+0x34)[0xaee6f4]
/lib64/libpthread.so.0(+0xf5e0)[0x7f5b9ac685e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f5b9901c1f7]
/lib64/libc.so.6(abort+0x148)[0x7f5b9901d8e8]
build/X86/gem5.fast[0x6627df]
build/X86/gem5.fast[0x95d518]
build/X86/gem5.fast(_ZN10BaseKvmCPU13handleKvmExitEv+0x249)[0xc10859]
build/X86/gem5.fast[0xbade3c]
build/X86/gem5.fast(_ZN10EventQueue10serviceOneEv+0x91)[0xad4fc1]
build/X86/gem5.fast(_Z9doSimLoopP10EventQueue+0xa0)[0xb5b110]
build/X86/gem5.fast(_Z8simulatem+0x1f3)[0xb5b563]
build/X86/gem5.fast[0x93867d]
build/X86/gem5.fast[0x939c65]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x730a)[0x7f5b9a56b0ca]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x663c)[0x7f5b9a56a3fc]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x67bd)[0x7f5b9a56a57d]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x67bd)[0x7f5b9a56a57d]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f5b9a56d002]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5513)[0x7f5b9a5692d3]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x663c)[0x7f5b9a56a3fc]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f5b9a56d002]
/lib64/libpython2.7.so.1.0(+0x10043f)[0x7f5b9a58643f]
/lib64/libpython2.7.so.1.0(PyRun_StringFlags+0x65)[0x7f5b9a5872a5]
build/X86/gem5.fast(_Z6m5MainiPPc+0x5f)[0xad327f]
build/X86/gem5.fast(main+0x33)[0x5ddf23]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f5b99008c05]
build/X86/gem5.fast[0x5df8fc]
--- END LIBC BACKTRACE ---
Aborted (core dumped)
build/X86/gem5.fast -d ~/tmp/output1/ configs/example/fs.py
--mem-size=2GB --disk-image=linux-x86.img --cpu-type=X86KvmCPU
Any idea? thanks in advance.
best,
Da
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

Jason Lowe-Power

2018-02-18 21:50:35 UTC

Permalink

Hi Da,

You can look at some of the other scripts in my github for examples of how
to fast forward with KVM.

I suggest that you *not* use fs.py since you know what you want to do. You
should write a script like my runkvm.py that runs gem5 exactly how you
need. fs/se.py were originally created as *examples* of how to write a
runscript. They've grown beyond that, though :).

I don't know exactly what is required and what isn't. You can play around
with the scripts and find out.

Cheers,
Jason

Post by Da Zhang
Hi Jason,
Thanks a lot! Your scripts work!
I am able to scale up to 40 cores on a compute node with 20 physical cores
/ 40 logical cores with your scripts. So I think the fs.py doesn't do
multithreaded mode correctly for running with kvm cpu. Some of your code
are commented as "required" for running kvm, is that all I need to make
fs.py work?
Moreover, can I use --fast-forward combined with kvm? Our goal is to run a
program with kvm to an interesting point and then switch to a detailed cpu
for experiments. Can we specify the number of instructions to fast forward
(as with --fast-forward option)? So that we can avoid the long wait time
for things like initialization.
best,
Da Zhang

Post by Jason Lowe-Power
Hi Da,
You likely need to enable gem5's multithreaded mode to get many CPUs to
boot correctly. I've had success with up to 32 cores on a 4-core 8-thread
system. I'm not sure if fs.py automatically does this correctly or not. See
I haven't rebased in a couple of months).
Cheers,
Jason

Post by Da Zhang
Hey Jason,
The package works.
However, I encountered performance issues with increased CPU number.
The performance was very great with up to 4 CPUs ("-n", I am assuming
"number of CPUs" equal to "number of cores" for testing multithreading
workload later). However, the system fails to boot (or maybe it was just
too slow) when I scale it >= 5 CPUs. Any ideas or suggestions?
2.40GHz, 10 physic cores each (and 40 logical cores in total). I only used
the fs.py script with --cpu-type=X86KvmCPU, --mem-size=8GB and -n 4.
Moreover, the ancient Linux kernel and image from gem5 website have no
performance problem with increased CPUs.
best,
Da Zhang

Post by Da Zhang
Hi Jason
The package works (I used the second one)! And it also works with the
package you provided (
https://gem5-review.googlesource.com/c/public/gem5/+/7301) in my
another email to fix the keyboard and mouse issue for running later
linux kernel and ubuntu. (However, there are some conflicts in this
package, and it is a little tricky to merge them.)
Now, I can run gem5 for linux kernel v4.8.13 and ubuntu 16.04.1 with
kvm support. And the speedup is so amazing. It used to take me 20 ~ 30
minutes to boot up the system without the kvm cpu. Now, it takes only
several seconds!!!
Thanks so much!
best,
Da

Post by Jason Lowe-Power
These patches "fix" the problem. However, they may not apply cleanly
to HEAD and they definitely are not cleanly implemented.
https://gem5-review.googlesource.com/c/public/gem5/+/7362
https://gem5-review.googlesource.com/c/public/gem5/+/7361
Cheers,
Jason

Post by Da Zhang
I am trying to run fs.py with kvm support which might help speedup
our simulation in full system mode. I find the cpu type X86KvmCPU which is
a "kvm-based hardware virtualized cpu". But running fs.py failed with the
panic: KVM: Failed to enter virtualized mode (hw reason: 0x80000021)
Memory Usage: 2416600 KBytes
Program aborted at tick 53418967500
--- BEGIN LIBC BACKTRACE ---
build/X86/gem5.fast(_Z15print_backtracev+0x1f)[0xaee60f]
build/X86/gem5.fast(_Z12abortHandleri+0x34)[0xaee6f4]
/lib64/libpthread.so.0(+0xf5e0)[0x7f5b9ac685e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f5b9901c1f7]
/lib64/libc.so.6(abort+0x148)[0x7f5b9901d8e8]
build/X86/gem5.fast[0x6627df]
build/X86/gem5.fast[0x95d518]
build/X86/gem5.fast(_ZN10BaseKvmCPU13handleKvmExitEv+0x249)[0xc10859]
build/X86/gem5.fast[0xbade3c]
build/X86/gem5.fast(_ZN10EventQueue10serviceOneEv+0x91)[0xad4fc1]
build/X86/gem5.fast(_Z9doSimLoopP10EventQueue+0xa0)[0xb5b110]
build/X86/gem5.fast(_Z8simulatem+0x1f3)[0xb5b563]
build/X86/gem5.fast[0x93867d]
build/X86/gem5.fast[0x939c65]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x730a)[0x7f5b9a56b0ca]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x663c)[0x7f5b9a56a3fc]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x67bd)[0x7f5b9a56a57d]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x67bd)[0x7f5b9a56a57d]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f5b9a56d002]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5513)[0x7f5b9a5692d3]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x663c)[0x7f5b9a56a3fc]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f5b9a56cefd]
/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f5b9a56d002]
/lib64/libpython2.7.so.1.0(+0x10043f)[0x7f5b9a58643f]
/lib64/libpython2.7.so.1.0(PyRun_StringFlags+0x65)[0x7f5b9a5872a5]
build/X86/gem5.fast(_Z6m5MainiPPc+0x5f)[0xad327f]
build/X86/gem5.fast(main+0x33)[0x5ddf23]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f5b99008c05]
build/X86/gem5.fast[0x5df8fc]
--- END LIBC BACKTRACE ---
Aborted (core dumped)
build/X86/gem5.fast -d ~/tmp/output1/ configs/example/fs.py
--mem-size=2GB --disk-image=linux-x86.img --cpu-type=X86KvmCPU
Any idea? thanks in advance.
best,
Da
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users