[gem5-users] gem5 Compute GPU model

[Trinayan Baruah]

Hi,

In the wiki page for the gem5 AMD GPU model it mentions that the simulator
supports https://github.com/AMDComputeLibraries/ComputeApps apps.

I tried to run a couple of apps there and they all had the same error.
Basically I built the model as instructed and then did something like this.


./build/GCN3_X86/gem5.opt ./configs/example/apu_se.py -c
/home/ComputeApps/lulesh-amp/lulesh .

The error I got is "sysconf(NPROCESSORS_CONF) failed.

: No such file or directory

request to allocate mask for invalid number; abort

: No such file or directory"



The other apps also have the same issue. Could anyone point me as to what
is the issue here or is my command incorrect?.


Thanks


Trinayan

[Gutierrez, Anthony]

This is a problem with X86 SE mode simulation, and not the GPU model itself.

It looks like there is sysconf call that is failing. I think a syscall trace may be useful as it can show you which system calls are being used, and which files the application is trying to open. I believe this error can happen when an open() call fails to find a certain file that should hold some configuration information.

Have you tested with ROCm 1.6.x? That is the only version we support in gem5.

Tony

From: gem5-users <gem5-users-***@gem5.org> On Behalf Of Trinayan Baruah
Sent: Tuesday, August 21, 2018 3:18 PM
To: m5-***@m5sim.org
Subject: [gem5-users] gem5 Compute GPU model

Hi,

In the wiki page for the gem5 AMD GPU model it mentions that the simulator supports https://github.com/AMDComputeLibraries/ComputeApps apps.

I tried to run a couple of apps there and they all had the same error. Basically I built the model as instructed and then did something like this.

./build/GCN3_X86/gem5.opt ./configs/example/apu_se.py -c /home/ComputeApps/lulesh-amp/lulesh .

The error I got is "sysconf(NPROCESSORS_CONF) failed.

: No such file or directory

request to allocate mask for invalid number; abort

: No such file or directory"





The other apps also have the same issue. Could anyone point me as to what is the issue here or is my command incorrect?.



Thanks



Trinayan

[Matt Sinclair]

Hi Tony & Trinayan,

I believe Tsung Tai has fixed this problem and is pushing a patch soon.
Not sure if he can reply to this and provide Trinayan either with a beta
version of the patch or some guidance about what he changed. Tsung Tai,
please correct me if I'm wrong.

Thanks,
Matt

On Thu, Aug 30, 2018 at 9:56 AM Gutierrez, Anthony <
***@amd.com> wrote:

> This is a problem with X86 SE mode simulation, and not the GPU model
> itself.
>
>
>
> It looks like there is sysconf call that is failing. I think a syscall
> trace may be useful as it can show you which system calls are being used,
> and which files the application is trying to open. I believe this error can
> happen when an open() call fails to find a certain file that should hold
> some configuration information.
>
>
>
> Have you tested with ROCm 1.6.x? That is the only version we support in
> gem5.
>
>
>
> Tony
>
>
>
> *From:* gem5-users <gem5-users-***@gem5.org> *On Behalf Of *Trinayan
> Baruah
> *Sent:* Tuesday, August 21, 2018 3:18 PM
> *To:* m5-***@m5sim.org
> *Subject:* [gem5-users] gem5 Compute GPU model
>
>
>
> Hi,
>
>
>
> In the wiki page for the gem5 AMD GPU model it mentions that the simulator
> supports https://github.com/AMDComputeLibraries/ComputeApps apps.
>
>
>
> I tried to run a couple of apps there and they all had the same error.
> Basically I built the model as instructed and then did something like
> this.
>
>
>
> ./build/GCN3_X86/gem5.opt ./configs/example/apu_se.py -c
> /home/ComputeApps/lulesh-amp/lulesh .
>
>
>
> The error I got is "sysconf(NPROCESSORS_CONF) failed.
>
> : No such file or directory
>
> request to allocate mask for invalid number; abort
>
> : No such file or directory"
>
>
>
>
>
> The other apps also have the same issue. Could anyone point me as to what
> is the issue here or is my command incorrect?.
>
>
>
> Thanks
>
>
>
> Trinayan
>
>
>
>
> _______________________________________________
> gem5-users mailing list
> gem5-***@gem5.org
> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

[Matt Sinclair]

Hi Trinayan,

To follow up on Tony's question, what version of ROCm 1.6 do you have
installed? Did you checkout the roc-1.6.x branches for them? Either way,
can you explain a little bit about how you installed them?

We are working with Tsung to get his patches posted, but in the meantime
let me try to recreate the solution with you. I am looking through my
notes when we helped Tsung fixed this problem, and I believe I've narrowed
down what the solution to this problem is. First, to confirm your problem
is the same as his, can you get a SyscallAll trace (–debug-flags=SyscallAll
-e -r)? When Tsung was seeing this problem, his SyscallAll trace showed
that the open syscall was pointing to the wrong location for
"/proc/self/status", "/sys/devices/system/node", and
"/sys/devices/system/cpu." Hopefully your trace will show the same. If
so, then we can try and guide you through the fix Tsung made (or his patch
will be posted by then, hopefully).

Matt

On Thu, Aug 30, 2018 at 10:29 AM Matt Sinclair <***@gmail.com>
wrote:

> Hi Tony & Trinayan,
>
> I believe Tsung Tai has fixed this problem and is pushing a patch soon.
> Not sure if he can reply to this and provide Trinayan either with a beta
> version of the patch or some guidance about what he changed. Tsung Tai,
> please correct me if I'm wrong.
>
> Thanks,
> Matt
>
> On Thu, Aug 30, 2018 at 9:56 AM Gutierrez, Anthony <
> ***@amd.com> wrote:
>
>> This is a problem with X86 SE mode simulation, and not the GPU model
>> itself.
>>
>>
>>
>> It looks like there is sysconf call that is failing. I think a syscall
>> trace may be useful as it can show you which system calls are being used,
>> and which files the application is trying to open. I believe this error can
>> happen when an open() call fails to find a certain file that should hold
>> some configuration information.
>>
>>
>>
>> Have you tested with ROCm 1.6.x? That is the only version we support in
>> gem5.
>>
>>
>>
>> Tony
>>
>>
>>
>> *From:* gem5-users <gem5-users-***@gem5.org> *On Behalf Of *Trinayan
>> Baruah
>> *Sent:* Tuesday, August 21, 2018 3:18 PM
>> *To:* m5-***@m5sim.org
>> *Subject:* [gem5-users] gem5 Compute GPU model
>>
>>
>>
>> Hi,
>>
>>
>>
>> In the wiki page for the gem5 AMD GPU model it mentions that the
>> simulator supports https://github.com/AMDComputeLibraries/ComputeApps
>> apps.
>>
>>
>>
>> I tried to run a couple of apps there and they all had the same error.
>> Basically I built the model as instructed and then did something like
>> this.
>>
>>
>>
>> ./build/GCN3_X86/gem5.opt ./configs/example/apu_se.py -c
>> /home/ComputeApps/lulesh-amp/lulesh .
>>
>>
>>
>> The error I got is "sysconf(NPROCESSORS_CONF) failed.
>>
>> : No such file or directory
>>
>> request to allocate mask for invalid number; abort
>>
>> : No such file or directory"
>>
>>
>>
>>
>>
>> The other apps also have the same issue. Could anyone point me as to what
>> is the issue here or is my command incorrect?.
>>
>>
>>
>> Thanks
>>
>>
>>
>> Trinayan
>>
>>
>>
>>
>> _______________________________________________
>> gem5-users mailing list
>> gem5-***@gem5.org
>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
>
>

[Trinayan Baruah]

Hi Tony and Matt,

Thanks for the info and sorry for posting the question again as I didn't
see that it was already replied to today.

Tony, to answer your question i do have rocm 1.6 installed. So maybe the
patch that Matt is mentioning will help to resolve this. I will see if and
any file is being opened that results in this error. Finally I see some
user specific paths in env variable in apu_se.py . Not sure if it has
anything to do with the problem but are we supposed to change those paths?.

Looking forward to receive the patch. Thanks.

Best,
Trinayan

[Gutierrez, Anthony]

Yes, you need to make sure that the LD_LIBRARY_PATH and other env vars are set appropriately for your system. In particular the paths to the ROCm libs.

Also, if you use a version of the ROCt library that is different than the one we use you’ll need to copy the kfd_ioctl.h header that comes with the thunk into src/dev/hsa so the ioctl command codes matchup.



From: gem5-users <gem5-users-***@gem5.org> On Behalf Of Trinayan Baruah
Sent: Thursday, August 30, 2018 6:09 PM
To: gem5-***@gem5.org
Subject: Re: [gem5-users] gem5 Compute GPU model

Hi Tony and Matt,

Thanks for the info and sorry for posting the question again as I didn't see that it was already replied to today.

Tony, to answer your question i do have rocm 1.6 installed. So maybe the patch that Matt is mentioning will help to resolve this. I will see if and any file is being opened that results in this error. Finally I see some user specific paths in env variable in apu_se.py . Not sure if it has anything to do with the problem but are we supposed to change those paths?.

Looking forward to receive the patch. Thanks.

Best,
Trinayan

[Trinayan Baruah]

Hi Matt,

Thanks for the response. I installed 1.6 version based on the instructions
provided by Joseph Greathouse here(
https://github.com/RadeonOpenCompute/ROCm/issues/517). To be specific I
used the package titled apt_1.6.0.tar.bz2 linked there. I built a local
package and installed it. I can see the files installed in /opt/rocm.

Keep in mind that I am not using a machine that has a AMD GPU installed as
I don't think it is required to run the simulator. So although I can
compile the apps from the ComputeApps repo, I cannot really run them there
because of lack of a Rocm supported GPU.

At your recommendation I tried the debug flags and indeed I can see a lot
of opens failing starting from this

61936500: system.cpu0: T0 : syscall open: failed -> path:/etc/ld.so.preload
(inferred from:/etc/ld.so.preload)

61936500: system.cpu0: T0 : syscall open returns -2

71543500: system.cpu0: T0 : syscall open called w/arguments
140737488347968, 524288, 140737354129768, 140737351990768, 0,
140737488348511

71543500: system.cpu0: T0 : syscall open: failed ->
path:/opt/rocm/hcc/bin/../lib/libdl.so.2 (inferred
from:/opt/rocm/hcc/bin/../lib/libdl.so.2)

I can share the full log with you if you need to have a look at it.

Now not really sure why this is happening but yes a lot of open calls are
failing.

Finally wanted to point this out. apu_se.py seems to have some env
variables directed to some rocm , gcc and libunwind directories and these
paths seem to be user specific to some extent. Not sure if I need to modify
them. In any case let me know what is the next natural step to debug this
problem.

Thanks a lot for the help.

Best,
Trinayan

[Matt Sinclair]

Hi Trinayan,

I have almost exclusively installed ROCm on machines without attached GPUs
of any kind (for running in gem5), so yes it is not necessary to have an
AMD GPU installed. As Tony recommended, you should definitely update the
paths in apu_se.py. For example, if you install ROCm in /opt/rocm/, then
you need to update the LD_LIBRARY_PATH lines starting here (
https://gem5.googlesource.com/amd/gem5/+/agutierr/master-gcn3-staging/configs/example/apu_se.py#458)
accordingly (including where you have unwind and gcc installed). From the
snippet you showed me, it appears that your problems are related to this.

After fixing this, you will potentially run into the problems Tsung faced,
but for now this is definitely the first thing you need to fix.

Potentially related: I have never tried installing the ROCm stuff via
apt-get like Joe recommended so I don't know if that will work or not. It
may end up that you'll need to manually installed them by checking out the
repos on Github (e.g., https://github.com/RadeonOpenCompute/hcc) and
specifically checkout out the roc-1.6.x branches like the instructions
specify. But for now please update the LD_LIBRARY_PATH variables like Tony
specified and we'll see what happens.

Thanks,
Matt

On Fri, Aug 31, 2018 at 2:19 PM Trinayan Baruah <***@gmail.com> wrote:

> Hi Matt,
>
> Thanks for the response. I installed 1.6 version based on the instructions
> provided by Joseph Greathouse here(
> https://github.com/RadeonOpenCompute/ROCm/issues/517). To be specific I
> used the package titled apt_1.6.0.tar.bz2 linked there. I built a local
> package and installed it. I can see the files installed in /opt/rocm.
>
> Keep in mind that I am not using a machine that has a AMD GPU installed as
> I don't think it is required to run the simulator. So although I can
> compile the apps from the ComputeApps repo, I cannot really run them there
> because of lack of a Rocm supported GPU.
>
> At your recommendation I tried the debug flags and indeed I can see a lot
> of opens failing starting from this
>
> 61936500: system.cpu0: T0 : syscall open: failed ->
> path:/etc/ld.so.preload (inferred from:/etc/ld.so.preload)
>
> 61936500: system.cpu0: T0 : syscall open returns -2
>
> 71543500: system.cpu0: T0 : syscall open called w/arguments
> 140737488347968, 524288, 140737354129768, 140737351990768, 0,
> 140737488348511
>
> 71543500: system.cpu0: T0 : syscall open: failed ->
> path:/opt/rocm/hcc/bin/../lib/libdl.so.2 (inferred
> from:/opt/rocm/hcc/bin/../lib/libdl.so.2)
>
> I can share the full log with you if you need to have a look at it.
>
> Now not really sure why this is happening but yes a lot of open calls are
> failing.
>
> Finally wanted to point this out. apu_se.py seems to have some env
> variables directed to some rocm , gcc and libunwind directories and these
> paths seem to be user specific to some extent. Not sure if I need to modify
> them. In any case let me know what is the next natural step to debug this
> problem.
>
> Thanks a lot for the help.
>
> Best,
> Trinayan
> _______________________________________________
> gem5-users mailing list
> gem5-***@gem5.org
> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

[Trinayan Baruah]

Hi Matt,

I finally tried out Tony's suggestion and updated the paths. Sorry for the
delay as I did not have access to a PC the past few days. It seems the .so
files that are being showed in the syscall trace are not actually present
in the directories where it is being searched. So as a next step as per
your suggestion. I will try to pull all 1.6.x related repo mentioned in the
wiki and build and install. Maybe that should help. Thanks for the help.

Best,
Trinayan

[Trinayan Baruah]

Hi Matt and Tony,

I am somewhat confused about the whole process of installing rocm for the
simulator. Matt mentioned I am suppose to get the 1.6 branch of the
different components and install them individually for the simulator.
I got the thunk and runtime to be installed and working on installing hcc
now. Given that I did not do a regular rocm installation, hcc installation
is having trouble finding the headers and library paths.
If you have any notes could you share the exact commands or list you
followed for installing rocm for the simulator if you could. In addition
does the kernel driver have to be also installed from
https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver to run the
simulator?
Again if you could share some list of commands or the process you would
follow to get the simulator up and running on a new system it would really
help a lot. Thanks ahead.

Best,
Trinayan

[Gutierrez, Anthony]

Trinayan,

You don’t need the ROCk component because it is emulated in the simulator. You only need the packages mentioned on the Wiki (ROCr/ROCt/HCC).

While, you need to build the components yourself, as opposed to using apt-get, I recommend installing the components in the default location if possible (/opt/rocm) this way the cmake files for each package will know where to find things. I think by using make install after building, things will be installed with the correct directory structure.

If you cannot install in /opt/rocm (because you lack root or something like that), then I would suggest installing everything in a known location using cmake install prefix. Something like -DCMAKE_INSTALL_PREFIX=/path/to/local/rocm/install when invoking cmake should do this for you.
Tony

From: gem5-users <gem5-users-***@gem5.org> On Behalf Of Trinayan Baruah
Sent: Monday, September 03, 2018 6:29 PM
To: gem5-***@gem5.org
Subject: Re: [gem5-users] gem5 Compute GPU model

Hi Matt and Tony,

I am somewhat confused about the whole process of installing rocm for the simulator. Matt mentioned I am suppose to get the 1.6 branch of the different components and install them individually for the simulator.
I got the thunk and runtime to be installed and working on installing hcc now. Given that I did not do a regular rocm installation, hcc installation is having trouble finding the headers and library paths.
If you have any notes could you share the exact commands or list you followed for installing rocm for the simulator if you could. In addition does the kernel driver have to be also installed from https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver to run the simulator?
Again if you could share some list of commands or the process you would follow to get the simulator up and running on a new system it would really help a lot. Thanks ahead.

Best,
Trinayan

[Trinayan Baruah]

Hi Tony and Matt,

With your help, I got all the components installed and can see the
simulation going ahead compared to before. So thanks for making it happen.

Right now though the simulation fails with the error "fatal: syscall
sched_setaffinity (#203) unimplemented." Again this is the lulesh
benchmark from the ComputeApps repository. I do understand the error but
not sure what to do about it for now

Did you get this error when running any application?. Could you please let
me know of any potential fixes to this problem?.

Thanks again.

Best,
Trinayan

[Matt Sinclair]

I have not seen this error on RHEL. The ultimate solution would be to
implement the syscall. Tsung got around the error by changing the
sched_setaffinity syscall from unimplemented to ignored it
syscall_emul.hh. I don't think we want to make that change in the mainline
code, but for now you could try making the same change as Tsung and seeing
if it allows you to make progress.

Matt

On Tue, Sep 4, 2018 at 8:02 PM Trinayan Baruah <***@gmail.com> wrote:

> Hi Tony and Matt,
>
> With your help, I got all the components installed and can see the
> simulation going ahead compared to before. So thanks for making it happen.
>
> Right now though the simulation fails with the error "fatal: syscall
> sched_setaffinity (#203) unimplemented." Again this is the lulesh
> benchmark from the ComputeApps repository. I do understand the error but
> not sure what to do about it for now
>
> Did you get this error when running any application?. Could you please let
> me know of any potential fixes to this problem?.
>
> Thanks again.
>
> Best,
> Trinayan
> _______________________________________________
> gem5-users mailing list
> gem5-***@gem5.org
> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

[Trinayan Baruah]

Hi Tony and Matt,

I changed the system calls file to ignore the unimplemented syscalls that
were arising namely sched_get_affinity, sched_set_affinity and the
simulation progresses more than before.

First of all the simulation hangs. After I check the simout, I see some
open calls failing when trying to load some libraries. Maybe the hang is
because of this. But the surprising thing is that these libraries are not
on my system at all. Things like libhsa-ext-finalize64.so.1,
libhsa-ext-image64.so.1 , libhsa-amd-aqlprofile64.so.1 namely. I am not
sure how to get these because they were not installed when I installed the
components required for the simulator. Could you let me know how to resolve
this? Thank you.

Best,
Trinayan

[Gutierrez, Anthony]

Hi Trinayan,


I'm not aware of any deadlock/livelock bugs that would cause this. Is you simulation getting to a GPU kernel launch? It could be an issue with the GPU pipeline, something in SE mode, or the Ruby protocol. A trace would be useful to help you identify the source of the problem.


The following flags may be useful:


For the GPU:

--debug-flags=GPUALL


For the protocol:

--debug-flags=ProtocolTrace,Ruby


For syscalls:

--debug-flag=SyscallAll


--ExecAll may also be useful for the CPU's execution trace.


Tony

________________________________
From: gem5-users <gem5-users-***@gem5.org> on behalf of Trinayan Baruah <***@gmail.com>
Sent: Thursday, September 6, 2018 3:35:55 PM
To: gem5-***@gem5.org
Subject: Re: [gem5-users] gem5 Compute GPU model

Hi Tony and Matt,

I changed the system calls file to ignore the unimplemented syscalls that were arising namely sched_get_affinity, sched_set_affinity and the simulation progresses more than before.

First of all the simulation hangs. After I check the simout, I see some open calls failing when trying to load some libraries. Maybe the hang is because of this. But the surprising thing is that these libraries are not on my system at all. Things like libhsa-ext-finalize64.so.1, libhsa-ext-image64.so.1 , libhsa-amd-aqlprofile64.so.1 namely. I am not sure how to get these because they were not installed when I installed the components required for the simulator. Could you let me know how to resolve this? Thank you.

Best,
Trinayan

[Trinayan Baruah]

Hi Tony and Matt,

On using the flag GPUALL I notice on simout that the dispatcher prints
"Launching 0 kernels", "Returning 0 kernels". Basically the kernel launches
are not happening right ?. So could this be issue with the runtime?.

Again if you could look at my previous question, I mentioned that the
syscall trace showed there are failed open calls on libraries like
libhsa-ext-finalize64.so.1,
libhsa-ext-image64.so.1 , libhsa-amd-aqlprofile64.so.1 . These libraries
were not installed by the components mentioned on the wiki for the
simulator and therefore the open calls fail. Does your syscall trace show
these libraries trying to be opened?. I just want to make sure the
necessary components are there since the kernel launches seem to be failing
right now and it may be tied to these things.

Thanks for all the help

Best,
Trinayan

[Gutierrez, Anthony]

If you look in the source, that print is called during exec() for the dispatcher. It simply means that for that cycle there are no kernel launches. It’s not unusual to see that print at the beginning showing 0 kernel launches.

Those libraries do not matter as they are extensions, and they are not likely causing a hang.



From: gem5-users <gem5-users-***@gem5.org> On Behalf Of Trinayan Baruah
Sent: Saturday, September 08, 2018 12:51 PM
To: gem5-***@gem5.org
Subject: Re: [gem5-users] gem5 Compute GPU model

Hi Tony and Matt,

On using the flag GPUALL I notice on simout that the dispatcher prints "Launching 0 kernels", "Returning 0 kernels". Basically the kernel launches are not happening right ?. So could this be issue with the runtime?.

Again if you could look at my previous question, I mentioned that the syscall trace showed there are failed open calls on libraries like libhsa-ext-finalize64.so.1, libhsa-ext-image64.so.1 , libhsa-amd-aqlprofile64.so.1 . These libraries were not installed by the components mentioned on the wiki for the simulator and therefore the open calls fail. Does your syscall trace show these libraries trying to be opened?. I just want to make sure the necessary components are there since the kernel launches seem to be failing right now and it may be tied to these things.

Thanks for all the help

Best,
Trinayan

[Trinayan Baruah]

Hi Tony and Matt,

The syscall trace shows that after running for a while


syscall open: failed -> path:/home/trinayan/gem5/
m5out/fs/sys/bus/event_source/devices/amd_iommu/events (inferred
from:/sys/bus/event_source/devices/amd_iommu/events)


syscall open: failed -> path:/dev/shm/trinayan/gem5_
9797/sem.hsakmt_semaphore




After that the simulation trace shows only the following line and it just
prints this line on and on.


system.cpu00: T0 : syscall read needs retry


Are the above opens that failed relevant?. If so how could this be resolved?


Thanks again.


Best,

Trinayan

[Gutierrez, Anthony]

Thos open call seem important, but there isn’t enough information here for me to understand what your actual problem is, or how it could be addressed.

My guess would be your application has some copies that need to be removed. You should make your kernel use pointer-based data structures, because gem5 only models APUs.

Tony

From: gem5-users <gem5-users-***@gem5.org> On Behalf Of Trinayan Baruah
Sent: Tuesday, September 11, 2018 3:35 PM
To: gem5-***@gem5.org
Subject: Re: [gem5-users] gem5 Compute GPU model

Hi Tony and Matt,

The syscall trace shows that after running for a while



syscall open: failed -> path:/home/trinayan/gem5/m5out/fs/sys/bus/event_source/devices/amd_iommu/events (inferred from:/sys/bus/event_source/devices/amd_iommu/events)



syscall open: failed -> path:/dev/shm/trinayan/gem5_9797/sem.hsakmt_semaphore







After that the simulation trace shows only the following line and it just prints this line on and on.



system.cpu00: T0 : syscall read needs retry



Are the above opens that failed relevant?. If so how could this be resolved?



Thanks again.



Best,

Trinayan

[Trinayan Baruah]

Hi Tony and Matt,

Could I have a answer to this?