Linux NUMA physical memory layout

Just looked through the pseudo files in the procfs.

/proc/zoneinfo has some interesting information.

There are two zones in my current system: DMA, normal. We also have 2 nodes, and thus the file is shown as:

• Node0 DMA
• Node0 DMA32
• Node0 normal
• Node1 normal

Each section has how many free pages, present pages there are. At the end of the section there is the start_pfn that shows us the physical address of the beginning of the zone.

Thus we can approximate the physical address space of our system by using the start_pfn * PAGE_SIZE and also using the number of present pages.

NUMA Linux Memory Allocation

Scope of Memory Policies

System Default Policy

Hard coded into the kernel. On system boot it uses interleaved. After bootup it uses local allocation.

Task/Process Policy

Per task policy. Task policies are inherited to child processes. Thus applications like numactl uses this property to propogate the task policy to the child process.

In multi-threaded situation where other threads exist only the thread that calls the MEMORY_POLICY_APIS will set its memory policy. All other exisitng threads will retain the prior policy.

The policy only affects memory allocation after the time the policy is set. All allocations before the change are not affected.

VMA Policy

Only applies to anonymous pages. File mapped VMAs will ignore the VMA policy if it is set to MAP_SHARED. If it is MAP_PRIVATE VMA policy will only be enforced on a write to the mapping (CoW).

VMA policies are shared by threads of the same address space. VMA policies do not persist accross exec() calls (as the address pace is wiped)

Shared Policy

Similar to VMA policy but it is shared among processes. Some more details, but skipped due to irrelevance to my work.

Memory Policies

Default Mode

Specifies the current scope does not follow a policy, fall back to larger scope’s policy. At the root it’ll follow the system policy.

Bind Mode

Memory allocated from the nodes specified by the policy. Proximity is considered first, and if enough free space exists for the closest memory node (to the allocation requestor) it’ll be granted

Preferred Mode

Allocation will be attempted from the preferred (single) node. If it fails, the nodes will be searched for free space in nearest first fashion.
Local allocation is a preferred mode where the node that initiates a page fault is the preferred node.

Interleaved Mode

For Anonymous/shared pages: Node set is indexed using the page offset to the VMA. (Address % node_nums). The indexed node is requested for a page as in Preferered mode. And if it fails follows preferred mode style.

Page cache pages: A node counter is used to wrap around and try to spread out pages among the nodes (that are specified).

 

Reference: What is Linux Memory Policy

 

Zedboard Linux Hello world execution failure

No such file or directory!

On trying to execute the hello_world.elf file I ran into a ./hello_world.elf: No such file or directory error.

The problem was due to a missing interpreter that was provided in the elf file.

readelf -a hello_world.elf

results in a big output.

If yo check the Program Headers section you see the following output:

<br />Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
EXIDX 0x0004b4 0x000104b4 0x000104b4 0x00008 0x00008 R 0x4
PHDR 0x000034 0x00010034 0x00010034 0x00100 0x00100 R E 0x4
INTERP 0x000134 0x00010134 0x00010134 0x00019 0x00019 R 0x1
[Requesting program interpreter: /lib/ld-linux-armhf.so.3]
LOAD 0x000000 0x00010000 0x00010000 0x004c0 0x004c0 R E 0x10000
LOAD 0x0004c0 0x000204c0 0x000204c0 0x0011c 0x00120 RW 0x10000
DYNAMIC 0x0004cc 0x000204cc 0x000204cc 0x000e8 0x000e8 RW 0x4
NOTE 0x000150 0x00010150 0x00010150 0x00044 0x00044 R 0x4
GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0x10

The Requesting program interpreter is our problem here.

That file is not available on the zerboard’s /lib directory.

Thus just simply create a symbolic link of the interpreter on the zedboard

ln -s /lib/ld-2.17.so /lib/ld-linux-armhf.so.3

 

Xilinx SDK Hello World Build Errors

Cannot find libz.so.1

Got an error in the Xilinx SDK where the error message was that the arm gcc compiler was missing libz.so.1

Not sure if this will help but I tried installing the 32-bit libz library

apt-get install zlib1g:i386

Cannot find -lxil

Seems like somebody else had a problem here

And the third message had the answer:

I wasn’t able to solve it. It was a simple project, so I just remade it and it worked.

So I cleaned the build and remade it.

Driving FPGA IPs from Linux

 few links that’ll help me get started:

Zedboard.org forum

http://fpgacpu.wordpress.com/2013/05/24/yet-another-guide-to-running-linaro-ubuntu-desktop-on-xilinx-zynq-on-the-zedboard/

http://fpgacpu.wordpress.com/2013/05/28/how-to-design-and-access-a-memory-mapped-device-part-one/

http://fpgacpu.wordpress.com/2013/05/28/how-to-design-and-access-a-memory-mapped-device-part-two/

Forum Posts:

http://zedboard.com/content/how-communicate-fpga-linux

http://zedboard.org/content/interrupt-handler-kernel-module

By Sven Andersson

http://svenand.blogdrive.com/archive/203.html

Xilinx Wiki code

http://www.wiki.xilinx.com/Linux+User+Mode+Pseudo+Driver

Generating DTS and DTIs from Xilinx SDK

Xilinx Tutorial

DTB compiling help

When building the kernel with the uImage we need a mkimage which is in the uboot. Also uboot requires dtc which is available on the Fetch sources.

At the moment the kernel doesn’t boot after Starting kernel ...

Building the Kernel

The Xilinx manual makes the kernel load at 0x208000 however the u-boot boots from 0x8000; thus changed the

make ARCH=arm xilinx_zynq_defconfig
make ARCH=arm menuconfig
make ARCH=arm uImage LOADADDR=0x00008000

Also, changed the linux-xlnx branch from master to the xilinx-v2016.2 tag.

More problems: Nice forum post

https://forums.xilinx.com/t5/Embedded-Linux/Zynq-Linux-Freeze-on-Memory-Mapped-Peripheral-Write/td-p/246156

This post was helpful. It first identified that a problem may be in the FSBL

From the u-boot if the following command does not work, then we may need to recompile the FSBL

md.l 0x43c00000

If this does not work, the translation table in the FSBL must be incorrect. (This may be because we use the UCR-Bar’s files.Build the FSBL with the boot.bif in the UCR-Bar repo.

This page tells us how to create the FSBL Boot image, and This page how to build the FSBL from our IP configuration.

Done! – The process

• Thus we need to create a FSBL that allows us to access the actual physical addresses that are mapped for the IP.
  • FSBL is created by the SDK (new project (standalone) -> FSBL(template))
  • Package the FSBL and bitstream into a Boot.bin by using the `bootgen` tool and refer to [boot.bif](https://github.com/ucb-bar/fpga-images-zedboard/blob/968b9dd258c2dfef2d9108335889b993737f5650/boot.bif) file.
• Our device tree needs to map the IP address for the UIO
• [Enable the FPGA clk](https://forums.xilinx.com/t5/Embedded-Linux/Zynq-Linux-Freeze-on-Memory-Mapped-Peripheral-Write/m-p/685323#M15566) on the device tree

 

Vivado HLS (2016.02) co-simulation build error on Ubuntu 14.04 (crti.o not found)

Vivado HLS 2016.02 has an error when building binary files (for co-simulation) on Ubuntu 14.04. (Don’t know about other distributions, or other versions of Ubuntu)

I got my help from this blog post.

The problem I got was as follows:

INFO: [HLS 200-10] In directory '/home/username/Vivado_HLS_projects/XorGenerator/XorGenerator/sim/wrapc'
clang: warning: argument unused during compilation: '-fno-builtin-isinf'
clang: warning: argument unused during compilation: '-fno-builtin-isnan'
INFO: [APCC 202-3] Tmp directory is /tmp/apcc_db_username/160961476017075859177
INFO: [APCC 202-1] APCC is done.
Generating cosim.tv.exe
In file included from apatb_do_xor.cpp:18:0:
/opt/Xilinx/Vivado_HLS/2016.2/include/ap_stream.h:70:2: warning: #warning AP_STREAM macros are deprecated. Please use hls::stream<> from "hls_stream.h" instead. [-Wcpp]
/usr/bin/ld: cannot find crt1.o: No such file or directory
/usr/bin/ld: cannot find crti.o: No such file or directory

So the problem was that crt1.o and crti.o are note found. Just to give you a hint, these files are under /usr/lib/x86_64-linux_gnu

However instead of adding that directory as the build path or Library path (which I think is the intuition and the right way to do it…) The workaround in the blog post is to copy the crti.o, crt1.o, and crtn.o file to the Vivado HLS's gcc directories.

The path has changed since Vivado HLS 2012.02 as in the post.

Now the path is at /opt/Xilinx/Vivado_HLS/2016.2/lnx64/tools/gcc/lib/gcc/x86_64-unknown-linux-gnu/4.6.3/

Thus you will want to execute the following command:

cp /usr/lib/x86_64-linux-gnu/crt?.o /opt/Xilinx/Vivado_HLS/2016.2/lnx64/tools/gcc/lib/gcc/x86_64-unknown-linux-gnu/4.6.3/

Cheers 🙂

Debugging Gem5 with GDB (SE mode)

In order to debug gem5, we don’t need to hook on via the port that is printed when executing the simulator

0: system.remote_gdb.listener: listening for remote gdb #0 on port 7000

Rather just run with gdb --args ./build/X86/gem5.opt ...

Zedboard Block RAM Size

The Zedboard’s device name is Z-7020; and if we take a look at the data sheet of the Zynq-7000, we can find that the Block RAM size is 4.9Mb, or 140 x 36Kb blocks of BRAM.

FIFO can be used as a bridge between the PS and PL, as either will have different frequencies. To prevent data loss, the FIFO is used as a shared buffer as a synchronization circuit. (Ref)

Synology Gitlab Setup SSL over Let’s Encrypt

With Let’s Encrypt and Synology, we need to take an extra step to setup certificates in the Gitlab persistent data.

First off, the synology certificates seem to be at /usr/syno/etc/certificate/system/default

I’m referring to the instructions on github and by P. Behnke.

Also, the gitlab certs directory should be placed at /volume1/docker/gitlab/certs

Three files are required in all, gitlab.key, gitlab.crt, and dhparam.pem. If any of them does not exist, it won’t work.

Generate dhparam.pem by referring to the github info.

openssl dhparam -dsaparam -out dhparam.pem 2048

Also append the fullchain.crt & cert.pem into gitlab.crt. Change the name of privkey.pem into gitlab.key

Thus in the end you’ll have the following three files:

• dhparam.pem
• gitlab.crt
• gitlab.key

Restart the gitlab container, and you should check to see if you get a certificate not found error. If you don’t, then you should be set to go 🙂

Note: synology also has the dh-param files ready. (It takes very long to generate the file, I generated it on my workstation and ferried it over). Anwyays, you can find the synology generated files in the following path: /usr/syno/etc/ssl

Update: When you a SSL certification verification failed error

When using the HTTPS protocol the SSL verification sometimes seems to fail. The reason seems to be gnuTLS being picky about the order of the certificates.

fatal: unable to access ‘https://hostname:port/username/repo.git’ server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

Thus instead of the order that is shown in P.Behnke‘s blog, reverse the order of the fullchain and cert as follows:

$ cat fullchain.crt cert.pem > gitlab.crt

After restarting the docker containers, all seems to work.

Setting up Gitlab on Synology (via Docker)

Synology provides Gitlab from their package manager. However when you log on you see that the build version is too old, and Gitlab asks for upgrade ASAP.

Thus, I checked the docker registry, and downloaded the sameersbn/gitlab.

Synology seems to use Docker API v1 instead of v2, and so you can’t see all the tags. Thus logon to the synology device over ssh, and execute the following command

docker pull sameersbn/gitlab:8.12.3

The 8.12.3 was the latest tag. There is also another tag that is latest and so you may just pull from that tag.

Then you get a totally new docker image on the docker explorer on synology DSM. From here we need to copy some environment variables from the synology gitlab docker.

From the provided 8.6.2 to the 8.12.3, obviously there were a lot of change, and we need to add two more environment variables before gitlab boots properly.

This github issue helps us with the issue. Basically find the .secret file, and the secret value needs to be set as the GITLAB_SECRETS_OTP_KEY_BASE and GITLAB_SECRETS_SECRET_KEY_BASE. After you add these values gitlab should boot up.

Its a shame that the synology version of docker does not suport the pulling the latest version, and we have to do this manually. But still, synology did provide quite a lot of useful variables to refer to.

Finally set GITLAB_HTTPS to true to forward the GITLAB_PORT to HTTPS.

You should be set to go!