did you had a look at Jami-daemon sous Mobian arm64 (pinephone) ?
it address similar concerns on aarch64 and provides a hack to compile it on limited target platform aka
sed -i -e 's/-j\${NPROC}/-j1/' CMakeList.txt # to avoid killing the pinephone
maybe same solution will work on R-Pi!