I am trying to use tensorflow with GPU acceleration on Manjaro 23.0.3 with KDE.
Unfortunately I ran into the following problem:
The python-tensorflow-* version is: 2.13.0-1
The python-keras version is: 2.12.0-2
But tensorflow 2.13 apparently requires keras 2.13 or throws exactly this error.
When I create a pipenv enviroment and install tf 2.12 and keras 2.12 or 2.13 and 2.13 my code works, but for some reason my gpu is then not recognized.
So my questions are:
can I somehow install keras 2.13?
or can I get my pipenv somehow to recognize my gpu?
python-tensorflow-* packages have been been flagged out-of-date since September 30. You’ll either have to wait for the maintainer to update the packages or do it yourself (which will be difficult given the amount of dependencies).
If you installed CUDA from the repositories, you’ll likely to have version 12.x installed. The tensorflow packages are compiled against 11.8, so it does not work.
However, newer versions might work because they now bring their own runtime.
Inside your virtualenv, you can use pip install tensorflow[and-cuda] and test if it works. (This is for tf 2.14, not 2.13)
Alrighty! Thx for the tip with the package! I just build a python-keras 2.13.1 package and at least this works now
Here’s the code for other unfortunate souls encountering the same problem. Just paste the following code (original code here) into a PKGBUILD file, then run makepkg, followed by sudo pacman -U python-keras-2.13.1-2-any.pkg.tar.zst
# Maintainer: Konstantin Gizdov <arch at kge dot pw>
_pkgbase='keras'
pkgbase="python-${_pkgbase}"
pkgname=("${pkgbase}")
pkgdesc='A deep learning API written in Python, running on top of the machine learning platform TensorFlow'
url='https://keras.io/'
license=('Apache')
pkgver=2.13.1
pkgrel=2
arch=('any')
source=("https://files.pythonhosted.org/packages/2e/f3/19da7511b45e80216cbbd9467137b2d28919c58ba1ccb971435cb631e470/keras-2.13.1-py3-none-any.whl"
"${_pkgbase}-${pkgver}-LICENSE::https://raw.githubusercontent.com/keras-team/${_pkgbase}/v${pkgver}/LICENSE")
makedepends=('python-build' 'python-installer' 'python-wheel')
depends=('absl-py' 'python' 'python-h5py' 'python-keras-preprocessing' 'python-numpy' 'python-pandas' 'python-pillow'
'python-pydot' 'python-scipy' 'python-six' 'python-tensorflow' 'python-yaml' 'tensorboard')
b2sums=('2598665c63080e63a41aa8d024da768825ac1533830bee60ce2ae95bf24df2f45bca02b629e507b11029e90ad1af6a50fa5c69ebf934422013a7a3bea4aba8aa'
'dc6395f606b09f8a2fa6e8d28f8436a9b0d2ee7e43b4033542c55eb1bf26e9e6c01fd53770e825b9e996ef15fd2eb77f1e0524d4fc1a3e8bf52d72de3adbd653')
package () {
python -m installer --destdir="$pkgdir" *.whl
install -Dm 644 "${_pkgbase}-${pkgver}-LICENSE" "${pkgdir}/usr/share/licenses/${pkgname}/LICENSE"
}
But the problem with the virtual envs not recognizing my gpu still persists. So any help here would be very much appreciated.
My pleasure
$ nvidia-smi
Tue Oct 10 20:30:37 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.113.01 Driver Version: 535.113.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA TITAN Xp Off | 00000000:02:00.0 On | N/A |
| 23% 25C P8 14W / 250W | 597MiB / 12288MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 670 G /usr/lib/Xorg 270MiB |
| 0 N/A N/A 753 G /usr/bin/kwalletd5 3MiB |
| 0 N/A N/A 826 G /usr/bin/ksmserver 3MiB |
| 0 N/A N/A 828 G /usr/bin/kded5 3MiB |
| 0 N/A N/A 829 G /usr/bin/kwin_x11 35MiB |
| 0 N/A N/A 866 G /usr/bin/plasmashell 46MiB |
| 0 N/A N/A 898 G ...b/polkit-kde-authentication-agent-1 3MiB |
| 0 N/A N/A 900 G /usr/lib/xdg-desktop-portal-kde 3MiB |
| 0 N/A N/A 1009 G /usr/bin/msm_kde_notifier 3MiB |
| 0 N/A N/A 1011 G /usr/lib/kdeconnectd 3MiB |
| 0 N/A N/A 1031 G /usr/bin/kaccess 3MiB |
| 0 N/A N/A 1034 G /usr/bin/pamac-tray-plasma 3MiB |
| 0 N/A N/A 1404 G /usr/lib/baloorunner 3MiB |
| 0 N/A N/A 1409 G /usr/bin/systemsettings 18MiB |
| 0 N/A N/A 1525 G /usr/lib/firefox/firefox 176MiB |
| 0 N/A N/A 1963 G /usr/bin/dolphin 3MiB |
+---------------------------------------------------------------------------------------+
I tried this already, but unfortunately without success:
[~/dev/projects/test_tf_214]$ pipenv shell
Launching subshell in virtual environment...
. /home/wurstkopf/.local/share/virtualenvs/test_tf_214-DyH0khk9/bin/activate
[~/dev/projects/test_tf_214]$ . /home/wurstkopf/.local/share/virtualenvs/test_tf_214-DyH0khk9/bin/activate
(test_tf_214) [~/dev/projects/test_tf_214]$ python
Python 3.11.5 (main, Oct 10 2023, 05:03:41) [GCC 13.2.1 20230801] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2023-10-10 20:39:05.333342: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-10-10 20:39:05.365187: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-10-10 20:39:05.365218: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-10-10 20:39:05.365250: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-10-10 20:39:05.371660: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-10-10 20:39:05.371865: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-10-10 20:39:06.326297: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
>>> print (tf.__version__)
2.14.0
>>>
Yes - I ran pipenv shell and then installed tf via pip install tensorflow.
I would also prefer to use py-/pipenv over globally installed packages. At least I can run stuff for now with gpu, but it would be cool to get this also working with pipenv. Any idea what to check or do next?
I think the error mentioned in the first post occurred already when I only had a tf package installed. I am not sure rn. I can test it tomorrow.
The problem was (or is) that the cuda package is version 12.2 and pip install tensorflow[and-cuda] installs tensorflow 2.13.1 but with specifying the version for 2.14.0 at least the GPU is recognized. Unfortunately cudnn, cublas and cufft from 12.2 aren’t supported in 2.14.0. But yeah so far so good
Innocent question: As I am fairly new to arch (as you might can tell), I am wondering, if in general it would make sense to update driver packages at the point in time when they can actually be used? Or am I not seeing something here?
Thanks for the help!
ps. headsup tf 2.14 only runs on python 3.10 as the required tensorrt package is not yet available for python 3.11
Tensorrt is a requirement for tensorflow 2.14.0 and it’s not ready for python 3.11
As described here. I ran into the same problem.
And, yeah, basically nothing is available for cuda 12.2 - that’s why I asked if it wouldn’t make more sense to wait until it’s actually usable before updating the arch package.
