Tensorflow-CUDA breaks Python and R sessions

after the recent update, Tensorflow-CUDA stopped working on my device. It only breaks while on GPU. I am using optimus-manager and in intel mode, the code runs and learns but obviously without a GPU support, and on Nvidia mode the session breaks. Only the code that actually requires GPU support (training the model) breaks the session, creation and compilation of the model works fine.

R output:

2019-04-05 18:40:35.708966: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-04-05 18:40:35.734224: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2894255000 Hz
2019-04-05 18:40:35.734621: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x1129c8a0 executing computations on platform Host. Devices:
2019-04-05 18:40:35.734640: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2019-04-05 18:40:35.805393: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-04-05 18:40:35.806258: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x115c6280 executing computations on platform CUDA. Devices:
2019-04-05 18:40:35.806273: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): GeForce GTX 960M, Compute Capability 5.0
2019-04-05 18:40:35.806582: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: 
name: GeForce GTX 960M major: 5 minor: 0 memoryClockRate(GHz): 1.176
pciBusID: 0000:01:00.0
totalMemory: 3.95GiB freeMemory: 3.64GiB
2019-04-05 18:40:35.806596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-04-05 18:40:36.217346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-04-05 18:40:36.217381: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2019-04-05 18:40:36.217401: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2019-04-05 18:40:36.217727: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3360 MB memory) -> physical GPU (device: 0, name: GeForce GTX 960M, pci bus id: 0000:01:00.0, compute capability: 5.0)

With occasional:

Couldn't open CUDA library libcublas.so.10.1. LD_LIBRARY_PATH: 

which seams to be the key to the problem.

Python output:

WARNING:tensorflow:From /usr/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Train on 48000 samples, validate on 12000 samples
Epoch 1/50

and the session terminates.

My tensorflow installation:

python --version
Python 3.7.3
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:17_PST_2019
Cuda compilation tools, release 10.1, V10.1.105

Everyting worked fine before the update on 03-Apr-2019.

I am not the only one.

Workaround by partus:
export LD_LIBRARY_PATH=/opt/cuda/lib64
to ~/.bashrs or ~/.zshrc or ~/.profile

Your ~/.profile should contain:

export CUDA_HOME=/usr/local/cuda
export PATH

Change it to:

export CUDA_HOME=/opt/cuda      
export PATH

After reboot check:


Forum kindly sponsored by Bytemark