It’s been several days since the release of Ubuntu 16.04, and since I’ve bought a new graphic card, I think it’s necessary to have some fun with it. Obviously I underestimated the difficulty of adapting the Keras with a Theano backend on the new system. It took me a day to figure out what went wrong and searching for any solution, for I am so green on the practical mechanism behind the library (I only know how to use it).
Generally, the problem I met is like:
ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: ('nvcc return status', 1, 'for cmd', 'nvcc -shared -O3 -m64 -Xcompiler -DCUDA_NDARRAY_CUH=c72d035fdf91890f3b36710688069b2e,-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,-fPIC,-fvisibility=hidden -Xlinker -rpath,/home/joseph/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-2.7.11+-64/cuda_ndarray -I/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/theano/gof -o /home/joseph/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-2.7.11+-64/cuda_ndarray/cuda_ndarray.so mod.cu -L/usr/lib -lcublas -lpython2.7 -lcudart') WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu is not available (error: cuda unavailable)
when I launched theano on some tasks with GPU enabled and specified.
If you are in a hurry, click here to jump directly to the solution.
Setting Up CUDA & Theano/Keras Environment
I am just going to recall some interesting procedure before the problem came up.
The Graphic Card Driver
Everybody knows that we need to download the CUDA toolkit before we can actually communicate with the GPU, or at least you need to install the driver of your graphic card so that you may get more fluent experience when watching some high resolution videos…
OK, I don’t know since when did Ubuntu had the option of “Additional Driver”, but it is convenient to install an official driver this way. It is located in “System Settings”->”Software & Updates”->”Additional Drivers”. You can see something like this:
Simply select the “proprietary, tested” option and click “Apply Changes” below, and then wait until it is done. This is the easiest way in my opinion to install the driver of the graphic card.
And of course there’s another way to install the driver, which is attached in the CUDA toolkit installer.
The CUDA Toolkit
The latest version of CUDA toolkit 7.5 can be found here. The “friendly” nVIDIA offers various approaches to set up the toolkit, but at first I prefer the “.deb” method because I am lazzzzy.
I was aware of a known issue on Ubuntu 16.04 that third-party debs cannot be installed, but there are workarounds and it is nearly the same as what nVIDIA told you:
# approach 1: set up the toolkit via a .deb (doesn't work) ~$ sudo dpkg -i cuda-repo-ubuntu1504-7-5-local_7.5-18_amd64.deb ~$ sudo apt-get update ~$ sudo apt-get install cuda
While everything went on well until the last step, it told me that there no such pacakge as “cuda”…… Well I did found something called “nvidia-cuda-toolkit” in the Ubuntu official repository, and it conflicts with the current .deb file. Rationally speaking, the package from the repository should work, which means you may simply use the following command to finish the toolkit installation:
# approach 2: set up the toolkit via the Ubuntu repository (should work) ~$ sudo apt-get install nvidia-cuda-toolkit ~$ sudo apt-get install nvidia-cuda-*
In case of missing something, the second command above will install “nvidia-cuda-dev”, “nvidia-cuda-doc”, “nvidia-cuda-gdb” together with “nvidia-cuda-toolkit”. And after that, the toolkit should work.
Obviously the “approach 1″ does not work for me, and it seems that the “approach 2″ should work, but I didn’t try the second approach, instead I try the third approach – through a “.run” file.
# approach 3: set up the toolkit via a .run file (it worked) ~$ sudo chmod a+x cuda_7.5.18_linux.run ~$ sudo cuda_7.5.18_linux.run --override
Thanks to Dr Donald Kinghorn’s article, the “–override” parameter allows the installation script to ignore the incompatibility of compilers (CUDA works with gcc/g++ version <= 4.9, while the version on Ubuntu 16.04 reaches 5.3+).
If you have installed the driver from the “Additional Drivers” dialogue, then you does not need to install the driver in the script. Otherwise, you may need to shutdown all x-window related programs for the driver installer to run, which means if you want to install the graphic card driver from the “.run” file, press “Ctrl+Alt+F3″ to switch to commandline mode and use the commands below instead of the above one:
# approach 3-1: set up the toolkit and the driver via a .run file ~$ sudo init 3 ~$ sudo chmod a+x cuda_7.5.18_linux.run ~$ sudo cuda_7.5.18_linux.run --override
“approach 3-1″ will allow you to install the driver all together with the toolkit, theoretically, because I hadn’t tried this way.
Environment of CUDA Toolkit
You may refer to Dr Donald Kinghorn’s article for a quick setup, or follow the instruction from the “.run” file as below.
# set up CUDA environment ~$ sudo vim ~/.bashrc export PATH=/usr/local/cuda-7.5/bin:$PATH export LD_LIBRARY_PATH=/usr/local/cuda-7.5/lib64:$LD_LIBRARY_PATH ~$ sudo source ~/.bashrc
And then you can open a new terminal and type “nvcc” to see whether the environment is set correctly.
Solving gcc/g++ Incompatibilities
Even though you skip the compatibility check in the installation script, you still need a compatible compiler for CUDA to work with. So first of all make sure you have one:
# install gcc/g++ 4.9 ~$ sudo apt-get install gcc-4.9 g++-4.9
Then we may establish a soft link of the specific version for the CUDA binaries folder:
# soft link for gcc/g++ ~$ sudo ln -s /usr/bin/gcc-4.9 /usr/local/cuda-7.5/bin/gcc ~$ sudo ln -s /usr/bin/g++-4.9 /usr/local/cuda-7.5/bin/g++
Then CUDA will seek for gcc/g++ 4.9 when needed.
Installing Theano and Keras
This will be a lot easier since there are already tutorials.
First-Time Compiling Failure & The Solution
At first I found my own program on GPU does not work, with the error message mentioned at the beginning, I then refer to the example of the Keras library. I tried the mnist example from Keras, still the same error. Then I ran a CUDA test from the CUDA-7.5-Samples, and it passed, which means that CUDA itself is OK!
So the problem must be in something related to Theano, a simple deduction.
It is important to know what exactly the problem is. I went to seach using the keyword “Theano Ubuntu 16.04 failed to compile cuda_ndarray.cu”, and an issue on github hit.
This solution is from an issue on github. The problem occured probably because of the incompatibility of compilers and a bug in the version of libc. Since we have solved the incompatibility of compilers, we may continue with the problem of libc. By adding an extra theano flag “nvcc.flags=-D_FORCE_INLINES” on first time compiling will solve the problem.
# extra flag when first time compiling ~$ THEANO_FLAGS=device=gpu0,nvcc.flags=-D_FORCE_INLINES python ./mnist.py
And you just need to specify the extra flag once, because Theano precompiles once after the cache is empty, which means next time you can just run the program with what you used to run with.
Still Not Working?
Well it may still be the problem of incompatibility of compilers. The issue on github also provides how to solve this problem and maybe that would help.