首页 1 2 3 4 5 6 7

NVIDIA DIGITS 5.1-dev学习笔记之安装过程记录：Windows10 x64位系统、 MicroSoft Caffe Master、CUDA 8.0 、Python 2.7

今天成功在windows下配置成功了英伟达的DIGITS，记录一下问题解决过程。
环境简介： Windows10_x64 CUDA 8.0 / CUDA 7.5 Python2.7 Microsoft-Caffe-master
Github DIGITS: https://github.com/NVIDIA/DIGITS/blob/master/docs/BuildDigitsWindows.md 点击打开链接
完全按照此步骤操作不会出现问题，尤其是关于Python package的版本问题，详见DIGITS-master目录下的requirements.txt：

Pillow>=2.3.0,<=3.1.2
numpy>=1.8.1,<=1.11.0
scipy>=0.13.3,<=0.17.0
protobuf>=2.5.0,<=2.6.1
six>=1.5.2,<=1.10.0
requests>=2.2.1,<=2.9.1
gevent>=1.0,<=1.1.0
gevent-websocket==0.9.3
Flask==0.10.1
Flask-WTF>=0.11,<=0.12
wtforms>=2.0,<=2.1
Flask-SocketIO==2.6
setuptools>=3.3,<=20.7.0
lmdb==0.87
h5py>=2.2.1,<=2.6.0
pydot>=1.0.28,<=1.0.29
psutil>=1.2.1,<=3.4.2
matplotlib>=1.3.1,<=1.5.1
scikit-fmm>=0.0.9

版本号一定确保和上述一致。

BUG 1 : 与google.protobuf 有关

**
我记得是在加载from google.protobuf import _symbol 指令时（大概就是这个指令吧），问题是由于我原来安装的时ptotobuf 2.5.0，版本有点低，然后我更新到2.6.1后，就没问题了。ptotobuf在windows下的 python支持编译方法请自行google。

BUG 2 : 当我运行时

I:\DIGITS-master>python -m digits
  ___ ___ ___ ___ _____ ___
 |   \_ _/ __|_ _|_   _/ __|
 | |) | | (_ || |  | | \__ \
 |___/___\___|___| |_| |___/ -dev

A valid Caffe installation was not found on your system.
Use the envvar CAFFE_ROOT to indicate a valid installation.
Traceback (most recent call last):
  File "I:\Python27\lib\runpy.py", line , in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "I:\Python27\lib\runpy.py", line , in _run_code
    exec code in run_globals
  File "I:\DIGITS-master\digits\__main__.py", line , in <module>
    main()
  File "I:\DIGITS-master\digits\__main__.py", line , in main
    import digits.config
  File "digits\config\__init__.py", line , in <module>
    from . import (  # noqa
  File "digits\config\caffe.py", line , in <module>
    executable, version, flavor = load_from_path()
  File "digits\config\caffe.py", line , in load_from_path
    version, flavor = get_version_and_flavor(executable)
  File "digits\config\caffe.py", line , in get_version_and_flavor
    version = parse_version(version_string)
  File "digits\utils\__init__.py", line , in parse_version
    return pkg_resources.SetuptoolsVersion(v)
  File "I:\Python27\lib\site-packages\pkg_resources\_vendor\packaging\version.py", line , in __init__
    raise InvalidVersion("Invalid version: '{0}'".format(version))
pkg_resources._vendor.packaging.version.InvalidVersion: Invalid version: 'CAFFE_VERSION'

然后，我看了下caffe.py这个文件。
caffe.py：
关键信息我都红色标记了。

from __future__ import absolute_import

import imp
import os
import platform
import re
import subprocess
import sys

from . import option_list
from digits import device_query
from digits.utils import parse_version


def load_from_envvar(envvar):
    """
    Load information from an installation indicated by an environment variable
    """
    value = os.environ[envvar].strip().strip("\"' ")
#此处需要修改路径，于CAFFE_HOME对应
    if platform.system() == 'Windows':
        #executable_dir = os.path.join(value, 'install', 'bin')
        executable_dir = os.path.join(value)
        #python_dir = os.path.join(value, 'install', 'python')
        python_dir = os.path.join(value, 'pycaffe')
    else:
        executable_dir = os.path.join(value, 'build', 'tools')
        python_dir = os.path.join(value, 'python')

    try:
        executable = find_executable_in_dir(executable_dir)
        if executable is None:
            raise ValueError('Caffe executable not found at "%s"'
                             % executable_dir)
        if not is_pycaffe_in_dir(python_dir):
            raise ValueError('Pycaffe not found in "%s"'
                             % python_dir)
        import_pycaffe(python_dir)
        version, flavor = get_version_and_flavor(executable)
    except:
        print ('"%s" from %s does not point to a valid installation of Caffe.'
               % (value, envvar))
        print 'Use the envvar CAFFE_ROOT to indicate a valid installation.'
        raise
    return executable, version, flavor


def load_from_path():
    """
    Load information from an installation on standard paths (PATH and PYTHONPATH)
    """
    try:
        executable = find_executable_in_dir()
        if executable is None:
            raise ValueError('Caffe executable not found in PATH')
        if not is_pycaffe_in_dir():
            raise ValueError('Pycaffe not found in PYTHONPATH')
        import_pycaffe()
        version, flavor = get_version_and_flavor(executable)
    except:
        print 'A valid Caffe installation was not found on your system.'
        print 'Use the envvar CAFFE_ROOT to indicate a valid installation.'
        raise
    return executable, version, flavor


def find_executable_in_dir(dirname=None):
    """
    Returns the path to the caffe executable at dirname
    If dirname is None, search all directories in sys.path
    Returns None if not found
    """
    if platform.system() == 'Windows':
        exe_name = 'caffe.exe'
    else:
        exe_name = 'caffe'

    if dirname is None:
        dirnames = [path.strip("\"' ") for path in os.environ['PATH'].split(os.pathsep)]
    else:
        dirnames = [dirname]

    for dirname in dirnames:
        path = os.path.join(dirname, exe_name)
        if os.path.isfile(path) and os.access(path, os.X_OK):
            return path
    return None


def is_pycaffe_in_dir(dirname=None):
    """
    Returns True if you can "import caffe" from dirname
    If dirname is None, search all directories in sys.path
    """
    old_path = sys.path
    if dirname is not None:
        sys.path = [dirname]  # temporarily replace sys.path
    try:
        imp.find_module('caffe')
    except ImportError:
        return False
    finally:
        sys.path = old_path
    return True


def import_pycaffe(dirname=None):
    """
    Imports caffe
    If dirname is not None, prepend it to sys.path first
    """
    if dirname is not None:
        sys.path.insert(, dirname)
        # Add to PYTHONPATH so that build/tools/caffe is aware of python layers there
        os.environ['PYTHONPATH'] = '%s%s%s' % (
            dirname, os.pathsep, os.environ.get('PYTHONPATH'))

    # Suppress GLOG output for python bindings
    GLOG_minloglevel = os.environ.pop('GLOG_minloglevel', None)
    # Show only "ERROR" and "FATAL"
    os.environ['GLOG_minloglevel'] = '2'

    # for Windows environment, loading h5py before caffe solves the issue mentioned in
    # https://github.com/NVIDIA/DIGITS/issues/47#issuecomment-206292824
    import h5py  # noqa
    try:
        import caffe
    except ImportError:
        print 'Did you forget to "make pycaffe"?'
        raise

    # Strange issue with protocol buffers and pickle - see issue #32
    sys.path.insert(, os.path.join(
        os.path.dirname(caffe.__file__), 'proto'))

    # Turn GLOG output back on for subprocess calls
    if GLOG_minloglevel is None:
        del os.environ['GLOG_minloglevel']
    else:
        os.environ['GLOG_minloglevel'] = GLOG_minloglevel


def get_version_and_flavor(executable):
    """
    Returns (version, flavor)
    Should be called after import_pycaffe()
    """
    version_string = get_version_from_pycaffe()
    if version_string is None:
        version_string = get_version_from_cmdline(executable)
    if version_string is None:
        version_string = get_version_from_soname(executable)

    if version_string is None:
        raise ValueError('Could not find version information for Caffe build ' +
                         'at "%s". Upgrade your installation' % executable)
    #这部分代码没用，但是会出现bug，我就注释了
    #version = parse_version(version_string)

    #if parse_version(0, 99, 0) > version > parse_version(0, 9, 0):
    #    flavor = 'NVIDIA'
    #    minimum_version = '0.11.0'
    #    if version < parse_version(minimum_version):
    #        raise ValueError(
    #            'Required version "%s" is greater than "%s". Upgrade your installation.'
    #            % (minimum_version, version_string))
    #else:
    #    flavor = 'BVLC'
    flavor = 'BVLC'
    return version_string, flavor


def get_version_from_pycaffe():
    try:
        from caffe import __version__ as version
        return version
    except ImportError:
        return None


def get_version_from_cmdline(executable):
    command = [executable, '-version']
    p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    if p.wait():
        print p.stderr.read().strip()
        raise RuntimeError('"%s" returned error code %s' % (command, p.returncode))

    pattern = 'version'
    for line in p.stdout:
        if pattern in line:
            return line[line.find(pattern) + len(pattern) + :].strip()
    return None


def get_version_from_soname(executable):
    command = ['ldd', executable]
    p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    if p.wait():
        print p.stderr.read().strip()
        raise RuntimeError('"%s" returned error code %s' % (command, p.returncode))

    # Search output for caffe library
    libname = 'libcaffe'
    caffe_line = None
    for line in p.stdout:
        if libname in line:
            caffe_line = line
            break

    if caffe_line is None:
        raise ValueError('libcaffe not found in linked libraries for "%s"'
                         % executable)

    # Read the symlink for libcaffe from ldd output
    symlink = caffe_line.split()[]
    filename = os.path.basename(os.path.realpath(symlink))

    # parse the version string
    match = re.match(r'%s(.*)\.so\.(\S+)$' % (libname), filename)
    if match:
        return match.group()
    else:
        return None
#看这里，看这里，一个路径问题
#我们需要在环境变量里声明一下，CAFFE_ROOT 或者 CAFFE_HOME都可以，指向caffe编译后的 ./Build/x64/Release
if 'CAFFE_ROOT' in os.environ:
    executable, version, flavor = load_from_envvar('CAFFE_ROOT')
elif 'CAFFE_HOME' in os.environ:
    executable, version, flavor = load_from_envvar('CAFFE_HOME')
else:
    executable, version, flavor = load_from_path()

option_list['caffe'] = {
    'executable': executable,
    'version': version,
    'flavor': flavor,
    'multi_gpu': (flavor == 'BVLC' or parse_version(version) >= parse_version(, )),
    'cuda_enabled': (len(device_query.get_devices()) > ),
}

然后就跑起来吧！！！

贴图看效果：
1. 训练ing

这里写图片描述
2. 测试，可以看特征图，参数啊啊啊（Nvidia DIGITS大法好）

这里写图片描述

下面是安装步骤！！！

Build DIGITS on Windows

Limitation

DIGITS for Windows depends on Windows branch of BVLC Caffe. The following layers, required for DetectNet feature, are not implemented in that branch.

detectnet_transform_layer
l1_loss_layer
As a result, DIGITS for Windows does not support DetectNet. To run DIGITS with DetectNet, please use NV-Caffe 0.15 or above on Ubuntu.

Prerequisites

Python2
CUDA 7.5
CuDNN 5.1
Caffe
Graphviz

Installing prerequisites

Python2

Download and install Python 2.7.11 64bit from Python’s official site (https://www.python.org/ftp/python/2.7.11/python-2.7.11.amd64.msi). Please select Add Python Path during installation.

Download numpy, scipy, matplotlib, scikit-image, h5py from Unofficial Windows Binaries for Python Extension Packages webpage at (http://www.lfd.uci.edu/~gohlke/pythonlibs/). Remember to download correct version (2.7) and architecture (64-bit).

Additionally, download gevent v1.0.2 at the same site. Run command prompt (cmd.exe) as administrator, and issue the following commands.

python -m pip install cython
python -m pip install numpy-1.11.0+mkl-cp27-cp27m-win_amd64.whl
python -m pip install scipy-0.17.0-cp27-none-win_amd64.whl
python -m pip install matplotlib-1.5.1-cp27-none-win_amd64.whl
python -m pip install scikit_image-0.12.3-cp27-cp27m-win_amd64.whl
python -m pip install h5py-2.6.0-cp27-cp27m-win_amd64.whl

If the installation process complains compiler not found, you need to install Microsoft Visual C++ Compiler for Python 2.7, downloaded at (https://www.microsoft.com/en-us/download/details.aspx?id=44266). We recommend installing it by

msiexec /i VCForPython27.msi ALLUSERS=1
After that compiler is installed, finish the above python -m pip install commands.

At this moment, do not install gevent yet. We need to install it after installing DIGITS.

CUDA 7.5

CUDA 7.5 can be obtained at NVIDIA CUDA (https://developer.nvidia.com/cuda-downloads). Please select Windows 7 to download.

CuDNN 5.1

Download CuDNN 5.1 at NVIDIA website (https://developer.nvidia.com/cudnn). Please select CuDNN 5.1 for CUDA 7.5.

Caffe

Caffe can be obtained at (https://github.com/bvlc/caffe/tree/windows). Note you need to install Visual Studio 2013 to build Caffe. Before building it, enable Python support, CUDA and CuDNN by following instructions on the same page. Because we are using Official CPython, please change the value of PythonDir tag from C:\Miniconda2\ to C:\PYTHON27\ (assume your CPython installation is the default C:\PYTHON27). After building it, configure your Python environment to include pycaffe, which is described at (https://github.com/bvlc/caffe/tree/windows#remark). Your caffe.exe will be inside Build\x64\Release directory (if you made release build).

Graphviz

Graphviz is available at (www.graphviz.org/Download.PHP). Please note this site is not always available online. The installation directory can not contain space, so don’t install it under the regular ‘c:\Program Files (x86)’ directory. Try something like ‘c:\graphviz’ instead. When the installation directory contains space, pydot could not launch the dot.exe file, even it has no problem finding it. Add the c:\graphviz\bin directory to your PATH.

Installing DIGITS

Clone DIGITS from github.com (https://github.com/nvidia/digits). From the command prompt (run as administrator) and cd to DIGITS directory. Then type

python -m pip install -r requirements.txt

You may see error about Pillow, like ValueError: jpeg is required unless explicitly disabled using –disable-jpeg, aborting If this happens, download Pillow Windows Installer (Pillow-3.1.1.win-amd64-py2.7.exe) at https://pypi.python.org/pypi/Pillow/3.1.1 and run the exectuables. After installing Pillow in the above way, run

python -m pip install -r requirements.txt

again.

After the above command, check if all required Python dependencies are met by comparing requirements.txt and output of the following command.

python -m pip list

If gevent is not v1.0.2, install it from the whl file, downloaded previously from (http://www.lfd.uci.edu/~gohlke/pythonlibs/).

python -m pip install gevent-1.0.2-cp27-none-win_amd64.whl

It should uninstall the gevent you had, and install gevent 1.0.2.

Because readline is not available in Windows, you need to install one additional Python package.

python -m pip install pyreadline

Running DIGITS

First, check if caffe executable is included in your PATH environment variable. If not, add it.

set PATH=%PATH%;MY_CAFFE_ROOT\Build\x64\Release

Replace MY_CAFFE_ROOT with your local caffe directory.

Launch DIGITS devserver with the following command:

python digits-devserver

Point your browser to localhost:5000. You should be able to see DIGITS.

Troubleshooting

DIGITS crashes when trying to classify images with * Show visualizations and statistics *

This issue should have been resolved. However, if you still encounter this issue, this seems related to different hdf5 DLL binding between pycaffe and h5py. The DLL used by pycaffe was pulled from nuget, and its version is 1.8.15.2. Slightly older than the DLL in h5py. A temporary solution is to load h5py before pycaffe. To force loading h5py before pycaffe, you can either add one line at the beginning of digits-devserver file, or import h5py just before import caffe in digits/config/caffe_option.py.

import readline causes ImportError

Change import readline in digits\config\prompt.py to

try:
    import readline
except ImportError:
    import pyreadline as readline

DIGITS complains Torch binary not found in PATH

Currently, DIGITS does not support Torch on Windows platform.

NVIDIA DIGITS 5.1-dev学习笔记之安装过程记录：Windows10 x64位系统 、 MicroSoft Caffe Master、CUDA 8.0 、Python 2.7

BUG 1 : 与google.protobuf 有关

BUG 2 : 当我运行时

NVIDIA DIGITS 5.1-dev学习笔记之安装过程记录：Windows10 x64位系统、 MicroSoft Caffe Master、CUDA 8.0 、Python 2.7