使用 Python Api 读取大型 DBFS 装入的文件Reading large DBFS-mounted files using Python APIs

本文介绍如何解决使用本地 Python Api 读取大型 DBFS 装入的文件时发生的错误。This article explains how to resolve an error that occurs when you read large DBFS-mounted files using local Python APIs.

问题Problem

如果将文件夹装载到 dbfs:// PYTHON API (如pandas)中的文件,并读取大于2gb 的文件,则会看到以下错误:If you mount a folder onto dbfs:// and read a file larger than 2GB in a Python API like pandas, you will see following error:

/databricks/python/local/lib/python2.7/site-packages/pandas/parser.so in pandas.parser.TextReader.__cinit__ (pandas/parser.c:3427)()
/databricks/python/local/lib/python2.7/site-packages/pandas/parser.so in pandas.parser.TextReader._setup_parser_source (pandas/parser.c:6883)()
IOError: Initializing from file failed

原因Cause

之所以发生此错误,是因为 Python 方法中一个用于读取文件的参数是一个有符号的 int,文件的长度是一个 int,如果该对象是大于2GB 的文件,则长度可以大于最大的带符号整数。The error occurs because one argument in the Python method to read a file is a signed int, the length of the file is an int, and if the object is a file larger than 2GB, the length can be larger than maximum signed int.

解决方案Solution

将文件从移 dbfs:// 到本地文件系统 (file://)Move the file from dbfs:// to local file system (file://). 然后阅读使用 Python API。Then read using the Python API. 例如:For example:

  1. 将文件从复制 dbfs://file://Copy the file from dbfs:// to file://:

    %fs cp dbfs:/mnt/large_file.csv file:/tmp/large_file.csv
    
  2. 读取pandas API 中的文件:Read the file in the pandas API:

    import pandas as pd
    pd.read_csv('file:/tmp/large_file.csv',).head()