sql server - Hadopp Polybase: the query processor encountered an unexpected error during the processing of a remote query phase

Question

I am new to Polybase, but I have been able to create an external data source and external table from SQL Server polybase to Hadoop HortonWorks 3.1 with Kerberos and HA enabled on.

When I try to have a select query from my external table the below error is thrown:

Internal Query Processor Error: The query processor encountered an unexpected error during the processing of a remote query phase.

I have already read the documents and possible reasons which states that I need to restart my Polybase services after installation but I have already restarted all SQL server related services plus polybase ones. Also, my polybase Data Movement service sometimes stops by itself; however while it is running, I face the same error.

Here is my external data source

CREATE EXTERNAL DATA SOURCE [Hive_test3] 
WITH 
(TYPE = HADOOP,
LOCATION = N'hdfs://xxxxxxxxxx:8020', 
RESOURCE_MANAGER_LOCATION = N'xxxxxxxxx:8032',
CREDENTIAL = [HadoopUser2])

and my external table as:

CREATE EXTERNAL TABLE [dbo].[municipal_test]
(
    [id] [int] NULL,
    [city_id] [int] NULL,
    [name] [varchar](250) NULL
)
WITH (DATA_SOURCE = [Hive_test3],
LOCATION = N'/user/xxxx/xxxx.csv',
FILE_FORMAT = [csvFileFormat],
REJECT_TYPE = VALUE,REJECT_VALUE = 500)

and even the following query fails:

SELECT 1
FROM 
[dbo].[municipal_test]

Can you please kindly help me out?

question from:https://stackoverflow.com/questions/65917121/hadopp-polybase-the-query-processor-encountered-an-unexpected-error-during-the

Answer 1

Please check you've done the following:

1. Configured 'polybase enabled'
2. Configured 'hadoop connectivity'
3. Modified 'yarn-site.xml'
4. Created the master key encryption
5. In Hadoop you've created the user at the OS level and granted folder permissions to it

Also, please provide the code for your external file format (it was not included when I wrote this answer). Here is additional information regarding Kerberos: https://docs.microsoft.com/en-us/sql/relational-databases/polybase/polybase-configuration, and here is additional information for troubleshooting: https://docs.microsoft.com/en-us/sql/relational-databases/polybase/polybase-troubleshoot-connectivity.