delphi - TSQLQuery only streams first 1MB of data correctly for large strings

Question

(see Edit #1 with stack trace and Edit #2 with workaround at end of post)

While troubleshooting TSQLQuery.FieldByName().AsString -> TStringStream Corrupts Data, I found that a TSQLQuery.FieldByName().AsBytes will only stream exactly 1MB of varchar(max) data correctly.

• Using WireShark, I verified that the data is all being handed to the Delphi app correctly.
• I verified that it always writes out the correct number of bytes to the output file, but any bytes that exceed exactly 1MB are null bytes.
• Additionally, TSQLQuery.FieldByName().AsString and .AsWideString also exhibit the same behavior.

What would cause .AsBytes to supply the correct number of bytes to the TFileStream, but null all bytes that exceed 1MB?

Test Case

This test case creates two output files. Plus14.txt is 1MB + 14 bytes. Plus36.txt is 1MB + 36 bytes. In both cases, the bytes more than 1MB are null byte values. I even tried a 16MB string. The first 1MB of the output file was correct; the next 15MB were all null bytes.

SQL Server

use tempdb
go
create procedure RunMe
as
  declare @s1 varchar(max), @s2 varchar(max)

  set @s1 = '0123456789ABCDEF'
  set @s2 = @s1 + @s1 + @s1 + @s1 + @s1 + @s1 + @s1 + @s1 -- 128 bytes
  set @s1 = @s2 + @s2 + @s2 + @s2 + @s2 + @s2 + @s2 + @s2 -- 1,024 bytes
  set @s2 = @s1 + @s1 + @s1 + @s1 + @s1 + @s1 + @s1 + @s1 -- 8,192 bytes
  set @s1 = @s2 + @s2 + @s2 + @s2 + @s2 + @s2 + @s2 + @s2 -- 65,536 bytes
  set @s2 = @s1 + @s1 + @s1 + @s1 + @s1 + @s1 + @s1 + @s1 -- 524,288 bytes
  set @s1 = @s2 + @s2                                     -- 1,048,576 bytes

  set @s2 = @s1 + 'this is a test'                        -- 1MB + 14 bytes
  set @s1 = @s1 + 'of the emergency broadcasting system'  -- 1MB + 36 bytes

  select @s2 as Plus14, @s1 as Plus36
go
grant execute on RunMe to public
go

Delphi DFM

Default form, with this TSQLConnection dropped on it (and one TButton):

object SQLConnection1: TSQLConnection
  DriverName = 'MSSQL'
  GetDriverFunc = 'getSQLDriverMSSQL'
  LibraryName = 'dbxmss.dll'
  LoginPrompt = False
  Params.Strings = (
    'User_Name=user'
    'Password=password'
    'SchemaOverride=%.dbo'
    'DriverUnit=Data.DBXMSSQL'

      'DriverPackageLoader=TDBXDynalinkDriverLoader,DBXCommonDriver160.' +
      'bpl'

      'DriverAssemblyLoader=Borland.Data.TDBXDynalinkDriverLoader,Borla' +
      'nd.Data.DbxCommonDriver,Version=16.0.0.0,Culture=neutral,PublicK' +
      'eyToken=91d62ebb5b0d1b1b'

      'MetaDataPackageLoader=TDBXMsSqlMetaDataCommandFactory,DbxMSSQLDr' +
      'iver160.bpl'

      'MetaDataAssemblyLoader=Borland.Data.TDBXMsSqlMetaDataCommandFact' +
      'ory,Borland.Data.DbxMSSQLDriver,Version=16.0.0.0,Culture=neutral' +
      ',PublicKeyToken=91d62ebb5b0d1b1b'
    'GetDriverFunc=getSQLDriverMSSQL'
    'LibraryName=dbxmss.dll'
    'VendorLib=sqlncli10.dll'
    'VendorLibWin64=sqlncli10.dll'
    'HostName=localhost'
    'Database=tempdb'
    'MaxBlobSize=-1'
    'LocaleCode=0000'
    'IsolationLevel=ReadCommitted'
    'OSAuthentication=False'
    'PrepareSQL=True'
    'BlobSize=-1'
    'ErrorResourceFile='
    'OS Authentication=True'
    'Prepare SQL=False')
  VendorLib = 'sqlncli10.dll'
  Left = 8
  Top = 8
end

Delphi PAS

The code for the TButton.OnClick:

procedure TForm1.Button1Click(Sender: TObject);
var qry: TSQLQuery;

  procedure save(str: string);
  var data: TBytes; fs: TFileStream;
  begin
    fs := TFileStream.Create(Format('c:\%s.txt', [str]), fmCreate);
    try
      data := qry.FieldByName(str).AsBytes;
      if data <> nil then
        fs.WriteBuffer(data[0], Length(data));
    finally
      FreeAndNil(fs);
    end;
  end;

begin
  SQLConnection1.Open;
  qry := TSQLQuery.Create(nil);
  try
    qry.MaxBlobSize := -1;
    qry.SQLConnection := SQLConnection1;
    qry.SQL.Text := 'set nocount on; exec RunMe';
    qry.Open;
    save('Plus14');
    save('Plus36');
  finally
    FreeAndNil(qry);
  end;
  SQLConnection1.Close;
end;

<<< Edit #1 - Stack Trace >>>

I traced through Embarcadero's code and found the place where the null bytes first appear.

• FMethodTable.FDBXRow_GetBytes
• Data.DBXDynalink.TDBXDynalinkByteReader.GetBytes(0,0,(...),0,1048590,True)
• Data.SqlExpr.TCustomSQLDataSet.GetFieldData(1,$7EC80018)
• Data.SqlExpr.TCustomSQLDataSet.GetFieldData(???,$7EC80018)
• Data.DB.TDataSet.GetFieldData($66DB18,$7EC80018,True)
• Data.SqlExpr.TSQLBlobStream.ReadBlobData
• Data.SqlExpr.TSQLBlobStream.Read((no value),1048590)
• System.Classes.TStream.ReadBuffer((no value),1048590) 1MB + 14b
• Data.DB.TBlobField.GetAsBytes
• Unit1.save('Plus14')

When FDBXRow_GetBytes returns, Value: TBytes is 1048590 bytes, with null values set for the last 14 bytes.

I'm not sure what to try next. Any help is greatly appreciated.

<<< Edit #2 - Workaround >>>

I set SQLConnection1.MaxBlobSize := 2097152, and now all bytes are stream to the output files correctly. So the problem only seems to occur when .MaxBlobSize = -1.

The urgency to fix the issue is gone now that I found a workaround. However, I would still like to get -1 to work if possible since the values from my database will sometimes exceed 50 megs. So any suggestions or help is still appreciated.

<<< Edit #3 - Bug Report >>>

I filed a bug report with Embarcadero (QC #108475). I will report back once the bug has been acknowledged / fixed.

<<< Edit #4 - Escalated Bug Report >>>

I found today that using this workaround will sometimes causes a TClientDataSet to throw an EOleException with the text 'Catastrophic Failure'. Apparently a TClientDataSet prefers a MaxBlobSize := '-1';. Consequently, I escalated the bug report at Embarcadero. Hopefully they will provide a fix or a better workaround for this soon.

Answer 1

I was able to work around the issue. It may be possible to just set the TSQLConnection properties like this:

sqlcon.Params.Values['MaxBlobSize'] := '250000000'; // 250 megs
sqlcon.Params.Values['BlobSize'] := '-1';

But I set a lot more properties than these by using a DBXRegDB unit to setup all TSQLConnection components. I don't use the IDE's property editor... Some of the settings from the DBXRegDB unit may also be required for the workaround to work (I don't know for sure). I'll include the DBXRegDB unit + instructions how to use it just in case.

• Add the DBXRegDB unit to the .dpr file.
• Add DBXRegDB to the form's uses clause.
• Execute the following code, passing it the TSQLConnection component on your form:

.

procedure SetupMSSqlConnection(const sqlcon: TSQLConnection; const hostname, port, maxcon, dbname, username, password: string);
begin
  sqlcon.Params.Clear;
  sqlcon.DriverName := 'MSSQL_Con';
  sqlcon.VendorLib := sqlcon.Params.Values[TDBXPropertyNames.VendorLib];
  sqlcon.LibraryName := sqlcon.Params.Values[TDBXPropertyNames.LibraryName];
  sqlcon.GetDriverFunc := sqlcon.Params.Values[TDBXPropertyNames.GetDriverFunc];

  sqlcon.Params.Values[TDBXPropertyNames.HostName] := hostname;
  sqlcon.Params.Values[TDBXPropertyNames.Port]     := port;
  sqlcon.Params.Values[TDBXPropertyNames.Database] := dbname;
  sqlcon.Params.Values[TDBXPropertyNames.UserName] := username;
  sqlcon.Params.Values[TDBXPropertyNames.Password] := password;
end;

Lastly, set the TSQLQuery.MaxBlobSize to '0', so that it will automatically copy the value from its TSQLConnection.

Here is the DBXRegDB unit, for those who want to use it. I adapted it from something I found here: DBX without deploying DBXDrivers.ini. Make sure you do not set the BlobSize to 250 megs or you will get out of memory errors.

unit DBXRegDB;

interface

implementation

uses
  DBXCommon, DBXDynalinkNative, DBXMSSQL, Forms, Classes;

type
  TDBXInternalDriver = class(TDBXDynalinkDriverNative)
  public
    constructor Create(DriverDef: TDBXDriverDef); override;
  end;

  TDBXInternalProperties = class(TDBXProperties)
  public
    constructor Create(DBXContext: TDBXContext); override;
  end;

{ TDBXInternalDriver }

constructor TDBXInternalDriver.Create(DriverDef: TDBXDriverDef);
begin
  inherited Create(DriverDef, TDBXDynalinkDriverLoader);
  InitDriverProperties(TDBXInternalProperties.Create(DriverDef.FDBXContext));
end;

{ TDBXInternalProperties }

constructor TDBXInternalProperties.Create(DBXContext: TDBXContext);
begin
  inherited Create(DBXContext);

  Values[TDBXPropertyNames.SchemaOverride]         := '%.dbo';
  Values[TDBXPropertyNames.DriverUnit]             := 'DBXMSSQL';
  Values[TDBXPropertyNames.DriverPackageLoader]    := 'TDBXDynalinkDriverLoader,DBXCommonDriver160.bpl';
  Values[TDBXPropertyNames.DriverAssemblyLoader]   := 'Borland.Data.TDBXDynalinkDriverLoader,Borland.Data.DbxCommonDriver,Version=15.0.0.0,Culture=neutral,PublicKeyToken=91d62ebb5b0d1b1b';
  Values[TDBXPropertyNames.MetaDataPackageLoader]  := 'TDBXMsSqlMetaDataCommandFactory,DbxMSSQLDriver160.bpl';
  Values[TDBXPropertyNames.MetaDataAssemblyLoader] := 'Borland.Data.TDBXMsSqlMetaDataCommandFactory,Borland.Data.DbxMSSQLDriver,Version=15.0.0.0,Culture=neutral,PublicKeyToken=91d62ebb5b0d1b1b';
  Values[TDBXPropertyNames.GetDriverFunc]          := 'getSQLDriverMSSQL';
  Values[TDBXPropertyNames.LibraryName]            := 'dbxmss.dll';
  Values[TDBXPropertyNames.VendorLib]              := 'sqlncli10.dll';
  Values[TDBXPropertyNames.HostName]               := 'ServerName';
  Values[TDBXPropertyNames.Database]               := 'Database Name';
  Values[TDBXPropertyNames.MaxBlobSize]            := '250000000';
  Values['LocaleCode']                             := '0000';
  Values[TDBXPropertyNames.IsolationLevel]         := 'ReadCommitted';
  Values['OSAuthentication']                       := 'False';
  Values['PrepareSQL']                             := 'True';
  Values[TDBXPropertyNames.UserName]               := 'user';
  Values[TDBXPropertyNames.Password]               := 'password';
  Values['BlobSize']                               := '-1';
  Values[TDBXPropertyNames.ErrorResourceFile]      := '';
  Values['OS Authentication']                      := 'False';
  Values['Prepare SQL']                            := 'True';
  Values[TDBXPropertyNames.ConnectTimeout]         := '30';
end;

var
  InternalConnectionFactory: TDBXMemoryConnectionFactory;

initialization

  TDBXDriverRegistry.RegisterDriverClass('MSSQL_Con', TDBXInternalDriver);
  InternalConnectionFactory := TDBXMemoryConnectionFactory.Create;
  InternalConnectionFactory.Open;
  TDBXConnectionFactory.SetConnectionFactory(InternalConnectionFactory);

end.