I've got a pile of json lines files that I'm trying to process into a set of Sql Server database tables. Can't use OPENJSON() because jsonlines is not valid in total, only line by line. Plus the files are mammoth so I was trying to work up a way to process it line by line without reading the whole file in as a CLOB and then parsing it.
My first thought was to use OPENROWSET with a format file that declared only one column - one giant string. Then I was going to put a cursor on that and loop through.
The individual bits are working great, but when I put a cursor on an OPENROWSET like this, the cursor fetches one line and stops (breaks out of the WHILE loop).
I googled for a while but couldn't find anything precisely like this.
Is this an inherent limitation of trying to use OPENROWSET and a cursor together? Or are there some different cursor options I could use that might handle it better?
I was hoping for an all-in-sql approach, but I suppose I could knock together some script that will read the lines and feed them in a query individually.
Thanks
DECLARE @line nvarchar(max)
DECLARE JsonLCursor CURSOR LOCAL FAST_FORWARD for
SELECT [Object] FROM OPENROWSET (BULK 'G:\input\944.jsonl', FORMATFILE='G:\jsonl.fmt') as j
OPEN JsonLCursor
FETCH NEXT FROM JsonLCursor INTO @Line
WHILE @@FETCH_STATUS = 0
BEGIN
EXEC [dbo].[JsonLProcess] @line = @line
FETCH NEXT FROM JsonLCursor INTO @Line
END
CLOSE JsonLCursor
DEALLOCATE JSonLCursor
JsonL.fmt
14.0
1
1 SQLCHAR 0 1000000 "\r\n" 1 Object Latin1_General_100_CI_AS_SC_UTF8
[]
beginning and end, and perhaps a comma at the end of each line? Either way, you should be able to useOPENJSON
or whatever code you have in that stored procedure, directly on theOPENROWSET
result, which would be far easier and perform better. Please add the code for the procedure in the question