Read before start of stream error upon Recover #766

shiming-tp · 2022-11-15T09:24:04Z

shiming-tp
Nov 15, 2022

I'm storing the logs and checkpoints on Azure BLOB storage. In the initial run, everything works fine, but once it has run for about 24 hours non-stop with constant upserts, it'll crash with "Attempt to read before start of stream" error when _h.Recover() is executed.

The app runs on a web app with scaling in place and all the instances call will load the same BLOB contanier. I'm not sure what is the cause of the error, so any guidance will be useful.

                            var log = new AzureStorageDevice(STORAGE_STRING, BASE_CONTAINER, "", "hlog.log");
                            var objlog = new AzureStorageDevice(STORAGE_STRING, BASE_CONTAINER, "", "hlog.obj.log");
                            var checkpointManager = new DeviceLogCommitCheckpointManager(
                                new AzureStorageNamedDeviceFactory(STORAGE_STRING),
                                new DefaultCheckpointNamingScheme($"{BASE_CONTAINER}/checkpoints/"));

                            var logSettings = new LogSettings
                            {
                                LogDevice = log,
                                ObjectLogDevice = objlog,
                                ReadCacheSettings = true ? new ReadCacheSettings() : null,
                                // Uncomment below for low memory footprint demo
                                // PageSizeBits = 12, // (4K pages)
                                // MemorySizeBits = 20 // (1M memory for main log)
                            };
                            _h = new FasterKV<string, string>(
                                size: 1L << 20,
                                logSettings: logSettings,
                                checkpointSettings: new CheckpointSettings { CheckpointManager = checkpointManager, RemoveOutdated = true }
                            );
                            
                            bool recovered = true;

                            try
                            {
                                _h.Recover();
                            }
                            catch
                            {
                                recovered = false;
                            }

badrishc · 2022-11-15T12:18:14Z

badrishc
Nov 15, 2022
Maintainer

Can you run in debug mode for recovery, ideally with debug bits of FASTER, and post the call stack at the time of the recover exception?

0 replies

badrishc · 2022-11-15T12:19:43Z

badrishc
Nov 15, 2022
Maintainer

If this doesn't help, if you could share the checkpoint and log files along with a sample program that reproduces the error when recovering from those files, we can take a look.

0 replies

shiming-tp · 2022-11-17T11:24:55Z

shiming-tp
Nov 17, 2022
Author

Thanks @badrishc . This is the trace log

System.IO.IOException
  HResult=0x80131620
  Message=An attempt was made to move the position before the beginning of the stream.
  Source=System.Private.CoreLib
  StackTrace:
   at System.IO.MemoryStream.Seek(Int64 offset, SeekOrigin loc)
   at FASTER.core.GenericAllocator`2.Deserialize(Byte* raw, Int64 ptr, Int64 untilptr, Record`2[] src, Stream stream)
   at FASTER.core.GenericAllocator`2.AsyncReadPageWithObjectsCallback[TContext](UInt32 errorCode, UInt32 numBytes, Object context)
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
   at System.Threading.Tasks.AwaitTaskContinuation.<>c.<.cctor>b__17_0(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask)
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

  This exception was originally thrown at this call stack:
    [External Code]

Let me come back to you on the sample code as I made some changes to attempt to see if it fixes the problem, and the issue just resurfaced. Will get back next week.

0 replies

shiming-tp · 2022-11-18T01:46:43Z

shiming-tp
Nov 18, 2022
Author

Hi @badrishc , sample program here.

https://tripanzee-my.sharepoint.com/:u:/g/personal/shiming_travelprologue_com/EV-pdzYjKzNAiQeMij3si5QBIBHAqyGCc9Uv_Ij8dXGlNw?e=GXRnQz

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Read before start of stream error upon Recover #766

{{title}}

Replies: 4 comments

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Read before start of stream error upon Recover #766

shiming-tp Nov 15, 2022

Replies: 4 comments

badrishc Nov 15, 2022 Maintainer

badrishc Nov 15, 2022 Maintainer

shiming-tp Nov 17, 2022 Author

shiming-tp Nov 18, 2022 Author

shiming-tp
Nov 15, 2022

badrishc
Nov 15, 2022
Maintainer

badrishc
Nov 15, 2022
Maintainer

shiming-tp
Nov 17, 2022
Author

shiming-tp
Nov 18, 2022
Author