Repository navigation
Crashes in zend_accel_inheritance_cache_find since upgrading to 8.1.3 due to corrupt on-disk file cache #8143
Description
Description
I dont't have a minimal reproducer for this but an application which is based on sabre/dav running on Windows Server 2016 using IIS
Since upgrading from 8.1.2 to 8.1.3 I get frequent 500 errors.
In the event log I find
Log Name: Application
Source: Application Error
Date: 23.02.2022 11:26:34
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: PSDEVMB.test.example.com
Description:
Faulting application name: php-cgi.exe, version: 8.1.3.0, time stamp: 0x62160b66
Faulting module name: php_opcache.dll, version: 8.1.3.0, time stamp: 0x62160b62
Exception code: 0xc0000005
Fault offset: 0x00000000000043a3
Faulting process id: 0xfd0
Faulting application start time: 0x01d8289fd314a2a7
Faulting application path: C:\Program Files\PHP\v8.1\php-cgi.exe
Faulting module path: C:\Program Files\PHP\v8.1\ext\php_opcache.dll
Report Id: f5dc9377-9cfc-46be-ad05-5ad40b0fafe7
Faulting package full name:
Faulting package-relative application ID:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2022-02-23T10:26:34.226734200Z" />
<EventRecordID>44517</EventRecordID>
<Channel>Application</Channel>
<Computer>PSDEVMB.test.example.com</Computer>
<Security />
</System>
<EventData>
<Data>php-cgi.exe</Data>
<Data>8.1.3.0</Data>
<Data>62160b66</Data>
<Data>php_opcache.dll</Data>
<Data>8.1.3.0</Data>
<Data>62160b62</Data>
<Data>c0000005</Data>
<Data>00000000000043a3</Data>
<Data>fd0</Data>
<Data>01d8289fd314a2a7</Data>
<Data>C:\Program Files\PHP\v8.1\php-cgi.exe</Data>
<Data>C:\Program Files\PHP\v8.1\ext\php_opcache.dll</Data>
<Data>f5dc9377-9cfc-46be-ad05-5ad40b0fafe7</Data>
<Data>
</Data>
<Data>
</Data>
</EventData>
</Event>
The relevant opcache configuration from php.ini:
[opcache]
zend_extension=php_opcache.dll
opcache.enable_cli=1
opcache.enable=1
opcache.enable_file_override=1
opcache.file_cache="C:\Windows\temp\"
opcache.file_cache_fallback=1
I ran a git bisect since 8.1.2 was working fine.
The first faulty commit is 78fd573
Attaching a debugger to a running php-cgi.exe process, I see
Exception thrown at 0x00007FF992294F38 in php-cgi.exe: Microsoft C++ exception: std::out_of_range at memory location 0x00000013415FB250.
Exception thrown at 0x00007FF992294F38 in php-cgi.exe: Microsoft C++ exception: std::out_of_range at memory location 0x00000013415FB250.
Exception thrown at 0x00007FF992294F38 in php-cgi.exe: Microsoft C++ exception: std::out_of_range at memory location 0x00000013415FB250.
Exception thrown at 0x00007FF992294F38 in php-cgi.exe: Microsoft C++ exception: std::out_of_range at memory location 0x00000013415FB250.
Exception thrown at 0x00007FF956D743A3 (php_opcache.dll) in php-cgi.exe: 0xC0000005: Access violation reading location 0x0000015600000012.
PHP Version
PHP 8.1.3
Operating System
Windows Server 2016 / IIS
Activity
mbiebl commentedon Feb 23, 2022
Disabling the PHP opcache file_cache by commenting out the line
seems to avoid the issue.
cmb69 commentedon Feb 23, 2022
That's somewhat unlikely, since that commit only changed some version numbers (it can be tricky to use
git bisect, if the file cache is enabled). Anyhow, please provide a stack backtrace.mbiebl commentedon Feb 23, 2022
Correct, but it's reproducible. I'm also puzzled by that. I was wondering if the build system uses different flags when built with a -dev version?
cmb69 commentedon Feb 23, 2022
Ah, a -dev version produces a different zend_system_id, so that uses different OPcache SHM and file cache.
Anyhow, please provide a stack backtrace. :)
mbiebl commentedon Feb 23, 2022
16 remaining items
cristicotet commentedon Mar 2, 2022
This also occurs on CentOS Linux release 7.9.2009 (Core)
Linux vm 3.10.0-1160.59.1.el7.x86_64 # 1 SMP Wed Feb 23 16:47:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
php81-php-fpm-8.1.3-1.el7.remi.x86_64
cmb69 commentedon Mar 2, 2022
@cristicotet, please try without Xdebug.
cristicotet commentedon Mar 2, 2022
It works fine after disabling xdebug.
This only happened on the staging server (where xdebug is enabled) with php 8.1, it's not the first time.
Last time it was resolved by upgrading from a previous minor version 8.1.x to 8.1.3. Probably this also triggered clearing of cached files?!?
Disabling xdebug maybe also triggered clearing of cached files and may occur again? If it does I will come back with a new backtrace.
cmb69 commentedon Mar 2, 2022
Running with and without Xdebug is supposed to use two distinct file caches.
[-]Crashes in php_opcache.dll since upgrading to 8.1.3[/-][+]Crashes in php_opcache.dll since upgrading to 8.1.3 due to corrupt on-disk file cache[/+]cristicotet commentedon Mar 10, 2022
It happened again with xdebug disabled:
strace
[-]Crashes in php_opcache.dll since upgrading to 8.1.3 due to corrupt on-disk file cache[/-][+]Crashes in zend_accel_inheritance_cache_find since upgrading to 8.1.3 due to corrupt on-disk file cache[/+]mbiebl commentedon Apr 1, 2022
One more finding how I managed to corrupt the cache:
I was using fiddler https://www.telerik.com/fiddler to debug the php application.
Once I stopped using fiddler, the segfaults started to happen.
So I assume by using fiddler I managed to corrupt / break the opcode cache.
cristicotet commentedon Jun 7, 2022
For version 8.1.6 it happens a lot more often and it doesn't go away if I clear opcache
turchanov commentedon Nov 30, 2022
We also experience crashes due to segmentation fault in
zend_accel_inheritance_cache_findin php 8.1.12 (also in 8.1.10, 8.1.8, 8.1.6).Previous attempts to debug this issue showed that
entryparamenter tozend_accel_inheritance_cache_findis valid upon the entry to the function, but during iteration over chain links, a corrupted entry encountered at some point.Execution environment
We use php-fpm with opcache and opcache filecache. This bug happens erratically (and thus is very difficult to reproduce) when we deploy a new code release using the following scheme: we pre-warm opcache by referencing files in a new code release (which creates opcache filecache entries), and then we switch current revision symlink to a new release and do php-fpm reload (that is opcache is populated from filecache).
This bug is not specific to a single host, but to a lot of machines (20+ hosts)
cristicotet commentedon Dec 2, 2022
We also switch to new releases by updating a symlink, maybe this confuses opcache.
After updating symlink we call opcache_clear() by making an curl api call (to make sure that opcache_clear() is called in the correct context: nginx + php-fpm).
From what I remember, the crash is not happening immediately after deploying new version but much later (even days later).
Currently at version 8.1.12, it happens less often than 8.1.6 (don't know exactly when it started to happen less often).
GreenReaper commentedon Feb 7, 2023
I ran into what seems to be this while upgrading between Debian-compatible builds of 8.1.15 provided by the Sury repository. Clearing out the file cache fixed the segmentation fault, which seems to have arisen on the 'entry->parent` line mentioned.
strngr commentedon Apr 20, 2023
Here are the steps to reproduce that work for my php-fpm:
siege -b -c 50 http://endpoint/kill -USR2 <PID of php-fpm master process>My PHP version:
Backtrace:
Strace of failing worker:
dstogov commentedon Oct 11, 2023
This should be fixed via 90f2e76