Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Started to get ORA-27103: internal error consistently with latest-faststart #79

Closed
ipsips opened this issue Dec 9, 2024 · 9 comments
Closed

Started to get ORA-27103: internal error consistently with latest-faststart #79

ipsips opened this issue Dec 9, 2024 · 9 comments
Assignees
@gvenzl
Labels
bug Something isn't working

Comments

@ipsips
Copy link

ipsips commented Dec 9, 2024

Started to get ORA-27103: internal error consistently with latest-faststart since this Monday (Dec 9th) morning. Things were still running smoothly last Friday. Noticed a new version was published over the weekend on Docker Hub. Did some regression slip in?

Got things working with 23.5-faststart for now.

Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '.*DATABASE IS READY TO USE!.*\s'
	at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:47)
	at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52)
	at org.testcontainers.oracle.OracleContainer.waitUntilContainerStarted(OracleContainer.java:81)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:490)
	... 91 common frames omitted
09:19:12.089 [Test worker] ERROR tc.gvenzl/oracle-free:latest-faststart -- Log output from the failed container:
CONTAINER: starting up...
CONTAINER: first database startup, initializing...
CONTAINER: starting up Oracle Database...

LSNRCTL for Linux: Version 23.0.0.0.0 - for Oracle Cloud and Engineered Systems on 09-DEC-2024 09:09:12

Copyright (c) 1991, 2024, Oracle.  All rights reserved.

Starting /opt/oracle/product/23ai/dbhomeFree/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 23.0.0.0.0 - for Oracle Cloud and Engineered Systems
System parameter file is /opt/oracle/product/23ai/dbhomeFree/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/81aa229c35db/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_FREE)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC_FOR_FREE)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 23.0.0.0.0 - for Oracle Cloud and Engineered Systems
Start Date                09-DEC-2024 09:09:12
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Default Service           FREE
Listener Parameter File   /opt/oracle/product/23ai/dbhomeFree/network/admin/listener.ora
Listener Log File         /opt/oracle/diag/tnslsnr/81aa229c35db/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_FREE)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
The listener supports no services
The command completed successfully
ORA-27103: internal error
Additional information: 16878
Additional information: 1
@europ
Copy link

europ commented Dec 10, 2024 •
edited
Loading

Same issue in slim image ghcr.io/gvenzl/oracle-free:23-slim.

Workaround: downgrade image to 23.5-slim tag

@gvenzl gvenzl self-assigned this Dec 10, 2024
@gvenzl gvenzl added the bug Something isn't working label Dec 10, 2024
@gvenzl
Copy link
Owner

gvenzl commented Dec 10, 2024 •
edited
Loading

I'm sorry to hear that and apologize for the inconvenience, @ipsips and @europ!

The error shown occurs during database startup and is generated by the database itself.

Could you please provide the output from docker exec <container name> cat /opt/oracle/diag/rdbms/*/*/trace/alert*log and upload it here?

@europ
Copy link

europ commented Dec 11, 2024 •
edited
Loading

Image: ghcr.io/gvenzl/oracle-free:23.6-slim

File: /opt/oracle/diag/rdbms/*/*/trace/alert*log

[oracle@37a841c0123b ~]$ cat /opt/oracle/diag/rdbms/*/*/trace/alert*log
2024-12-11T12:39:36.960155+00:00
Adjusting the default value of parameter "parallel_max_servers" from 0 to 1.
Starting ORACLE instance (normal) (OS id: 73)
2024-12-11T12:39:36.973203+00:00
****************************************************
 Sys-V shared memory will be used for creating SGA
 ****************************************************
2024-12-11T12:39:36.974441+00:00
**********************************************************************
2024-12-11T12:39:36.974529+00:00
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)

2024-12-11T12:39:36.974689+00:00
 Domain name: docker/37a841c0123b49c58af7fc4151a9d72eebe8c5d96a7fa8bd939d12046f2ea716
2024-12-11T12:39:36.974786+00:00
 Per process system memlock (soft) limit = 64K
2024-12-11T12:39:36.974871+00:00
 Expected per process system memlock (soft) limit to lock
 instance MAX SHARED GLOBAL AREA (SGA) into memory: 1532M
2024-12-11T12:39:36.975036+00:00
 Available system pagesizes:
  4K, 2048K
2024-12-11T12:39:36.975205+00:00
 Supported system pagesize(s):
2024-12-11T12:39:36.975288+00:00
  PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s)
2024-12-11T12:39:36.975372+00:00
        4K       Configured               5           391555        NONE
2024-12-11T12:39:36.975507+00:00
     2048K                0             766                0        NONE
2024-12-11T12:39:36.975611+00:00
RECOMMENDATION:
2024-12-11T12:39:36.975700+00:00
 1. For optimal performance, configure system with expected number
 of pages for every supported system pagesize prior to the next
 instance restart operation.
2024-12-11T12:39:36.975952+00:00
 2. Increase per process memlock (soft) limit to at least 1532MB
 to lock 100% of SHARED GLOBAL AREA (SGA) pages into physical memory
2024-12-11T12:39:36.976113+00:00
**********************************************************************
Using an SGA granule size of 16MB
Memory protection keys enabled.
Hugetlbfs Dynamic SGA NOT enabled.
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Using LOG_ARCHIVE_DEST_1 parameter default value as /opt/oracle/product/23ai/dbhomeFree/dbs/arch
.... (PID:): Enable RFS client   [kcrlc.c:609]
Memory protection keys enabled on shared strand buffers
Autotune of undo retention is turned on.
Log buffer boundaries=(0xc0000000, 0xc0452000)
IMODE=BR
ILAT =35
NOTE: Using default ASM root directory ASM
Oracle Database 23ai Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
Version 23.6.0.24.10.
ORACLE_HOME:    /opt/oracle/product/23ai/dbhomeFree
System name:	Linux
Node name:	37a841c0123b
Release:	5.4.17-2136.327.2.el7uek.x86_64
Version:	#2 SMP Fri Jan 5 14:53:41 PST 2024
Machine:	x86_64
VM name:	KVM CPUID feature flags: 0x01003afb
Using parameter settings in server-side spfile /opt/oracle/product/23ai/dbhomeFree/dbs/spfileFREE.ora
System parameters with non-default values:
  processes                = 200
  cpu_count                = "2"
  nls_language             = "AMERICAN"
  nls_territory            = "AMERICA"
  sga_target               = 1536M
  control_files            = "/opt/oracle/oradata/FREE/control01.ctl"
  control_files            = "/opt/oracle/oradata/FREE/control02.ctl"
  db_block_size            = 8192
  compatible               = "23.6.0"
  _instance_recovery_bloom_filter_size= 1048576
  undo_tablespace          = "UNDOTBS1"
  fast_start_parallel_rollback= "FALSE"
  remote_login_passwordfile= "EXCLUSIVE"
  dispatchers              = "(PROTOCOL=TCP) (SERVICE=FREEXDB)"
  shared_servers           = 0
  local_listener           = ""
  job_queue_processes      = 1
  db_name                  = "FREE"
  open_cursors             = 300
  pga_aggregate_target     = 512M
  _dmm_blas_library        = "libora_netlib.so"
  control_management_pack_access= "DIAGNOSTIC+TUNING"
  spatial_vector_acceleration= FALSE
  diagnostic_dest          = "/opt/oracle"
  enable_pluggable_database= TRUE
  common_user_prefix       = ""
  mle_prog_languages       = "OFF"
2024-12-11T12:39:37.145967+00:00
Errors in file /opt/oracle/diag/rdbms/free/FREE/trace/FREE_ora_73.trc:
ORA-27180: failed to create memory protection key
Linux-x86_64 Error: 1: Operation not permitted
Additional information: 16720

File: opt/oracle/diag/rdbms/free/FREE/trace/FREE_ora_73.trc

[oracle@37a841c0123b ~]$ cat /opt/oracle/diag/rdbms/free/FREE/trace/FREE_ora_73.trc
Trace file /opt/oracle/diag/rdbms/free/FREE/trace/FREE_ora_73.trc
Oracle Database 23ai Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
Version 23.6.0.24.10
Build label:    RDBMS_23.6.0.24.10DBRU_LINUX.X64_241017.LA
ORACLE_HOME:    /opt/oracle/product/23ai/dbhomeFree
System name:	Linux
Node name:	37a841c0123b
Release:	5.4.17-2136.327.2.el7uek.x86_64
Version:	#2 SMP Fri Jan 5 14:53:41 PST 2024
Machine:	x86_64
Storage:	?
CLID:	U
Instance name: FREE
Instance number: 0 <none>
Database name: N/A
Database unique name: N/A
Database id: N/A
Redo thread mounted by this instance: 0 <none>
Oracle process number: 0
Unix process pid: 73, NID: 4026532330, image:


*** 2024-12-11T12:39:36.960201+00:00
SGA (id: 1) creation time at startup: 3 ms
I:2024-12-11 12:39:36.985 : [    GPNP][    GPNP]  clsgpnp_Init init failed. Error: CLSGPNP_ERR (1) .
I:2024-12-11 12:39:36.988 : [ CLSINET][ CLSINET]  Failed to initialize clsgpnp context grv 1
I:2024-12-11 12:39:36.989 : [    GPNP][    GPNP]  clsgpnp_Init init failed. Error: CLSGPNP_ERR (1) .
I:2024-12-11 12:39:36.990 : [ CLSINET][ CLSINET]  Failed to initialize clsgpnp context grv 1
Lockdown sysfs /sys/kernel/security/lockdown doesn't exist. Set to default KSCXCAPTYPEKERNELSEC_NONE
KSIPC: KSIPC SERVICE ENV 0x0
KSIPC SERVICE MASK MGAON:[INETOFF]:[IPCLWON]:[GRPAM]:[PRON]:[TOPOON]:[DLLON]:[KSGLOFF][RSTFWOFF]:[RDMAPINGON]:[SHREGON]:MEMCBON:[STATSFWON]:[SNOFF]:[DGOFF]:[ORDNFY]:[RDSTCPBSON][SHREGODPON]:[SIGSAFEON]:[THRPOOLON]:[VNICOFF]
KSNMON: Initial Config: enable=0x7fffffff, maxflow=65535, minstsint=5000
KSNMON: Runtime Config: service enabled=0x3ff
kcrlnfy: Enable RFS client   [kcrlc.c:609]
kcrfwy: minimum sleep (disk) is 1058 usecs (overhead is 58 usecs)
init table done
Running with 1 strand for Non-Enterprise Edition
Running without dynamic strands for Non-Enterprise Edition
NOTE: Cluster configuration type = NONE [2]
sskgm_mpkey_create failed: 1
kcrfw_mpkey_init: Failed to initialize memory protection key
ORA-27180: failed to create memory protection key
Linux-x86_64 Error: 1: Operation not permitted
Additional information: 16720
<error barrier> at 0x7ffe3628a0b0 placed kcrfw.c@39186
sskgm_mpkey_set_prot invalid key:0
Protecting shared strand buffer via memory protection key failed
KJHA:2phase 010 opi_error:27103

Docker info:


Client:
 Debug Mode: false

Server:
 Containers: 15
  Running: 2
  Paused: 0
  Stopped: 13
 Images: 954
 Server Version: 19.03.11-ol
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7eba5930496d9bbe375fdf71603e610ad737d2b2
 runc version: 51d5e94
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.17-2136.327.2.el7uek.x86_64
 Operating System: Oracle Linux Server 7.9
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 125.3GiB
 Name: xxxxxxx
 ID: xxxxxxx
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: xxxxxxx
 HTTPS Proxy: xxxxxxx
 No Proxy: xxxxxxx
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: API is accessible on http://127.0.0.1:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
WARNING: bridge-nf-call-ip6tables is disabled
Registries:

@gvenzl
Copy link
Owner

gvenzl commented Dec 12, 2024

Thank you, @europ!

As you have probably already deduced yourself, this is a regression in Oracle Database Free itself. I have contacted Oracle Development for further help with this - stay tuned!

@gvenzl
Copy link
Owner

gvenzl commented Dec 14, 2024

Hey @europ, hey @ipsips,

I spoke with Oracle Database Development and they have informed me that the memory protection key kernel feature can be disabled with the underscore parameter "_enable_memory_protection_keys"=FALSE.

Not yet knowing exactly what causes the failure in some environments (the 23.6 containers build and run successfully in my x86_64 and aarch64 environments), I have rolled out this parameter change across all 23.6 images and re-uploaded them.

You should be able to use them now, please do let me know whether that is the case.

gvenzl added a commit that referenced this issue Dec 14, 2024
@gvenzl
Fix for #79 - disable memory protection keys in 23.6
b6e89b0 
Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com>
@gvenzl gvenzl mentioned this issue Dec 14, 2024
Merged
gvenzl added a commit that referenced this issue Dec 14, 2024
@gvenzl
Fix for #79 (#80)
5b7d68b 
* Fix for #79 - disable memory protection keys in 23.6

Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com>

---------

Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com>
@gvenzl
Copy link
Owner

gvenzl commented Dec 14, 2024

I just spotted a bug that this parameter still doesn't get set for the full image. Fixing that now.

@gvenzl
Copy link
Owner

gvenzl commented Dec 15, 2024

All fixed and uploaded

@europ
Copy link

europ commented Dec 17, 2024 •
edited
Loading

Hello.

Thank you for your quick feedback/updates and especially for the extremely fast hotfix. Fantastic!

I can confirm that ghcr.io/gvenzl/oracle-free:latest and ghcr.io/gvenzl/oracle-free:23-slim perfectly works on the above mentioned platform (OL 7.9 x86_64 with 5.4.17 UEK). The issue was resolved, thank you for your help.

@gvenzl
Copy link
Owner

gvenzl commented Dec 20, 2024

Hey @europ,

Great, I'm glad to hear it!

Thanks again for using these images and Happy Holidays!

@gvenzl gvenzl closed this as completed Dec 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gvenzl gvenzl

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ipsips @gvenzl @europ