HANA DB Startup Troublrshooting

 This Page Explains Basic troubleshooting steps 

Checks to perform :


The first thing that is needed to be determined, is if the SAP HANA database is running. To do this run:

ps -ef | grep hdb



If the HANA database is running the following processes will be present

hdbnameserver

hdbpreprocessor

hdbcompileserver

hdbindexserver

Please ensure that processes are being run by the correct <SID>adm user in case they have multiple HANA's running on the system

If you see the running processes then please review the System Hang section.

 

To see if the HANA database will start, try via putty going to /usr/sap/<SID>/HDB<instance#> and running

HDB start

 

If this fails go to /usr/sap/<SID>/HDB<instance#>/exe here you can try and run the processes manually

usually, you will only need to call ./hdbnameserver and then the ./hdbindexserver and continue with the

rest if it is successful, if it is successful the issue could be with hdbdaemon or sapstartsvr and you

will check the associated logs.

 

Check the HANA trace files in the following location /usr/sap/<SID>/HDB<instance#>/<server name>/trace : command =/usr/sap/<SID>/HDB<instance#>cdtrace

or create a full system dump by following SAP Note 1732157 - Collecting diagnosis information for SAP HANA

 

The order of checking the trace files should be first the daemon, nameserver, indexserver, compileserver, and preprocessor (statistics server would not cause the system to stop starting).

However, after checking the indexserver, you should be able to see where the error lies


How to collect system dump or diagnosis information:


From HANA Studio: Go to Diagnosis Files Tab if in case of system is not up click "open diagnosis Mode"


Can select or unselect the "Include System views" option and can change Days .Default =7days


Once the files have been collected click on the list option.




Now click on Download Collection and save in the desktop.


Now Downloading System dump From OS Level 

Once logged on to SIDADM execute "cdpy"  


Now execute "python fullsysteminfodump.py" like below and it will take time to get a dump.

for Tenant DB:- python fullsysteminfodump.py -t SID

By default, it collects for 7 days but we can specify the days also for example below

python fullsysteminfodump.py -t  SID -days=14


Now go to directory /usr/sap/SID/SYS/global/sapcontrol/snapshots


Issues and Reported Problems

Common issues that we can see are:

Authorization Issues :

In the trace, we will see the message 'not authorized' in the trace, in this scenario check as the <SID>adm

user and see if that user can make a file in the location specified in the trace to verify this. If you

cannot create the file run the chmod command on the folder to allow reading and writing (ie chmod 764)

Disk Full Error :

In this, the trace file will contain the words 'rc=24 no space left on device errors' for this, please

review SAP Note 2083715 - Analyzing log volume full situations

 Ex: DiskFullEvent on Log Volume (workaround)

Sol: Stop the database --> we will have to copy the log file from /hana/log/SID/mnt00001/ to a temporary location and symbolic link to the log directory.-->Start the DB and wait until a backup happens and then reclaim the log volume.

1679938 for complete details.

Corrupt Log Segments

The trace file will say something like cannot find or cannot read a log segment at a hexadecimal address, the only resolution to a corrupt log segment is to do a recovery that does not involve that log segment

 

Missing Log Segments

In the trace we will see 'Cannot open file "/<path_to_missing_logsegment>/logsegment_000_XXXXXXXX.dat", rc=2: No such file or directory

for this please review SAP Note 1788692 - Index Server crash due to missing LogSegment file

 

Hardware issue

There is no generic line in the trace would point to hardware, but if the issue is OS related or a disk cannot mount please follow the

hardware portion of the survival guide

 

HANA Not Starting after a failed hdblcm rename (hdbrename)

When you try to start HANA it fails with "process hdbdaemon HDB Daemon not running". No daemon, nameserver, or indexserver trace is created which indicates that it hasn't even gotten to the point of trying to start the services.

SAP Note 2142432 - SAP HANA does not start after a failed attempt to rename the HANA SID

System Crash

An SAP incident will have to be made with a full system dump (SAP Note 1732157 - Collecting diagnosis information for SAP HANA)

HANA up but SAP system not starting:

Check if a connection is possible to the database by running

R3trans -d

this will end with a return code. RC <8 is a successful connection to the database but rc=12 would be a failure. 

Check the trans.log which is produced to see further details about why the ABAP side of the SAP system could not connect to the database.

Here are some examples of common issues when R3trans d results in r=12

 

  1. Your HANA DB rev is SPS9 (rev 90 or higher) and you see something similar to what is listed below:
    "

    4 ETW000  [     dev trc,00000]  Database release is HDB 1.00.090.00.1413897729                            54  0.055046 4 ETW000  [dbhdbsql.cpp,00000]  *** ERROR => Using non supported HANA version: 1.00.090.00.1413897729 4 ETW000  [dbhdbsql.cpp,00000]  *** ERROR => Min. version for this release must be 1.00.62

    " 

     Please see SAP Note 1952701  - DBSL supports new SAP HANA SP9 version number

Timezone and DST issues:

The system may come up but have dumps of ZDATE_LARGE_TIME_DIFF

Follow the guidelines at: http://scn.sap.com/docs/DOC-58741  

SAP Note 1932132 - SAP HANA: Large time difference between the application server and HANA database

SAP KBA  2137138 - Timezone name incorrect after DST switch

Related Documents

 For DST preparations: http://scn.sap.com/docs/DOC-58741

  

Related SAP Notes/KBAs

SAP Note 1732157 - Collecting diagnosis information for SAP HANA

SAP Note 2083715 - Analyzing log volume full situations

SAP Note 1788692 - Index Server crash due to missing LogSegment file

Comments

Post a Comment

Popular Posts

SAP Logon Configuration(Load Balancing)

Image
Logon groups (or work groups) are configured to dynamically distribute the load being processed by the dialog work processes. Setting up logon groups helps in uniform distribution of the work load across the available instances. While logging on using a logon group, the ABAP message server is contacted to identify the instance with best performance statistics within the selected logon group . A report runs in SAP every 5 minutes which determines the load across each server and updates in the memory area of the message server. This information will be used by SAP GUI to determine the best instance to distribute the next user.   SAP Logon Process Let’s start with discussing an overview of SAP Logon process. High-level understanding of the process will help to explain details of SAP Logon configuration. SAP Logon Process Overview When user(s) accessing SAP system via  SAP GUI , the request is forwarded into ABAP dispatcher for the direct logon. If SAP GUI is using logon group, th...
Read more

EHP Upgrade (SUM)

Image
  Introduction: SAP Technical Upgrade is a periodic project that is implemented across companies to upgrade their SAP system to the latest released version. Most of the upgrade activities are done by the technical team, and functional consultants' role is limited and mostly confined to regression testing. Reasons for SAP Technical Upgrade : Since technology and businesses evolve, companies need to maintain a technological infrastructure that can support that change. This is the major reason why companies should undertake a technical upgrade of their SAP system. Of course, there are many secondary reasons, the significant one being the fact that like all software vendors SAP supports a released version of its software only for a limited period. Now we can go through the detailed phases included in the SUM upgrade process. latest SPAM version and Kernal also considered prerequisites  Prerequisites : 1. Check SYSTEM user is De-active, if yes, it should be activated. Passwords f...
Read more

Interview Questions

Image
1. Startup profile from 7.3? Answer: Now the Start profile contents are merged with the Instance profile. With help of the new Instance, profile SAP processes are started and at the same time, instance-specific parameters are read. 2. SAP Application server connects with Database using database client. 3. Difference b/w profiles? Default.PFL - default profile Instance Profile - SID_DVEBMGS"nr"_Hostname location = usr/sap/SID/SYS/Profile 4. Below are the common errors in SAP transport management Return code 4:  Imported with warnings Generation of program, colomns or row missing . Return code 8 :  Imported with errors Syntax error Program generation error Dictionary activation error or method execution error Return code 12 :  Indicates Import Cancelled Program cancelled due to job Import cancelled due to object missing Import cancelled due to object not active Return code 18 :  Indicates Import Cancelled Due to system down while import Due to user expired during impo...
Read more

SNOTE Implementation and FAQ

Image
 SAP Note : SAP Maintains a knowledge base of problems and resolutions which are accessed from market place. www.service.sap.com/notes.SAP provides resolutions in the form of a note which is a number. Note can be searched on number (if we know)or we can search with the problem code, error number etc. Note provides info. Reg the problem, as follows 1.Problem 2.Pre-Requisites 3.Cause of the problem 4.Solution, Corrections, attachments, and note may redirect one or more notes. Notes of 3 types : 1.Informative note Which consists of details to solve the problem 2.Corrective note This provides changes to the data dictionary elements or repository objects. 3.Manual note: Some of the steps will be present in notes to solve the issue.   Procedure : Read the note carefully and check for validity, Pre-Implementation steps , Post-Implementation steps and Pre-requisite Notes. Do the manual pre-implementation steps in case the Note sugges...
Read more

SAP Startup Troubleshooting

Image
  1. IS DATABASE UP AND RUNNING? This is an  absolute  prerequisite for SAP SYSTEM to start. In order to test this, operating system command  R3trans -d   can be executed (as SIDadm user). Any return code different than 0 means that DB cannot be reached by DIALOG work processes, and so SAP system will not initiate. The following return indicates that the database is up, running, and reachable by SAP work processes: <hostname>:SIDadm > R3trans -d This is R3trans version X.xx (release 742 - 18.11.14 - 20:14:09). unicode enabled version R3trans finished (0000). -->There is  NO POINT  in moving the analysis further while the return code is different than zero. --> In case the return code is different than zero, please involve your DBA/Network/OS teams. 2 . IS SAP STARTUP AGENT (SAPSTARTSRV) RUNNING? There must be  ONLY ONE  sapstartsrv process running for each instance. Examples for each platform below: Examples...
Read more

HANA Installation & upgrade

Image
  Software Download: SAP HANA System Types : Ø   Single-host system - One  SAP HANA  instance on one host Ø   Multiple-host (distributed) system - Multiple  SAP HANA  instances distributed over multiple hosts, with one instance per host. SAP HANA Deployment Types : Tenant Databases : The tenant database deployment type makes it possible to run several SAP HANA instances on the same hardware in a production environment. Multiple Components on One Database (MCOD) : MCOD deployments are characterized by multiple applications on one SAP HANA system. SAP supports deploying and running multiple applications on a single SAP HANA production database  only  for packaged applications and scenarios listed on the "White List" included in SAP Note 1661202. Virtualized : SAP HANA systems can be run on virtual machines with restrictions to the hypervisor (including logical partitions). Multiple Components on One System (MCOS): MCOS...
Read more

SAP HANA Backup

Image
  SAP HANA Persistence To ensure optimal performance, the SAP HANA database holds most of its data in-memory. However, it still uses persistent storage to provide a fallback in case of failure.   During normal database operation, data is automatically saved from memory to disk at regular savepoints . Additionally, all data changes are recorded in the redo log . The redo log is saved from memory to disk with each committed database transaction. After a power failure , you can restart the database as you would with any disk-based database. It then returns to its last consistent state by replaying the redo log since the last savepoints . Although savepoints and log writing protect your data against power failures, savepoints do not help if the persistent storage itself is damaged . To protect against data loss because of disk failures, backups are required . Backups save the payload (the actual data) of the data area and log area to different locations.   Types o...
Read more

SAP Transport Management System (STMS)

Image
TMS is used to move, manage and control copy development objects and customizing settings in an orderly fashion across SAP Landscape. STMS [SAP Transport Management System] Domain Controller–:  It is used to control the landscape. There will be only one domain controller in the landscape. Initially, it is set to development and later can be moved to on higher available system such as PRD/QAS/Pre7/ production but to keep things simple most of the cases we keep DEV as Domain Controller Member Systems–: The other systems are included in the domain referred to as member systems.  How to Configure   STMS Log on to the SAP system, which is decided to be the Domain Controller, in client  000  and enter the transaction code  STMS. If there is no Domain Controller already, a system will prompt you to create one. When the Transport Domain is created for the first time, the following activities happen in the background: Initiation of the Transport Domain / Landsc...
Read more

Set Up SAP HANA System Replication with hdbnsutil

Image
  Procedure:                                               SITE -A Enable system replication on the primary system(SITE A) as follows: In the Administration editor of SAP HANA studio, choose the  Configuration  tab and ensure that  log_mode  is set to "normal" in the  persistence  section of the  global.ini  file. Log mode  normal  means that log segments must be backed up. Log mode  overwrite  means log segments are freed by the savepoint (therefore only useful for test installations without backup and recovery). Do an initial data backup or create a storage snapshot. In multiple-container systems, the system database and all tenant databases must be backed up. As  <sid>adm  on the command line enable the primary for system replication and give it a logical name with the following command. Th...
Read more

SAP Kernel Upgrade in Distributed System

Image
What is the kernel of an SAP system? The kernel is a central program that acts as an interface between the SAP application and operating system. These files help startup the R/3 system, initialize the memory, create buffers and start managing the requests from users and effectively utilizing of hardware resources. The kernel is also responsible for starting and stopping all the application services like dispatcher, message server, collector, etc. Why Kernel Upgrade? SAP Kernel is the core of the application. Like all other applications, the Kernel contains the executable files (.EXE files for stating various processes in SAP). The Kernel is the heart of the operating system. It contains those files which are used to run every event in SAP. E.g.|: starting database, shutdowns of the database, starting sap, shutdown of sap, saposcol, to uncar the sap files etc. That's the reason why when a Kernel upgrade is done it means new versions of the various EXE files replace the older version...
Read more