I am not able to start the Cassandra instance. It is throwing below error while checking the Cassandra status :
apigee-service apigee-cassandra status
opt/apigee/apigee-service-4.17.05-0.0.572/bin/../lib/service-functions.sh: fork: retry: Resource temporarily unavailable
/opt/apigee/apigee-service-4.17.05-0.0.572/bin/../lib/service-functions.sh: fork: retry: Resource temporarily unavailable
Also in the Cassandra log I can see below error :
Failed to flush writer .no space left on device.
We do have enough disk space on the machine . Not sure what's causing this error.
Also should it affect the Edge-UI. As we are not able to access UI.
Can you share more information on the size of your node?
How many cassandra nodes do you have (size of your ring)?
Are the other C* nodes good and is there any difference between these nodes? @Pranjali
I get this info on my org:
/opt/apigee/apigee-cassandra-2.1.16-0.0.1049/bin/nodetool status
Datacenter: dc-1
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns Host ID Token Rack
UN 10.54.16.174 713.93 MB ? 4beb1271-****-****-****-e6b54031134e 0 ra-1
Not sure how to get other details.
Also, paste the output of
1. `df -h`
2. sudo lsblk -l
3. value for `data_file_directories` from cassandra.yaml file
4. value for `commitlog_directory` from cassandra.yaml file
Hi Vaibhav,
Please see the details below:
1. `df -h`
2. sudo lsblk -l
As I don't have root access, can not execute this command.
3. value for `data_file_directories` from cassandra.yaml file\
data_file_directories:
/opt/apigee/data/apigee-cassandra/data
4. value for `commitlog_directory` from cassandra.yaml file
commitlog_directory:
/opt/apigee/data/apigee-cassandra/commitlog
Hi @vaibhav :
Please see the details below :
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-root
5.8G 946M 4.6G 17% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 976M 175M 735M 20% /boot
/dev/mapper/rootvg-home
240M 3.6M 220M 2% /home
/dev/mapper/rootvg-opt
13G 8.5G 3.2G 73% /opt
/dev/mapper/rootvg-osmf
3.9G 182M 3.5G 5% /osmf/mgmt
/dev/mapper/rootvg-scheduler
213M 2.1M 196M 2% /osmf/mgmt/scheduler
/dev/mapper/rootvg-usr
4.8G 1.4G 3.2G 31% /usr
/dev/mapper/rootvg-var
5.8G 1.1G 4.4G 21% /var
none 1.0G 21M 1004M 3% /tmp
/dev/mapper/appvg-applv1
9.7G 2.6G 6.6G 28% /oraclient
/dev/mapper/appvg-applv2
20G 2.7G 16G 15% /software/bea
/dev/mapper/appvg-applv3
40G 3.2G 34G 9% /wls_domains
/dev/mapper/appvg-applv5
25G 44M 24G 1% /software/bea_hwb2brtp1
/dev/mapper/appvg-applv6
25G 44M 24G 1% /wls_domains/hwb2brtp1
/dev/mapper/appvg-applv7
30G 44M 28G 1% /wls_domains/hwb2brtp1_logs
/dev/mapper/appvg-applv8
30G 44M 28G 1% /wls_domains/hwb2brtp1_filestores
/dev/mapper/appvg-applv4
20G 12G 7.3G 62% /wls_domains_logs
sudo lsblk -l
Password:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
sda 8:0 0 35G 0 disk
sda1 8:1 0 1G 0 part /boot
sda2 8:2 0 34G 0 part
rootvg-root (dm-0) 252:0 0 6G 0 lvm /
rootvg-osmf (dm-11) 252:11 0 4G 0 lvm /osmf/mgmt
rootvg-scheduler (dm-12) 252:12 0 224M 0 lvm /osmf/mgmt/scheduler
rootvg-opt (dm-13) 252:13 0 12.5G 0 lvm /opt
rootvg-usr (dm-14) 252:14 0 5G 0 lvm /usr
rootvg-var (dm-15) 252:15 0 6G 0 lvm /var
rootvg-home (dm-16) 252:16 0 256M 0 lvm /home
sdb 8:16 0 5G 0 disk
swapvg-swap (dm-1) 252:1 0 5G 0 lvm [SWAP]
sdc 8:32 0 215G 0 disk
sdc1 8:33 0 215G 0 part
appvg-applv1 (dm-2) 252:2 0 9.9G 0 lvm /oraclient
appvg-applv2 (dm-3) 252:3 0 19.9G 0 lvm /software/bea
appvg-applv3 (dm-4) 252:4 0 39.9G 0 lvm /wls_domains
appvg-applv4 (dm-5) 252:5 0 20G 0 lvm /wls_domains_logs
appvg-applv5 (dm-6) 252:6 0 24.9G 0 lvm /software/bea_hwb2brtp1
appvg-applv6 (dm-7) 252:7 0 24.9G 0 lvm /wls_domains/hwb2brtp1
appvg-applv7 (dm-8) 252:8 0 29.9G 0 lvm /wls_domains/hwb2brtp1_logs
appvg-applv8 (dm-9) 252:9 0 29.9G 0 lvm /wls_domains/hwb2brtp1_filestores
appvg-applv9 (dm-10) 252:10 0 14.9G 0 lvm /appl/b2b/support
sr0 11:0 1 1024M 0 rom
less /opt/apigee/apigee-cassandra-2.1.16-0.0.1049/conf/cassandra.yaml
commitlog_directory: /opt/apigee/data/apigee-cassandra/commitlog
data_file_directories - /opt/apigee/data/apigee-cassandra/data
please let me know if you need more information.
Hi Pranjali
add to /etc/security/limits.conf file (you need root or sudo to edit it) the following lines:
apigee soft memlock unlimited apigee hard memlock unlimited apigee soft nofile 32768 apigee hard nofile 65536 apigee soft as unlimited apigee hard as unlimited apigee - nproc 32768
Also restart cassandra. Report back your new findings !
Will help you if this not fully resolves and we will also look at available disk space on partition.
Basically the error message says, Cassandra cannot find enough disk space to flush the memtables to sstables into the disk.
In the df -h output, could you help us identify which is the data and commitlog directory for Cassandra ?
typically data directory is /opt/apigee/data/apigee-cassandra/data
Alternatively, you can do this
cd /opt/apigee/data/apigee-cassandra/data df -h .
Check for available disk space